Advertisement

g-binary: A New Non-parameterized Code for Improved Inverted File Compression

  • Ilias Nitsos
  • Georgios Evangelidis
  • Dimitrios Dervos
Conference paper
  • 497 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2736)

Abstract

The inverted file is a popular and efficient method for indexing text databases and is being used widely in information retrieval applications. As a result, the research literature is rich in models (global and local) that describe and compress inverted file indexes. Global models compress the entire inverted file index using the same method and can be distinguished in parameterized and non-parameterized ones. The latter utilize fixed codes and are applicable to dynamic collections of documents. Local models are always parameterized in the sense that the method they use makes assumptions about the distribution of each and every word in the document collection of the text database. In the present study, we examine some of the most significant integer compression codes and propose g-binary, a new non-parameterized coding scheme that combines the Golomb codes and the binary representation of integers. The proposed new coding scheme does not introduce any extra computational overhead when compared to the existing non-parameterized codes. With regard to storage utilization efficiency, experimental runs conducted on a number of TREC text database collections reveal an improvement of about 6% over the existing non-parameterized codes. This is an improvement that can make a difference for very large text database collections.

Keywords

Binary Representation Index Compression Good Compression Inverted List Bernoulli Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Blandford, D., Blelloch, G.: Index Compression through Document Reordering. In: Proceedings of the Data Compression Conference (2002)Google Scholar
  2. 2.
    Bookstein, A., Klein, S.T., Raita, T.: Model based concordance compression. In: Storer and Cohn., pp. 82–91 (1992)Google Scholar
  3. 3.
    Elias, P.: Universal codeword sets and representations of the integers. IEEE Transactions on Information Theory IT–21, 194–203 (1975)CrossRefMathSciNetGoogle Scholar
  4. 4.
    Fox, E., Harman, D., Baeza-Yates, R., Lee, W.: Inverted Files. In: Frakes, W., Baeza-Yates, R. (eds.) Information Retrieval: Data Structures and Algorithms, ch. 3, pp. 28–43. Prentice-Hall, Englewood Cliffs (1992)Google Scholar
  5. 5.
    Gallager, R.G., van Voorhis, D.C.: Optimal source codes for geometrically distributed alphabets. IEEE Transactions on Information Theory IT–21, 228–230 (1975)CrossRefGoogle Scholar
  6. 6.
    Golomb, S.W.: Run-length Encodings. IEEE Transactions on Information Theory IT–21, 399–401 (1966)CrossRefMathSciNetGoogle Scholar
  7. 7.
    Huffman, D.A.: A method for the construction of minimum redundancy codes. Procedures IRE 40(9), 1098–1101 (1952)CrossRefGoogle Scholar
  8. 8.
    Moffat, A., Stuiver, L.: Exploiting clustering in inverted file compression. In: Storer and Cohn., 82–91 (1996)Google Scholar
  9. 9.
    Moffat, A., Zobel, J.: Paremeterised Compression for Sparse Bitmaps. In: 15th Ann Int’l SIGIR, Denmark, pp. 274–285 (1992)Google Scholar
  10. 10.
    Scholer, F., Williams, H.E., Yiannis, J., Zobel, J.: Compression of Inverted Indexes for Fast Query Evaluation. In: SIGIR, Finland, pp. 222–229 (2002)Google Scholar
  11. 11.
    Schuegraf, E.J.: Compression of large inverted files with hyperbolic term distribution. Information Processing and Managemant 12, 377–384 (1976)zbMATHCrossRefGoogle Scholar
  12. 12.
    Teuhola, J.: A compression method for clustered bit-vectors. Information Processing Letters 7(2), 308–311 (1978)zbMATHCrossRefMathSciNetGoogle Scholar
  13. 13.
    Williams, H.E., Zobel, J.: Compressing Integers for Fast File Access. The Computer Journal 42, 193–201 (1999)CrossRefGoogle Scholar
  14. 14.
    Witten, I.H., Moffat, A., Bell, T.C.: Managing Gigabytes. In: Compressing and Indexing Documents and Images. Academic Press, London (1999)Google Scholar
  15. 15.
    Zobel, J., Moffat, A., Ramamohanarao, K.: Inverted Files Versus Signature Files for Text Indexing. ACM Transactions on Database Systems 23, 369–410 (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Ilias Nitsos
    • 1
  • Georgios Evangelidis
    • 1
  • Dimitrios Dervos
    • 2
  1. 1.Department of Applied InformaticsUniversity of MacedoniaThessalonikiGreece
  2. 2.Department of Information TechnologyTEIThessalonikiGreece

Personalised recommendations