SCM: Structural Contexts Model for Improving Compression in Semistructured Text Databases

  • Joaquín Adiego
  • Gonzalo Navarro
  • Pablo de la Fuente
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2857)


We describe a compression model for semistructured documents, called Structural Contexts Model, which takes advantage of the context information usually implicit in the structure of the text. The idea is to use a separate semiadaptive model to compress the text that lies inside each different structure type (e.g., different XML tag). The intuition behind the idea is that the distribution of all the texts that belong to a given structure type should be similar, and different from that of other structure types. We test our idea using a word-based Huffman coding, which is the standard for compressing large natural language textual databases, and show that our compression method obtains significant improvements in compression ratios. We also analyze the possibility that storing separate models may not pay off if the distribution of different structure types is not different enough, and present a heuristic to merge models with the aim of minimizing the total size of the compressed database. This technique gives an additional improvement over the plain technique. The comparison against existing prototypes shows that our method is a competitive choice for compressed text databases. Finally, we show how to apply SCM over text chunks, which allows one to adjust the different word frequencies as they change across the text collection.


Text Compression Compression Model Semistructured Documents 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [BCC+00]
    Buchsbaum, A.L., Caldwell, D.F., Ward Church, K., Fowler, G.S., Muthukrishnan, S.: Engineering the compression of massive tables: an experimental approach. In: Symposium on Discrete Algorithms, pp. 175–184 (2000)Google Scholar
  2. [BSTW86]
    Bentley, J., Sleator, D., Tarjan, R., Wei, V.: A locally adaptive data compression scheme. Communications of the ACM 29, 320–330 (1986)zbMATHCrossRefMathSciNetGoogle Scholar
  3. [Che01]
    Cheney, J.: Compressing XML with multiplexed hierarchical PPM models. In: Proc. Data Compression Conference (DCC 2001), p. 163 (2001)Google Scholar
  4. [DPS99]
    Dvorský, J., Pokorný, J., Snásel, V.: Word-based compression methods and indexing for text retrieval systems. In: Eder, J., Rozman, I., Welzer, T. (eds.) ADBIS 1999. LNCS, vol. 1691, pp. 75–84. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  5. [Har95]
    Harman, D.: Overview of the Third Text REtrieval Conference. In: Proc. Third Text REtrieval Conference (TREC-3), pp. 1–19. NIST Special Publication 500-207 (1995) Google Scholar
  6. [Hea78]
    Heaps, H.S.: Information Retrieval - Computational and Theoretical Aspects. Academic Press, London (1978)zbMATHGoogle Scholar
  7. [Huf52]
    Huffman, D.A.: A method for the construction of minimum-redundancy codes. Proc. Inst. Radio Engineers 40(9), 1098–1101 (1952)Google Scholar
  8. [LS00]
    Buchsbaum, A.L., Caldwell, D.F., Ward Church, K., Fowler, G.S., Muthukrishnan, S.: Engineering the compression of massive tables: an experimental approach. In: Symposium on Discrete Algorithms, pp. 175–184 (2000)Google Scholar
  9. [MNZB00]
    Silva de Moura, E., Navarro, G., Ziviani, N., Baeza-Yates, R.: Fast and flexible word searching on compressed text. ACM Transactions on Information Systems 18(2), 113–139 (2000)CrossRefGoogle Scholar
  10. [Mof89]
    Moffat, A.: Word-based text compression. Software - Practice and Experience 19(2), 185–198 (1989)CrossRefGoogle Scholar
  11. [MW01]
    Moffat, A., Wan, R.: RE-store: A system for compressing, browsing and searching large documents. In: Proc. 8th Intl. Symp. on String Processing and Information Retrieval (SPIRE 2001), pp. 162–174 (2001)Google Scholar
  12. [NMN+00]
    Navarro, G., Silva de Moura, E., Neubert, M., Ziviani, N., Baeza-Yates, R.: Adding compression to block addressing inverted indexes. Information Retrieval 3(1), 49–77 (2000)CrossRefGoogle Scholar
  13. [Sha48]
    Shannon, C.: A mathematical theory of communication. Bell Syst. Tech. J. 27, 398–403 (1948)MathSciNetGoogle Scholar
  14. [TCB90]
    Witten, I.H., Bell, T.C., Cleary, J.G.: Text Compression. Prentice Hall, Englewood Cliffs (1990)Google Scholar
  15. [TH02]
    Tolani, P., Haritsa, J.R.: XGRIND: A query-friendly XML compressor. In: ICDE (2002),
  16. [WMB99]
    Witten, I.H., Moffat, A., Bell, T.C.: Managing Gigabytes, 2nd edn. Morgan Kaufmann Publishers, Inc., San Francisco (1999)Google Scholar
  17. [ZL77]
    Ziv, J., Lempel, A.: An universal algorithm for sequential data compression. IEEE Trans. on Information Theory 23(3), 337–343 (1977)zbMATHCrossRefMathSciNetGoogle Scholar
  18. [ZMNBY00]
    Ziviani, N., Moura, E., Navarro, G., Baeza-Yates, R.: Compression: A key for next-generation text retrieval systems. IEEE Computer 33(11), 37–44 (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Joaquín Adiego
    • 1
  • Gonzalo Navarro
    • 2
  • Pablo de la Fuente
    • 1
  1. 1.Departamento de InformáticaUniversidad de ValladolidValladolidEspaña
  2. 2.Departamento de Ciencias de la ComputaciónUniversidad de ChileSantiagoChile

Personalised recommendations