SCM: Structural Contexts Model for Improving Compression in Semistructured Text Databases
- 403 Downloads
We describe a compression model for semistructured documents, called Structural Contexts Model, which takes advantage of the context information usually implicit in the structure of the text. The idea is to use a separate semiadaptive model to compress the text that lies inside each different structure type (e.g., different XML tag). The intuition behind the idea is that the distribution of all the texts that belong to a given structure type should be similar, and different from that of other structure types. We test our idea using a word-based Huffman coding, which is the standard for compressing large natural language textual databases, and show that our compression method obtains significant improvements in compression ratios. We also analyze the possibility that storing separate models may not pay off if the distribution of different structure types is not different enough, and present a heuristic to merge models with the aim of minimizing the total size of the compressed database. This technique gives an additional improvement over the plain technique. The comparison against existing prototypes shows that our method is a competitive choice for compressed text databases. Finally, we show how to apply SCM over text chunks, which allows one to adjust the different word frequencies as they change across the text collection.
KeywordsText Compression Compression Model Semistructured Documents
Unable to display preview. Download preview PDF.
- [BCC+00]Buchsbaum, A.L., Caldwell, D.F., Ward Church, K., Fowler, G.S., Muthukrishnan, S.: Engineering the compression of massive tables: an experimental approach. In: Symposium on Discrete Algorithms, pp. 175–184 (2000)Google Scholar
- [Che01]Cheney, J.: Compressing XML with multiplexed hierarchical PPM models. In: Proc. Data Compression Conference (DCC 2001), p. 163 (2001)Google Scholar
- [Har95]Harman, D.: Overview of the Third Text REtrieval Conference. In: Proc. Third Text REtrieval Conference (TREC-3), pp. 1–19. NIST Special Publication 500-207 (1995) Google Scholar
- [Huf52]Huffman, D.A.: A method for the construction of minimum-redundancy codes. Proc. Inst. Radio Engineers 40(9), 1098–1101 (1952)Google Scholar
- [LS00]Buchsbaum, A.L., Caldwell, D.F., Ward Church, K., Fowler, G.S., Muthukrishnan, S.: Engineering the compression of massive tables: an experimental approach. In: Symposium on Discrete Algorithms, pp. 175–184 (2000)Google Scholar
- [MW01]Moffat, A., Wan, R.: RE-store: A system for compressing, browsing and searching large documents. In: Proc. 8th Intl. Symp. on String Processing and Information Retrieval (SPIRE 2001), pp. 162–174 (2001)Google Scholar
- [TCB90]Witten, I.H., Bell, T.C., Cleary, J.G.: Text Compression. Prentice Hall, Englewood Cliffs (1990)Google Scholar
- [TH02]Tolani, P., Haritsa, J.R.: XGRIND: A query-friendly XML compressor. In: ICDE (2002), http://citeseer.nj.nec.com/503319.html
- [WMB99]Witten, I.H., Moffat, A., Bell, T.C.: Managing Gigabytes, 2nd edn. Morgan Kaufmann Publishers, Inc., San Francisco (1999)Google Scholar
- [ZMNBY00]Ziviani, N., Moura, E., Navarro, G., Baeza-Yates, R.: Compression: A key for next-generation text retrieval systems. IEEE Computer 33(11), 37–44 (2000)Google Scholar