Abstract
We present a new compression format for natural language texts, allowing both exact and approximate search without decompression. This new code —called End-Tagged Dense Code— has some advantages with respect to other compression techniques with similar features such as the Tagged Huffman Code of [Moura et al., ACM TOIS 2000]. Our compression method obtains (i) better compression ratios, (ii) a simpler vocabulary representation, and (iii) a simpler and faster encoding. At the same time, it retains the most interesting features of the method based on the Tagged Huffman Code, i.e., exact search for words and phrases directly on the compressed text using any known sequential pattern matching algorithm, efficient word-based approximate and extended searches without any decoding, and efficient decompression of arbitrary portions of the text. As a side effect, our analytical results give new upper and lower bounds for the redundancy of d-ary Huffman codes.
This work is partially supported by CICYT grant (#TEL99-0335-C04), CYTED VII.19 RIBIDI Project, and (for the third author) Fondecyt Grant 1-020381.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
M. D. Araújo, G. Navarro, and N. Ziviani. Large text searching allowing errors. In R. Baeza-Yates, editor, Proc. 4th South American Workshop on String Processing (WSP’97), pages 2–20. Carleton University Press, 1997.
R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval. Addison-Wesley, 1999.
T. C. Bell, J. G. Cleary, and I. H. Witten. Text Compression. Prentice Hall, 1990.
D. A. Huffman. A method for the construction of minimum-redundancy codes. Proc. Inst. Radio Eng., 40(9):1098–1101, September 1952.
B. Mandelbrot. An informational theory of the statistical structure of language. In Proc. Symp. on Applications of Communication Theory, pages 486–500, 1952.
D. Manstetten. Tight bounds on the redundancy of Huffman codes. IEEE Trans. on Information Theory, 38(1):144–151, January 1992.
A. Moffat. Word-based text compression. Software — Practice and Experience, 19(2): 185–198, 1989.
A. Moffat and A. Turpin. On the implementation of minimum-redundancy prefix codes. In Proc. Data Compression Conference, pages 170–179, 1996.
G. Navarro, E. Silva de Moura, M. Neubert, N. Ziviani, and R. Baeza-Yates. Adding compression to block addressing inverted indexes. Information Retrieval, 3(1):49–77, 2000.
G. Navarro and M. Raffinot. Flexible Pattern Matching in Strings — Practical online search algorithms for texts and biological sequences. Cambridge University Press, 2002.
G. Navarro and J. Tarhio. Boyer-moore string matching over ziv-lempel compressed text. In Proc. 11th Annual Symposium on Combinatorial Pattern Matching (CPM’2000), LNCS 1848, pages 166–180, 2000.
R. De Prisco and A. De Santis. On lower bounds for the redundancy of optimal codes. Designs, Codes and Cryptography, 15(1):29–45, 1998.
E. Silva de Moura, G. Navarro, N. Ziviani, and R. Baeza-Yates. Fast and flexible word searching on compressed text. ACM Transactions on Information Systems, 18(2):113–139, April 2000.
I. Witten, A. Moffat, and T. Bell. Managing Gigabytes. Morgan Kaufmann Publishers, second edition, 1999.
G. Zipf. Human Behaviour and the Principle of Least Effort. Addison-Wesley, 1949.
J. Ziv and A. Lempel. A universal algorithm for sequential data compression. IEEE Transactions on Information Theory, 23(3):337–343, 1977.
J. Ziv and A. Lempel. Compression of individual sequences via variable-rate coding. IEEE Transactions on Information Theory, 24(5):530–536, 1978.
N. Ziviani, E. Silva de Moura, G. Navarro, and R. Baeza-Yates. Compression: A key for next-generation text retrieval systems. IEEE Computer, 33(11):37–44, 2000.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Brisaboa, N.R., Iglesias, E.L., Navarro, G., Paramá, J.R. (2003). An Efficient Compression Code for Text Databases. In: Sebastiani, F. (eds) Advances in Information Retrieval. ECIR 2003. Lecture Notes in Computer Science, vol 2633. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-36618-0_33
Download citation
DOI: https://doi.org/10.1007/3-540-36618-0_33
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-01274-0
Online ISBN: 978-3-540-36618-8
eBook Packages: Springer Book Archive