Abstract
Web archives, query and proxy logs, and so on, can all be very large and highly repetitive; and are accessed only sporadically and partially, rather than continually and holistically. This type of data is ideal for compression-based archiving, provided that random-access to small fragments of the original data can be achieved without needing to decompress everything. The recent RLZ (relative Lempel Ziv) compression approach uses a semi-static model extracted from the text to be compressed, together with a greedy factorization of the whole text encoded using static integer codes. Here we demonstrate more precisely than before the scenarios in which RLZ excels. We contrast RLZ with alternatives based on block-based adaptive methods, including approaches that “prime” the encoding for each block, and measure a range of implementation options using both hard-disk (HDD) and solid-state disk (SSD) drives. For HDD, the dominant factor affecting access speed is the compression rate achieved, even when this involves larger dictionaries and larger blocks. When the data is on SSD the same effects are present, but not as markedly, and more complex trade-offs apply.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
https://github.com/Cyan4973/lz4, accessed 27 July 2015.
- 2.
http://ir.dcs.gla.ac.uk/test_collections/gov2-summary.htm, 27 July 2015.
- 3.
Our concerns in this regard have been communicated to the authors of [5].
References
Bergman, A., Zohar, E.: Compressing Yahoo mail. In: Proceedings of the DCC, pp. 223–232 (2015)
Ferrada, H., Gagie, T., Gog, S., Puglisi, S.J.: Relative Lempel-Ziv with constant-time random access. In: Moura, E., Crochemore, M. (eds.) SPIRE 2014. LNCS, vol. 8799, pp. 13–17. Springer, Heidelberg (2014)
Fiala, E.R., Greene, D.H.: Data compression with finite windows. Commun. ACM 32(4), 490–505 (1989)
Gog, S., Beller, T., Moffat, A., Petri, M.: From theory to practice: plug and play with succinct data structures. In: Gudmundsson, J., Katajainen, J. (eds.) sea 2014. LNCS, vol. 8504, pp. 326–337. Springer, Heidelberg (2014)
Hoobin, C., Puglisi, S.J., Zobel, J.: Relative Lempel-Ziv factorization for efficient storage and retrieval of web collections. PVLDB 5(3), 265–273 (2011)
Moffat, A., Turpin, A.: Compression and Coding Algorithms. Kluwer, Boston (2002)
Tong, J., Wirth, A., Zobel, J.: Principled dictionary pruning for low-memory corpus compression. In: Proceedings of the SIGIR, pp. 283–292 (2014)
Webber, W., Moffat, A.: In search of reliable retrieval experiments. In: Proceedings of the 10th Australasian Document Computing Symposium, pp. 26–33 (2005)
Williams, H.E., Zobel, J.: Compressing integers for fast file access. Comput. J. 42(3), 193–201 (1999)
Acknowledgments
This work was supported under the Australian Research Council’s Discovery Projects scheme (project DP140103256).We have had access to the code of Hoobin et al. while working on this project, and we thank them for making it available.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Petri, M., Moffat, A., Nagesh, P.C., Wirth, A. (2015). Access Time Tradeoffs in Archive Compression. In: Zuccon, G., Geva, S., Joho, H., Scholer, F., Sun, A., Zhang, P. (eds) Information Retrieval Technology. AIRS 2015. Lecture Notes in Computer Science(), vol 9460. Springer, Cham. https://doi.org/10.1007/978-3-319-28940-3_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-28940-3_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-28939-7
Online ISBN: 978-3-319-28940-3
eBook Packages: Computer ScienceComputer Science (R0)