PLZMA: A Parallel Data Compression Method for Cloud Computing
Recent decades have seen the rapid development of cloud computing, resulting in a huge breakthrough for people to handle the data produced every second and everywhere. Meanwhile, data compression is becoming increasingly important, due to its great potential in benefiting both the network transportation and the storage. Based on the urgent demand in high-efficient compression method with balanced performance in both merits of compression time and ratio, this paper presents PLZMA, a parallel design of LZMA. Process-level and thread-level parallelisms are implemented according to the algorithm of LZMA, which have gained great improvement in compression time, while ensuring a fair compression ratio. Experimental results on real-world application showed that PLZMA is able to achieve more balanced performance over other famous methods. The parallel design is able to achieve a performance speedup of 8\(\times \) over the serial baseline, using 12 threads.
KeywordsData compression Parallel computing LZMA
L. Gan, and J. Xu are supported by the National Natural Science Foundation of China (grant no. 61702297); and the China Postdoctoral Science Foundation (grant no. 2016M601031).
H. Fu, and X. Wang are supported by the National Key Research & Development Plan of China (grant no. 2017YFA0604500), the National Natural Science Foundation of China (grant no. 91530323, 41661134014, 41504040 and 61361120098); and the Tsinghua University Initiative Scientific Research Program (grant no. 20131089356).
G. Yang, and J. Yang are supported by the National Key Research & Development Plan of China (grant no. 2016YFA0602200).
X. Huang is supported by a grant from the State’s Key Project of Research and Development Plan (2016YFB0201100) and the National Natural Science Foundation of China (41375102).
- 2.Motley, C.F.: Telecommunication data compression apparatus and method, April 13 2004. US Patent 6,721,282Google Scholar
- 3.Yan, C., Zhang, Y., Dai, F., Li, L.: Highly parallel framework for HEVC motion estimation on many-core platform. In: Data Compression Conference (DCC), pp. 63–72. IEEE (2013)Google Scholar
- 5.Gan, L., Fu, H., Mencer, O., Luk, W., Yang, G.: Data flow computing in geoscience applications. Adv. Comput. 104, 125–158 (2017)Google Scholar
- 6.Burrows, M., Wheeler, D.J.: A block-sorting lossless data compression algorithm (1994)Google Scholar
- 7.Deutsch, P.L.: Deflate compressed data format specification version 1, 3 (1996)Google Scholar
- 9.Gristwood, T., Fineran, P.C., Everson, L., Salmond, G.P.C.: PigZ, a TetR/AcrR family repressor, modulates secondary metabolism via the expression of a putative four-component resistance-nodulation-cell-division efflux pump, zrpadbc, in serratia sp. atcc 39006. Mol. Microbiol. 69(2), 418–435 (2008)CrossRefGoogle Scholar
- 10.Adiego, J., Fuente, P.D.L.: Merging prediction by partial matching with structural contexts model, p. 522 (2004)Google Scholar
- 11.Berners-Lee, T., Fielding, R., Frystyk, H.: Hypertext transfer protocol-http/1.0. Technical report (1996)Google Scholar
- 12.Woods, J.: PPP deflate protocol (1996)Google Scholar
- 13.Boutell, T.: PNG (portable network graphics) specification version 1.0. (1997)Google Scholar
- 14.Deutsch, P., Gailly, J.-L.: Zlib compressed data format specification version 3.3. Technical report (1996)Google Scholar
- 15.Zhu, W., Xu, J., Ding, W., Shi, Y.: Adaptive LZMA-based coding for screen content. In: Picture Coding Symposium, pp. 373–376 (2013)Google Scholar
- 16.Kärkkäinen, J.: Fast BWT in small space by blockwise suffix sorting. Elsevier Science Publishers Ltd. (2007)Google Scholar
- 17.Culler, M., Dunfield, N.M., Weeks, J.R.: Snappy, a computer program for studying the geometry and topology of 3-manifolds (2017)Google Scholar
- 18.Pavlov, I.: Lzma sdk (software development kit) (2007)Google Scholar
- 19.Reinhold, L.M.: Quicklz websiteGoogle Scholar
- 20.Oberhumer, M.F.X.J.: Lzo-a real-time data compression library (2008). http://www.oberhumer.com/opensource/lzo/
- 22.Lembayung, W.: Comparative analysis on the izarc compression process and 7-zip (2011)Google Scholar