EDA-78: A Novel Error Detection Algorithm for Lempel-Ziv-78 Compressed Data

  • Beom Kwon
  • Myongsik Gong
  • Sanghoon LeeEmail author


This paper presents an error detection algorithm for Lempel-Ziv-78 (LZ78) compressed data. LZ78 data compression involves dictionary coding and aims to compress the original data without loss. For detecting bit errors in compressed data, methods such as parity bit and Hamming code approaches have been applied. However, in these conventional methods, the insertion of additional bits is required for error detection, increasing the data redundancy. For error detection of LZ78 compressed data, we introduced four unique properties of LZ78 compressed data and developed an algorithm that detects bit errors in LZ78 compressed data according to these properties, without the insertion of additional bits. The proposed algorithm, which is called EDA-78 (Error Detection Algorithm for LZ78 compressed data), achieved an error detection rate of nearly 100% in the case of six or more bit errors. However, when the number of bit errors was smaller than six, the error detection rate was degraded. To overcome this drawback, we employed parity bits, significantly improving the error detection rate for a small number of bits.


Lossless compression Dictionary coding Lempel-Ziv-78 Error detection Bitstream error 



This work was supported by the research fund of Signal Intelligence Research Center supervised by Defense Acquisition Program Administration and Agency for Defense Development of Korea.


  1. 1.
    Kwon, B., Gong, M., Huh, J., & Lee, S. (2018). Identification and restoration of LZ77 compressed data using a machine learning approach. In Proceeding Asia-Pacific signal and information processing association annual summit and conference (APSIPA ASC), pp. 1787–1790.Google Scholar
  2. 2.
    Song, H., Kwon, B., Lee, S., & Lee, S. (2019). Dictionary based compression type classification using a CNN architecture. In Proceeding Asia-Pacific signal and information processing association annual summit and conference (APSIPA ASC), pp. 1–5.Google Scholar
  3. 3.
    Kostina, V. (2017). Data compression with low distortion and finite blocklength. IEEE Transactions on Information Theory, 63(7), 4268–4285.MathSciNetCrossRefGoogle Scholar
  4. 4.
    Movassagh, M., & Kabal, P. (2016). Scalable audio coding using trellis-based optimized joint entropy coding and quantization. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(12), 2288–2300.CrossRefGoogle Scholar
  5. 5.
    Lee, H., & Lee, S. (2006). Visual entropy gain for wavelet image coding. IEEE Signal Processing Letters, 13(9), 553–556.CrossRefGoogle Scholar
  6. 6.
    Ha, H., Oh, T., & Lee, S. (2009). Macroblock-based frequency selective weighting for visual scalable video coding of H. 264/AVC. IEEE Transactions on Broadcasting, 55(3), 559–568.CrossRefGoogle Scholar
  7. 7.
    Schmidhuber, J., & Heil, S. (1996). Sequential neural text compression. IEEE Transactions on Neural Networks, 7(1), 142–146.CrossRefGoogle Scholar
  8. 8.
    Moffat, A., Zobel, J., & Sharman, N. (1997). Text compression for dynamic document databases. IEEE Transactions on Knowledge and Data Engineering, 9(2), 302–313.CrossRefGoogle Scholar
  9. 9.
    Abel, J., & Teahan, W. (2005). Universal text preprocessing for data compression. IEEE Transactions on Computers, 54(5), 497–507.CrossRefGoogle Scholar
  10. 10.
    Ziv, J., & Lempel, A. (1978). Compression of individual sequences via variable-rate coding. IEEE Transactions on Information Theory, 24(5), 530–536.MathSciNetCrossRefGoogle Scholar
  11. 11.
    Chi, C. H. (1998). Study on multi-lingual LZ77 and LZ78 text compression. In Proceeding IEEE data compression conference (DCC), p. 533.Google Scholar
  12. 12.
    Bannai, H., Inenaga, S., & Takeda, M. (2012). Efficient LZ78 factorization of grammar compressed text. In Proceeding international symposium on string processing and information retrieval (SPIRE), pp. 86–98.Google Scholar
  13. 13.
    Li, M., & Sleep, R. (2005). Genre classification via an LZ78-based string kernel. In Proceeding International Society for Music Information Retrieval (ISMIR), pp. 252–259.Google Scholar
  14. 14.
    Bannai, H., Inenaga, S., & Takeda, M. (2006). Image classification via LZ78 based string kernel: A comparative study. In Proceeding Pacific-Asia conference on knowledge discovery and data mining (PAKDD), pp. 704–712.Google Scholar
  15. 15.
    Moghaddam, A., & Kabir, E. (2009). Dynamic and memory efficient web page prediction model using LZ78 and LZW algorithms. In Proceeding IEEE international CSI computer conference (CSICC), pp. 676–681.Google Scholar
  16. 16.
    Alam, M. R., Reaz, M. B. I., & Ali, M. M. (2012). SPEED: An inhabitant activity prediction algorithm for smart homes. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 42(4), 985–990.CrossRefGoogle Scholar
  17. 17.
    Das, S. K., & Cook, D. J. (2004). Health monitoring in an agent-based smart home by activity prediction. In Proceeding international conference on smart homes and health telematics (ICOST), pp. 3–14.Google Scholar
  18. 18.
    Zhang, H., & Dong, Y. N. (2006). Mobility prediction model based link stability metric for wireless ad hoc networks. In Proceeding IEEE international conference on wireless communications, networking and mobile computing (WiCOM), pp. 1–4.Google Scholar
  19. 19.
    Kwon, B., Park, J., & Lee, S. (2015). Virtual MIMO broadcasting transceiver design for multi-hop relay networks. Digital Signal Processing, 46, 97–107.MathSciNetCrossRefGoogle Scholar
  20. 20.
    Kwon, B., Park, J., & Lee, S. (2015). A target position decision algorithm based on analysis of path departure for an autonomous path keeping system. Wireless Personal Communications, 83(3), 1843–1865.CrossRefGoogle Scholar
  21. 21.
    Kwon, B., Kim, S., Lee, H., & Lee, S. (2015). A downlink power control algorithm for long-term energy efficiency of small cell network. Wireless Networks, 21(7), 2223–223.CrossRefGoogle Scholar
  22. 22.
    Kwon, B., Kim, S., Jeon, D., & Lee, S. (2016). Iterative interference cancellation and channel estimation in evolved multimedia broadcast multicast system using filter-bank multicarrier-quadrature amplitude modulation. IEEE Transactions on Broadcasting, 62(4), 864–875.CrossRefGoogle Scholar
  23. 23.
    Kwon, B., Kim, S., & Lee, S. (2017). Scattered reference symbol-based channel estimation and equalization for FBMC-QAM systems. IEEE Transactions on Communications, 65(8), 3522–3537.Google Scholar
  24. 24.
    Kwon, B., & Lee, S. (2017). Effective interference nulling virtual MIMO broadcasting transceiver for multiple relaying. IEEE Access, 5, 20695–20706.CrossRefGoogle Scholar
  25. 25.
    Kwon, B., Kim, J., Lee, K., Lee, Y., Park, S., & Lee, S. (2017). Implementation of a virtual training simulator based on 360° multi-view human action recognition. IEEE Access, 5, 12496–12511.CrossRefGoogle Scholar
  26. 26.
    Kwon, B., & Lee, S. (2018). Cross-antenna interference cancellation and channel estimation for MISO-FBMC/QAM-based eMBMS. Wireless Networks, 24(8), 3281–3293.CrossRefGoogle Scholar
  27. 27.
    Hamming, R. W. (1950). Error detecting and error correcting codes. Bell Labs Technical Journal, 29(2), 147–160.MathSciNetCrossRefGoogle Scholar
  28. 28.
    Ziv, J., & Lempel, A. (1997). A universal algorithm for sequential data compression. IEEE Transactions on Information Theory, 23(3), 337–343.MathSciNetCrossRefGoogle Scholar
  29. 29.
    Welch, T. A. (1984). A technique for high-performance data compression. Computer, 6(17), 8–19.CrossRefGoogle Scholar
  30. 30.
    Storer, J. A., & Szymanski, T. G. (1982). Data compression via textual substitution. Journal of the ACM, 29(4), 928–951.MathSciNetCrossRefGoogle Scholar
  31. 31.
    Lonardi, S., Szpankowski, W., & Ward, M. D. (2007). Error resilient LZ’77 data compression: Algorithms, analysis, and experiments. IEEE Transactions on Information Theory, 53(5), 1799–1813.MathSciNetCrossRefGoogle Scholar
  32. 32.
    Korosec, T., & Tomazic, S. (2012). An adaptive-parity error-resilient LZ’77 compression algorithm. Informacije MIDEM—Journal of Microelectronics, Electronic Components and Materials, 42(1), 29–35.Google Scholar
  33. 33.
    Wu, Y., Lonardi, S., & Szpankowski, W. (2006). Error-resilient LZW data compression. In Proceeding IEEE data compression conference (DCC), pp. 193–202.Google Scholar
  34. 34.
    Kitakami, M., & Kawasaki, T. (2009). Burst error recovery method for LZSS coding. IEICE Transactions on Information and Systems, 92(12), 2439–2444.CrossRefGoogle Scholar
  35. 35.
    Kwon, B., Gong, M., & Lee, S. (2017). Novel error detection algorithm for LZSS compressed data. IEEE Access, 5, 8940–8947.CrossRefGoogle Scholar
  36. 36.
    Bell, T., Witten, I. H., & Cleary, J. G. (1989). Modeling for text compression. ACM Computing Surveys, 21(4), 557–591.CrossRefGoogle Scholar
  37. 37.
    Arnold, R., & Bell, T. (1997). A corpus for the evaluation of lossless compression algorithms. In Proceeding IEEE data compression conference (DCC), pp. 201–210.Google Scholar
  38. 38.
    Kwon, B., & Lee, S. (2019). Error detection algorithm for Lempel-Ziv-77 compressed data. Jornal of Communications and Networks, 21(2), 100–112.CrossRefGoogle Scholar
  39. 39.
    Mstafa, R. J., & Elleithy, K. M. (2014). A highly secure video steganography using Hamming code (7, 4). In Proceeding IEEE long island systems. Applications and technology conference (LISAT), pp. 1–6.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Electrical and Electronic EngineeringYonsei UniversitySeoulSouth Korea

Personalised recommendations