Rate Distortion Theory and Data Compression

  • Toby Berger
Part of the International Centre for Mechanical Sciences book series (CISM, volume 166)


In this introductory lecture we present the rudiments of rate distortion theory, the branch of information theory that treats data compression problems. The rate distortion function is defined and a powerful iterative algorithm for calculating it is described. Shannon’s source coding theorems are stated and heuristically discussed.


Mean Square Error Linear Code Data Compression Code Word Average Mutual Information 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    SHANNON, C.E., (1948). “A Mathematical Theory of Communication”, BSTJ, 27, 379–423, 623–656.Google Scholar
  2. [2]
    SHANNON, C.E., (1959) “Coding Theorems for a Discrete Source with a Fidelity Criterion”, IRE Nat’l. Cony. Rec., Part 4, 142–163.Google Scholar
  3. [3]
    BLAHUT, R.E., (1972) “Computation of Channel Capacity and Rate-distortion Functions”, Trans. IEEE, IT-18, 460–473.MathSciNetGoogle Scholar
  4. [4]
    PINKSTER, M.S., (1963) “Sources of Messages”, Problemy Peredecii Informatsii. 14, 5–20.Google Scholar
  5. [5]
    GAL LAGER, R.G., (1968) “Information Theory and Reliable Communication”, Wiley, New York.MATHGoogle Scholar
  6. [6]
    BERGER, T., (1968) “Rate Distortion Theory for Sources with Abstract Alphabets and Memory”, Information and Control, 13, 254–273.CrossRefMATHMathSciNetGoogle Scholar
  7. [7]
    GOBLICK, T.J., Jr. (1969) “A Coding Theorem for Time-Discrete Analog Data Sources”, Trans. IEEE, IT-15, 401–407.MathSciNetGoogle Scholar
  8. [8]
    BERGER, T., (1971) “Rate Distortion Theory. A Mathematical Basis for Data Compression”, Prentice-Hall, Englewood Cliffs, N.Y.Google Scholar
  9. [9]
    GRAY, R.M., and L.D. DAVISSON (1973) “Source Coding Without Ergodicity”, Presented at 1973 IEEE Intern. Symp. on Inform. Theory, Ashkelon, Israel.Google Scholar
  10. [10]
    GOBLICK, T.J., Jr. (1962) “Coding for a Discrete Information Source with a Distortion Measure”, Ph.D.Dissertation, Elec. Eng. Dept. M. I. T. Cambridge, Mass.Google Scholar
  11. [11]
    KOLMOGOROV, A.N., (1956) “On the Shannon Theory of Information Transmission in the Case of Continuous Signals”, Trans. IEEE, IT-2, 102–108.Google Scholar
  12. [12]
    HAMMING, R.W., (1950) “Error Detecting and Error Correcting Codes”, BSTJ, 29, 147–160.MathSciNetGoogle Scholar
  13. [13]
    BERLEKAMP, E.R., (1968) “Algebraic Coding Theory”, McGraw-Hill, N.Y.Google Scholar
  14. [14]
    BERGER, T., and J.A. VAN DER HORST (1973) “BCH Source Codes”, Submitted to IEEE Trans. on Information Theory.Google Scholar
  15. [15]
    POSNER, E.C., (1968) In Man H.B.“Error Correcting Codes”, Wiley, N.Y. Chapter 2.Google Scholar
  16. [16]
    JELINEK, F., (1969) “Tree Encoding of Memoryless Time-Discrete Sources with a Fidelity Criterion”, Trans. IEEE, IT-15, 584–590.MathSciNetGoogle Scholar
  17. [17]
    JELINEK, F., and J.B. ANDERSON (1971) “Instrumentable Tree Encoding and Information Sources”, Trans. IEEE, IT-17, 118–119.Google Scholar
  18. [18]
    ANDERSON, J.B., and F. JELINEK (1973) “A Two-Cycle Algorithm for Source Coding with a Fidelity Criterion”, Trans. IEEE, IT-19, 77–92.MathSciNetGoogle Scholar
  19. [19]
    GALLAGER, R.G., (1973) “Tree Encoding for Symmetric Sources with a Distortion Measure”, Presented at 1973 IEEE Int’l. Symp. on Information Theory, Ashkelon, Israel.Google Scholar
  20. [20]
    VITERBI, A.J., and J.K. OMURA (1974) “Trellis Encoding of Memoryless Discrete-Time Sources with a Fidelity Criterion”, Trans. IEEE, IT-20, 325–332.MathSciNetGoogle Scholar
  21. [21]
    BERGER, T., R.J. DICK and F. JELINEK (1974) “Tree Encoding of Gaussian Sources”, Trans. IEEE, IT-20, 332–336.MathSciNetGoogle Scholar
  22. [22]
    BERGER, T., F. JELINEK and J.K. WOLF (1972) “Permutation Codes for Sources”, Trans. IEEE, IT-18, 160–169Google Scholar
  23. [23]
    SLEPIAN, D., (1965) “Permutation Modulation”, Proc. IEEE, 53, 228–236.CrossRefGoogle Scholar
  24. [24]
    BERGER, T., (1972) “Optimum Quantizers and Permutation Codes”, Trans. IEEE, IT-18, 759–765.Google Scholar
  25. [25]
    BERGER, T., (1973) “Information - Singular Random Processes”, Presented at Third International Symposium on Information Theory, Tallinn, Estonia, USSR.Google Scholar

Copyright information

© Springer-Verlag Wien 1975

Authors and Affiliations

  • Toby Berger
    • 1
  1. 1.School of Electrical EngineeringCornell UniversityIthacaUSA

Personalised recommendations