Universal Source Coding
The basic purpose of data compression is to massage a data stream to reduce the average bit rate required for transmission or storage by removing unwanted redundancy and/or unnecessary precision. A mathematical formulation of data compression providing figures of merit and bounds on optimal performance was developed by Shannon [1,2] both for the case where a perfect compressed reproduction is required and for the case where a certain specified average distortion is allowable. Unfortunately, however, Shannon’s probabilistic approach requires advance precise knowledge of the statistical description of the process to be compressed - a demand rarely met in practice. The coding theorems only apply, or are meaningful, when the source is stationary and ergodic.
We here present a tutorial description of numerous recent approaches and results generalizing the Shannon approach to unknown statistical environments. Simple examples and empirical results are given to illustrate the essential ideas.
KeywordsStationary Source Average Mutual Information Source Block Message Block Average Distortion
Unable to display preview. Download preview PDF.
- Coding Theorems for a Discrete Source with a Fidelity Criterion“, in IRE Nat.Conv.Rec., pt. 4, pp. 142–163, 1959.Google Scholar
- GRAY, R.M., and DAVISSON, L.D., “Source Coding Theorems without the Ergodic Assumption”, IEEE Trans. IT, July 1974.Google Scholar
- GRAY, R.M., and DAVISSON, L.D., “The Ergodic Decomposition of Discrete Stationary Sources”, IEEE Trans. IT, September 1974.Google Scholar
- GRAY, R.M., NEUHOFF, D., and SHIELDS, P., “A Generalization of Ornstein’s d Metric with Applications to Information Theory”, Annals of Probability (to be published).Google Scholar
- NEUHOFF, D., GRAY, R.M. and DAVISSON, L.D., “Fixed Rate Universal Source Coding with a Fidelity Criterion”, submitted to IEEE Trans. IT.Google Scholar
- PURSLEY, M.B., “Coding Theorems for Non-Ergodic Sources and Sources with Unknown Parameters”, USC Technical Report, February 1974.Google Scholar
- GALLAGER, R.G., “Information Theory and Reliable Communication”, New York, Wiley, 1968, ch. 9.Google Scholar
- BERGER, T., “Rate Distortion Theory: A Mathematical Basis for Data Compression”, Englewood Cliffs, New Jersey, Prentice-Hall.Google Scholar
- NEUHOFF, D., Ph.D. Research, Stanford University, 1973.Google Scholar
- GRAY, R.M., and DAVISSON, L.D. “A Mathematical Theory of Data Compression (?)”, USCEE Report, September 1974.Google Scholar