Abstract
The principles of maximum entropy (ME) and minimum relative entropy (MRE) are information-theoretic methods for estimating unknown probability distributions based on information about their expected values [Elsasser, 1937; Jaynes, 1957; Kullback, 1959; Csiszár, 1975; Shore and Johnson, 1980]. MRE differs from ME by taking into account an initial estimate of the unknown distribution. Both ME and MRE are used in a variety of successful applications, but widespread use has been hindered by some unresolved issues. One issue is the choice of state space in which to express problems and their solutions. Another issue is the choice and interpretation of initial estimates for MRE. A third issue is the identity of the “correct” expression to use for entropy when ME is applied to spectrum analysis and image enhancement. This paper shows that these issues are interrelated, and presents results that help to resolve them.
The full version of this paper has been submitted to the IEEE Transactions on Information Theory.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Abies, J. G. (1974), “Maximum entropy spectral analysis” Astron. Astro-phys. Supp. 15, pp. 383–393.
Burg, J. P. (1967), “Maximum Entropy Spectral Analysis,” presented at the 37th Annual Meeting, Society of Exploration Geophysicists, Oklahoma City, Okla.
Burg, J. P. (1975), “Maximum Entropy Spectral Analysis,” Ph.D. Dissertation, Stanford University, Stanford, Calif. (University Microfilms No. AAD75–25, 499 ).
Csiszar, I. (1975), “I-divergence geometry of probability distributions and minimization problems,” Ann. Prob. 3, pp. 146–158.
Elsasser, W. M. (1937), “On quantum measurements and the role of the uncertainty relations in statistical mechanics,” Phys. Rev. 52, pp. 987–999.
Frieden, B. R. (1972), “Restoring with maximum likelihood and maximum entropy,” J. Opt. Soc. Am. 62, pp. 511–518.
Frieden, B. R. (1980), “Statistical models for the image restoration problem,” Comp. Graphics Image Processing 12, pp. 40–59.
Gordon, R., and G. T. Herman (1971), “Reconstruction of pictures from their projections,” Quart. Bull. Center for Theor. Biol. 4, pp. 71–151.
Gray, R. M., A. H. Gray, Jr., G. Rebolledo, and J. E. Shore (1981), “Rate-distortion speech coding with a minimum discrimination information distortion measure,” IEEE Trans. Inf. Theory IT-27, pp. 708–721.
Gull, S. F., and G. J. Daniell (1978), “Image reconstruction from incomplete and noisy data,” Nature 272, pp. 686–690.
Jaynes, E. T. (1957), “Information theory and statistical mechanics I,” Phys. Rev. 106, pp. 620–630.
Kay, S. M., and S. L. Marple, Jr. (1981), “Spectrum analysis—a modern perspective,” Proc. IEEE 69, pp. 1380–1419.
Kullback, S. (1959), Information Theory and Statistics, Wiley, New York ( Dover, New York, 1968 ).
Lacoss, R. T. (1971), “Data adaptive spectral analysis methods,” Geophys. 36, pp. 661–675.
Markel, J. D, and A. H. Gray, Jr. (1976), Linear Prediction of Speech, Springer-Verlag, New York.
Nadeu, C., E. Sanvicente, and M. Bertran (1981), “A new algorithm for spectral estimation,” Proc. International Conf. on DSP, pp. 463–470.
Ortigueira, M. D., R. Garcia-Gomez, and J. M. Tribolet (1981), “An iterative algorithm for maximum flatness spectral analysis,” Proc. International Conf. on DSP, pp. 810–818.
Papoulis, A. (1981), “Maximum entropy and spectral estimation: a review,” IEEE Trans. Acoust., Speech, Sig. Processing ASSP-29, pp. 1176–1186.
Robinson, E. A. (1982), “A historical perspective of spectrum estimation,” Proc. IEEE 76, pp. 885–907.
Shore, J. E. (1981), “Minimum cross-entropy spectral analysis,” IEEE Trans. Acoust, Speech, Sig. Processing ASSP-29, pp. 230–237.
Shore, J. E., and R. W. Johnson (1980), “Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy,” IEEE Trans. Inf. Theory IT-26, pp. 26–37. (See also comments and corrections in IEEE Trans. Inf. Theory IT-29, p. 942, 1983.)
Shore, J. E., and R. W. Johnson (1984), “Which is the better entropy expression for speech processing: -S log S or log S?” IEEE Trans. Acoust., Speech, Sig. Processing ASSP-32, pp. 129–137.
Skilling, J., and S. F. Gull (1985), “Algorithms and applications,” in Maximum-Entropy and Bayesian Methods in Inverse Problems, C. Ray Smith and W. T. Grandy, eds., D. Reidel, Dordrecht, pp. 83–132.
Smylie, D. E., G. K. C. Clarke, and T. J. Ulrych (1973), “Analysis of irregularities in the earth’s rotation,” Meth. Comp. Phys. 13, pp. 391–431.
Ulrych, T. J., and T. N. Bishop (1975), “Maximum entropy spectral analysis and autoregressive decomposition,” Rev. Geophys. Space Phys. 43, pp. 183–200.
VanDenBos, A. (1971), “Alternative interpretation of maximum entropy spectral analysis,” IEEE Trans. Inf. Theory IT-17, pp. 493–494.
Wernecke, S. J. (1977), “Two-dimensional maximum entropy reconstruction of radio brightness,” Radio Sci. 12, pp. 831–844.
Wernecke, S. J., and L. D’Addario (1977), “Maximum entropy image reconstruction,” IEEE Trans. Computers C-26, pp. 351–364.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1987 D. Reidel Publishing Company
About this paper
Cite this paper
Shore, J.E. (1987). State Spaces and Initial Estimates in Minimum Relative-Entropy Inversion with Application to Spectrum Analysis and Image Enhancement. In: Smith, C.R., Erickson, G.J. (eds) Maximum-Entropy and Bayesian Spectral Analysis and Estimation Problems. Fundamental Theories of Physics, vol 21. Springer, Dordrecht. https://doi.org/10.1007/978-94-009-3961-5_3
Download citation
DOI: https://doi.org/10.1007/978-94-009-3961-5_3
Publisher Name: Springer, Dordrecht
Print ISBN: 978-94-010-8257-0
Online ISBN: 978-94-009-3961-5
eBook Packages: Springer Book Archive