Abstract
This chapter provides a historical perspective of the concept of entropy, Shannon’s reasoning, and the axioms that justify the principles of entropy optimization, namely, the maximum entropy and minimum cross-entropy principles. The mathematical forms of various entropy optimization problems are also discussed along with references to the existing literature. The chapter consists of two sections. Section 1.1 introduces the concept of entropy, and Section 1.2 classifies different entropy optimization problems to be studied in this book.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Baierlein, R., “How Entropy Got Its Name,” American Journal of Physics, Vol. 60, 1992, p. 1151.
Ben-Tal, A., Teboulle, M., and Charnes, A., “The Role of Duality in Optimization Problems involving Entropy Functionals with Applications to Information Theory,” Journal of Optimization Theory and Applications, Vol. 58, 1988, pp. 209–223.
Bregman, L.M., “The Relaxation Method of Finding the Common Point of Convex Sets and its Application to the Solution of Problems in Convex Programming,” U.S.S.R. Computational Mathematics and Mathematical Physics, Vol. 7, 1967, pp. 200–217.
Censor, Y., Elfving, T., and Herman, G.T., “Special Purpose Algorithms for Linearly Constrained Optimization,” Maximum-Entropy and Bayesian Spectral Analysis and Estimation Problems, edited by C.R. Smith and G.J. Erickson, D. Reidel Publishing Company, 1987, pp. 241–254.
Censor, Y., De Pierro, A.R., Elfving, T., Herman, G.T., and Iusem, A.N., “On Maximization of Entropies and Generalization of Bregman’s Method of Convex Programming,” Technical Report MIPG 113, Medical Image Processing Group, Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, PA., 1986.
Clausius, R., “Ueber Verschiedene fur die Anwendung Bequeme Formen der Hauptgleichungen der Mechanischen Warmetheorie,” Annalen der Physik und Chemie, Vol. 125, 1865, pp. 353–400.
Cozzolino, J.M., and Zahner, M.J., “The Maximum Entropy Distribution of the Future Market Price of a Stock,” Operations Research, Vol. 21, 1973, pp. 1200–1211.
Fang, S.-C., Peterson, E.L., and Rajasekera, J.R., “Minimum Cross-Entropy Analysis with Entropy-Type Constraints,” Journal of Computational and Applied Mathematics, Vol. 39, 1992, pp. 165–178.
Fang, S.-C., and Rajasekera, J.R., “Quadratically Constrained Minimum Cross-Entropy Analysis,” Mathematical Programming, Vol. 44, 1989, pp. 85–96.
Fang, S.-C., and Rajasekera, J.R., “Minimum Cross-Entropy Analysis with Convex Constraints,” Information and Computation, Vol. 116, 1995, pp. 304–311.
Fang, S.-C., and Tsao, H.-S.J., “An Unconstrained Convex Programming Approach to Solving Convex Quadratic Programming Problems,” Optimization, Vol. 27, 1993, pp. 235–243.
Fang, S.-C., and Tsao, H.-S.J., “Linear-Constrained Entropy Maximization Problem with Quadratic Cost and its Applications to Transportation Planning Problems,” Transportation Science, Vol. 29, 1995, pp. 353–365.
Guiasu, S., Information Theory with Applications, McGraw-Hill, New York, 1977.
Guiasu, S., “Maximum Entropy Condition in Queueing Theory,” Journal of Operational Research Society, Vol. 37, 1986, pp. 293–301.
Gordon, R., Bender, R., and Herman, G. T., “Algebraic Reconstruction Techniques (ART) for Three Dimensional Electron Microscopy and X-ray Photography,” Journal of Theoretical Biology, Vol. 29, 1970, pp. 471–481.
Jaynes, E.T., “Information Theory and Statistical Mechanics II,” Physics Review, Vol. 108, 1957, pp. 171–190.
Johnson, R.W., and Shore, J.E., “Minimum Cross-entropy Spectral Analysis of Multiple Signals,” IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. ASSP-31, 1983, pp. 574–582.
Kapur, J.N., and Kesavan, H.K., Entropy Optimization Principles with Applications, Academic Press, Boston, 1992.
Kumar, V., Hoshino, K., and Kumar, U., “An Application of the Entropy Maximization Approach in Shopping Area Planning,” International Journal of General Systems, Vol. 16, 1989, pp. 25–42.
Lent, A., “A Convergent Algorithm for Maximum Entropy Image Restoration with a Medical X-Ray Application,” in Image Analysis and Evaluation, edited by Shaw, R., Society of Photographic Scientists and Engineers (SPSE), Washington, D.C., 1977, pp. 249–257.
Rajasekera, J.R., and Fang, S.-C., “Deriving an Unconstrained Convex Program for Linear Programming,” Journal of Optimization Theory and Applications, Vol. 75, 1992, pp. 603–612.
Scott, C.H., and Jefferson T.R., “Entropy Maximizing Models of Residential Location via Geometric Programming,” Geographical Analysis, Vol. 9, 1977, pp. 181–187.
Shannon, C.E., and Weaver, W., The Mathematical Theory of Communication, University of Illinois Press, Urbana, Illinois, 1962.
Shore, J.E., “Minimum Cross-entropy Spectral Analysis,” IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. ASSP-29, 1981, pp. 230–237.
Tribus, M., Rational Descriptions, Decisions, and Designs, Pergamon Press, New York, 1969.
Tribus, M., “An Engineer Looks at Bayes,” in Maximum-Entropy and Bayesian Methods in Science and Engineering, Vol. 1: Foundations, edited by Gary J. Erickson and C. Ray Smith, Kluwer Academic Publishers, Dordrecht, The Netherlands, 1988, pp. 31–52.
Tsao, H.-S.J., Fang, S.-C., and Lee, D.N., “On the Optimal Entropy Analysis,” European Journal of Operational Research, Vol. 59, 1992, pp. 324–329.
Wang, Y., and Lu, W., “Multicriterion Maximum Entropy Image Reconstruction from Projections,” IEEE Transactions on Medical Imaging, Vol. 11, 1992, pp. 70–75.
Wilson, A.G., “A Statistical Theory of Spatial Distribution Models,” Transportation Research, Vol. 1, 1967, pp. 253–269.
Wilson, A.G., Entropy in Urban and Regional Planning, Pion, London, 1970.
Yamada, M., and Rajasekera, J.R., “Portfolio Re-balancing with the Entropy Criteria,” Report No. 310, QUICK Research Institute Corp., Tokyo, 1993.
Zhang, J., and Brockett, P.L., “Quadratically Constrained Information Theoretic Analysis,” SIAM Journal on Applied Mathematics, Vol. 47, 1987, pp. 871–885.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 1997 Springer Science+Business Media New York
About this chapter
Cite this chapter
Fang, SC., Rajasekera, J.R., Tsao, HS.J. (1997). Introduction to Entropy and Entropy Optimization Principles. In: Entropy Optimization and Mathematical Programming. International Series in Operations Research & Management Science, vol 8. Springer, Boston, MA. https://doi.org/10.1007/978-1-4615-6131-6_1
Download citation
DOI: https://doi.org/10.1007/978-1-4615-6131-6_1
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4613-7810-5
Online ISBN: 978-1-4615-6131-6
eBook Packages: Springer Book Archive