Foundations: Indifference, Independence & MaxEnt

  • Manfred Schramm
  • Michael Greiner
Conference paper
Part of the Fundamental Theories of Physics book series (FTPH, volume 70)


Through completing an under specified probability model, Maximum Entropy (MaxEnt) supports non-monotonic inferences. Some major aspects of how this is done by MaxEnt can be understood from the background of two principles of rational decision: the concept of Indifference and the concept of Independence. In a formal specification MaxEnt can be viewed as (conservative) extension of these principles; so these principles shed light on the “magical” decisions of MaxEnt. But the other direction is true as well: Since MaxEnt is a “correct” representation of the set of models (Concentration Theorem), it elucidates these two principles (e.g. it can be shown, that the knowledge of independences can be of very different information-theoretic value). These principles and their calculi are not just arbitrary ideas: When extended to work with qualitative constraints which are modelled by probability intervals, each calculus can be successfully applied to V.Lifschitz’s Benchmarks of Non-Monotonic Reasoning and is able to infer some instances of them ([Lifschitz88]). Since MaxEnt is strictly stronger than the combination of the two principles, it yields a powerful tool for decisions in situations of incomplete knowledge. To give an example, a well-known problem of statistical inference (Simpson’s Paradox) will serve as an illustration throughout the paper.


Maximum Entropy Undirected Graph Linear Constraint Propositional Logic Elementary Event 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [Adams75]
    E.W. Adams, “The Logic of Conditionals”, D.Reidel Dordrecht Netherlands, 1975.zbMATHGoogle Scholar
  2. [Bacchus90]
    F. Bacchus, “Lp — A Logic for Statistical Information”, Uncertainty in Artificial Intelligence 5, pp. 3–14, Elsevier Science, ed.: M. Henrion, R.D. Shachter, L.N. Kanal, J.F. Lemmer, 1990.Google Scholar
  3. [Bacchus94]
    F. Bacchus, A.J. Grove, J.Y. Halpern, D. Koller, “From Statistical Knowledge Bases to Degrees of Belief”, Technical Report(available via ftp at logos.uwaterloo. ca: /pub/bacchus), 1994.Google Scholar
  4. [Blyth73]
    C. Blyth, “Simpson’s Paradox und mutually favourable Events”, Journal of the American Statistical Association, Vol. 68, p. 746, 1973.MathSciNetzbMATHCrossRefGoogle Scholar
  5. [Cheeseman88]
    P. Cheeseman, “An Inquiry into Computer Understanding”, Computational Intelligence, Vol. 4, pp. 58 – 66, 1988.CrossRefGoogle Scholar
  6. [Cox79]
    R.T. Cox, “Of Inference and Inquiry — An Essay in Inductive Logic”, in: The Maximum Entropy Formalism, MIT Press, ed.: Levine & Tribus, pp. 119–167, 1979.Google Scholar
  7. [Howson93]
    C. Howson, P. Urbach. Urbach,“Scientific Reasoning: The Bayesian Approach”, 2nd Edition, Open Court, 1993.Google Scholar
  8. [Jaynes78]
    E.T. Jaynes, “Where do we stand on Maximum Entropy?”, 1978, in: E.T. Jaynes: Papers on Probability, Statistics and Statistical Physics, pp. 210 – 314, Kluwer Academic Publishers, ed.: R.D. Rosenkrantz, 1989.Google Scholar
  9. [Jaynes82]
    E.T. Jaynes, “On the Rationale of Maximum-Entropy Methods”, Proceedings of the IEEE, Vol. 70, No. 9, pp. 939 – 952, 1982.CrossRefGoogle Scholar
  10. [Lewis59]
    P.M. Lewis, “Approximating Probability Distributions to Reduce Storage Requirements”, Information and Control 2, pp. 214–225, 1959.MathSciNetzbMATHCrossRefGoogle Scholar
  11. [Lifschitz88]
    V. Lifschitz, “Benchmark Problems for Formal nonmonotonic Reasoning”, Lecture Notes in Artificial Intelligence Non-Monotonie Reasoning, Vol. 346, pp. 202–219, ed.: Reinfrank et al., 1988.Google Scholar
  12. [Neapolitan90]
    R.E. Neapolitan, “Probabilistic Reasoning in Expert Systems: Theory and Algorithms”, John Wiley & Sons, 1990.Google Scholar
  13. [Paris89]
    J.B. Paris, A. Vencovska, “On the Applicability of Maximum Entropy to Inexact Reasoning”, Int. Journal of approximate reasoning, Vol. 3, pp. 1 – 34, 1989.MathSciNetzbMATHCrossRefGoogle Scholar
  14. [Paris90]
    J.B. Paris, A. Vencovska, “A note on the Inevitability of Maximum Entropy”, Int. Journal of approximate reasoning, Vol. 4, pp. 183–223, 1990.MathSciNetzbMATHCrossRefGoogle Scholar
  15. [Pearl88]
    J. Pearl, “Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference”, Kaufmann, San Mateo, CA, 1988.Google Scholar
  16. [Shore80]
    J.E. Shore, R.W. Johnson, “Axiomatic Derivation of the Principle of Maximum Entropy and the Principle of Minimum Cross Entropy”, IEEE Transactions on Information Theory, Vol. IT-26, No. 1, pp. 26– 37, 1980.MathSciNetCrossRefGoogle Scholar
  17. [Skilling88]
    J. Skilling, “The Axioms of Maximum Entropy, Maximum-Entropy and Bayesian Methods in Science and Engineering, Vol. 1 – Foundations”, Kluwer Academic, ed.: G.J. Erickson, C.R. Smith, Seattle Univ. Washington, 1988.Google Scholar

Copyright information

© Kluwer Academic Publishers 1996

Authors and Affiliations

  • Manfred Schramm
    • 1
  • Michael Greiner
    • 1
  1. 1.Institut für InformatikTechnischen Universität MünchenGermany

Personalised recommendations