Characterizing the Principle of Minimum Cross-Entropy

Abstract

Probability theory provides a sound and convenient machinery to be used for knowledge representation and automated reasoning (see, for instance, [Cox46, DPT90, DP91b, LS88, Pea88, TGK92]). In many cases, only relatively few relationships between relevant variables are known, due to incomplete information. Or maybe, an abstractional representation is intended, incorporating only fundamental relationships. In both cases, the knowledge explicitly stated is not sufficient to determine uniquely a probability distribution. One way to cope with this indetermination is to calculate upper and lower bounds for probabilities (cf. [Nil86, TGK92, DPT90]). This method, however, brings about two problems: Sometimes the inferred bounds are quite bad, and one has to handle intervals instead of single values.

An alternative way that provides best expectation values for the unknown probabilities and guarantees a logically sound reasoning is to use the principle of maximum entropy resp. the principle of minimum cross entropy to represent all available probabilistic knowledge by a unique distribution (see Section 2.5; cf. [Sho86, Kul68, Jay83a, GHK94]). Here we assume the available knowledge to constitute of a (consistent) set R of conditionals, each equipped with a probability, usually providing only incomplete probabilistic knowledge.

The aim of this chapter is to establish a direct and constructive link between probabilistic conditionals and their suitable representation via distributions, taking prior knowledge into account if necessary.

We develope the following four principles which mark the corner-stones for using quantified conditionals consistently for probabilistic knowledge representation and updating:

• (P1) The principle of conditional preservation: this is to express that prior conditional dependencies shall be preserved “as far as possible” under adaptation;

• (P2) the idea of a functional concept which underlies the adaptation and which allows us to calculate a posterior distribution from prior and new knowledge

• (P3) the principle of logical coherence ¹: posterior distributions shall be used coherently as priors for further inferences

• (P4) the principle of representation invariance: the resulting distribution shall not depend upon the actual probabilistic representation of the new information.