A Generalized Iterative Scaling Algorithm for Maximum Entropy Model Computations Respecting Probabilistic Independencies

Abstract

Maximum entropy distributions serve as favorable models for commonsense reasoning based on probabilistic conditional knowledge bases. Computing these distributions requires solving high-dimensional convex optimization problems, especially if the conditionals are composed of first-order formulas. In this paper, we propose a highly optimized variant of generalized iterative scaling for computing maximum entropy distributions. As a novel feature, our improved algorithm is able to take probabilistic independencies into account that are established by the principle of maximum entropy. This allows for exploiting the logical information given by the knowledge base, represented as weighted conditional impact systems, in a very condensed way.

Proofs of Results

Proposition 1

Let $\mathcal {R}$ be a consistent knowledge base, and let $\{\mathcal {G}_1,\ldots ,\mathcal {G}_k\}$ be a syntax partition for $\mathcal {R}$. For all $\omega \in \varOmega$,

$$ \mathcal {P}^\mathsf {ME}_\mathcal {R}(\omega )=\prod _{j=1}^k \mathcal {P}^\mathsf {ME}_\mathcal {R}(\omega _{\mathcal {G}_j}). $$

Proof

We give a proof for those cases in which the representation (5) of $\mathcal {P}^\mathsf {ME}_\mathcal {R}$ exists. The normalizing constant can be written as $\alpha _0=\sum _{\omega \in \varOmega }\prod _{i=1}^m \alpha _i^{f_i(\omega )}$ where $f_{X}(C)$ abbreviates $(1-p_i)\cdot \mathsf {ver}_{X}(C)-p_i\cdot \mathsf {fal}_X(C)$ for any ground formula $C\in \mathsf {FOL}$. Further, let $\mathfrak {R}=\{R^1_G,\ldots ,R^k_G\}$ be a $\{\mathcal {G}_1,\ldots ,\mathcal {G}_k\}$-respecting decomposition of $\mathcal {R}$ with $R^j_G=\{R^j_1,\ldots ,R^j_n\}$ for $j=1,\ldots ,k$. Then, $\alpha _0=\prod _{j=1}^k \alpha _0^j$ holds where $\alpha _0^j=\sum _{\omega _j\in \varOmega _{\mathcal {G}_j}}\prod _{i=1}^m \alpha _i^{f_i(\omega _j)}$. For $\omega \in \varOmega \setminus \varOmega ^0$, it follows that

$$\begin{aligned} \mathcal {P}^\mathsf {ME}_\mathcal {R}(\omega ) =&\alpha _0 \prod _{i=1}^m \alpha _i^{f_i(\omega )} = \alpha _0 \prod _{i=1}^m \prod _{j=1}^k \alpha _i^{f_{R^j_i}(\omega _{\mathcal {G}_j})} \\ =&\prod _{j=1}^k \Big [\left( \alpha _0^j \prod _{i=1}^m \alpha _i^{f_{R^j_i}(\omega _{\mathcal {G}_j})}\right) \cdot \prod _{l\ne j} \underbrace{\left( \sum _{\omega '_l\in \varOmega _{\mathcal {G}_l}} \alpha _0^l \prod _{i=1}^m \alpha _i^{f_{R^l_i}(\omega '_l)} \right) }_{=1}\Big ] \\ =&\prod _{j=1}^k \Big ( \sum _{\begin{array}{c} \omega '\in \varOmega \\ \omega '\,{\models }\,\omega _{\mathcal {G}_j} \end{array}} \alpha _0 \prod _{i=1}^m \prod _{l=1}^k \alpha _i^{f_{R^l_i}(\omega _{\mathcal {G}_l})}\Big ) = \prod _{j=1}^k \Big ( \sum _{\begin{array}{c} \omega '\in \varOmega \\ \omega '\,{\models }\,\omega _{\mathcal {G}_j} \end{array}} \alpha _0 \prod _{i=1}^m \alpha _i^ {f_i(\omega ')} \Big ) \\ =&\prod _{j=1}^k \mathcal {P}^\mathsf {ME}_\mathcal {R}(\omega _{\mathcal {G}_j}). \end{aligned}$$

If $\omega \in \varOmega ^0$, there is a deterministic conditional $r=(B|A)[p]\in \mathcal {R}$ and an index $l\in \{1,\ldots ,k\}$ such that $\mathsf {ver}_{\mathsf {Grnd}(r)}(\omega _{\mathcal {G}_l})>0$ if $p=0$ and $\mathsf {fal}_{\mathsf {Grnd}(r)}(\omega _{\mathcal {G}_l})>0$ if $p=1$. As a consequence, every $\omega '$ with $\omega '\,{\models }\,\omega _{\mathcal {G}_l}$ is a null-world, and

$$ \prod _{j=1}^k \mathcal {P}^\mathsf {ME}_\mathcal {R}(\omega _{\mathcal {G}_j})= \left( \sum _{\omega '\,{\models }\,\omega _{\mathcal {G}_l}} \mathcal {P}^\mathsf {ME}_\mathcal {R}(\omega ')\right) \cdot \prod _{j\ne l} \mathcal {P}^\mathsf {ME}_{\mathcal {R}}(\omega _{\mathcal {G}_j})=0\cdot \prod _{j\ne l} \mathcal {P}^\mathsf {ME}_{\mathcal {R}}(\omega _{\mathcal {G}_j})=0 $$

as required. $\square$

Proposition 2

Let $\mathcal {R}$ be a knowledge base, let $\mathfrak {G}$ be a syntax partition for $\mathcal {R}$, and let $\mathfrak {R}$ be a $\mathfrak {G}$-respecting decomposition of $\mathcal {R}$ as described above. If $\omega \in \varOmega$ is not a null-world, then

$$\begin{aligned} {\varvec{\gamma }}_{\mathcal {R}_G}(\omega )=\big (( \sum _{j=1}^k (\gamma _{R^j_i}(\omega _{\mathcal {G}_j})_i)_1, \sum _{j=1}^k (\gamma _{R^j_i}(\omega _{\mathcal {G}_j})_i)_2 )\big )_{i=1,\ldots ,m}. \end{aligned}$$

If $\omega$ is a null-world, then ${\varvec{\gamma }}_{\mathcal {R}^j_G}(\omega _{\mathcal {G}_j})$ is undefined for at least one $j\in \{1,\ldots ,k\}$.

Proof

Let $\omega \in \varOmega \setminus \varOmega ^0$. By definition, $\varvec{\gamma }_{\mathcal {R}_\mathcal {G}}(\omega )=((\mathsf {ver}_i(\omega ),\mathsf {fal}_i(\omega )))_{i=1,\ldots ,m}$. Since $\mathfrak {R}$ is a $\mathfrak {G}$-respecting decomposition of $\mathcal {R}$, $\mathsf {ver}_{i}(\omega )=\sum _{j=1}^k \mathsf {ver}_{R^j_i}(\omega _{\mathcal {G}_j})$ as well as $\mathsf {fal}_{i}(\omega )=\sum _{j=1}^k \mathsf {fal}_{R^j_i}(\omega _{\mathcal {G}_j})$ hold for $i=1,\ldots ,n$, and hence, in particular, this holds for $i=1,\ldots ,m$ (since $m\le n$). By applying the definition of $\varvec{\gamma }_{R^j_i}(\omega _{\mathcal {G}_j})$, the proposition follows. As syntax partitions also take deterministic conditionals into account, the statement concerning null-worlds follows immediately. $\square$