Distributed Estimation of Mixture Models

Abstract

The contribution deals with sequential distributed estimation of global parameters of normal mixture models, namely mixing probabilities and component means and covariances. The network of cooperating agents is represented by a directed or undirected graph, consisting of vertices taking observations, incorporating them into own statistical knowledge about the inferred parameters and sharing the observations and the posterior knowledge with other vertices. The aim to propose a computationally cheap online estimation algorithm naturally disqualifies the popular (sequential) Monte Carlo methods for the associated high computational burden, as well as the expectation-maximization (EM) algorithms for their difficulties with online settings requiring data batching or stochastic approximations. Instead, we proceed with the quasi-Bayesian approach, allowing sequential analytical incorporation of the (shared) observations into the normal inverse-Wishart conjugate priors. The posterior distributions are subsequently merged using the Kullback–Leibler optimal procedure.

Appendix

Below we give several useful definitions and lemmas regarding the Bayesian estimation of exponential family distributions with conjugate priors [9]. The proofs are trivial. Their application to the normal model and normal inverse-gamma prior used in Sect. 3.4 follows.

Definition 1 (Exponential family distributions and conjugate priors).

Any distribution of a random variable y parameterized by θ with the probability density function of the form

$$\displaystyle{ p(y\vert \theta ) = f(y)g(\theta )\exp \left \{\eta (\theta )^{\intercal }T(y)\right \}, }$$

where f, g, η, and T are known functions, is called an exponential family distribution. η ≡ η(θ) is its natural parameter, T(y) is the (dimension preserving) sufficient statistic. The form is not unique.

Any prior distribution for θ is said to be conjugate to p(y | θ), if it can be written in the form

$$\displaystyle{ \pi (\theta \vert \xi,\nu ) = q(\xi,\nu )g(\theta )^{\nu }\exp \left \{\eta (\theta )^{\intercal }\xi \right \}, }$$

where q is a known function and the hyperparameters ν ∈ ℝ ⁺ and ξ is of the same shape as T(y).

Lemma 1 (Bayesian update with conjugate priors).

Bayes’ theorem

$$\displaystyle{ \pi (\theta \vert \xi _{t},\nu _{t}) \propto p(y_{t}\vert \theta )\pi (\theta \vert \xi _{t-1},\nu _{t-1}) }$$

yields the posterior hyperparameters as follows:

$$\displaystyle{ \xi _{t} =\xi _{t-1} + T(y_{t})\qquad \text{and}\qquad \nu _{t} =\nu _{t-1} + 1. }$$

Lemma 2.

The normal model

$$\displaystyle{ p(y_{t}\vert \mu,\sigma ^{2}) = \frac{(\sigma ^{2})^{-\frac{1} {2} }} {\sqrt{2\pi }} \exp \left \{-\frac{1} {2\sigma ^{2}}(y_{t}-\mu )^{2}\right \} }$$

where μ,σ ² are unknown can be written in the exponential family form with

$$\displaystyle\begin{array}{rcl} \eta = \left ( \frac{\mu } {\sigma ^{2}}, \frac{-1} {2\sigma ^{2}}, \frac{-\mu ^{2}} {2\sigma ^{2}} \right )^{\intercal },\qquad T(y_{ t}) = \left (y,y^{2},1\right )^{\intercal },\qquad g(\eta ) = \left (\sigma ^{2}\right )^{-\frac{1} {2} }.& & {}\\ \end{array}$$

Lemma 3.

The normal inverse-gamma prior distribution for μ,σ ² with the (nonnatural) real scalar hyperparameters m, and positive s,a,b, having the density

$$\displaystyle{ p(\mu,\sigma ^{2}\vert m,s,a,b) = \frac{b^{a}(\sigma ^{2})^{a+1+\frac{1} {2} }} {\sqrt{2\pi }s\varGamma (a)} \exp \left \{-\frac{1} {\sigma ^{2}} \left [b + \frac{1} {2s}(m-\mu )^{2}\right ]\right \} }$$

can be written in the prior-conjugate form with

$$\displaystyle{ \xi _{t} = \left (\frac{m} {s}, \frac{m^{2}} {s} + 2b, \frac{1} {s}\right )^{\intercal }. }$$

Lemma 4.

The Bayesian update of the normal inverse-gamma prior following the previous lemma coincides with the ‘ordinary’ well-known update of the original hyperparameters,

$$\displaystyle{ \begin{array}{ll} s_{t}^{-1} & = s_{t-1}^{-1} + 1, \\ m_{t} & = s_{t}\left (\frac{m_{t-1}} {s_{t-1}} + y_{t}\right ),\end{array} \qquad \qquad \begin{array}{ll} a_{t}& = a_{t-1} + \frac{1} {2}, \\ b_{t} & = b_{t-1} + \frac{1} {2}\left (\frac{m_{t-1}^{2}} {s_{t-1}} -\frac{m_{t}^{2}} {s_{t}} + y_{t}^{2}\right ).\end{array} }$$

Definition 2 (Kullback–Leibler divergence).

Let f(x), g(x) be two probability density functions of a random variable x, f absolutely continuous with respect to g. The Kullback–Leibler divergence is the nonnegative functional

$$\displaystyle{ \mathrm{D}(f\vert \vert g) = \mathbb{E}_{f}\left [\log \frac{f(x)} {g(x)}\right ] =\int f(x)\log \frac{f(x)} {g(x)}dx, }$$

(3.12)

where the integration domain is the support of f. The Kullback–Leibler divergence is a premetric; it is zero if f = g almost everywhere, it does not satisfy the triangle inequality nor is it symmetric.