Finite Inverted Beta-Liouville Mixture Models with Variational Component Splitting

Abstract

Use of mixture models to statistically approximate data has been an interesting topic of research in unsupervised learning methods. Mixture models based on exponential family of distributions have gained popularity in recent years. In this chapter, we introduce a finite mixture model based on Inverted Beta-Liouville distribution which has a higher degree of freedom to provide a better fit for the data. We use a variational learning framework to estimate the parameters which decreases the computational complexity of the model. We handle the problem of model selection with a component splitting approach which is an added advantage as it is done within the variational framework. We evaluate our model against some challenging applications like image clustering, speech clustering, spam image detection, and software defect detection.

Appendix: Proof of Eqs. (10.23), (10.24), (10.25) and (10.26)

From Eq. 10.16 we can write the logarithm of the joint as:

$$\displaystyle \begin{aligned} \ln p\big(\mathcal{X},\mathcal{Z}\big) = & \sum_{i=1}^N\sum_{j=1}^MZ_{ij}\Bigg[\ln\frac{\Gamma\big(\sum_{l=1}^{D}\alpha_{jl}\big)}{\prod_{l=1}^{D}\Gamma\big(\alpha_{jl}\big)} + \ln\frac{\Gamma(\alpha_j+\beta_j)}{\Gamma(\alpha_j)\Gamma(\beta_j)} + \sum_{l=1}^D\big(\alpha_{jl}-1\big)\ln X_{il} \\ &+\beta_j\ln\lambda_j + \Big(\alpha_j-\sum_{i=1}^D\alpha_{jl}\Big)\ln\Big(\sum_{i=1}^DX_{il}\Big)\\ &- \big(\alpha_j +\beta_j\big)\ln\Big(\lambda_j +\sum_{i=1}^DX_{il}\Big)\Bigg]\\ &+ \sum_{i=1}^N\Bigg[\sum_{j=1}^sZ_{ij}\ln\pi_j + \sum_{j=s+1}^MZ_{ij}\ln\pi_j^*\Bigg]-\big(M-s\big)\ln\Bigg[1-\sum_{k=1}^s\pi_k\Bigg]\\ &+\ln\frac{\Gamma\big(\sum_{j=s+1}^{M}c_{j}\big)}{\prod_{j=s+1}^{M}\Gamma\big(c_{j}\big)} +\sum_{j=s+1}^M\big(c_j-1\big)\ln\bigg[\pi_j^*-\Big(1-\sum_{k=1}^s\pi_k\Big)\bigg]\\ &+ \sum_{j=1}^M\sum_{l=1}^D u_{jl}\ln \nu_{jl} - \ln\Gamma\big(u_{jl}\big) + \big(u_{jl} - 1\big)\ln\alpha_{jl} - \nu_{jl}\alpha_{jl}\\ &+ \sum_{j=1}^M p_{j}\ln q_{j} - \ln\Gamma\big(p_{j}\big) + \big(p_{j} - 1\big)\ln\alpha_{j} - q_{j}\alpha_{j}\\ &+ \sum_{j=1}^M g_{j}\ln h_{j} - \ln\Gamma\big(g_{j}\big) + \big(g_{j} - 1\big)\ln\beta_{j} - h_{j}\beta_{j}\\ &+ \sum_{j=1}^M s_{j}\ln t_{j} - \ln\Gamma\big(s_{j}\big) + \big(s_{j} - 1\big)\ln\lambda_{j} - t_{j}\lambda_{j} \end{aligned} $$

(10.57)

To derive the variational solutions of each parameter, we consider the logarithm with respect to each of the parameter assuming the rest of the parameters to be constant. This is explained in the following subsections.

1.1 Variational Solution for Q(Z) Eq. (10.23)

The logarithm with respect to Q(Z _i) on the joint is given by:

$$\displaystyle \begin{aligned}{{}} \ln Q\big(Z_i\big) = & \sum_{j=1}^MZ_{ij}\Bigg[R_j + S_j+ \sum_{l=1}^D\big(\alpha_{jl}-1\big)\ln X_{il} +\beta_j\ln\lambda_j \\ &+ \Big(\alpha_j-\sum_{i=1}^D\alpha_{jl}\Big)\ln\Big(\sum_{i=1}^DX_{il}\Big) - \big(\alpha_j +\beta_j\big)T_{ij}\Bigg]\\ &+\Bigg[\sum_{j=1}^sZ_{ij}\ln\pi_j + \sum_{j=s+1}^MZ_{ij}\ln\pi_j^*\Bigg]\\ =& \sum_{j=1}^s\bigg[\ln\pi_j +R_j + S_j+ \sum_{l=1}^D\big(\alpha_{jl}-1\big)\ln X_{il} +\beta_j\ln\lambda_j\\ &+ \Big(\alpha_j-\sum_{i=1}^D\alpha_{jl}\Big)\ln\Big(\sum_{i=1}^DX_{il}\Big) - \big(\alpha_j +\beta_j\big)T_{ij}\bigg]\\ &\sum_{j=s+1}^M\bigg[\langle\ln\pi_j^*\rangle +R_j + S_j+ \sum_{l=1}^D\big(\alpha_{jl}-1\big)\ln X_{il} +\beta_j\ln\lambda_j \end{aligned} $$

(10.58)

$$\displaystyle \begin{aligned} &+ \Big(\alpha_j-\sum_{i=1}^D\alpha_{jl}\Big)\ln\Big(\sum_{i=1}^DX_{il}\Big) - \big(\alpha_j +\beta_j\big)T_{ij}\bigg] \end{aligned} $$

(10.59)

where

$$\displaystyle \begin{aligned} R_j = \bigg\langle\ln\frac{\Gamma\big(\sum_{l=1}^{D}\alpha_{jl}\big)}{\prod_{l=1}^{D}\Gamma\big(\alpha_{jl}\big)}\bigg\rangle,\; S_j = \bigg\langle\ln\frac{\Gamma(\alpha_j+\beta_j)}{\Gamma(\alpha_j)\Gamma(\beta_j)}\bigg\rangle,\; T_{ij} = \bigg\langle\ln\Big(\lambda_j +\sum_{i=1}^DX_{il}\Big)\bigg\rangle \end{aligned} $$

(10.60)

R _j, S _j, and T _ij are intractable in the above equations. Due to this reason we use second order Taylor series approximation for R _j and S _j and first order Taylor series approximation for T _ij. the equations are given in Eqs. (10.30), (10.31), and (10.32), respectively. It is a notable fact that (10.58) is of the form:

$$\displaystyle \begin{aligned}{{}} \ln Q\big(\mathcal{Z}\big) =\sum_{i=1}^N\Bigg[ \sum_{j=1}^sZ_{ij}\ln\tilde{r}_{ij} + \sum_{j=s+1}^MZ_{ij}\ln \tilde{r}_{ij}^*\Bigg] + \text{const} \end{aligned} $$

(10.61)

given

$$\displaystyle \begin{aligned} \ln\tilde{r}_{ij} = &\;\ln \pi_j + R_j+S_j+\Big(\overline{\alpha}_j- \sum_{l=1}^D\overline{\alpha}_{jl}\Big)\ln\Big(\sum_{l=1}^DX_{il}\Big) + \overline{\beta}_j\langle\ln\lambda_j\rangle\\ &+\sum_{l=1}^D\Big[\big(\overline{\alpha}_{jd}-1\big)\ln X_{id}\Big]-\big(\overline{\alpha}+\overline{\beta}\big)T_{ij} \end{aligned} $$

(10.62)

$$\displaystyle \begin{aligned} \ln\tilde{r}_{ij}^* = &\;\langle\ln \pi_j^*\rangle + R_j+S_j+\Big(\overline{\alpha}_j- \sum_{l=1}^D\overline{\alpha}_{jl}\Big)\ln\Big(\sum_{l=1}^DX_{il}\Big) + \overline{\beta}_j\langle\ln\lambda_j\rangle\\ &+\sum_{l=1}^D\Big[\big(\overline{\alpha}_{jd}-1\big)\ln X_{id}\Big]-\big(\overline{\alpha}+\overline{\beta}\big)T_{ij} \end{aligned} $$

(10.63)

By taking the exponentiation of Eq. (10.58) we can write:

$$\displaystyle \begin{aligned} Q\big(\mathcal{Z}\big) \propto \prod_{i=1}^{N}\Bigg[\prod_{j=1}^{s}\tilde{r}_{ij}^{Z_{ij}}\prod_{j=s+1}^{M}\tilde{r}_{ij}^{*Z_{ij}}\Bigg] \end{aligned} $$

(10.64)

Normalizing this equation we can write the variational solution of $Q\big (\mathcal {Z}\big )$ as

$$\displaystyle \begin{aligned} Q\big(\mathcal{Z}\big) \propto \prod_{i=1}^{N}\Bigg[\prod_{j=1}^{s}r_{ij}^{Z_{ij}}\prod_{j=s+1}^{M}r_{ij}^{*Z_{ij}}\Bigg] \end{aligned} $$

(10.65)

where r _ij and $r_{ij}^*$ can be obtained from Eqs. (10.28) and (10.29). Also, we can say that <Z _ij> = r _ij for j = 1, …, s and $\big <Z_{ij}^*\big >= r_{ij}^*$ for j = s + 1, …, M

1.2 Proof of Eq. (10.24): Variational Solution of Q(π ^∗)

Similarly, the logarithm of the variational solution Q(π ^∗) is given as

$$\displaystyle \begin{aligned} \ln Q\big(\pi_j^*\big) = &\big<\ln p\big(\mathcal{X},\Theta\big)\big>_{\Theta \ne \pi_j^*}\\ =&\sum_{i=1}^N\big<Z_{ij}\big>\ln \pi_j^* + \big(c_j - 1\big) \ln \pi_j^* + \text{const}\\ =&\ln \pi_j^*\Bigg[\sum_{i=1}^N\big<Z_{ij}\big>+ c_j - 1 \Bigg] + \text{const} \end{aligned} $$

(10.66)

This equation shows that it has the same logarithmic form as that of Eq. (10.15). So we can write the variational solution of Q(π ^∗) as

$$\displaystyle \begin{aligned} Q\big(\mathbf{\pi}^*\big) = \Bigg(1 - \sum_{k=1}^s\pi_k\Bigg)^{-M+s} \frac{\Gamma\big(\sum_{j=s+1}^Mc_j^*\big)}{\prod_{j=s+1}^M\Gamma\big(c_j^*\big)}\prod_{j=s+1}^M\Bigg(\frac{\pi_j^*}{1-\sum_{k=1}^s\pi_k}\Bigg)^{c_j^*-1} \end{aligned} $$

(10.67)

where

$$\displaystyle \begin{aligned} c_j^* = \sum_{i=1}^N \big<Z_{ij}\big>+ c_j \end{aligned} $$

(10.68)

$\big <Z_{ij}\big >= r_{ij}^*$ in the above equation.

1.3 Proof of Eq. (10.25): Variational Solution of Q(α)

As in the other two cases the logarithm of the variational solution Q(α _jl) is given by

$$\displaystyle \begin{aligned}{{}} \ln Q\big(\alpha_{jl}\big) = &\big<\ln p\big(\mathcal{X},\Theta\big)\big>_{\Theta \ne \alpha_{jl}}\\ =&\sum_{i=1}^N\big<Z_{ij}\big>\Big[\mathcal{J}\big(\alpha_{jl}\big)+\alpha_{jl}\ln X_{il} -\alpha_{jl}\ln\Big(\sum_{i=1}^DX_{il}\Big)\Big]\\ &+\big(u_{jl}-1 \big) \ln \alpha_{jl} - \nu_{jl}\alpha_{jl} + \text{const} \end{aligned} $$

(10.69)

where

$$\displaystyle \begin{aligned} \mathcal{J}\big(\alpha_{jl}\big) = \Bigg<\ln \frac{\Gamma\big(\alpha_{jl}+\sum_{s \ne l}^{D+1}\alpha_{js}\big)}{\Gamma\big(\alpha_{jl}\big)\prod_{s \ne l}^{D+1}\Gamma\big(\alpha_{js}\big)}\Bigg>_{\Theta \ne \alpha_{jl}} \end{aligned} $$

(10.70)

Similar to what we encountered in the case of R _j the equation for $\mathcal {J}\big (\alpha _{jl}\big )$ is also intractable. We solve this problem finding the lower bound for the equation by calculating the first-order Taylor expansion with respect to $\overline {\alpha }_{jl}$. The calculated lower bound is given by

$$\displaystyle \begin{aligned} \mathcal{L}\big(\alpha_{jl}\big) \ge\,\, & \overline{\alpha}_{jl} \ln \alpha_{jl}\Bigg[\psi\Bigg(\sum_{l=1}^{D+1}\overline{\alpha}_{jl}\Bigg)-\psi\big(\overline{\alpha}_{jl}\big)+ \sum_{s \ne l}^{D+1}\overline{\alpha}_{js}\\ &\times\psi'\Bigg(\sum_{l=1}^{D+1}\overline{\alpha}_{jl}\Bigg)\big(\big<\ln \alpha_{js}\big>-\ln\overline{\alpha}_{js}\big)\Bigg] + \text{const} \end{aligned} $$

(10.71)

This approximation is also found to be a strict lower bound of $\mathcal {L}\big (\alpha _{jl}\big )$. Substituting this equation for lower bound in Eq. (10.69)

$$\displaystyle \begin{aligned} \ln Q\big(\alpha_{jl}\big) = &\sum_{i=1}^N\langle Z_{ij}\rangle\overline{\alpha}_{jl}\ln\alpha_{jl}\Bigg[\psi\bigg(\sum_{l=1}^D\overline{\alpha}_{jl}\bigg) -\psi\big(\overline{\alpha}_{jl}\big)\\ &+ \psi'\Big(\sum_{l=1}^D\overline{\alpha}_{jl}\Big)\sum_{d \ne l}^D\Big(\langle\ln\alpha_{jl}\rangle- \ln\overline{\alpha}_{jl}\Big)\overline{\alpha}_{jl}\Bigg]\\ &+\sum_{i=1}^N\alpha_{jl}\langle Z_{ij}\rangle\Bigg[\ln X_{il} - \ln\Big(\sum_{l=1}^DX_{il}\Big)\Bigg] + \text{const} \end{aligned} $$

(10.72)

This equation can be rewritten as

$$\displaystyle \begin{aligned}{{}} \ln Q\big(\alpha_{jl}\big) = \ln \alpha_{jl}\big(u_{jl}+\varphi_{jl} - 1\big) - \alpha_{jl}\big(\nu_{jl}-\vartheta_{jl}\big) + \text{const} \end{aligned} $$

(10.73)

where

$$\displaystyle \begin{aligned} \varphi_{jl} =&\sum_{i=1}^N\langle Z_{ij}\rangle\overline{\alpha}_{jl}\Bigg[\psi\bigg(\sum_{l=1}^D\overline{\alpha}_{jl}\bigg) -\psi\big(\overline{\alpha}_{jl}\big)\\ &+ \psi'\Big(\sum_{l=1}^D\overline{\alpha}_{jl}\Big)\sum_{d \ne l}^D\Big(\langle\ln\alpha_{jl}\rangle- \ln\overline{\alpha}_{jl}\Big)\overline{\alpha}_{jl}\Bigg] \psi'\Bigg(\sum_{l=1}^{D+1}\overline{\alpha}_{jl}\Bigg)\big(\big<\ln \alpha_{js}\big>-\ln\overline{\alpha}_{js}\big)\Bigg] \end{aligned} $$

(10.74)

$$\displaystyle \begin{aligned} \vartheta_{jl} =& \sum_{i=1}^N\langle Z_{ij}\rangle\Bigg[\ln X_{il} - \ln\Big(\sum_{l=1}^DX_{il}\Big)\Bigg] \end{aligned} $$

(10.75)

Eq. (10.73) is the logarithmic form of a Gamma distribution. If we exponentiate both the sides, we get

$$\displaystyle \begin{aligned} Q\big(\alpha_{jl}\big) \propto \alpha_{jl}^{u_{jl}+\varphi_{jl} - 1}e^{-\big(\nu_{jl}-\vartheta_{jl}\big)\alpha_{jl}} \end{aligned} $$

(10.76)

This leaves us with the optimal solution for the hyper-parameters u _jl and ν _jl given by

$$\displaystyle \begin{aligned} u_{jl}^* = u_{jl} + \varphi_{jl},\,\,\,\, \nu_{jl}^* = \nu_{jl}-\vartheta_{jl} \end{aligned} $$

(10.77)

By following the same procedure we can get the variational solutions for Q(α), Q(β), and Q(λ).