Online Variational Learning for Medical Image Data Clustering

Abstract

Data mining is an extensive area of research involving pattern discovery and feature extraction which is applied in various critical domains. In clinical aspect, data mining has emerged to assist the clinicians in early detection, diagnosis, and prevention of diseases. Advances in computational methods have led to implementation of machine learning in multi-modal clinical image analysis. One recent method is online learning where data become available in a sequential order, thus sequentially updating the best predictor for the future data at each step, as opposed to batch learning techniques which generate the best predictor by learning the entire data set at once.

In this chapter, we have examined and analysed multi-modal medical images by developing an unsupervised machine learning algorithm based on online variational inference for finite inverted Dirichlet mixture model. Our prime focus was to validate the developed approach on medical images. We do so by implementing the algorithm on both synthetic and real data sets. We test the algorithm’s ability to detect challenging real world diseases, namely brain tumour, lung tuberculosis, and melanomic skin lesion.

Appendix

1.1 Proof of Eq. (11.17): Variational Solution of $Q\big ( \mathcal {Z}\big )$

For the variational solution Q _s(Θ _s), the general expression expressed as:

$$\displaystyle \begin{aligned} \ln Q_s(\varTheta_s) = \big<\ln p(X,\varTheta)\big>_{j\neq s} + const \end{aligned} $$

(11.46)

where const is an additive term representing every term that is independent of Q _s(Θ _s). Now consider the joint distribution in Eq. (11.10), the variational solution for $Q\big (\mathcal {Z}\big )$ can be derived as follows:

$$\displaystyle \begin{aligned}{{}} \ln Q\big(\mathcal{Z}\big) = \alpha_{ij} \Bigg[ \ln\pi_j + \mathcal{R}_j + \sum^{D+1}_{l=1}(\alpha_{jl} - 1) \ln X_{il} \Bigg] + const \end{aligned} $$

(11.47)

Where

$$\displaystyle \begin{aligned} \mathcal{R}_j = \Bigg<\ln \frac{\varGamma (\sum^{D+1}_{l=1}\alpha_{jl}}{\prod_{{D+1}_{l=1}} \varGamma(\alpha_{jl})} \Bigg>_{\alpha_{jl},\ldots \alpha_{jD+1}} \end{aligned} $$

(11.48)

and

$$\displaystyle \begin{aligned} \alpha_{jl} = \big<\alpha_{jl} \big> = \frac{u_{jl}}{v_{jl}} \end{aligned} $$

(11.49)

Since we don’t have a closed form solution for $\mathcal {R}_{j}$, therefore it is not possible to directly apply the variational inference. Therefore, in order to provide traceable approximations, the second-order Taylor’s expansion is used to approximate the expected values of parameters α _j [14]. Hence, considering the logarithm form of (11.6), Eq. (11.47) can be written as

$$\displaystyle \begin{aligned} \ln Q \big( \mathcal{Z} \big) = \sum^ N_{i=1} \sum^M_{j=1} \mathcal{Z}_{ij} \ln \rho_{ij} + const \end{aligned} $$

(11.50)

where

$$\displaystyle \begin{aligned} \ln \rho_{ij} = \ln \pi_{j} + \mathcal{R}_{j} + \sum^D_{l=1}(\alpha_{jl} - 1) \ln X_{il} \end{aligned} $$

(11.51)

Since all the term without $\mathcal {Z}_{ij}$ can be added to the constant, it possible to show that

$$\displaystyle \begin{aligned} Q\big(\mathcal{Z}\big) \propto \prod^N_{i=1} \prod^M_{j=1} \rho_{ij}^{Z_{ij}} \end{aligned} $$

(11.52)

To find the exact formula for $Q(\mathcal {Z})$, Eq. (11.53) should be normalized and the calculation can be expressed as

$$\displaystyle \begin{aligned} Q\big(\mathcal{Z}\big)= \prod^N_{i=1} \prod^M_{j=1} r_{ij}^{Z_{ij}} \end{aligned} $$

(11.53)

where

$$\displaystyle \begin{aligned} r_{ij} = \frac{\rho_{ij}}{\sum^M_{j=1}\rho_{ij}} \end{aligned} $$

(11.54)

It is noteworthy that $\sum ^M_{j=1} r_{ij} = 1$, thus the result for $Q(\mathcal {Z})$ is

$$\displaystyle \begin{aligned} \big<\mathcal{Z}_{ij}\big> = r_{ij} \end{aligned} $$

(11.55)

1.2 Proof of Eqs. (11.18), (11.22) and (11.23)

Assuming the parameters α _jl are independent in a mixture model with M components, we can factorize Q(α) as

$$\displaystyle \begin{aligned} Q(\alpha) = \prod^M_{j=1}\prod^{D+1}_{l=1} Q(\alpha_{jl}) \end{aligned} $$

(11.56)

We compute the variational solution for the Q(α _jl ) by using Eq. (11.16) instead of using the gradient method. The logarithm of the variational solution Q(α _jl ) is given by,

$$\displaystyle \begin{aligned}{{}} \ln Q\big(\alpha_{jl}\big) = &\big<\ln p\big(\mathcal{X},\varTheta\big)\big>_{\varTheta \ne \alpha_{jl}}\\ =&\sum_{i=1}^N\big<Z_{ij}\big>\mathcal{J}\big(\alpha_{jl}\big)+\alpha_{jl}\sum_{i=1}^N\big<Z_{ij}\big>\ln X_{il} - \alpha_{jl} \ln\Bigg(1+\sum_{l=1}^{D+1}X_{il}\Bigg)\\ &+\big(u_{jl}-1 \big) \ln \alpha_{jl} - \nu_{jl}\alpha_{jl} + \text{const} \end{aligned} $$

(11.57)

where,

$$\displaystyle \begin{aligned} \mathcal{J}\big(\alpha_{jl}\big) = \Bigg<\ln \frac{\varGamma\big(\alpha_{jl}+\sum_{s \ne l}^{D+1}\alpha_{js}\big)}{\varGamma\big(\alpha_{jl}\big)\prod_{s \ne l}^{D+1}\varGamma\big(\alpha_{js}\big)}\Bigg>_{\varTheta \ne \alpha_{jl}} \end{aligned} $$

(11.58)

Similar to what we encountered in the case of R _j, the equation for $\mathcal {J}\big (\alpha _{jl}\big )$ is also intractable. We solve this problem finding the lower bound for the equation by calculating the first-order Taylor expansion with respect to $\overline {\alpha }_{jl}$. The calculated lower bound is given by [44],

$$\displaystyle \begin{aligned} \mathcal{J}\big(\alpha_{jl}\big) \ge\,\, & \overline{\alpha}_{jl} \ln \alpha_{jl}\Bigg[\psi\Bigg(\sum_{l=1}^{D+1}\overline{\alpha}_{jl}\Bigg)-\psi\big(\overline{\alpha}_{jl}\big)+ \sum_{s \ne l}^{D+1}\overline{\alpha}_{js}\\ &\times\psi'\Bigg(\sum_{l=1}^{D+1}\overline{\alpha}_{jl}\Bigg)\big(\big<\ln \alpha_{js}\big>-\ln\overline{\alpha}_{js}\big)\Bigg] + \text{const} \end{aligned} $$

(11.59)

Substituting this equation for lower bound in Eq. (11.57)

$$\displaystyle \begin{aligned} \ln Q\big(\alpha_{jl}\big) = &\sum_{i=1}^N\big<Z_{ij}\big>\overline{\alpha}_{jl} \ln \alpha_{jl}\Bigg[\psi\Bigg(\sum_{l=1}^{D+1}\overline{\alpha}_{jl}\Bigg)-\psi\big(\overline{\alpha}_{jl}\big)\\ &+ \sum_{s \ne l}^{D+1}\overline{\alpha}_{js} \psi'\Bigg(\sum_{l=1}^{D+1}\overline{\alpha}_{jl}\Bigg)\big(\big<\ln \alpha_{js}\big>-\ln\overline{\alpha}_{js}\big)\Bigg]\\ & +\alpha_{jl}\sum_{i=1}^N\big<Z_{ij}\big>\ln X_{il} - \alpha_{jl} \ln\Bigg(1+\sum_{l=1}^{D+1}X_{il}\Bigg)\\ &+\big(u_{jl}-1 \big) \ln \alpha_{jl} - \nu_{jl}\alpha_{jl} + \text{const} \end{aligned} $$

(11.60)

This equation can be rewritten as,

$$\displaystyle \begin{aligned}{{}} \ln Q\big(\alpha_{jl}\big) = \ln \alpha_{jl}\big(u_{jl}+\varphi_{jl} - 1\big) - \alpha_{jl}\big(\nu_{jl}-\vartheta_{jl}\big) + \text{const} \end{aligned} $$

(11.61)

where,

$$\displaystyle \begin{aligned} \varphi_{jl} =&\sum_{i=1}^N\big<Z_{ij}\big>\overline{\alpha}_{jl} \Bigg[\psi\Bigg(\sum_{l=1}^{D+1}\overline{\alpha}_{jl}\Bigg)-\psi\big(\overline{\alpha}_{jl}\big)\\ &+ \sum_{s \ne l}^{D+1}\overline{\alpha}_{js} \psi'\Bigg(\sum_{l=1}^{D+1}\overline{\alpha}_{jl}\Bigg)\big(\big<\ln \alpha_{js}\big>-\ln\overline{\alpha}_{js}\big)\Bigg] \end{aligned} $$

(11.62)

$$\displaystyle \begin{aligned} \vartheta_{jl} =& \sum_{i=1}^N\big<Z_{ij}\big>\Bigg[\ln X_{il}- \ln \Bigg(1+\sum_{l=1}^D X_{il}\Bigg)\Bigg] \end{aligned} $$

(11.63)

Equation (11.61) is the logarithmic form of a gamma distribution. If we exponentiate both the sides, we get,

$$\displaystyle \begin{aligned} Q\big(\alpha_{jl}\big) \propto \alpha_{jl}^{u_{jl}+\varphi_{jl} - 1}e^{-\big(\nu_{jl}-\vartheta_{jl}\big)\alpha_{jl}} \end{aligned} $$

(11.64)

This leaves us with the optimal solution for the hyper-parameters u _jl and ν _jl given by,

$$\displaystyle \begin{aligned} u_{jl}^* = u_{jl} + \varphi_{jl},\,\,\,\, \nu_{jl}^* = \nu_{jl}-\vartheta_{jl} \end{aligned} $$

(11.65)

1.3 Proof of Eq. (11.27)

We calculate the mixing coefficients value π by maximizing the lower bound w.r.t to π. It is essential to include Lagrangian term in the lower bound because of the constraint $\sum ^M_{j=1} \pi _j = 1$. Then, solving for the derivative w.r.t π _j and setting the result to zero, we have [44]

$$\displaystyle \begin{aligned} {{}} \frac{\partial\mathcal{L}(Q)}{\partial \pi_{j}} &= \frac{\partial\mathcal{L}(Q)}{\partial \pi_{j}} \sum^N_{i=1}\sum^M_{j=1} r_{ij} \ln \pi_{j} + \lambda \Bigg(\sum^M_{j=1}\pi_j - 1 \Bigg) \\ &= \sum^N_{i=1}r_{ij}(1/\pi_j)+ \lambda = 0 \end{aligned} $$

(11.66)

$$\displaystyle \begin{aligned} {{}} \Rightarrow \sum^N_{i=1}r_{ij} = -\lambda \pi_j \end{aligned} $$

(11.67)

By taking the sum of both sides of Eq. (11.67) over j, we can obtain λ = −N. Then substituting the value of λ Eq. (11.66), we can obtain

$$\displaystyle \begin{aligned} \pi_j = \frac{1}{N} \sum^N_{i=1} r_{ij} \end{aligned} $$

(11.68)