Modelling multilevel data in multimedia: A hierarchical factor analysis approach

Abstract

Multimedia content understanding research requires rigorous approach to deal with the complexity of the data. At the crux of this problem is the method to deal with multilevel data whose structure exists at multiple scales and across data sources. A common example is modeling tags jointly with images to improve retrieval, classification and tag recommendation. Associated contextual observation, such as metadata, is rich that can be exploited for content analysis. A major challenge is the need for a principal approach to systematically incorporate associated media with the primary data source of interest. Taking a factor modeling approach, we propose a framework that can discover low-dimensional structures for a primary data source together with other associated information. We cast this task as a subspace learning problem under the framework of Bayesian nonparametrics and thus the subspace dimensionality and the number of clusters are automatically learnt from data instead of setting these parameters a priori. Using Beta processes as the building block, we construct random measures in a hierarchical structure to generate multiple data sources and capture their shared statistical at the same time. The model parameters are inferred efficiently using a novel combination of Gibbs and slice sampling. We demonstrate the applicability of the proposed model in three applications: image retrieval, automatic tag recommendation and image classification. Experiments using two real-world datasets show that our approach outperforms various state-of-the-art related methods.

Appendix

Sampling u _i

This section provides details for sampling u _i when the primary medium has real-valued features and the associated media have categorical features. The primary and associated media features are modeled using Gaussian and categorical distributions. This is the model assumed throughout our experiments using real datasets.

Due to the use of Dirichlet process prior, the number of groups (J) can increase/decrease with data. For the existing groups, i.e. for u _i = 1, … , J, the conditional Gibbs posterior of u _i is given as

$$\begin{array}{@{}rcl@{}} &&p\left(u_{i}\mid\ldots\right)\propto\left\{ \frac{{\Pi}_{\{l\mid\mathbf{C}^{l,i}>0\}}{\Pi}_{t=0}^{d_{li}-1}\left(a_{\psi}+n_{u_{i}}^{-i,l}+t\right)}{{\Pi}_{t=0}^{n_{c_{i}}-1}\left[{\sum}_{\{l\mid\mathbf{C}^{l,i}>0\}}\left(a_{\psi}+n_{u_{i}}^{-i,l}\right)+t\right]}\right\}\\ &&{\kern3.4pc} \times\left\{ \frac{n_{u_{i}}^{-i}}{\xi_{0}+N-1}\delta_{\psi_{u_{i}}}\right\} p\left(\mathbf{Z}^{i,:}\mid\mathbf{Z}^{-i,u_{i}},\beta\right)p\left(\mathbf{W}^{i,:}\mid\mathbf{W}^{-i,u_{i}}\right) \end{array} $$

(28)

where a _ψ comes from ψ _j ∼ Dirichlet (a _ψ ). The number d _{l i} is the frequency of l–th feature and $n_{c_{i}}$ is the total number of features in i–th item of the associated medium.

For a new group, i.e. when u _i = J + 1, the conditional Gibbs posterior of u _i is given as

$$ p\left(u_{i}\mid\ldots\right)\propto\left\{ \left(a_{\psi}\right)^{n_{c_{i}}}\frac{\Gamma\left(La_{\psi}\right)}{\Gamma\left(La_{\psi}+n_{c_{i}}\right)}\right\} \times\left\{ \frac{\xi_{0}}{\xi_{0}+N-1}\right\} \times p\left(\mathbf{Z}^{i,:}\mid\mathbf{Z}^{-i,u_{i}},\beta\right)p\left(\mathbf{W}^{i,:}\mid\mathbf{W}^{-i,u_{i}}\right) $$

(29)

In above expressions, the predictive distribution of Z ^{i, :} is given as

$$ p\left(\mathbf{Z}^{i,:}\mid\mathbf{Z}^{-i,u_{i}},\beta\right)={\Pi}_{k=1}^{K^{\dagger}}\frac{\Gamma\left(\alpha_{u_{i}}\right){\Gamma}\left(\alpha_{u_{i}}\beta_{(k)}+f_{u_{i}}^{k}\right){\Gamma}\left(\alpha_{u_{i}}\bar{\beta}_{(k)}+N_{u_{i}}-f_{u_{i}}^{k}\right)}{\Gamma\left(\alpha_{u_{i}+N_{u_{i}}}\right){\Gamma}\left(\alpha_{u_{i}}\beta_{(k)}\right){\Gamma}\left(\alpha_{u_{i}}\bar{\beta}_{(k)}\right)} $$

(30)

where $f_{u_{i}}^{k}\triangleq \sum \limits _{i^{\prime }}\mathbf {Z}_{u_{i}}^{i^{\prime },k}$. The predictive distribution of W ^{i, :} is given as

$$ p\left(\mathbf{W}^{i,:}\mid\mathbf{W}^{-i,u_{i}}\right)=\left(2\pi\right)^{-\frac{K^{\dagger}}{2}}\frac{D\left(s_{u_{i}}^{-i},m_{u_{i}}^{-i},\nu_{u_{i}}^{-i},{\Delta}_{u_{i}}^{-i}\right)}{D\left(s_{0},m_{0},\nu_{0},{\Delta}_{0}\right)} $$

(31)

where D denotes the normalization constant of the Normal-Wishart distribution while s, m, ν and Δ are the parameters characterizing the distribution. Using the parameter set (s ₀, m ₀, ν ₀, Δ₀) for the prior distribution, the posterior set $\left (s_{u_{i}}^{-i},m_{u_{i}}^{-i},\nu _{u_{i}}^{-i},{\Delta }_{u_{i}}^{-i}\right )$ is given as $s_{u_{i}}^{-i}=s_{0}+N_{u_{i}}^{-i},~m_{u_{i}}^{-i}=\frac {s_{0}m_{0}+{\sum }_{i^{\prime }\in S_{-i}}\mathbf {W}^{i^{\prime },:}}{s_{0}+N_{u_{i}}^{-i}}~\nu _{u_{i}}^{-i}=\nu _{0}+N_{u_{i}}^{-i}$ and ${\Delta }_{u_{i}}^{-i}={\Delta }_{0}+\mathbf {W}^{-i,u_{i}}\left (\mathbf {W}^{-i,u_{i}}\right )^{\mathsf {T}}+s_{0}m_{0}\left (m_{0}\right )^{\mathsf {T}}-s_{u_{i}}^{-i}m_{u_{i}}^{-i}\left (m_{u_{i}}^{-i}\right )^{\mathsf {T}}$ and $S_{-i}\triangleq \left \{ i^{\prime }|u_{i^{\prime }}=u_{i},i^{\prime }\neq i\right \}$.