Weakly-supervised region annotation for understanding scene images

Abstract

Scene image understanding has drawn much attention for its intriguing applications in the past years. In this paper, we propose a unified probabilistic graphical model called Topic-based Coherent Region Annotation (TCRA) for weakly-supervised scene region annotation. The multiscale over-segmented regions within a scene image are considered as the “words” of our topic model, which impose neighborhood contextual constraints on topic level through spatial MRF modeling, and incorporate an annotation reasoning mechanism for learning and inferring region labels automatically. Mean field variational inference is provided for model learning. The proposed TCRA has the following two main advantages for understanding natural scene images. First, spatial information of multiscale over-segmented regions is explicitly modeled to obtain coherent region annotations. Second, only image-level labels are needed for automatically inferring the label of every region within the scene. This is particularly helpful in reducing human burden on manually labeling pixel-level semantics in the scene understanding research. Thus, given a scene image that has no textual prior, the regions in it can be automatically labeled using the learned TCRA model. The experimental results conducted on three benchmarks consisting of the MSRCORID image dataset, the UIUC Events image dataset and the SIFT FLOW dataset show that the proposed model outperforms the recent state-of-the-art methods.

Appendix

Here we briefly describe how to derive the equations in our EM algorithm. We use Maximum Likelihood Estimation (MLE) to estimate both the latent variables and the parameters. For the joint probability distribution or likelihood p(X), where X indicates the observations, we aim at maximizing the equivalent log likelihood lnp(X). Let L denote the latent variables, we use Jensen’s inequality as follows:

$$\begin{array}{@{}rcl@{}} \log P(\mathbf{X})&=&\log\sum\limits_{\mathbf{L}}{P(\mathbf{L,X})}=\log \sum\limits_{\mathbf{L}}{\frac{p(\mathbf{L},\mathbf{X})q(\mathbf{L})}{q(\mathbf{L})}}\\ &\geq&\sum\limits_{\mathbf{L}}{q(\mathbf{L})\log{\frac{p(\mathbf{L},\mathbf{X})}{q(\mathbf{L})}}} =E_{q}{\log{p(\mathbf{L},\mathbf{X})}}-E_{q}{\log{q(\mathbf{L})}} \end{array} $$

(16)

To be tractable, we need maximize the lower bound of p(X). In our TCRA model, we substitute $\prod \limits _{d = 1}^{D} {p\left ({{\theta ^{d}},{{\boldmath {z}}^{d}},\beta ,{{\boldmath {r}}^{d}},\pi ,{{\boldmath {w}}^{d}},{{\boldmath {y}}^{d}}|\alpha ,\eta ,\rho ,\delta } \right )}$ in (5) and the variational posterior distribution q(β, π, 𝜃, z, y) in (6) into the above equation.

In variational inference, to optimize the proper latent variables, namely, { 𝜃, z, β, π, y}, we use the Lagrangian with constraints and set the derivative to zero. Denote the lower bound of the log-likelihood as LL, then the update (10) of latent variable $\phi _{nk}^{d}$ can be derived as follows:

$$\begin{array}{@{}rcl@{}} LL_{[\phi_{nk}^{d}]}^{(1)}&=&E_{q}\{\log p(z_{nk}^{d}|\theta^{d},\delta)\} \\ &=&E_{q}\{z_{nk}^{d} \log {{\theta_{k}^{d}}}+\delta\sum\limits_{m~n}{z_{nk}^{d} z_{mk}^{d}}\} + constant \\ &=&\phi_{nk}^{d} E\left[\log{{\theta_{k}^{d}}}|\gamma^{d}\right] + \delta \sum\limits_{n~m}{\phi_{nk}^{d} \phi_{mk}^{d}} + constant \end{array} $$

(17)

and

$$ \frac{\partial LL_{[\phi_{nk}^{d}]}^{(1)}}{\partial \phi_{nk}^{d}}=E \left[ \log{{\theta_{k}^{d}}} | \gamma^{d} \right] + \delta \sum\limits_{m \in {\mathcal{N}}(n)}{\phi_{mk}^{d}} $$

(18)

Next,

$$\begin{array}{@{}rcl@{}} LL_{[\phi_{nk}^{d}]}^{(2)}&=&E_{q} \left\{ \log \prod\limits_{f=1}^{N_{f}}{ p({r_{n}^{d,f} | {z_{n}^{d}} , {\beta^{f}}}) } \right\} \\ &=& E_{q} \left\{ r_{nv}^{d,f} z_{nk}^{d} \sum\limits_{f=1}^{N_{f}} \sum\limits_{v=1}^{V_{f}} { \log \beta_{kv}^{f} } \right\} \\ &=& r_{nv}^{d,f} \phi_{nk}^{d} \sum\limits_{f=1}^{N_{f}} \sum\limits_{v=1}^{V_{f}} {E \left[ \log \beta_{k,v}^{f} | {\lambda_{k}^{f}} \right]} \end{array} $$

(19)

and

$$ \frac{\partial LL_{[\phi_{nk}^{d}]}^{(2)}}{\partial \phi_{nk}^{d}} = r_{nv}^{d,f} \sum\limits_{f=1}^{N_{f}} \sum\limits_{v=1}^{V_{f}} {E \left[ \log \beta_{k,v}^{f} | {\lambda_{k}^{f}} \right]} $$

(20)

Then,

$$\begin{array}{@{}rcl@{}} LL_{[\phi_{nk}^{d}]}^{(3)} &=& E_{q} \left\{ p({w_{m}^{d}} | y_{mn}^{d}, z_{nk}^{d}, \pi) \right\} \\ &=& E_{q} \left\{ \sum\limits_{m=1}^{M^{d}} \sum\limits_{u=1}^{U} {y_{mn}^{d} z_{nk}^{d} w_{mn}^{d} \log \pi_{ku}} \right\}\\ &=&\sum\limits_{m=1}^{M^{d}} \sum\limits_{u=1}^{U} {\kappa_{mn}^{d} \phi_{nk}^{d} w_{mn}^{d} E \left[ \log \pi_{ku} | \xi_{k} \right]} \end{array} $$

(21)

and

$$ \frac{\partial LL_{[\phi_{nk}^{d}]}^{(3)}}{\partial \phi_{nk}^{d}} = \sum\limits_{m=1}^{M^{d}} \sum\limits_{u=1}^{U} {\kappa_{mn}^{d} w_{mn}^{d} E \left[ \log \pi_{ku} | \xi_{k} \right]} $$

(22)

The last term is:

$$\begin{array}{@{}rcl@{}} -LL_{[\phi_{nk}^{d}]}^{(4)} &=& E_{q} \left\{ q(z_{nk}^{d} | \phi_{nk}^{d} ) \right\} + \lambda (\sum\limits_{k=1}^{K}{\phi_{nk}^{d}}-1)\\ &=& \phi_{nk}^{d} \log{\phi_{nk}^{d}} + \lambda (\sum\limits_{k=1}^{K}{\phi_{nk}^{d}}-1) \end{array} $$

(23)

and

$$ \frac{\partial LL_{[\phi_{nk}^{d}]}^{(4)}}{\partial \phi_{nk}^{d}} = \log{\phi_{nk}^{d}} + 1 + \lambda $$

(24)

Finally, by combining all the above terms and setting the derivative to zero:

$$ \frac{\partial LL_{[\phi_{nk}^{d}]}}{\partial \phi_{nk}^{d}}=\frac{\partial LL_{[\phi_{nk}^{d}]}^{(1)}}{\partial \phi_{nk}^{d}} +\frac{\partial LL_{[\phi_{nk}^{d}]}^{(2)}}{\partial \phi_{nk}^{d}}+\frac{\partial LL_{[\phi_{nk}^{d}]}^{(3)}}{\partial \phi_{nk}^{d}} +\frac{\partial LL_{[\phi_{nk}^{d}]}^{(4)}}{\partial \phi_{nk}^{d}} = 0 $$

(25)

we can get the update (10). Through similar ways, the other update equations can be derived. Thus in the E step, we can update one latent variable while fixing the others in each iteration. This process can be repeated from topic level to scene image level for parsing the semantics of images.