Face detection and facial expression recognition using simultaneous clustering and feature selection via an expectation propagation statistical learning framework

Abstract

In this paper, we focus on developing a novel framework which can be effectively used for both face detection (i.e. discriminate faces from non-face patterns) and facial expression recognition. The proposed statistical framework is based on a Dirichlet process mixture of generalized Dirichlet (GD) distributions used to model local binary pattern (LBP) features. Our method is built on nonparametric Bayesian analysis where the determination of the number of clusters is sidestepped by assuming an infinite number of mixture components. An unsupervised feature selection scheme is also integrated with the proposed nonparametric framework to improve modeling performance and generalization capabilities. By learning the proposed model using an expectation propagation (EP) inference approach, all the involved model parameters and feature saliencies can be evaluated simultaneously in a single optimization framework. Furthermore, the proposed framework is extended by adopting a localized feature selection scheme which has shown, according to our results, superior performance, to determine the most important facial features, as compared to the global one. The effectiveness and utility of the proposed method is illustrated through extensive empirical results using both synthetic data and two challenging applications involving face detection, and facial expression recognition.

Appendices

Appendix A: Proof of equation (30)

The partial derivative of ln Z _i with respect to the hyperparameter $a_j^{\setminus i}$ is calculated by

$$ \begin{array}{rll} \nabla_{a_j}^{\setminus i}\ln Z_i &=& \frac{1}{Z_i}\int f_i(\Lambda)\frac{q^{\setminus i}(\Lambda)}{q^{\setminus i}\left(\phi_j^{\setminus i}\right)}\frac{\partial}{\partial a_j^{\setminus i}}q^{\setminus i}\left(\phi_j^{\setminus i}\right) d\Lambda \\ &=&\int \frac{f_i(\Lambda)q^{\setminus i}(\Lambda)}{Z_i}\left[\ln \phi^{\setminus i}_j + \Psi\left(a_j^{\setminus i}+b_j^{\setminus i}\right)-\Psi\left(a_j^{\setminus i}\right)\right]d\Lambda \\ &=&\int \widehat{p}(\Lambda)\left[\ln \phi^{\setminus i}_j + \Psi\left(a_j^{\setminus i}+b_j^{\setminus i}\right)-\Psi\left(a_j^{\setminus i}\right)\right]d\Lambda \\ &=&E_{\widehat{p}}\left[\ln\phi_j\right] + \Psi\left(a_j^{\setminus i}+b_j^{\setminus i}\right) - \Psi\left(a_j^{\setminus i}\right) \\ &=&\Psi\left(a_j^\ast\right)-\Psi\left(a_j^\ast+b_j^\ast\right) + \Psi\left(a_j^{\setminus i}+b_j^{\setminus i}\right) - \Psi\left(a_j^{\setminus i}\right) \end{array} $$

(39)

Appendix B: Calculating $\nabla_{b_j}^{\setminus i}\ln Z_i$, $\nabla_{c_1}^{\setminus i}\ln Z_i$, $\nabla_{\vec{u}_{jl}}^{\setminus i}\ln Z_i$, $\nabla_{A_{jl}}^{\setminus i}\ln Z_i$, $\nabla_{\rho_l}^{\setminus i}\ln Z_i$ and $\nabla_{B_l}^{\setminus i}\ln Z_i$

We can calculate the partial derivative of ln Z _i with respect to the hyperparameter $b_j^{\setminus i}$ as

$$ \begin{array}{rll} \nabla_{b_j}^{\setminus i}\ln Z_i &=& \frac{1}{Z_i}\int f_i(\Lambda)\frac{q^{\setminus i}(\Lambda)}{q^{\setminus i}\left(\phi_j^{\setminus i}\right)}\frac{\partial}{\partial b_j^{\setminus i}}q^{\setminus i}\left(\phi_j^{\setminus i}\right) d\Lambda \\ &=&\int \frac{f_i(\Lambda)q^{\setminus i}(\Lambda)}{Z_i}\left[\ln \left(1-\phi^{\setminus i}_j\right) + \Psi\left(a_j^{\setminus i}+b_j^{\setminus i}\right)-\Psi\left(b_j^{\setminus i}\right)\right]d\Lambda \\ &=&\int \widehat{p}(\Lambda)\left[\ln \left(1-\phi^{\setminus i}_j\right) + \Psi\left(a_j^{\setminus i}+b_j^{\setminus i}\right)-\Psi\left(b_j^{\setminus i}\right)\right]d\Lambda \\ &=&E_{\widehat{p}}\left[\ln\left(1-\phi_j\right)\right] + \Psi\left(a_j^{\setminus i}+b_j^{\setminus i}\right) - \Psi\left(b_j^{\setminus i}\right) \\ &=&\Psi(b_j^\ast)-\Psi\left(a_j^\ast+b_j^\ast\right) + \Psi\left(a_j^{\setminus i}+b_j^{\setminus i}\right) - \Psi\left(b_j^{\setminus i}\right) \end{array} $$

(40)

Next, the partial derivative of ln Z _i with respect to the hyperparameter $c_1^{\setminus i}$ is calculated by

$$ \begin{array}{rll} \nabla_{c_1}^{\setminus i}\ln Z_i &=& \frac{1}{Z_i}\int f_i(\Lambda)\frac{q^{\setminus i}(\Lambda)}{q^{\setminus i}\left(\epsilon_{l_1}^{\setminus i}\right)}\frac{\partial}{\partial c_1^{\setminus i}}q^{\setminus i}\left(\epsilon_{l_1}^{\setminus i}\right) d\Lambda\nonumber\\ &=&\int \widehat{p}(\Lambda)\left[\ln \epsilon_{l_1} + \Psi\left(c_1^{\setminus i}+c_2^{\setminus i}\right)-\Psi\left(c_1^{\setminus i}\right)\right]d\Lambda\nonumber\\ &=&E_{\widehat{p}}[\ln \epsilon_{l_1}] + \Psi\left(c_1^{\setminus i}+c_2^{\setminus i}\right)-\Psi\left(c_1^{\setminus i}\right)\nonumber\\ &=&\Psi(c_1^{\ast})- \Psi\left(c_1^\ast+c_2^\ast\right) + \Psi\left(c_1^{\setminus i}+c_2^{\setminus i}\right)-\Psi\left(c_1^{\setminus i}\right) \end{array} $$

(41)

Similarly, we can compute $\nabla_{c_2}^{\setminus i}\ln Z_i$ as

$$ \nabla_{c_2}^{\setminus i}\ln Z_i = \Psi\left(c_2^{\ast}\right)- \Psi\left(c_1^\ast+c_2^\ast\right) + \Psi\left(c_1^{\setminus i}+c_2^{\setminus i}\right)-\Psi\left(c_2^{\setminus i}\right) $$

(42)

Then, the partial derivative of ln Z _i with respect to the hyperparameter $\boldsymbol{\mu}_{jl}^{\setminus i}$ is given by

$$ \begin{array}{rll} \nabla_{\boldsymbol{\mu}_{jl}}^{\setminus i}\ln Z_i &=& \frac{1}{Z_i}\int f_i(\Lambda)\frac{q^{\setminus i}(\Lambda)}{q^{\setminus i}\left(\theta_{jl}^{\setminus i}\right)}\frac{\partial}{\partial \boldsymbol{\mu}_{jl}^{\setminus i}}q^{\setminus i}\left(\theta_{jl}^{\setminus i}\right) d\Lambda\nonumber\\ &=&\int \frac{f_i(\Lambda)q^{\setminus i}(\Lambda)}{Z_i}\left[A_{jl}^{\setminus i}\theta_{jl}^{\setminus i}-A_{jl}^{\setminus i}\boldsymbol{\mu}_{jl}^{\setminus i}\right]d\Lambda\nonumber\\ &=&A_{jl}^{\setminus i}E_{\widehat{p}}[\theta_{jl}]-A_{jl}^{\setminus i}\boldsymbol{\mu}_{jl}^{\setminus i}\nonumber\\ &=&A_{jl}^{\setminus i}\boldsymbol{\mu}_{jl}^\ast-A_{jl}^{\setminus i}\boldsymbol{\mu}_{jl}^{\setminus i} \end{array} $$

(43)

We can also compute the partial derivative of ln Z _i with respect to the hyperparameter $A_{jl}^{\setminus i}$ as

$$ \begin{array}{rll} \nabla_{A_{jl}}^{\setminus i}\ln Z_i &=&\frac{1}{Z_i}\int f_i(\Lambda)\frac{q^{\setminus i}(\Lambda)}{q^{\setminus i}\left(\theta_{jl}^{\setminus i}\right)}\frac{\partial}{\partial A_{jl}^{\setminus i}}q^{\setminus i}\left(\theta_{jl}^{\setminus i}\right) d\Lambda \nonumber\\ &=& \int\frac{f_i(\Lambda)q^{\setminus i}(\Lambda)}{Z_i} \frac{1}{2}\left\{\left|\left(A_{jl}^{\setminus i}\right)^{-1}\right|-\left[\sum_{d=1}^2\left(\theta_{jld}^{\setminus i}\right)^2-2\theta_{jld}^{\setminus i}\mu_{jld}^{\setminus i}+\left(\mu_{jld}^{\setminus i}\right)^2\right] \right\}d\Lambda\nonumber\\ &=& \int \widehat{p}(\Lambda) \frac{1}{2}\left\{\left|\left(A_{jl}^{\setminus i}\right)^{-1}\right|-\left[\sum_{d=1}^2\left(\theta_{jld}^{\setminus i}\right)^2-2\theta_{jld}^{\setminus i}\mu_{jld}^{\setminus i}+\left(\mu_{jld}^{\setminus i}\right)^2\right] \right\}d\Lambda\nonumber\\ &=& \frac{1}{2}\left\{\left|\left(A_{jl}^{\setminus i}\right)^{-1}\right|-\left[\sum_{d=1}^2\left(\mu_{jld}^\ast\right)^2-2\mu_{jld}^\ast\mu_{jld}^{\setminus i}+\left(\mu_{jld}^{\setminus i}\right)^2\right] \right\} \end{array} $$

(44)

We then compute $\nabla_{\boldsymbol{\rho}_{l}}^{\setminus i}\ln Z_i$ and $\nabla_{B_{l}}^{\setminus i}\ln Z_i$ using similar ways as for $\nabla_{\boldsymbol{\mu}_{jl}}^{\setminus i}\ln Z_i$ and $\nabla_{A_{jl}}^{\setminus i}\ln Z_i$, such that

$$ \nabla_{\boldsymbol{\rho}_{l}}^{\setminus i}\ln Z_i = B_{l}^{\setminus i}\boldsymbol{\rho}_{l}^\ast-B_{l}^{\setminus i}\boldsymbol{\rho}_{jl}^{\setminus i} $$

(45)

$$ \nabla_{B_{l}}^{\setminus i}\ln Z_i = \frac{1}{2}\left\{\left|\left(B_{l}^{\setminus i}\right)^{-1}\right|-\left[\sum_{d=1}^2(\rho_{ld}^\ast)^2-2\rho_{ld}^\ast\rho_{ld}^{\setminus i}+\left(\rho_{ld}^{\setminus i}\right)^2\right] \right\} $$

(46)