Coupled Gaussian Process Regression for Pose-Invariant Facial Expression Recognition

  • Ognjen Rudovic
  • Ioannis Patras
  • Maja Pantic
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6312)


We present a novel framework for the recognition of facial expressions at arbitrary poses that is based on 2D geometric features. We address the problem by first mapping the 2D locations of landmark points of facial expressions in non-frontal poses to the corresponding locations in the frontal pose. Then, recognition of the expressions is performed by using any state-of-the-art facial expression recognition method (in our case, multi-class SVM). To learn the mappings that achieve pose normalization, we use a novel Gaussian Process Regression (GPR) model which we name Coupled Gaussian Process Regression (CGPR) model. Instead of learning single GPR model for all target pairs of poses at once, or learning one GPR model per target pair of poses independently of other pairs of poses, we propose CGPR model, which also models the couplings between the GPR models learned independently per target pairs of poses. To the best of our knowledge, the proposed method is the first one satisfying all: (i) being face-shape-model-free, (ii) handling expressive faces in the range from − 45° to + 45° pan rotation and from − 30° to + 30° tilt rotation, and (iii) performing accurately for continuous head pose despite the fact that the training was conducted only on a set of discrete poses.


Facial Expression Facial Expression Recognition Target Pair Landmark Point Facial Landmark 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Trans. Pattern Analysis and Machine Intelligence 31, 39–58 (2009)CrossRefGoogle Scholar
  2. 2.
    Vinciarelli, A., Pantic, M., Bourlard, H.: Social signal processing: Survey of an emerging domain. Image and Vision Computing 27, 1743–1759 (2009)CrossRefGoogle Scholar
  3. 3.
    Chang, Y., Vieira, M., Turk, M., Velho, L.: Automatic 3d facial expression analysis in videos. In: Proc. Int’l Workshop Analysis and Modelling of Faces and Gestures, pp. 293–307 (2005)Google Scholar
  4. 4.
    Sun, Y., Yin, L.: Facial expression recognition based on 3d dynamic range model sequences. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 58–71. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  5. 5.
    Sung, J., Kim, D.: Real-time facial expression recognition using staam and layered gda classifier. Image and Vision Computing 27, 1313–1325 (2009)CrossRefGoogle Scholar
  6. 6.
    Cheon, Y., Kim, D.: Natural facial expression recognition using differential-aam and manifold learning. Pattern Recognition 42, 1340–1350 (2009)zbMATHCrossRefGoogle Scholar
  7. 7.
    Zhu, Z., Ji, Q.: Robust real-time face pose and facial expression recovery. In: Proc. Int’l Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 681–688 (2006)Google Scholar
  8. 8.
    Wang, T.H., Lien, J.J.J.: Facial expression recognition system based on rigid and non-rigid motion separation and 3d pose estimation. Pattern Recognition 42, 962–977 (2009)CrossRefGoogle Scholar
  9. 9.
    Kumano, S., Otsuka, K., Yamato, J., Maeda, E., Sato, Y.: Pose-invariant facial expression recognition using variable-intensity templates. Int’l J. Computer Vision 83, 178–194 (2009)CrossRefGoogle Scholar
  10. 10.
    Chai, X., Shan, S., Chen, X., Gao, W.: Locally linear regression for pose-invariant face recognition. IEEE Trans. Image Processing 16, 1716–1725 (2007)CrossRefMathSciNetGoogle Scholar
  11. 11.
    Hu, Y., Zeng, Z., Yin, L., Wei, X., Tu, J., Huang, T.: A study of non-frontal-view facial expressions recognition. In: Proc. Int’l Conf. Pattern Recognition, pp. 1–4 (2008)Google Scholar
  12. 12.
    Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, Cambridge (2005)Google Scholar
  13. 13.
    Boyle, P., Frean, M.: Dependent gaussian processes. In: Advances in Neural Information Processing Systems, vol. 17, pp. 217–224. MIT Press, Cambridge (2005)Google Scholar
  14. 14.
    Murphy-Chutorian, E., Trivedi, M.M.: Head pose estimation in computer vision: A survey. IEEE Trans. Pattern Analysis and Machine Intelligence 31, 607–626 (2009)CrossRefGoogle Scholar
  15. 15.
    Rudovic, O., Patras, I., Pantic, M.: Facial expression invariant head pose normalization using gaussian process regression. In: Proceedings of IEEE Int’l Conf. Computer Vision and Pattern Recognition, vol. 3 (in Press, 2010)Google Scholar
  16. 16.
    Chen, T., Morris, J., Martin, E.: Gaussian process regression for multivariate spectroscopic calibration. Chemometrics and Intelligent Laboratory Systems 87, 59–71 (2007)CrossRefGoogle Scholar
  17. 17.
    Tresp, V., Taniguchi, M.: Combining estimators using non-constant weighting functions. In: Advances in Neural Information Processing Systems, pp. 419–426 (1995)Google Scholar
  18. 18.
    Tresp, V.: A bayesian committee machine. Neural Computing 12, 2719–2741 (2000)CrossRefGoogle Scholar
  19. 19.
    Julier, S.J., Uhlmann, J.K.: A non-divergent estimation algorithm in the presence of unknown correlations. In: Proc. American Control Conf., pp. 2369–2373 (1997)Google Scholar
  20. 20.
    Wang, J., Yin, L., Wei, X., Sun, Y.: 3d facial expression recognition based on primitive surface feature distribution. In: Proc. Int’l Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 1399–1406 (2006)Google Scholar
  21. 21.
    Gross, R., Matthews, I., Cohn, J., Kanade, T., Baker, S.: Multi-pie. Image and Vision Computing 28, 807–813 (2010)CrossRefGoogle Scholar
  22. 22.
    Cootes, T., Taylor, C.: Active shape models - smart snakes. In: Proc. British Machine Vision Conf., pp. 266–275 (1992)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Ognjen Rudovic
    • 1
  • Ioannis Patras
    • 2
  • Maja Pantic
    • 1
    • 3
  1. 1.Comp. DeptImperial CollegeLondonUK
  2. 2.Elec. Eng. DeptQueen Mary UniversityLondonUK
  3. 3.EEMCSUniversity of TwenteEnschedeThe Netherlands

Personalised recommendations