Consensus of Regression for Occlusion-Robust Facial Feature Localization

  • Xiang Yu
  • Zhe Lin
  • Jonathan Brandt
  • Dimitris N. Metaxas
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8692)


We address the problem of robust facial feature localization in the presence of occlusions, which remains a lingering problem in facial analysis despite intensive long-term studies. Recently, regression-based approaches to localization have produced accurate results in many cases, yet are still subject to significant error when portions of the face are occluded. To overcome this weakness, we propose an occlusion-robust regression method by forming a consensus from estimates arising from a set of occlusion-specific regressors. That is, each regressor is trained to estimate facial feature locations under the precondition that a particular pre-defined region of the face is occluded. The predictions from each regressor are robustly merged using a Bayesian model that models each regressor’s prediction correctness likelihood based on local appearance and consistency with other regressors with overlapping occlusion regions. After localization, the occlusion state for each landmark point is estimated using a Gaussian MRF semi-supervised learning method. Experiments on both non-occluded and occluded face databases demonstrate that our approach achieves consistently better results over state-of-the-art methods for facial landmark localization and occlusion detection.


Facial feature localization Consensus of Regression Occlusion detection Face alignment 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Asthana, A., Cheng, S., Zafeiriou, S., Pantic, M.: Robust discriminative response map fitting with constrained local models. In: CVPR (2013)Google Scholar
  2. 2.
    Belhumeur, P., Jacobs, D., Kriegman, D., Kumar, N.: Localizing parts of faces using a consensus of exemplars. In: CVPR (2011)Google Scholar
  3. 3.
    Burgos-Artizzu, X., Perona, P., Dollar, P.: Robust face landmark estimation under occlusion. In: ICCV (2013)Google Scholar
  4. 4.
    Cao, X., Wei, Y., Wen, F., Sun, J.: Face alignment by explicit shape regression. In: CVPR (2012)Google Scholar
  5. 5.
    Cheng, X., Sridharan, S., Saragih, J., Lucey, S.: Rank minimization across appearance and shape for aam ensemble fitting. In: ICCV (2013)Google Scholar
  6. 6.
    Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. In: Burkhardt, H., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1407, pp. 484–498. Springer, Heidelberg (1998)Google Scholar
  7. 7.
    Cootes, T., Taylor, C., Cooper, D., Graham, J.: Active shape models-their training and application. Computer Vision and Image Understanding 61(1), 38–59 (1995)CrossRefGoogle Scholar
  8. 8.
    Cootes, T.F., Ionita, M.C., Lindner, C., Sauer, P.: Robust and accurate shape model fitting using random forest regression voting. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VII. LNCS, vol. 7578, pp. 278–291. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  9. 9.
    Cristinacce, D., Cootes, T.: Automatic feature localization with constrained local models. Pattern Recognition 41(10), 3054–3067 (2007)CrossRefGoogle Scholar
  10. 10.
    Cristinacce, D., Cootes, T.: Boosted regression active shape models. In: BMVC (2007)Google Scholar
  11. 11.
    Dantone, M., Gall, J., Fanelli, G., Gool, L.: Real-time facial feature detection using conditional regression forests. In: CVPR (2012)Google Scholar
  12. 12.
    Dollar, P., Welinder, P., Perona, P.: Cascaded pose regression. In: CVPR (2010)Google Scholar
  13. 13.
    Huang, Y., Liu, Q., Metaxas, D.: A component based deformable model for generalized face alignment. In: ICCV (2007)Google Scholar
  14. 14.
    Le, V., Brandt, J., Lin, Z., Bourdev, L., Huang, T.S.: Interactive facial feature localization. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part III. LNCS, vol. 7574, pp. 679–692. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  15. 15.
    Liang, L., Xiao, R., Wen, F., Sun, J.: Face alignment via component-based discriminative search. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 72–85. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  16. 16.
    Martinez, B., Valstar, M., Binefa, X., Pantic, M.: Local evidence aggregation for regression-based faical point detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(5), 1149–1163 (2013)CrossRefGoogle Scholar
  17. 17.
    Matthews, I., Baker, S.: Active appearance models revisited. International Journal of Computer Vision 60(2), 135–164 (2004)CrossRefGoogle Scholar
  18. 18.
    Rivera, S., Martinez, A.: Learning deformable shape manifolds. Pattern Recognition 45(4), 1792–1801 (2012)CrossRefGoogle Scholar
  19. 19.
    Roh, M., Oguri, T., Kanade, T.: Face alignment robust to occlusion. In: Automatic Face and Gesture Recognition (2011)Google Scholar
  20. 20.
    Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: The first facial landmark localization challenge. In: ICCV Workshop (2013)Google Scholar
  21. 21.
    Saragih, J., Lucey, S., Cohn, J.: Deformable model fitting by regularized landmark mean-shift. International Journal of Computer Vision 91(2), 200–215 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  22. 22.
    Smith, B.M., Zhang, L.: Joint face alignment with non-parametric shape models. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part III. LNCS, vol. 7574, pp. 43–56. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  23. 23.
    Tzimiropoulos, G., Pantic, M.: Optimization problems for fast aam fitting in-the-wild. In: ICCV (2013)Google Scholar
  24. 24.
    Valstar, M., Martinez, B., Binefa, X., Pantic, M.: Facial point detection using boosted regression and graph models. In: CVPR (2010)Google Scholar
  25. 25.
    Wang, X., Han, T., Yan, S.: An hog-lbp human detector with partial occlusion handling. In: ICCV (2011)Google Scholar
  26. 26.
    Xiong, X., De la Torre, F.: Supervised descent method and its applications to face alignment. In: CVPR (2013)Google Scholar
  27. 27.
    Yang, F., Huang, J., Metaxas, D.N.: Sparse shape registration for occluded facial feature localization. In: Automatic Face and Gesture Recognition (2011)Google Scholar
  28. 28.
    Yang, H., Patras, I.: Sieving regression forest votes for facial feature detection in the wild. In: ICCV (2013)Google Scholar
  29. 29.
    Yu, X., Huang, J., Zhang, S., Yan, W., Metaxas, D.N.: Pose-free facial landmark fitting via optimized part mixtures and cascaded deformable shape model. In: ICCV (2013)Google Scholar
  30. 30.
    Yu, X., Yang, F., Huang, J., Metaxas, D.N.: Explicit occlusion detection based deformable fitting for facial landmark localization. In: Automatic Face and Gesture Recognition (2013)Google Scholar
  31. 31.
    Zhou, F., Brandt, J., Lin, Z.: Exemplar-based graph matching for robust facial landmark localization. In: ICCV (2013)Google Scholar
  32. 32.
    Zhu, X., Ghahramani, Z., Lafferty, J.: Semi-supervised learning using gaussian fields and harmonic functions. In: ICML (2003)Google Scholar
  33. 33.
    Zhu, X., Ramanan, D.: Face detection, pose estimation and landmark localization in the wild. In: CVPR (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Xiang Yu
    • 1
  • Zhe Lin
    • 2
  • Jonathan Brandt
    • 2
  • Dimitris N. Metaxas
    • 1
  1. 1.Rutgers UniversityPiscatawayUSA
  2. 2.Adobe ResearchSan JoseUSA

Personalised recommendations