Joint Face Alignment and 3D Face Reconstruction

  • Feng Liu
  • Dan Zeng
  • Qijun ZhaoEmail author
  • Xiaoming Liu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9909)


We present an approach to simultaneously solve the two problems of face alignment and 3D face reconstruction from an input 2D face image of arbitrary poses and expressions. The proposed method iteratively and alternately applies two sets of cascaded regressors, one for updating 2D landmarks and the other for updating reconstructed pose-expression-normalized (PEN) 3D face shape. The 3D face shape and the landmarks are correlated via a 3D-to-2D mapping matrix. In each iteration, adjustment to the landmarks is firstly estimated via a landmark regressor, and this landmark adjustment is also used to estimate 3D face shape adjustment via a shape regressor. The 3D-to-2D mapping is then computed based on the adjusted 3D face shape and 2D landmarks, and it further refines the 2D landmarks. An effective algorithm is devised to learn these regressors based on a training dataset of pairing annotated 3D face shapes and 2D face images. Compared with existing methods, the proposed method can fully automatically generate PEN 3D face shapes in real time from a single 2D face image and locate both visible and invisible 2D landmarks. Extensive experiments show that the proposed method can achieve the state-of-the-art accuracy in both face alignment and 3D face reconstruction, and benefit face recognition owing to its reconstructed PEN 3D face shapes.


Face alignment 3D face reconstruction Cascaded regression 



All correspondences should be forwarded to Dr. Qijun Zhao via This work is supported by the National Key Scientific Instrument and Equipment Development Projects of China (No. 2013YQ49087904).

Supplementary material

419978_1_En_33_MOESM1_ESM.pdf (6 mb)
Supplementary material 1 (pdf 6147 KB)


  1. 1.
    Asthana, A., Zafeiriou, S., Tzimiropoulos, G., Cheng, S., Pantic, M.: From pixels to response maps: discriminative image filtering for face alignment in the wild. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1312–1320 (2015)CrossRefGoogle Scholar
  2. 2.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: SIGGRAPH, pp. 187–194. ACM Press/Addison-Wesley Publishing Co. (1999)Google Scholar
  3. 3.
    Blanz, V., Vetter, T.: Face recognition based on fitting a 3D morphable model. IEEE Trans. Pattern Anal. Mach. Intell. 25(9), 1063–1074 (2003)CrossRefGoogle Scholar
  4. 4.
    Cao, C., Weng, Y., Lin, S., Zhou, K.: 3D shape regression for real-time facial animation. Trans. Graph. (TOG) 32(4), 41 (2013)zbMATHGoogle Scholar
  5. 5.
    Cao, C., Wu, H., Weng, Y., Shao, T., Zhou, K.: Real-time facial animation with image-based dynamic avatars. ACM Trans. Graph. (TOG) 35(4), 126 (2016)CrossRefGoogle Scholar
  6. 6.
    Cao, X., Wei, Y., Wen, F., Sun, J.: Face alignment by explicit shape regression. Int. J. Comput. Vision 107(2), 177–190 (2014)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Chu, B., Romdhani, S., Chen, L.: 3D-aided face recognition robust to expression and pose variations. In: CVPR, pp. 1907–1914. IEEE (2014)Google Scholar
  8. 8.
    Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 6, 681–685 (2001)CrossRefGoogle Scholar
  9. 9.
    Cootes, T.F., Lanitis, A.: Active shape models: evaluation of a multi-resolution method for improving image search. In: BMVC, pp. 327–338. Citeseer (1994)Google Scholar
  10. 10.
    Cristinacce, D., Cootes, T.: Automatic feature localisation with constrained local models. Pattern Recogn. 41(10), 3054–3067 (2008)CrossRefzbMATHGoogle Scholar
  11. 11.
    Cristinacce, D., Cootes, T.F.: Boosted regression active shape models. In: BMVC, pp. 1–10 (2007)Google Scholar
  12. 12.
    Drira, H., Ben Amor, B., Srivastava, A., Daoudi, M., Slama, R.: 3D face recognition under expressions, occlusions, and pose variations. IEEE Trans. Pattern Anal. Mach. Intell. 35(9), 2270–2283 (2013)CrossRefGoogle Scholar
  13. 13.
    Gong, X., Wang, G.: An automatic approach for pixel-wise correspondence between 3D faces. Hybrid Inf. Technol. 2, 198–205 (2006)Google Scholar
  14. 14.
    Han, H., Jain, A.K.: 3D face texture modeling from uncalibrated frontal and profile images. In: BTAS, pp. 223–230. IEEE (2012)Google Scholar
  15. 15.
    Hassner, T.: Viewing real-world faces in 3D. In: ICCV, pp. 3607–3614 (2013)Google Scholar
  16. 16.
    Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical Report 07–49, University of Massachusetts, Amherst (2007)Google Scholar
  17. 17.
    Jeni, L.A., Cohn, J.F., Kanade, T.: Dense 3D face alignment from 2D videos in real-time. In: FG. IEEE (2015)Google Scholar
  18. 18.
    Jourabloo, A., Liu, X.: Pose-invariant 3D face alignment. In: ICCV, pp. 3694–3702 (2015)Google Scholar
  19. 19.
    Jourabloo, A., Liu, X.: Large-pose face alignment via CNN-based dense 3D model fitting. In: CVPR, June 2016Google Scholar
  20. 20.
    Kemelmacher-Shlizerman, I., Basri, R.: 3D face reconstruction from a single image using a single reference face shape. IEEE Trans. Pattern Anal. Mach. Intell. 33(2), 394–405 (2011)CrossRefGoogle Scholar
  21. 21.
    Lee, D., Park, H., Yoo, C.D.: Face alignment using cascade Gaussian process regression trees. In: CVPR, pp. 4204–4212. IEEE (2015)Google Scholar
  22. 22.
    Lee, Y.J., Lee, S.J., Park, K.R., Jo, J., Kim, J.: Single view-based 3D face reconstruction robust to self-occlusion. EURASIP J. Adv. Sig. Process. 2012(1), 1–20 (2012)CrossRefGoogle Scholar
  23. 23.
    Liu, F., Zeng, D., Li, J., Zhao, Q.: Cascaded regressor based 3D face reconstruction from a single arbitrary view image. arXiv preprint arXiv:1509.06161 (2015)
  24. 24.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004)CrossRefGoogle Scholar
  25. 25.
    Matthews, I., Baker, S.: Active appearance models revisited. Int. J. Comput. Vision 60(2), 135–164 (2004)CrossRefGoogle Scholar
  26. 26.
    Qu, C., Monari, E., Schuchert, T., Beyerer, J.: Fast, robust and automatic 3D face model reconstruction from videos. In: AVSS, pp. 113–118. IEEE (2014)Google Scholar
  27. 27.
    Ren, S., Cao, X., Wei, Y., Sun, J.: Face alignment at 3000 FPS via regressing local binary features. In: CVPR, pp. 1685–1692. IEEE (2014)Google Scholar
  28. 28.
    Romdhani, S., Vetter, T.: Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior. In: CVPR, pp. 986–993. IEEE (2005)Google Scholar
  29. 29.
    Roth, J., Tong, Y., Liu, X.: Adaptive 3D face reconstruction from unconstrained photo collections. In: CVPR, June 2016Google Scholar
  30. 30.
    Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: the first facial landmark localization challenge. In: ICCVW, pp. 397–403. IEEE (2013)Google Scholar
  31. 31.
    Suwajanakorn, S., Kemelmacher-Shlizerman, I., Seitz, S.M.: Total moving face reconstruction. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part IV. LNCS, vol. 8692, pp. 796–812. Springer, Heidelberg (2014)Google Scholar
  32. 32.
    Suwajanakorn, S., Seitz, S.M., Kemelmacher-Shlizerman, I.: What makes tom hanks look like tom hanks. In: ICCV, pp. 3952–3960 (2015)Google Scholar
  33. 33.
    Tulyakov, S., Sebe, N.: Regressing a 3D face shape from a single image. In: ICCV, pp. 3748–3755. IEEE (2015)Google Scholar
  34. 34.
    Tzimiropoulos, G.: Project-out cascaded regression with an application to face alignment. In: CVPR, pp. 3659–3667. IEEE (2015)Google Scholar
  35. 35.
    Xiong, X., De la Torre, F.: Supervised descent method and its applications to face alignment. In: CVPR, pp. 532–539. IEEE (2013)Google Scholar
  36. 36.
    Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.J.: A 3D facial expression database for facial behavior research. In: FG, pp. 211–216. IEEE (2006)Google Scholar
  37. 37.
    Yu, X., Huang, J., Zhang, S., Yan, W., Metaxas, D.N.: Pose-free facial landmark fitting via optimized part mixtures and cascaded deformable shape model. In: ICCV, pp. 1944–1951. IEEE (2013)Google Scholar
  38. 38.
    Zhou, X., Leonardos, S., Hu, X., Daniilidis, K.: 3D shape estimation from 2D landmarks: a convex relaxation approach. In: CVPR, pp. 4447–4455. IEEE (2015)Google Scholar
  39. 39.
    Zhu, S., Li, C., Loy, C.C., Tang, X.: Face alignment by coarse-to-fine shape searching. In: CVPR, pp. 4998–5006 (2015)Google Scholar
  40. 40.
    Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: CVPR, pp. 2879–2886. IEEE (2012)Google Scholar
  41. 41.
    Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S.Z.: Face alignment across large poses: a 3D solution. In: CVPR, June 2016Google Scholar
  42. 42.
    Zhu, X., Lei, Z., Yan, J., Yi, D., Li, S.Z.: High-fidelity pose and expression normalization for face recognition in the wild. In: CVPR, pp. 787–796. IEEE (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.College of Computer ScienceSichuan UniversityChengduChina
  2. 2.Department of Computer Science and EngineeringMichigan State UniversityEast LansingUSA

Personalised recommendations