Deep 3D Facial Landmark Detection on Position Maps

  • Kangkang Gao
  • Shanming Yang
  • Keren FuEmail author
  • Peng Cheng
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11935)


3D facial landmark detection is a crucial step for many computer vision applications, such as 3D facial expression analysis, 3D face recognition, and 3D reconstruction. Pose variations, expression changes and self-occlusion yet make 3D facial landmark detection a very challenging task. In this paper, we propose a novel 3D Face Landmark Localization Network (3DLLN), which is robust to the above challenges. Different from existing methods, the proposed 3DLLN utilizes the position maps as an intermediate representation, from which 3DLLN detects 3D landmark coordinates. Further, we demonstrate the usage of a deep regression architecture to improve the accuracy and robustness of a large number of landmarks. The proposed scheme is evaluated on two public datasets FRGCv2 and BU_3DFE and achieves superior results to state-of-the-arts.


3D landmark detection Deep convolutional neural network Position map UV map 



This work was supported by the National Science Foundation of China, No. 61703077, U1833128, the Fundamental Research Funds for the Central Universities, No. YJ201755, and the Sichuan Science and Technology Major Projects (2018GZDZX0029).


  1. 1.
    Berretti, S., Amor, B.B., Daoudi, M., Del Bimbo, A.: 3D facial expression recognition using sift descriptors of automatically detected keypoints. Vis. Comput. 27(11), 1021 (2011)CrossRefGoogle Scholar
  2. 2.
    Blanz, V., Vetter, T., et al.: A morphable model for the synthesis of 3D faces. In: SIGGRAPH, vol. 99, pp. 187–194 (1999)Google Scholar
  3. 3.
    Booth, J., Zafeiriou, S.: Optimal UV spaces for facial morphable model construction. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 4672–4676. IEEE (2014)Google Scholar
  4. 4.
    Bulat, A., Tzimiropoulos, G.: Binarized convolutional landmark localizers for human pose estimation and face alignment with limited resources. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3706–3714 (2017)Google Scholar
  5. 5.
    Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1021–1030 (2017)Google Scholar
  6. 6.
    Chang, K.I., Bowyer, K.W., Flynn, P.J.: Multiple nose region matching for 3D face recognition under varying facial expression. IEEE Trans. Pattern Anal. Mach. Intell. 28(10), 1695–1700 (2006)CrossRefGoogle Scholar
  7. 7.
    Cihan Camgoz, N., Struc, V., Gokberk, B., Akarun, L., Alp Kindiroglu, A.: Facial landmark localization in depth images using supervised ridge descent. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 136–141 (2015)Google Scholar
  8. 8.
    Colombo, A., Cusano, C., Schettini, R.: 3D face detection using curvature analysis. Pattern Recogn. 39(3), 444–455 (2006)CrossRefGoogle Scholar
  9. 9.
    Creusot, C., Pears, N., Austin, J.: A machine-learning approach to keypoint detection and landmarking on 3D meshes. Int. J. Comput. Vis. 102(1–3), 146–179 (2013)CrossRefGoogle Scholar
  10. 10.
    Dibeklioğlu, H., Gökberk, B., Akarun, L.: Nasal region-based 3D face recognition under pose and expression variations. In: Tistarelli, M., Nixon, M.S. (eds.) ICB 2009. LNCS, vol. 5558, pp. 309–318. Springer, Heidelberg (2009). Scholar
  11. 11.
    Emambakhsh, M., Evans, A.N., Smith, M.: Using nasal curves matching for expression robust 3D nose recognition. In: 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS), pp. 1–8. IEEE (2013)Google Scholar
  12. 12.
    Fan, X., Jia, Q., Huyan, K., Gu, X., Luo, Z.: 3D facial landmark localization using texture regression via conformal mapping. Pattern Recogn. Lett. 83, 395–402 (2016)CrossRefGoogle Scholar
  13. 13.
    Feng, Y., Wu, F., Shao, X., Wang, Y., Zhou, X.: Joint 3D face reconstruction and dense alignment with position map regression network. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 534–551 (2018)CrossRefGoogle Scholar
  14. 14.
    Feng, Z.H., Hu, G., Kittler, J., Christmas, W., Wu, X.J.: Cascaded collaborative regression for robust facial landmark detection trained using a mixture of synthetic and real images with dynamic weighting. IEEE Trans. Image Process. 24(11), 3425–3440 (2015)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Grewe, C.M., Zachow, S.: Fully automated and highly accurate dense correspondence for facial surfaces. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9914, pp. 552–568. Springer, Cham (2016). Scholar
  16. 16.
    Harris, C.G., Stephens, M., et al.: A combined corner and edge detector. In: Alvey Vision Conference, vol. 15, pp. 10–5244. Citeseer (1988)Google Scholar
  17. 17.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  18. 18.
    Jahanbin, S., Choi, H., Bovik, A.C.: Passive multimodal 2-D+ 3-D face recognition using gabor features and landmark distances. IEEE Trans. Inf. Forensics Secur. 6(4), 1287–1304 (2011)CrossRefGoogle Scholar
  19. 19.
    Križaj, J., Emeršič, Ž., Dobrišek, S., Peer, P., Štruc, V.: Localization of facial landmarks in depth images using gated multiple ridge descent. In: 2018 IEEE International Work Conference on Bioinspired Intelligence (IWOBI), pp. 1–8. IEEE (2018)Google Scholar
  20. 20.
    Li, H., Huang, D., Morvan, J.M., Wang, Y., Chen, L.: Towards 3D face recognition in the real: a registration-free approach using fine-grained matching of 3D keypoint descriptors. Int. J. Comput. Vis. 113(2), 128–142 (2015)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Mian, A., Bennamoun, M., Owens, R.: An efficient multimodal 2D–3D hybrid approach to automatic face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 29(11), 1927–1943 (2007)CrossRefGoogle Scholar
  22. 22.
    Newell, A., Yang, K., Deng, J.: Stacked hourglass networks for human pose estimation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 483–499. Springer, Cham (2016). Scholar
  23. 23.
    Passalis, G., Perakis, P., Theoharis, T., Kakadiaris, I.A.: Using facial symmetry to handle pose variations in real-world 3D face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 33(10), 1938–1951 (2011)CrossRefGoogle Scholar
  24. 24.
    Paulsen, R.R., Juhl, K.A., Haspang, T.M., Hansen, T., Ganz, M., Einarsson, G.: Multi-view consensus CNN for 3D facial landmark placement. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11361, pp. 706–719. Springer, Cham (2019). Scholar
  25. 25.
    Perakis, P., Passalis, G., Theoharis, T., Kakadiaris, I.A.: 3D facial landmark detection under large yaw and expression variations. IEEE Trans. Pattern Anal. Mach. Intell. 35(7), 1552–1564 (2012)CrossRefGoogle Scholar
  26. 26.
    Segundo, M.P., Queirolo, C., Bellon, O.R., Silva, L.: Automatic 3D facial segmentation and landmark detection. In: 14th International Conference on Image Analysis and Processing (ICIAP 2007), pp. 431–436. IEEE (2007)Google Scholar
  27. 27.
    Segundo, M.P., Silva, L., Bellon, O.R.P., Queirolo, C.C.: Automatic face segmentation and facial landmark detection in range images. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 40(5), 1319–1330 (2010)CrossRefGoogle Scholar
  28. 28.
    Terada, T., Chen, Y.W., Kimura, R.: 3D facial landmark detection using deep convolutional neural networks. In: 2018 14th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), pp. 390–393. IEEE (2018)Google Scholar
  29. 29.
    Zhao, X., Dellandrea, E., Chen, L., Kakadiaris, I.A.: Accurate landmarking of three-dimensional facial data in the presence of facial expressions and occlusions using a three-dimensional statistical facial feature model. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 41(5), 1417–1428 (2011)CrossRefGoogle Scholar
  30. 30.
    Zhu, X., Liu, X., Lei, Z., Li, S.Z.: Face alignment in full pose range: a 3D total solution. IEEE Trans. Pattern Anal. Mach. Intell. 41(1), 78–92 (2019)CrossRefGoogle Scholar
  31. 31.
    Zulqarnain Gilani, S., Shafait, F., Mian, A.: Shape-based automatic detection of a large number of 3D facial landmarks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4639–4648 (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Kangkang Gao
    • 1
  • Shanming Yang
    • 2
  • Keren Fu
    • 1
    Email author
  • Peng Cheng
    • 3
  1. 1.College of Computer ScienceSichuan UniversityChengduChina
  2. 2.National Key Laboratory of Fundamental Science on Synthetic VisionSichuan UniversityChengduChina
  3. 3.College of Aeronautics and AstronauticsSichuan UniversityChengduChina

Personalised recommendations