Dense Semantic and Topological Correspondence of 3D Faces without Landmarks

  • Zhenfeng Fan
  • Xiyuan HuEmail author
  • Chen Chen
  • Silong Peng
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11220)


Many previous literatures use landmarks to guide the correspondence of 3D faces. However, these landmarks, either manually or automatically annotated, are hard to define consistently across different faces in many circumstances. We propose a general framework for dense correspondence of 3D faces without landmarks in this paper. The dense correspondence goal is revisited in two perspectives: semantic and topological correspondence. Starting from a template facial mesh, we sequentially perform global alignment, primary correspondence by template warping, and contextual mesh refinement, to reach the final correspondence result. The semantic correspondence is achieved by a local iterative closest point (ICP) algorithm of kernelized version, allowing accurate matching of local features. Then, robust deformation from the template to the target face is formulated as a minimization problem. Furthermore, this problem leads to a well-posed sparse linear system such that the solution is unique and efficient. Finally, a contextual mesh refining algorithm is applied to ensure topological correspondence. In the experiment, the proposed method is evaluated both qualitatively and quantitatively on two datasets including a publicly available FRGC v2.0 dataset, demonstrating reasonable and reliable correspondence results.


3D face Dense correspondence Point set registration 



All correspondences should be forwarded to X. Hu via This work is supported by the National Key R&D Program of China (2017YFC0803505) and the Open Project of National Engineering Laboratory for Forensic Science (2017NELKFKT02).


  1. 1.
    Corneanu, C.A., Simon, M.O., Cohn, J.F., Guerrero, S.E.: Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications. IEEE Trans. Pattern Anal. Mach. Intell. 38(8), 1548–1568 (2016)CrossRefGoogle Scholar
  2. 2.
    Matthews, I., Xiao, J., Baker, S.: 2d vs. 3d deformable face models: representational power, construction, and real-time fitting. Int. J. Comput. Vis. 75(1), 93–113 (2007)CrossRefGoogle Scholar
  3. 3.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3d faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 187–194. ACM Press/Addison-Wesley Publishing Co. (1999)Google Scholar
  4. 4.
    Mohammadzade, H., Hatzinakos, D.: Iterative closest normal point for 3d face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(2), 381–397 (2013)CrossRefGoogle Scholar
  5. 5.
    Tam, G.K., et al.: Registration of 3d point clouds and meshes: a survey from rigid to nonrigid. IEEE Trans. Vis. Comput. Gr. 19(7), 1199–1217 (2013)CrossRefGoogle Scholar
  6. 6.
    Pan, G., Zhang, X., Wang, Y., Hu, Z., Zheng, X., Wu, Z.: Establishing point correspondence of 3d faces via sparse facial deformable model. IEEE Trans. Image Process. 22(11), 4170–4181 (2013)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Gilani, S.Z., Mian, A., Eastwood, P.: Deep, dense and accurate 3d face correspondence for generating population specific deformable models. Pattern Recognit. 69, 238–250 (2017)CrossRefGoogle Scholar
  8. 8.
    Romdhani, S.: Face image analysis using a multiple features fitting strategy. Ph.D. thesis, University\_of\_Basel (2005)Google Scholar
  9. 9.
    Guo, J., Mei, X., Tang, K.: Automatic landmark annotation and dense correspondence registration for 3d human facial images. BMC Bioinf. 14(1), 232 (2013)CrossRefGoogle Scholar
  10. 10.
    Grewe, Carl Martin, Zachow, Stefan: Fully automated and highly accurate dense correspondence for facial surfaces. In: Hua, Gang, Jégou, Hervé (eds.) ECCV 2016. LNCS, vol. 9914, pp. 552–568. Springer, Cham (2016). Scholar
  11. 11.
    Creusot, C., Pears, N., Austin, J.: A machine-learning approach to keypoint detection and landmarking on 3d meshes. Int. J. Comput. Vis. 102(1–3), 146–179 (2013)CrossRefGoogle Scholar
  12. 12.
    Song, M., Tao, D., Sun, S., Chen, C., Maybank, S.J.: Robust 3d face landmark localization based on local coordinate coding. IEEE Trans. Image Process. 23(12), 5108–5122 (2014)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Zulqarnain Gilani, S., Shafait, F., Mian, A.: Shape-based automatic detection of a large number of 3d facial landmarks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4639–4648 (2015)Google Scholar
  14. 14.
    Phillips, P.J., et al.: Overview of the face recognition grand challenge. In: . IEEE Computer Society Conference on Computer vision and Pattern Recognition, 2005, CVPR 2005, Vol. 1, pp. 947–954. IEEE (2005)Google Scholar
  15. 15.
    Castellani, U., Bartoli, A.: 3d shape registration. In: 3D Imaging, Analysis and Applications, pp. 221–264. Springer (2012)Google Scholar
  16. 16.
    Blanz, V., Vetter, T.: Face recognition based on fitting a 3d morphable model. IEEE Trans. Pattern Anal. Mach. Intell. 25(9), 1063–1074 (2003)CrossRefGoogle Scholar
  17. 17.
    Patel, A., Smith, W.A.: 3d morphable face models revisited. In: IEEE Conference on Computer Vision and Pattern Recognition, 2009, CVPR 2009, pp. 1327–1334. IEEE (2009)Google Scholar
  18. 18.
    Amberg, B., Romdhani, S., Vetter, T.: Optimal step nonrigid icp algorithms for surface registration. In: IEEE Conference on Computer Vision and Pattern Recognition, 2007, CVPR’07, pp. 1–8. IEEE (2007)Google Scholar
  19. 19.
    Cheng, S., Marras, I., Zafeiriou, S., Pantic, M.: Statistical non-rigid icp algorithm and its application to 3d face alignment. Image Vis. Comput. 58, 3–12 (2017)CrossRefGoogle Scholar
  20. 20.
    Li, H., Sumner, R.W., Pauly, M.: Global correspondence optimization for non-rigid registration of depth scans. In: Computer graphics forum. Vol. 27, pp. 1421–1430. Wiley Online Library (2008)Google Scholar
  21. 21.
    Ferrari, C., Lisanti, G., Berretti, S., Del Bimbo, A.: A dictionary learning based 3d morphable shape model. IEEE Trans. Multimed. (2017)Google Scholar
  22. 22.
    Ferrari, C., Lisanti, G., Berretti, S., Del Bimbo, A.: Dictionary learning based 3d morphable model construction for face recognition with varying expression and pose. In: 2015 International Conference on 3D Vision (3DV), pp. 509–517. IEEE (2015)Google Scholar
  23. 23.
    Booth, J., Roussos, A., Zafeiriou, S., Ponniah, A., Dunaway, D.: A 3d morphable model learnt from 10,000 faces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5543–5552 (2016)Google Scholar
  24. 24.
    Booth, J., Roussos, A., Ponniah, A., Dunaway, D., Zafeiriou, S.: Large scale 3d morphable models. Int. J. Comput. Vis. 1–22 (2017)Google Scholar
  25. 25.
    Vlasic, D., Brand, M., Pfister, H., Popović, J.: Face transfer with multilinear models. In: ACM Transactions on Gaphics (TOG), vol. 24, pp. 426–433. ACM (2005)Google Scholar
  26. 26.
    Cheng, S., Marras, I., Zafeiriou, S., Pantic, M.: Active nonrigid icp algorithm. In: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Vol. 1, pp. 1–8. IEEE (2015)Google Scholar
  27. 27.
    Bolkart, T., Wuhrer, S.: A groupwise multilinear correspondence optimization for 3d faces. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3604–3612 (2015)Google Scholar
  28. 28.
    Zhang, C., Smith, W.A., Dessein, A., Pears, N., Dai, H.: Functional faces: Groupwise dense correspondence using functional maps. In: CVPR, pp. 5033–5041 (2016)Google Scholar
  29. 29.
    Sidorov, K.A., Richmond, S., Marshall, D.: Efficient groupwise non-rigid registration of textured surfaces. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2401–2408. IEEE (2011)Google Scholar
  30. 30.
    Bronstein, Alexander M., Bronstein, Michael M., Kimmel, Ron: Expression-invariant 3D face recognition. In: Kittler, Josef, Nixon, Mark S. (eds.) AVBPA 2003. LNCS, vol. 2688, pp. 62–70. Springer, Heidelberg (2003). Scholar
  31. 31.
    Besl, P.J., McKay, N.D.: A method for registration of 3-d shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992)CrossRefGoogle Scholar
  32. 32.
    Yang, J., Li, H., Jia, Y.: Go-icp: Solving 3d registration efficiently and globally optimally. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1457–1464 (2013)Google Scholar
  33. 33.
    Yang, J., Li, H., Campbell, D., Jia, Y.: Go-icp: a globally optimal solution to 3d icp point-set registration. IEEE Trans. Pattern Anal. Mach. Intell. 38(11), 2241–2254 (2016)CrossRefGoogle Scholar
  34. 34.
    Szeliski, R., Tonnesen, D.: Surface Modeling with Oriented Particle Systems. Vol. 26. ACM (1992)CrossRefGoogle Scholar
  35. 35.
    Linsen, L.: Point cloud representation. Univ., Fak. für Informatik, Bibliothek (2001)Google Scholar
  36. 36.
    Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.J.: A 3d facial expression database for facial behavior research. In: 7th International Conference on Automatic Face and Gesture Recognition, 2006, FGR 2006, pp. 211–216. IEEE (2006)Google Scholar
  37. 37.
    Savran, Arman, Alyüz, Neşe, Dibeklioğlu, Hamdi, Çeliktutan, Oya, Gökberk, Berk, Sankur, Bülent, Akarun, Lale: Bosphorus database for 3D face analysis. In: Schouten, Ben, Juul, Niels Christian, Drygajlo, Andrzej, Tistarelli, Massimo (eds.) BioID 2008. LNCS, vol. 5372, pp. 47–56. Springer, Heidelberg (2008). Scholar
  38. 38.
    Szeptycki, P., Ardabilian, M., Chen, L.: A coarse-to-fine curvature analysis-based rotation invariant 3d face landmarking. In: IEEE 3rd International Conference on Biometrics: Theory, Applications, and Systems, 2009, BTAS’09, pp. 1–6. IEEE (2009)Google Scholar
  39. 39.
    Segundo, M.P., Silva, L., Bellon, O.R.P., Queirolo, C.C.: Automatic face segmentation and facial landmark detection in range images. IEEE Trans. Syst. Man, Cybern. Part B (Cybernetics) 40(5), 1319–1330 (2010)CrossRefGoogle Scholar
  40. 40.
    Gupta, S., Markey, M.K., Bovik, A.C.: Anthropometric 3d face recognition. Int. J. Comput. Vis. 90(3), 331–349 (2010)CrossRefGoogle Scholar
  41. 41.
    Al-Osaimi, F., Bennamoun, M., Mian, A.: An expression deformation approach to non-rigid 3d face recognition. Int. J. Comput. Vis. 81(3), 302–316 (2009)CrossRefGoogle Scholar
  42. 42.
    Alyuz, N., Gokberk, B., Akarun, L.: Regional registration for expression resistant 3-d face recognition. IEEE Trans. Inf. Forensics Sec. 5(3), 425–440 (2010)CrossRefGoogle Scholar
  43. 43.
    Wang, Y., Liu, J., Tang, X.: Robust 3d face recognition by local shape difference boosting. IEEE Trans. Pattern Anal. Mach. Intell. 32(10), 1858–1870 (2010)CrossRefGoogle Scholar
  44. 44.
    Berretti, S., Del Bimbo, A., Pala, P.: 3d face recognition using isogeodesic stripes. IEEE Trans. Pattern Anal. Mach. Intell. 32(12), 2162–2177 (2010)CrossRefGoogle Scholar
  45. 45.
    Queirolo, C.C., Silva, L., Bellon, O.R., Segundo, M.P.: 3d face recognition using simulated annealing and the surface interpenetration measure. IEEE Trans. Pattern Anal. Mach. Intell. 32(2), 206–219 (2010)CrossRefGoogle Scholar
  46. 46.
    Spreeuwers, L.: Fast and accurate 3d face recognition. Int. J. Comput. Vis. 93(3), 389–414 (2011)CrossRefGoogle Scholar
  47. 47.
    Drira, H., Amor, B.B., Srivastava, A., Daoudi, M., Slama, R.: 3d face recognition under expressions, occlusions, and pose variations. IEEE Trans. Pattern Anal. Mach. Intell. 35(9), 2270–2283 (2013)CrossRefGoogle Scholar
  48. 48.
    Emambakhsh, M., Evans, A.: Nasal patches and curves for expression-robust 3d face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(5), 995–1007 (2017)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Institute of AutomationChinese Academy of SciencesBeijingChina
  2. 2.University of Chinese Academy of SciencesHuairouChina
  3. 3.Beijing Visytem Co. LtdHaidianChina

Personalised recommendations