Journal of Visualization

, Volume 21, Issue 4, pp 649–660 | Cite as

“Edutainment 2017” a visual and semantic representation of 3D face model for reshaping face in images

  • Jiang Du
  • Dan Song
  • Yanlong Tang
  • Ruofeng TongEmail author
  • Min Tang
Regular Paper


Reshaping faces is an interesting thing and a common demand in today’s media industry and cosmetic industry. Especially, the reshaped face images contribute a lot to special effects industry and plastic-beauty industry. In this paper, we provide a novel parametric representation of 3D face model that describes the face shape as a linear combination of semantic and non-semantic bases. The semantic (local) bases correspond to each individual face part and they are used to semantically edit each face part, while the non-semantic bases explain other shape variations to span the subspace of the face shape. First, we build a sparse and spatially localized parametric face model from a dataset of 3D face models by sparse principal component analysis. Second, to define the semantic bases, we train a regression model to correlate semantically significant values like nose height and mouth width. Finally, the bases of the resulting parametric face model is orthogonalized to all defined semantic bases by the Gram–Schmidt algorithm for generating the novel parametric representation of 3D face model. The novel representation can be applied in reshaping face in the images. The experiment results demonstrate that our representation not only retains the accuracy for 3D face reconstruction but also provides users a user-friendly tool to edit facial parts for desired facial shapes.

Graphical abstract


Image manipulation Image editing Semantic parameters Face image reshaping Deformation 



The research is supported in part by NSFC (61572424) and the People Programme (Marie Curie Actions) of the European Union’s Seventh Framework Programme FP7 (2007-2013) under REA Grant Agreement no. 612627-”AniNex”. Min Tang is supported in part by NSFC (61572423) and Zhejiang Provincial NSFC (LZ16F020003).


  1. Alvarez FJA, Parra EBB, Tubio FM (2017) Improving graphic expression training with 3D models. J Vis 20(4):889–904CrossRefGoogle Scholar
  2. Bach F, Jenatton R, Mairal J, Obozinski G (2012) Optimization with sparsity-inducing penalties. Found Trends Mach Learn 4(1):1–106CrossRefzbMATHGoogle Scholar
  3. Blanz V, Vetter T (1999) A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th annual conference on computer graphics and interactive techniques. ACM, pp 187–194Google Scholar
  4. Blanz V, Basso C, Poggio T, Vetter T (2003) Reanimating faces in images and video. Comput Gr Forum (Blackwell Publishing, Inc) 22(3):641–650CrossRefGoogle Scholar
  5. Blanz V, Scherbaum K, Vetter T, Seidel H-P (2004) Exchanging faces in images. Comput Gr Forum (Blackwell Publishing, Inc) 23(3):669–676CrossRefGoogle Scholar
  6. Cao C, Weng Y, Zhou S, Tong Y, Zhou K (2014) Facewarehouse: a 3d facial expression database for visual computing. Vis Comput Gr IEEE Trans 20(3):413–425CrossRefGoogle Scholar
  7. Chalas I, Urbanova P, Jurik V, Ferkova Z, Jandova M, Sochor J, Kozlikova B (2017) Generating various composite human faces from real 3D facial images. V Comput 33(4):443–458Google Scholar
  8. Chou J-K, Yang C-K, Gong S-D (2012) Face-off: automatic alteration of facial features. Multimed Tools Appl 56(3):569–596CrossRefGoogle Scholar
  9. Fan X, Chai Z, Feng Y, Wang Y, Wang S, Lou Z (2016) An efficient mesh-based face beautifier on mobile devices. Neurocomputing. 172:134–142CrossRefGoogle Scholar
  10. Golub GH, Vanloan CF (1996) Matrix computations (Johns Hopkins studies in mathematical sciences). The Johns Hopkins University Press, BaltimoreGoogle Scholar
  11. Guo Y, Zhang J, Cai J, Jiang B, Zheng J (2017) 3DFaceNet: real-time dense face reconstruction via synthesizing photo-realistic face images. arXiv:1708.00980
  12. Hasler N, Stoll C, Sunkel M, Rosenhahn B, Seidel H-P (2009) A statistical model of human pose and body shape. Comput Gr Forum 28(2):337–346CrossRefGoogle Scholar
  13. Kang D, Tian F, Sahandi R (2017) Interactive illustration of collage for children with folktale e-book. J Vis 20(3):639–650CrossRefGoogle Scholar
  14. Kavan L, Sloan PP, O’Sullivan C (2010) Fast and efficient skinning of animated meshes. Comput Gr Forum (Blackwell Publishing Ltd) 29(2):327–336CrossRefGoogle Scholar
  15. Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1867–1874Google Scholar
  16. Kemelmacher-Shlizerman I (2016) Transfiguring portraits. ACM Trans Gr (TOG) 35(4):94 (1–94:8) Google Scholar
  17. Klum S, Han H, Jain AK, Klara B (2013) Sketch based face recognition: forensic vs. composite sketches. In: Proceedings of the 6th international conference on biometrics. IEEE, pp 1–8Google Scholar
  18. Lewis JP, Anjyo K, Rhee T, Zhang M, Pighin FH, Deng Z (2014) Practice and theory of blendshape facial models. Eurographics (State Art Rep) 1(8)Google Scholar
  19. Leyvand T, Cohen-Or D, Dror G, Lischinski D (2008) Data-driven enhancement of facial attractiveness. ACM Trans Gr (TOG) 27(3):38Google Scholar
  20. Li H, Yu J, Ye Y, Bregler C (2013) Realtime facial animation with on-the-fly correctives. ACM Trans Gr (TOG) 32(4):42 (1–42:10) zbMATHGoogle Scholar
  21. Liao Q, Jin X, Zeng W (2012) Enhancing the symmetry and proportion of 3D face geometry. Vis Comput Gr IEEE Trans 18(10):1704–1716CrossRefGoogle Scholar
  22. Lu J, Sunkavalli K, Carr N, Hadap S, Forsyth D (2016) A visual representation for editing face images. arXiv:1612.00522
  23. Lu Z, Yang J, Liu Q (2017) Face image retrieval based on shape and texture feature fusion. Comput Visual Media 3(4):359–368CrossRefGoogle Scholar
  24. Ma DS, Correll J, Wittenbrink B (2015) The Chicago face database: a free stimulus set of faces and norming data. Behav Res Methods 47(4):1122–1135CrossRefGoogle Scholar
  25. Mansouri S, Ebrahimnezhad H (2016) Segmentation-based semi-regular remeshing of 3D models using curvature-adapted subdivision surface fitting. J Vis 19(1):141–155CrossRefGoogle Scholar
  26. Neumann T, Varanasi K, Wenger S, Wacker M, Magnor M, Theobalt C (2013) Sparse localized deformation components. ACM Trans Gr (TOG) 32(6):179Google Scholar
  27. Nirkin Y, Masi I, Tran AT, Hassner T, Medioni G (2017) On face segmentation, face swapping, and face perception. arXiv:1704.06729
  28. Paysan P, Knothe R, Amberg B (2009) A 3D face model for pose and illumination invariant face recognition. In: 6th IEEE international conference on advanced video and signal based surveillance (AVSS’09), IEEE, Genova, Italy, pp 296–301Google Scholar
  29. Ramamoorthi R, Hanrahan P (2001) An efficient representation for irradiance environment maps. In: Proceedings of the 28th annual conference on computer graphics and interactive techniques. ACM, pp 497–500Google Scholar
  30. Shamir A, Sorkine O (2009) Visual media retargeting. ACM SIGGRAPH ASIA 2009 courses. ACM, p 11Google Scholar
  31. Streuber S, Quiros-Ramirez MA, Hill MQ, Hahn CA, Zuffi S, O’Toole A, Black MJ (2016) Body talk: crowdshaping realistic 3D avatars with words. ACM Trans Gr (TOG) 35(4):54Google Scholar
  32. Suryanarayana GK, Dubey R (2017) Image analyses of supersonic air-intake buzz and control by natural ventilation. J Vis 1–17Google Scholar
  33. Thies J, Zollhofer M, Stamminger M, Theobalt C, Niebner M (2016) Face2face: real-time face capture and reenactment of RGB videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 2387–2395Google Scholar
  34. Tran AT, Hassner T, Masi I, Medioni G (2016) Regressing robust and discriminative 3D morphable models with a very deep neural network. arXiv:1612.04904
  35. Vlasic D, Brand M, Pfister H, Popovic J (2005) Face transfer with multilinear models. ACM Trans Gr (TOG) 24(3):426–433CrossRefGoogle Scholar
  36. Zhang Y, Lin W, Zhou B, Chen Z, Sheng B, Wu J (2014) Facial expression cloning with elastic and muscle models. J Vis Commun Image Represent 25(5):916–927CrossRefGoogle Scholar
  37. Zhou S, Fu H, Liu L, Cohen-Or D, Han X (2010) Parametric reshaping of human bodies in images. ACM Trans Gr (TOG) 29(4):126Google Scholar

Copyright information

© The Visualization Society of Japan 2018

Authors and Affiliations

  • Jiang Du
    • 1
  • Dan Song
    • 1
  • Yanlong Tang
    • 1
  • Ruofeng Tong
    • 1
    Email author
  • Min Tang
    • 1
  1. 1.State Key Lab of CAD&CGZhejiang UniversityHangzhouChina

Personalised recommendations