Learning 3DMM Deformation Coefficients for Rendering Realistic Expression Images

  • Claudio FerrariEmail author
  • Stefano Berretti
  • Pietro Pala
  • Alberto Del Bimbo
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11010)


Analysis of facial expressions is a task of increasing interest in Computer Vision, with many potential applications. However, collecting images with labeled expression for many subjects is a quite complicated operation. In this paper, we propose a solution that use a particular 3D morphable model (3DMM) that, starting from a neutral image of a target subject, is capable of producing a realistic expressive face image of the same subject. This is possible thanks to the fact the used 3DMM can effectively and efficiently fit to 2D images, and then deform itself under the action of deformation parameters that are learned expression-by-expression in a subject-independent manner. Ultimately, the application of such deformation parameters to the neutral model of a subject allows the rendering of realistic expressive images of the subject. In the experiments, we demonstrate that such deformation parameters can be learned even from a small set of training data using simple statistical tools; despite this simplicity, we show that very realistic subject-dependent expression renderings can be obtained with our method. Furthermore, robustness to cross dataset tests is also evidenced.


3D morphable model Deformation components learning Facial expression synthesis 


  1. 1.
    Amberg, B., Romdhani, S., Vetter, T.: Optimal step nonrigid ICP algorithms for surface registration. In: IEEE International Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, pp. 1–8, June 2007Google Scholar
  2. 2.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: ACM Conference on Computer Graphics and Interactive Techniques (1999)Google Scholar
  3. 3.
    Blanz, V., Vetter, T.: Face recognition based on fitting a 3D morphable model. IEEE Trans. Pattern Anal. Mach. Intell. 25(9), 1063–1074 (2003)CrossRefGoogle Scholar
  4. 4.
    Booth, J., Antonakos, E., Ploumpis, S., Trigeorgis, G., Panagakis, Y., Zafeiriou, S.: 3D face morphable models “in-the-wild”. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5464–5473, July 2017.
  5. 5.
    Booth, J., Roussos, A., Zafeiriou, S., Ponniahand, A., Dunaway, D.: A 3D morphable model learnt from 10,000 faces. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5543–5552 (2016)Google Scholar
  6. 6.
    Cosker, D., Krumhuber, E., Hilton, A.: Perception of linear and nonlinear motion properties using a FACS validated 3D facial model. In: ACM Applied Perception in Graphics and Vision (2010)Google Scholar
  7. 7.
    Cosker, D., Krumhuber, E., Hilton, A.: A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling. In: International Conference on Computer Vision (2011)Google Scholar
  8. 8.
    Cosker, D., Krumhuber, E., Hilton, A.: Perceived emotionality of linear and non-linear AUs synthesised using a 3D dynamic morphable facial model. In: Proceedings of the Facial Analysis and Animation, FAA 2015, p. 7:1. ACM (2015)Google Scholar
  9. 9.
    Dou, P., Shah, S.K., Kakadiaris, I.A.: End-to-end 3D face reconstruction with deep neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1503–1512, July 2017.
  10. 10.
    Ekman, P.: Facial expression and emotion. Am. Anthropol. 48(4), 384–392 (1992)Google Scholar
  11. 11.
    Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, CA (1978)Google Scholar
  12. 12.
    Ferrari, C., Lisanti, G., Berretti, S., Del Bimbo, A.: A dictionary learning-based 3D morphable shape model. IEEE Trans. Multimedia 19(12), 2666–2679 (2017). Scholar
  13. 13.
    Hu, G., et al.: Face recognition using a unified 3D morphable model. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 73–89. Springer, Cham (2016). Scholar
  14. 14.
    Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical report 07-49, University of Massachusetts, Amherst, October 2007Google Scholar
  15. 15.
    Huang, Y., Khan, S.M.: DyadGAN: generating facial expressions in dyadic interactions. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2259–2266, July 2017.
  16. 16.
    Huber, P., Kopp, P., Rätsch, M., Christmas, W.J., Kittler, J.: 3D face tracking and texture fusion in the wild. CoRR abs/1605.06764 (2016).
  17. 17.
    Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: IEEE Conference on Computer Vision and Pattern Recognition (2014)Google Scholar
  18. 18.
    Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: IEEE Conference on Computer Vision and Pattern Recognition-Workshops (2010)Google Scholar
  19. 19.
    Mairal, J., Bach, F., Ponce, J., Sapiro, G.: Online dictionary learning for sparse coding. In: International Conference on Machine Learning (2009)Google Scholar
  20. 20.
    Masi, I., Ferrari, C., Del Bimbo, A., Medioni, G.: Pose independent face recognition by localizing local binary patterns via deformation components. In: International Conference on Pattern Recognition (2014)Google Scholar
  21. 21.
    Patel, A., Smith, W.A.P.: 3D morphable face models revisited. In: IEEE Conference on Computer Vision and Pattern Recognition (2009)Google Scholar
  22. 22.
    Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition. In: IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 296–301 (2009)Google Scholar
  23. 23.
    Qiao, F., Yao, N., Jiao, Z., Li, Z., Chen, H., Wang, H.: Geometry-contrastive generative adversarial network for facial expression synthesis. CoRR abs/1802.01822 (2018).
  24. 24.
    Ramanathan, S., Kassim, A., Venkatesh, Y.V., Wah, W.S.: Human facial expression recognition using a 3D morphable model. In: International Conference on Image Processing (2006)Google Scholar
  25. 25.
    Romdhani, S., Vetter, T.: Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior. In: IEEE Conference on Computer Vision and Pattern Recognition (2005)Google Scholar
  26. 26.
    Savran, A., et al.: Bosphorus database for 3D face analysis. In: Schouten, B., Juul, N.C., Drygajlo, A., Tistarelli, M. (eds.) BioID 2008. LNCS, vol. 5372, pp. 47–56. Springer, Heidelberg (2008). Scholar
  27. 27.
    Tran, A.T., Hassner, T., Masi, I., Medioni, G.: Regressing robust and discriminative 3D morphable models with a very deep neural network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5163–5172, July 2017Google Scholar
  28. 28.
    Ujir, H., Spann, M.: Facial expression recognition using FAPs-based 3DMMM. In: Tavares, J., Natal Jorge, R. (eds.) Topics in Medical Image Processing and Computational Vision. LNCVB, vol. 8, pp. 33–47. Springer, Dordrecht (2013). Scholar
  29. 29.
    Yi, D., Lei, Z., Li, S.Z.: Towards pose robust face recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (2013)Google Scholar
  30. 30.
    Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.: A 3D facial expression database for facial behavior research. In: IEEE International Conference on Automatic Face and Gesture Recognition (2006)Google Scholar
  31. 31.
    Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S.Z.: Face alignment across large poses: a 3D solution. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Claudio Ferrari
    • 1
    Email author
  • Stefano Berretti
    • 1
  • Pietro Pala
    • 1
  • Alberto Del Bimbo
    • 1
  1. 1.Media Integration and Communication Center (MICC)University of FlorenceFlorenceItaly

Personalised recommendations