Advertisement

Rendering Realistic Subject-Dependent Expression Images by Learning 3DMM Deformation Coefficients

  • Claudio FerrariEmail author
  • Stefano Berretti
  • Pietro Pala
  • Alberto Del Bimbo
Conference paper
First Online:
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11130)

Abstract

Automatic analysis of facial expressions is now attracting an increasing interest, thanks to the many potential applications it can enable. However, collecting images with labeled expression for large sets of images or videos is a quite complicated operation that, in most of the cases, requires substantial human intervention. In this paper, we propose a solution that, starting from a neutral image of a subject, is capable of producing a realistic expressive face image of the same subject. This is possible thanks to the use of a particular 3D morphable model (3DMM) that can effectively and efficiently fit to 2D images, and then deform itself under the action of deformation parameters learned expression-by-expression in a subject-independent manner. Ultimately, the application of such deformation parameters to the neutral model of a subject allows the rendering of realistic expressive images of the subject. Experiments demonstrate that such deformation parameters can be learned from a small set of training data using simple statistical tools; despite this simplicity, very realistic subject-dependent expression renderings can be obtained. Furthermore, robustness to cross dataset tests is also evidenced.

Keywords

3D morphable model Deformation components learning Facial expression synthesis 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgments

The authors would like to thank Gabriele Barlacchi, Francesco Lombardi, Alessandro Sestini, and Alessandro Soci for developing and experimenting part of the code used in this work.

References

  1. 1.
    Amberg, B., Knothe, R., Vetter, T.: Expression invariant 3D face recognition with a morphable model. In: IEEE International Conference on Automatic Face and Gesture Recognition (2008)Google Scholar
  2. 2.
    Amberg, B., Romdhani, S., Vetter, T.: Optimal step nonrigid ICP algorithms for surface registration. In: IEEE International Conference on Computer Vision and Pattern Recognition, pp. 1–8. Minneapolis, MN, June 2007Google Scholar
  3. 3.
    Baranyi, R., Willinger, R., Lederer, N., Grechenig, T., Schramm, W.: Chances for serious games in rehabilitation of stroke patients on the example of utilizing the Wii Fit Balance Board. In: IEEE International Conference on Serious Games and Applications for Health (SeGAH), pp. 1–7, May 2013.  https://doi.org/10.1109/SeGAH.2013.6665319
  4. 4.
    Bejaoui, H., Ghazouani, H., Barhoumi, W.: Fully automated facial expression recognition using 3D morphable model and mesh-local binary pattern. In: Blanc-Talon, J., Penne, R., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2017. LNCS, vol. 10617, pp. 39–50. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-70353-4_4CrossRefGoogle Scholar
  5. 5.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: ACM Conference on Computer Graphics and Interactive Techniques (1999)Google Scholar
  6. 6.
    Blanz, V., Vetter, T.: Face recognition based on fitting a 3D morphable model. IEEE Trans Pattern Anal. Mach. Intell. 25(9), 1063–1074 (2003)CrossRefGoogle Scholar
  7. 7.
    Booth, J., Antonakos, E., Ploumpis, S., Trigeorgis, G., Panagakis, Y., Zafeiriou, S.: 3D face morphable models “in-the-wild”. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5464–5473, July 2017.  https://doi.org/10.1109/CVPR.2017.580
  8. 8.
    Booth, J., Roussos, A., Zafeiriou, S., Ponniahand, A., Dunaway, D.: A 3D morphable model learnt from 10,000 faces. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5543–5552 (2016)Google Scholar
  9. 9.
    Brunton, A., Bolkart, T., Wuhrer, S.: Multilinear wavelets: a statistical shape space for human faces. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 297–312. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10590-1_20CrossRefGoogle Scholar
  10. 10.
    Chang, F.J., Tran, A., Hassner, T., Masi, I., Nevatia, R., Medioni, G.: ExpNet: landmark-free, deep, 3d facial expressions. In: IEEE Conference on Automatic Face and Gesture Recognition (2018)Google Scholar
  11. 11.
    Cosker, D., Krumhuber, E., Hilton, A.: Perception of linear and nonlinear motion properties using a FACS validated 3D facial model. In: ACM Applied Perception in Graphics and Vision (2010)Google Scholar
  12. 12.
    Cosker, D., Krumhuber, E., Hilton, A.: A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling. In: International Conference on Computer Vision (2011)Google Scholar
  13. 13.
    Cosker, D., Krumhuber, E., Hilton, A.: Perceived emotionality of linear and non-linear AUs synthesised using a 3D dynamic morphable facial model. In: Proceedings of the Facial Analysis and Animation, FAA 2015, pp. 7:1–7:1. ACM (2015)Google Scholar
  14. 14.
    Dou, P., Shah, S.K., Kakadiaris, I.A.: End-to-end 3D face reconstruction with deep neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1503–1512, July 2017.  https://doi.org/10.1109/CVPR.2017.164
  15. 15.
    Ferrari, C., Lisanti, G., Berretti, S., Del Bimbo, A.: A dictionary learning-based 3D morphable shape model. IEEE Trans. Multimedia 19(12), 2666–2679 (2017).  https://doi.org/10.1109/TMM.2017.2707341CrossRefGoogle Scholar
  16. 16.
    Grupp, M., Kopp, P., Huber, P., Rätsch, M.: A 3D face modelling approach for pose-invariant face recognition in a human-robot environment. CoRR abs/1606.00474 (2016)Google Scholar
  17. 17.
    Hu, G., et al.: Face recognition using a unified 3D morphable model. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 73–89. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46484-8_5CrossRefGoogle Scholar
  18. 18.
    Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical report 07–49. University of Massachusetts, Amherst, October 2007Google Scholar
  19. 19.
    Huang, Y., Khan, S.M.: DyadGAN: generating facial expressions in dyadic interactions. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2259–2266, July 2017.  https://doi.org/10.1109/CVPRW.2017.280
  20. 20.
    Huber, P., Kopp, P., Rätsch, M., Christmas, W.J., Kittler, J.: 3D face tracking and texture fusion in the wild. CoRR abs/1605.06764 (2016). http://arxiv.org/abs/1605.06764
  21. 21.
    Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: IEEE Conference on Computer Vision and Pattern Recognition (2014)Google Scholar
  22. 22.
    Liarokapis, F., Debattista, K., Vourvopoulos, A., Petridis, P., Ene, A.: Comparing interaction techniques for serious games through brain-computer interfaces: a user perception evaluation study. Entertain. Comput. 5(4), 391–399 (2014).  https://doi.org/10.1016/j.entcom.2014.10.004. http://www.sciencedirect.com/science/article/pii/S1875952114000391CrossRefGoogle Scholar
  23. 23.
    Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: IEEE Conference on Computer Vision and Pattern Recognition-Workshops (2010)Google Scholar
  24. 24.
    Lüthi, M., Jud, C., Gerig, T., Vetter, T.: Gaussian process morphable models. CoRR abs/1603.07254 (2016). http://arxiv.org/abs/1603.07254
  25. 25.
    Mairal, J., Bach, F., Ponce, J., Sapiro, G.: Online dictionary learning for sparse coding. In: International Conference on Machine Learning (2009)Google Scholar
  26. 26.
    Masi, I., Ferrari, C., Del Bimbo, A., Medioni, G.: Pose independent face recognition by localizing local binary patterns via deformation components. In: International Conference on Pattern Recognition (2014)Google Scholar
  27. 27.
    Patel, A., Smith, W.A.P.: 3D morphable face models revisited. In: IEEE Conference on Computer Vision and Pattern Recognition (2009)Google Scholar
  28. 28.
    Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition. In: IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 296–301 (2009)Google Scholar
  29. 29.
    Qiao, F., Yao, N., Jiao, Z., Li, Z., Chen, H., Wang, H.: Geometry-contrastive generative adversarial network for facial expression synthesis. CoRR abs/1802.01822 (2018). http://arxiv.org/abs/1802.01822
  30. 30.
    Ramanathan, S., Kassim, A., Venkatesh, Y.V., Wah, W.S.: Human facial expression recognition using a 3D morphable model. In: International Conference on Image Processing (2006)Google Scholar
  31. 31.
    Romdhani, S., Vetter, T.: Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior. In: IEEE Conference on Computer Vision and Pattern Recognition (2005)Google Scholar
  32. 32.
    Savran, A., et al.: Bosphorus database for 3D face analysis. In: Schouten, B., Juul, N.C., Drygajlo, A., Tistarelli, M. (eds.) BioID 2008. LNCS, vol. 5372, pp. 47–56. Springer, Heidelberg (2008).  https://doi.org/10.1007/978-3-540-89991-4_6CrossRefGoogle Scholar
  33. 33.
    Tran, A.T., Hassner, T., Masi, I., Medioni, G.: Regressing robust and discriminative 3D morphable models with a very deep neural network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5163–5172, July 2017Google Scholar
  34. 34.
    Ujir, H., Spann, M.: Facial expression recognition using FAPs-based 3DMM. In: Tavares, J., Natal Jorge, R. (eds.) Topics in Medical Image Processing and Computational Vision. LNCS, pp. 33–47. Springer, Netherlands (2013).  https://doi.org/10.1007/978-94-007-0726-9_CrossRefGoogle Scholar
  35. 35.
    Yi, D., Lei, Z., Li, S.Z.: Towards pose robust face recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (2013)Google Scholar
  36. 36.
    Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.: A 3D facial expression database for facial behavior research. In: IEEE International Conference on Automatic Face and Gesture Recognition (2006)Google Scholar
  37. 37.
    Zakari, H.M., Ma, M., Simmons, D.: A review of serious games for children with autism spectrum disorders (ASD). In: Ma, M., Oliveira, M.F., Baalsrud Hauge, J. (eds.) SGDA 2014. LNCS, vol. 8778, pp. 93–106. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-11623-5_9CrossRefGoogle Scholar
  38. 38.
    Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S.Z.: Face alignment across large poses: a 3D solution. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Claudio Ferrari
    • 1
    Email author
  • Stefano Berretti
    • 1
  • Pietro Pala
    • 1
  • Alberto Del Bimbo
    • 1
  1. 1.Media Integration and Communication CenterUniversity of FlorenceFlorenceItaly

Personalised recommendations

Cite paper

Buy options