Advertisement

MobileFace: 3D Face Reconstruction with Efficient CNN Regression

  • Nikolai ChinaevEmail author
  • Alexander Chigorin
  • Ivan Laptev
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11132)

Abstract

Estimation of facial shapes plays a central role for face transfer and animation. Accurate 3D face reconstruction, however, often deploys iterative and costly methods preventing real-time applications. In this work we design a compact and fast CNN model enabling real-time face reconstruction on mobile devices. For this purpose, we first study more traditional but slow morphable face models and use them to automatically annotate a large set of images for CNN training. We then investigate a class of efficient MobileNet CNNs and adapt such models for the task of shape regression. Our evaluation on three datasets demonstrates significant improvements in the speed and the size of our model while maintaining state-of-the-art reconstruction accuracy.

Keywords

3D face reconstruction Morphable model CNN 

References

  1. 1.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: SIGGRAPH Conference Proceedings (1999)Google Scholar
  2. 2.
    Romdhani, S., Vetter, T.: Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior. In: Computer Vision and Pattern Recognition (CVPR) (2005)Google Scholar
  3. 3.
    Piotraschke, M., Blanz, V.: Automated 3D face reconstruction from multiple images using quality measures. In: Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  4. 4.
    Blanz, V., Vetter, T.: Face recognition based on fitting a 3D morphable model. IEEE Trans. Patt. Anal. Mach. Intell. 25, 1063–1074 (2003)CrossRefGoogle Scholar
  5. 5.
    Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., Nießner, M.: Face2Face: real-time face capture and reenactment of RGB videos. In: Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  6. 6.
    Garrido, P., et al.: Reconstruction of personalized 3D face rigs from monocular video. ACM Trans. Graph. (Presented at SIGGRAPH 2016) (2016)Google Scholar
  7. 7.
    Zhu, X., Yan, J., Yi, D., Lei, Z., Li, S.Z.: Discriminative 3D morphable model fitting. In: Automatic Face and Gesture Recognition (FG) (2015)Google Scholar
  8. 8.
    Jeni, L.A., Cohn, J.F., Kanade, T.: Dense 3D face alignment from 2D videos in real-time. In: Automatic Face and Gesture Recognition (FG) (2015)Google Scholar
  9. 9.
    Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S.Z.: Face alignment across large poses: a 3D solution. In: Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  10. 10.
    Tewari, A., et al.: MoFA: model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In: International Conference on Computer Vision (ICCV) (2017)Google Scholar
  11. 11.
    Jackson, A., Bulat, A., Argyriou, V., Tzimiropoulos, G.: Large pose 3D face reconstruction from a single image via direct volumetric CNN regression. In: International Conference on Computer Vision (ICCV) (2017)Google Scholar
  12. 12.
    Richardson, E., Sela, M., Kimmel, R.: 3D face reconstruction by learning from synthetic data. In: Fourth International Conference on 3D Vision (3DV) (2016)Google Scholar
  13. 13.
    Tran, A., Hassner, T., Masi, I., Medioni, G.: Regressing robust and discriminative 3D morphable models with a very deep neural network. In: Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  14. 14.
    Richardson, E., Sela, M., Or-El, R., Kimmel, R.: Learning detailed face reconstruction from a single image. In: Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  15. 15.
    Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: Computer Vision and Pattern Recognition (CVPR) (2014)Google Scholar
  16. 16.
    Xiong, X., De la Torre, F.: Supervised descent method and its applications to face alignment. In: Computer Vision and Pattern Recognition (CVPR) (2013)Google Scholar
  17. 17.
    Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In: International Conference on Computer Vision (ICCV) (2017)Google Scholar
  18. 18.
    Trigeorgis, G., Snape, P., Nicolaou, M.A., Antonakos, E., Zafeiriou, S.: Mnemonic descent method: a recurrent process applied for end-to-end face alignment. In: Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  19. 19.
    Bas, A., Smith, W.A.P.: What does 2D geometric information really tell us about 3D face shape? Arxiv preprint (2017)Google Scholar
  20. 20.
    Zhu, X., Lei, Z., Yan, J., Yi, D., Li, S.Z.: High-fidelity pose and expression normalization for face recognition in the wild. In: Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  21. 21.
    Fried, O., Shechtman, E., Goldman, D.B., Finkelstein, A.: Perspective-aware manipulation of portrait photos. ACM Trans. Graph. (Proc. SIGGRAPH) (2016)Google Scholar
  22. 22.
    Keller, M., Knothe, R., Vetter, T.: 3D reconstruction of human faces from occluding contours. In: Computer Vision/Computer Graphics Collaboration Techniques (2007)Google Scholar
  23. 23.
    Saito, S., Wei, L., Hu, L., Nagano, K., Li, H.: Photorealistic facial texture inference using deep neural networks. In: Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  24. 24.
    Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.: A 3D facial expression database for facial behavior research. In: Automatic Face and Gesture Recognition (FG) (2006)Google Scholar
  25. 25.
    Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: the first facial landmark localization challenge. In: International Conference on Computer Vision (ICCV) (2013)Google Scholar
  26. 26.
    Kemelmacher-Shlizerman, I., Basri, R.: 3D face reconstruction from a single image using a single reference face shape. IEEE Trans. Patt. Anal. Mach. Intell. 33, 394–405 (2011)CrossRefGoogle Scholar
  27. 27.
    Suwajanakorn, S., Kemelmacher-Shlizerman, I., Seitz, S.M.: Total moving face reconstruction. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 796–812. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10593-2_52CrossRefGoogle Scholar
  28. 28.
    Roth, J., Tong, Y., Liu, X.: Adaptive 3D face reconstruction from unconstrained photo collections. In: Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  29. 29.
    Tewari, A., et al.: Self-supervised multi-level face model learning for monocular reconstruction at over 250 Hz. arxiv preprint (2017)Google Scholar
  30. 30.
    Dou, P., Shah, S.K., Kakadiaris, I.A.: End-to-end 3D face reconstruction with deep neural networks. In: Computer Vision and Pattern Recognition (CVPR), vol. 5 (2017)Google Scholar
  31. 31.
    Cao, C., Weng, Y., Zhou, S., Tong, Y., Zhou, K.: FaceWarehouse: a 3D facial expression database for visual computing. IEEE Trans. Vis. Comput. Graph. 20, 413–425 (2014)CrossRefGoogle Scholar
  32. 32.
    DeVries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017)
  33. 33.
    Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications (2017)Google Scholar
  34. 34.
    Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  35. 35.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning (ICML) (2015)Google Scholar
  36. 36.
    Ramamoorthi, R., Hanrahan, P.: A signal-processing framework for inverse rendering. In: SIGGRAPH (2001)Google Scholar
  37. 37.
    Ramamoorthi, R., Hanrahan, P.: An efficient representation for irradiance environment maps. In: 28th Annual Conference on Computer Graphics and Interactive Techniques (2001)Google Scholar
  38. 38.
    Tulyakov, S., Sebe, N.: Regressing a 3D face shape from a single image. In: International Conference on Computer Vision (ICCV) (2015)Google Scholar
  39. 39.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  40. 40.
    Tran, A.T., Hassner, T., Iacopo, M., Paz, E., Nirkin, Y., Medioni, G.: Extreme 3D face reconstruction: looking past occlusions. arxiv preprint (2017)Google Scholar
  41. 41.
    Sela, M., Richardson, E., Kimmel, R.: Unrestricted facial geometry reconstruction using image-to-image translation. In: International Conference on Computer Vision (ICCV) (2017)Google Scholar
  42. 42.
    Bagdanov, A.D., Del Bimbo, A., Masi, I.: The florence 2D/3D hybrid face dataset. In: Proceedings of the 2011 Joint ACM Workshop on Human Gesture and Behavior Understanding (2011)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Nikolai Chinaev
    • 1
    Email author
  • Alexander Chigorin
    • 1
  • Ivan Laptev
    • 1
    • 2
  1. 1.VisionLabsAmsterdamThe Netherlands
  2. 2.Inria, WILLOW, Departement d’Informatique de l’Ecole Normale SuperieurePSL Research UniversityParisFrance

Personalised recommendations