Photorealistic Face Transfer in 2D and 3D Video

  • Daniel Merget
  • Philipp Tiefenbacher
  • Mohammadreza Babaee
  • Nikola Mitov
  • Gerhard Rigoll
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9358)


3D face transfer has been employed in a wide field of settings such as videoconferencing, gaming, or Hollywood movie production. State-of-the-art algorithms often suffer from a high sensitivity to tracking errors, require manual post-processing, or are overly complex in terms of computation time. Addressing these issues, we propose a lightweight system which is capable to transfer facial features in both 2D and 3D. This is accomplished by finding a dense correspondence between a source and target face, and then performing Poisson cloning. We solve the correspondence problem efficiently by a sparse initial registration and a subsequent warping, which is refined in a surface matching step using topological projections. Additional processing power is saved by converting extrapolation problems to simple interpolation problems without loss of precision. The final results are photorealistic face transfers in either 2D or 3D between arbitrary facial video streams.


  1. 1.
    Albarelli, A., Rodolà, E., Torsello, A.: Fast and accurate surface alignment through an isometry-enforcing game. Pattern Recogn. 48(7), 2209–2226 (2015)CrossRefzbMATHGoogle Scholar
  2. 2.
    Alexander, O., Rogers, M., Lambeth, W., Chiang, M., Debevec, P.: The digital emily project: photoreal facial modeling and animation. In: SIGGRAPH 2009 Courses, pp. 12:1–12:15. ACM (2009)Google Scholar
  3. 3.
    Blanz, V., Scherbaum, K., Seidel, H.P.: Fitting a morphable model to 3D scans of faces. In: International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8 (2007)Google Scholar
  4. 4.
    Borshukov, G., Piponi, D., Larsen, O., Lewis, J.P., Tempelaar-Lietz, C.: Universal capture - image-based facial animation for “The Matrix Reloaded". In: SIGGRAPH 2005 Courses. ACM (2005)Google Scholar
  5. 5.
    Bouaziz, S., Wang, Y., Pauly, M.: Online modeling for realtime facial animation. Trans. Graph. (SIGGRAPH) 32(4), 40:1–40:10 (2013)zbMATHGoogle Scholar
  6. 6.
    Cupisz, R.: Light probe interpolation using tetrahedral tessellations. In: Game Developers Conference (GDC) (2012).
  7. 7.
    Dale, K., Sunkavalli, K., Johnson, M.K., Vlasic, D., Matusik, W., Pfister, H.: Video face replacement. Trans. Graphics (SIGGRAPH) 30 (2011)Google Scholar
  8. 8.
    Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., Lischinski, D.: Coordinates for instant image cloning. Trans. Graph. (SIGGRAPH) 28(3), 67:1–67:9 (2009)Google Scholar
  9. 9.
    Indiegogo: Facerig (2013).
  10. 10.
    Kemelmacher-Shlizerman, I., Sankar, A., Shechtman, E., Seitz, S.M.: Being John Malkovich. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part I. LNCS, vol. 6311, pp. 341–353. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  11. 11.
    Kent, J.R., Carlson, W.E., Parent, R.E.: Shape transformation for polyhedral objects. Trans. Graph. (SIGGRAPH) 26(2), 47–54 (1992)Google Scholar
  12. 12.
    Kuster, C., Popa, T., Bazin, J.C., Gotsman, C., Gross, M.: Gaze correction for home video conferencing. Trans. Graph. (SIGGRAPH) 31(6), 1–6 (2012)CrossRefGoogle Scholar
  13. 13.
    Litke, N., Droske, M., Rumpf, M., Schröder, P.: An image processing approach to surface matching. In: Proceedings of the Eurographics Symposium on Geometry Processing. Eurographics Association (2005)Google Scholar
  14. 14.
    Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition. In: Proceedings of the Advanced Video and Signal-based Surveillance (AVSS). IEEE (2009)Google Scholar
  15. 15.
    Prez, P., Gangnet, M., Blake, A.: Poisson image editing. Trans. Graph. (SIGGRAPH) 22(3), 313–318 (2003)CrossRefGoogle Scholar
  16. 16.
    Savran, A., Alyüz, N., Dibeklioğlu, H., Çeliktutan, O., Gökberk, B., Sankur, B., Akarun, L.: Bosphorus database for 3D face analysis. In: Schouten, B., Juul, N.C., Drygajlo, A., Tistarelli, M. (eds.) BIOID 2008. LNCS, vol. 5372, pp. 47–56. Springer, Heidelberg (2008) CrossRefGoogle Scholar
  17. 17.
    Shiratori, T., Mahler, M., Trezevant, W., Hodgins, J.K.: Symposium on 3D User Interfaces (3DUI), pp. 59–66 (2013)Google Scholar
  18. 18.
    Singular Inversions Inc: Facegen modeller manual (2011).
  19. 19.
    Umeyama, S.: Least-squares estimation of transformation parameters between two point patterns. Trans. Pattern Anal. Mach. Intell. (PAMI) 13(4), 376–380 (1991)CrossRefGoogle Scholar
  20. 20.
    Uřičář, M., Franc, V., Thomas, D., Akihiro, S., Hlaváč, V.: Real-time multi-view facial landmark detector learned by the structured output SVM. In: BWILD 2015: Proceedings of the Automatic Face and Gesture Recognition Conference and Workshops. IEEE (2015)Google Scholar
  21. 21.
    Vlasic, D., Brand, M., Pfister, H., Popović, J.: Face transfer with multilinear models. Trans. Graph. (SIGGRAPH) 24(3), 426–433 (2005)CrossRefGoogle Scholar
  22. 22.
    Weise, T., Bouaziz, S., Li, H., Pauly, M.: Realtime performance-based facial animation. Trans. Graph. (SIGGRAPH) 30(4), 77:1–77:10 (2011)Google Scholar
  23. 23.
    Wu, Y., Ijiri, Y., Yang, M.H.: Multiple non-rigid surface detection and registration. In: International Conference on Computer Vision (ICCV), pp. 1992–1999 (2013)Google Scholar
  24. 24.
    Yoshizawa, S., Yokota, H.: Poisson image analogy: texture-aware seamless cloning. In: Eurographics - Posters. Eurographics Association (2013)Google Scholar
  25. 25.
    Zhou, E., Fan, H., Cao, Z., Jiang, Y., Yin, Q.: Extensive facial landmark localization with coarse-to-fine convolutional neural network. In: ICCV Workshop on 300 Faces in-the-Wild Challenge (2013)Google Scholar
  26. 26.
    Zhou, F., Brandt, J., Lin, Z.: Exemplar-based graph matching for robust facial landmark localization. In: International Conference on Computer Vision (ICCV) (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Open Access This chapter is distributed under the terms of the Creative Commons Attribution Noncommercial License, which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Authors and Affiliations

  • Daniel Merget
    • 1
  • Philipp Tiefenbacher
    • 1
  • Mohammadreza Babaee
    • 1
  • Nikola Mitov
    • 1
  • Gerhard Rigoll
    • 1
  1. 1.Institute for Human-Machine CommunicationTUMMunichGermany

Personalised recommendations