Abstract
3D face transfer has been employed in a wide field of settings such as videoconferencing, gaming, or Hollywood movie production. State-of-the-art algorithms often suffer from a high sensitivity to tracking errors, require manual post-processing, or are overly complex in terms of computation time. Addressing these issues, we propose a lightweight system which is capable to transfer facial features in both 2D and 3D. This is accomplished by finding a dense correspondence between a source and target face, and then performing Poisson cloning. We solve the correspondence problem efficiently by a sparse initial registration and a subsequent warping, which is refined in a surface matching step using topological projections. Additional processing power is saved by converting extrapolation problems to simple interpolation problems without loss of precision. The final results are photorealistic face transfers in either 2D or 3D between arbitrary facial video streams.
This is a preview of subscription content, log in via an institution.
References
Albarelli, A., Rodolà, E., Torsello, A.: Fast and accurate surface alignment through an isometry-enforcing game. Pattern Recogn. 48(7), 2209–2226 (2015)
Alexander, O., Rogers, M., Lambeth, W., Chiang, M., Debevec, P.: The digital emily project: photoreal facial modeling and animation. In: SIGGRAPH 2009 Courses, pp. 12:1–12:15. ACM (2009)
Blanz, V., Scherbaum, K., Seidel, H.P.: Fitting a morphable model to 3D scans of faces. In: International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8 (2007)
Borshukov, G., Piponi, D., Larsen, O., Lewis, J.P., Tempelaar-Lietz, C.: Universal capture - image-based facial animation for “The Matrix Reloaded". In: SIGGRAPH 2005 Courses. ACM (2005)
Bouaziz, S., Wang, Y., Pauly, M.: Online modeling for realtime facial animation. Trans. Graph. (SIGGRAPH) 32(4), 40:1–40:10 (2013)
Cupisz, R.: Light probe interpolation using tetrahedral tessellations. In: Game Developers Conference (GDC) (2012). http://gdcvault.com/play/1015312/Light-Probe-Interpolation-Using-Tetrahedral
Dale, K., Sunkavalli, K., Johnson, M.K., Vlasic, D., Matusik, W., Pfister, H.: Video face replacement. Trans. Graphics (SIGGRAPH) 30 (2011)
Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., Lischinski, D.: Coordinates for instant image cloning. Trans. Graph. (SIGGRAPH) 28(3), 67:1–67:9 (2009)
Indiegogo: Facerig (2013). http://www.facerig.com
Kemelmacher-Shlizerman, I., Sankar, A., Shechtman, E., Seitz, S.M.: Being John Malkovich. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part I. LNCS, vol. 6311, pp. 341–353. Springer, Heidelberg (2010)
Kent, J.R., Carlson, W.E., Parent, R.E.: Shape transformation for polyhedral objects. Trans. Graph. (SIGGRAPH) 26(2), 47–54 (1992)
Kuster, C., Popa, T., Bazin, J.C., Gotsman, C., Gross, M.: Gaze correction for home video conferencing. Trans. Graph. (SIGGRAPH) 31(6), 1–6 (2012)
Litke, N., Droske, M., Rumpf, M., Schröder, P.: An image processing approach to surface matching. In: Proceedings of the Eurographics Symposium on Geometry Processing. Eurographics Association (2005)
Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition. In: Proceedings of the Advanced Video and Signal-based Surveillance (AVSS). IEEE (2009)
Prez, P., Gangnet, M., Blake, A.: Poisson image editing. Trans. Graph. (SIGGRAPH) 22(3), 313–318 (2003)
Savran, A., Alyüz, N., Dibeklioğlu, H., Çeliktutan, O., Gökberk, B., Sankur, B., Akarun, L.: Bosphorus database for 3D face analysis. In: Schouten, B., Juul, N.C., Drygajlo, A., Tistarelli, M. (eds.) BIOID 2008. LNCS, vol. 5372, pp. 47–56. Springer, Heidelberg (2008)
Shiratori, T., Mahler, M., Trezevant, W., Hodgins, J.K.: Symposium on 3D User Interfaces (3DUI), pp. 59–66 (2013)
Singular Inversions Inc: Facegen modeller manual (2011). http://www.facegen.com
Umeyama, S.: Least-squares estimation of transformation parameters between two point patterns. Trans. Pattern Anal. Mach. Intell. (PAMI) 13(4), 376–380 (1991)
Uřičář, M., Franc, V., Thomas, D., Akihiro, S., Hlaváč, V.: Real-time multi-view facial landmark detector learned by the structured output SVM. In: BWILD 2015: Proceedings of the Automatic Face and Gesture Recognition Conference and Workshops. IEEE (2015)
Vlasic, D., Brand, M., Pfister, H., Popović, J.: Face transfer with multilinear models. Trans. Graph. (SIGGRAPH) 24(3), 426–433 (2005)
Weise, T., Bouaziz, S., Li, H., Pauly, M.: Realtime performance-based facial animation. Trans. Graph. (SIGGRAPH) 30(4), 77:1–77:10 (2011)
Wu, Y., Ijiri, Y., Yang, M.H.: Multiple non-rigid surface detection and registration. In: International Conference on Computer Vision (ICCV), pp. 1992–1999 (2013)
Yoshizawa, S., Yokota, H.: Poisson image analogy: texture-aware seamless cloning. In: Eurographics - Posters. Eurographics Association (2013)
Zhou, E., Fan, H., Cao, Z., Jiang, Y., Yin, Q.: Extensive facial landmark localization with coarse-to-fine convolutional neural network. In: ICCV Workshop on 300 Faces in-the-Wild Challenge (2013)
Zhou, F., Brandt, J., Lin, Z.: Exemplar-based graph matching for robust facial landmark localization. In: International Conference on Computer Vision (ICCV) (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Merget, D., Tiefenbacher, P., Babaee, M., Mitov, N., Rigoll, G. (2015). Photorealistic Face Transfer in 2D and 3D Video. In: Gall, J., Gehler, P., Leibe, B. (eds) Pattern Recognition. DAGM 2015. Lecture Notes in Computer Science(), vol 9358. Springer, Cham. https://doi.org/10.1007/978-3-319-24947-6_33
Download citation
DOI: https://doi.org/10.1007/978-3-319-24947-6_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-24946-9
Online ISBN: 978-3-319-24947-6
eBook Packages: Computer ScienceComputer Science (R0)