Abstract
Face views are particularly important in person-to-person communication. Differenes between the camera location and the face orientation can result in undesirable facial appearances of the participants during video conferencing. This phenomenon is particularly noticeable when using devices where the front-facing camera is placed in unconventional locations such as below the display or within the keyboard. In this paper, we take a video stream from a single RGB camera as input, and generate a video stream that emulates the view from a virtual camera at a designated location. The most challenging issue in this problem is that the corrected view often needs out-of-plane head rotations. To address this challenge, we reconstruct the 3D face shape and re-render it into synthesized frames according to the virtual camera location. To output the corrected video stream with natural appearance in real time, we propose several novel techniques including accurate eyebrow reconstruction, high-quality blending between the corrected face image and background, and template-based 3D reconstruction of glasses. Our system works well for different lighting conditions and skin tones, and can handle users wearing glasses. Extensive experiments and user studies demonstrate that our method provides high-quality results.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Monk, A. F.; Gale, C. A look is worth a thousand words: Full gaze awareness in video-mediated conversation. Discourse Processes Vol. 33, No. 3, 257–278, 2002.
Grayson, D. M.; Monk, A. F. Are You looking at me? Eye contact and desktop video conferencing. ACM Transactions on Computer-Human Interaction Vol. 10, No. 3, 221–243, 2003.
Mukawa, N.; Oka, T.; Arai, K.; Yuasa, M. What is connected by mutual gaze: User’s behavior in video-mediated communication. In: Proceedings of the Extended Abstracts on Human Factors in Computing Systems, 1677–1680, 2005.
Ishii, H.; Kobayashi, M. ClearBoard: A seamless medium for shared drawing and conversation with eye contact. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 525–532, 1992.
Okada, K. I.; Maeda, F.; Ichikawaa, Y.; Matsushita, Y. Multiparty videoconferencing at virtual social distance: MAJIC design. In: Proceedings of the ACM Conference on Computer Supported Cooperative Work, 385–393, 1994.
Matusik, W.; Buehler, C.; Raskar, R.; Gortler, S. J.; McMillan, L. Image-based visual hulls. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, 369–374, 2000.
Matusik, W.; Pfister, H. 3D TV: A scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Transactions on Graphics Vol. 23, No. 3, 814–824, 2004.
Kuster, C.; Popa, T.; Zach, C.; Gotsman, C.; Gross, M. H. FreeCam: A hybrid camera system for interactive free-viewpoint video. In: Vision, Modeling, and Visualization. Eisert, P.; Polthier, K.; Hornegger J. Eds. The Eurographics Association, 17–24, 2011.
Kuster, C.; Popa, T.; Bazin, J. C.; Gotsman, C.; Gross, M. Gaze correction for home video conferencing. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 174, 2012.
Giger, D.; Bazin, J. C.; Kuster, C.; Popa, T.; Gross, M. Gaze correction with a single webcam. In: Proceedings of the IEEE International Conference on Multimedia and Expo, 1–6, 2014.
Kononenko, D.; Lempitsky, V. Learning to look up: Realtime monocular gaze correction using machine learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4667–4675, 2015.
Hsu, C. F.; Wang, Y. S.; Lei, C. L.; Chen, K. T. Look at me! Correcting eye gaze in live video communication. ACM Transactions on Multimedia Computing, Communications, and Applications Vol. 15, No. 2, Article No. 38, 2019.
He, Z.; Spurr, A.; Zhang, X. C.; Hilliges, O. Photo-realistic monocular gaze redirection using generative adversarial networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 6931–6940, 2019.
Zhang, J. C.; Chen, J. J.; Tang, H.; Wang, W.; Yan, Y.; Sangineto, E., Sebe, N. Dual in-painting model for unsupervised gaze correction and animation in the wild. In: Proceedings of the 28th ACM International Conference on Multimedia, 1588–1596, 2020.
Thies, J.; Zollhöfer, M.; Stamminger, M.; Theobalt, C.; Nießner, M. Face2Face: Real-time face capture and reenactment of RGB videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2387–2395, 2016.
Fried, O.; Shechtman, E.; Goldman, D. B.; Finkelstein, A. Perspective-aware manipulation of portrait photos. ACM Transactions on Graphics Vol. 35, No. 4, Article No. 128, 2016.
Shu, Z. X.; Hadap, S.; Shechtman, E.; Sunkavalli, K.; Paris, S.; Samaras, D. Portrait lighting transfer using a mass transport approach. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 2, 2017.
Nagano, K.; Luo, H. W.; Wang, Z. J.; Seo, J.; Xing, J.; Hu, L. W.; Wei, L.; Li, H. Deep face normalization. ACM Transactions on Graphics Vol. 38, No. 6, Article No. 183, 2019.
Zollhöfer, M.; Thies, J.; Garrido, P.; Bradley, D.; Beeler, T.; Pérez, P.; Stamminger, M.; Nießner, M.; Theobalt, C. State of the art on monocular 3D face reconstruction, tracking, and applications. Computer Graphics Forum Vol. 37, No. 2, 523–550, 2018.
Blanz, V.; Vetter, T. A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, 187–194, 1999.
Cao, C.; Weng, Y. L.; Zhou, S.; Tong, Y. Y.; Zhou, K. FaceWarehouse: A 3D facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 3, 413–425, 2014.
Li, T. Y.; Bolkart, T.; Black, M. J.; Li, H.; Romero, J. Learning a model of facial shape and expression from 4D scans. ACM Transactions on Graphics Vol. 36, No. 6, Article No. 194, 2017.
Jiang, L.; Zhang, J. Y.; Deng, B. L.; Li, H.; Liu, L. G. 3D face reconstruction with geometry details from a single image. IEEE Transactions on Image Processing Vol. 27, No. 10, 4756–4770, 2018.
Richardson, E.; Sela, M. T.; Kimmel, R. 3D face reconstruction by learning from synthetic data. In: Proceedings of the 4th International Conference on 3D Vision, 460–469, 2016.
Zhu, X. Y.; Lei, Z.; Liu, X. M.; Shi, H. L.; Li, S. Z. Face alignment across large poses: A 3D solution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 146–155, 2016.
Richardson, E.; Sela, M. T.; Or-El, R.; Kimmel, R. Learning detailed face reconstruction from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5553–5562, 2017.
Jackson, A. S.; Bulat, A.; Argyriou, V.; Tzimiropoulos, G. Large pose 3D face reconstruction from a single image via direct volumetric CNN regression. In: Proceedings of the IEEE International Conference on Computer Vision, 1031–1039, 2017.
Sela, M. T.; Richardson, E.; Kimmel, R. Unrestricted facial geometry reconstruction using image-to-image translation. In: Proceedings of the IEEE International Conference on Computer Vision, 1585–1594, 2017.
Tewari, A.; Zollhöfer, M.; Kim, H.; Garrido, P.; Bernard, F.; Pérez, P.; Theobalt, C. MoFA: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In: Proceedings of the IEEE International Conference on Computer Vision, 3735–3744, 2017.
Tran, L.; Liu, X. Nonlinear 3D face morphable model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7346–7355, 2018.
Genova, K.; Cole, F.; Maschinot, A.; Sarna, A.; Vlasic, D.; Freeman, W. T. Unsupervised training for 3D morphable model regression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8377–8386, 2018.
Gecer, B.; Ploumpis, S.; Kotsia, I.; Zafeiriou, S. GANFIT: Generative adversarial network fitting for high fidelity 3D face reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1155–1164, 2019.
Guo, Y. D.; Zhang, J. Y.; Cai, J. F.; Jiang, B. Y.; Zheng, J. M. CNN-based real-time dense face reconstruction with inverse-rendered photo-realistic face images. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 41, No. 6, 1294–1307, 2019.
Deng, Y.; Yang, J. L.; Xu, S. C.; Chen, D.; Jia, Y. D.; Tong, X. Accurate 3D face reconstruction with weakly-supervised learning: From single image to image set. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 285–295, 2019.
Tewari, A.; Bernard, F.; Garrido, P.; Bharaj, G.; Elgharib, M.; Seidel, H. P.; Pérez, P.; Zollhöfer, M.; Theobalt, C. FML: Face model learning from videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10804–10814, 2019.
Garrido, P.; Zollhöfer, M.; Casas, D.; Valgaerts, L.; Varanasi, K.; Pérez, P.; Theobalt, C. Reconstruction of personalized 3D face rigs from monocular video. ACM Transactions on Graphics Vol. 35, No. 3, Article No. 28, 2016.
Cao, C.; Hou, Q. M.; Zhou, K. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Transactions on Graphics Vol. 33, No. 4, Article No. 43, 2014.
Zhai, D. M.; Liu, X. M.; Ji, X. Y.; Zhao, D. B.; Gao, W. Joint gaze correction and face beautification for conference video using dual sparsity prior. IEEE Transactions on Industrial Electronics Vol. 66, No. 12, 9601–9611, 2019.
Hassner, T.; Harel, S.; Paz, E.; Enbar, R. Effective face frontalization in unconstrained images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4295–4304, 2015.
Zhao, Y. J.; Huang, Z.; Li, T. Y.; Chen, W. K.; Legendre, C., Ren, X. L.; Shapiro, A.; Li, H. Learning perspective undistortion of portraits. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 7848–7858, 2019.
Yin, Y.; Jiang, S. Y.; Robinson, J. P.; Fu, Y. Dual-attention GAN for large-pose face frontalization. In: Proceedings of the 15th IEEE International Conference on Automatic Face and Gesture Recognition, 249–256, 2020.
Paysan, P.; Knothe, R.; Amberg, B.; Romdhani, S.; Vetter, T. A 3D face model for pose and illumination invariant face recognition. In: Proceedings of the 6th IEEE International Conference on Advanced Video and Signal Based Surveillance, 296–301, 2009.
Sumner, R. W.; Popović, J. Deformation transfer for triangle meshes. ACM Transactions on Graphics Vol. 23, No. 3, 399–405, 2004.
Müller, C. Spherical Harmonics. Springer Berlin Heidelberg, 1966.
He, K. M.; Zhang, X. Y.; Ren, S. Q.; Sun, J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778, 2016.
Bulat, A.; Tzimiropoulos, G. How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In: Proceedings of the IEEE International Conference on Computer Vision, 1021–1030, 2017.
Burt, P. J.; Adelson, E. H. A multiresolution spline with application to image mosaics. ACM Transactions on Graphics Vol. 2, No. 4, 217–236, 1983.
Kwatra, V.; Schödl, A.; Essa, I.; Turk, G.; Bobick, A. Graphcut textures. ACM Transactions on Graphics Vol. 22, No. 3, 277–286, 2003.
Boykov, Y.; Kolmogorov, V. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 26, No. 9, 1124–1137, 2004.
Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention. Lecture Notes in Computer Science, Vol. 9351. Navab, N.; Hornegger, J.; Wells, W.; Frangi, A. Eds. Springer Cham, 234–241, 2015.
Bouaziz, S.; Deuss, M.; Schwartzburg, Y.; Weise, T.; Pauly, M. Shape-up: Shaping discrete geometry with projections. Computer Graphics Forum Vol. 31, No. 5, 1657–1667, 2012.
Guillemot, C.; Le Meur, O. Image inpainting: Overview and recent advances. IEEE Signal Processing Magazine Vol. 31, No. 1, 127–144, 2014.
Kim, H.; Zollhöfer, M.; Tewari, A.; Thies, J.; Richardt, C.; Theobalt, C. InverseFaceNet: Deep monocular inverse face rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4625–4634, 2018.
Tewari, A.; Zollhoöfer, M.; Garrido, P.; Bernard, F.; Kim, H.; Pérez, P.; Theobalt, C. Self-supervised multi-level face model learning for monocular reconstruction at over 250 Hz. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2549–2559, 2018.
Tewari, A.; Bernard, F.; Garrido, P.; Bharaj, G.; Elgharib, M.; Seidel, H. P.; Pérez, P.; Zollhöfer, M.; Theobalt, C. FML: Face model learning from videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10804–10814, 2019.
Schroff, F.; Kalenichenko, D.; Philbin, J. FaceNet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 815–823, 2015.
Author information
Authors and Affiliations
Corresponding author
Additional information
Yudong Guo is a Ph.D. student in the School of Mathematical Sciences, University of Science and Technology of China (USTC). He obtained his bachelor degree from the same university in 2015. His research interests include computer vision and computer graphics.
Juyong Zhang is an associate professor in the School of Mathematical Sciences at USTC. He received his B.S. degree from USTC in 2006, and his Ph.D. degree from Nanyang Technological University, Singapore. His research interests include computer graphics, computer vision, and numerical optimization. He is an associate editor of The Visual Computer.
Yihua Chen is a master student at the School of Mathematical Sciences, USTC. He received his bachelor degree in mathematical sciences from USTC in 2020. His research interests include computer vision and computer graphics.
Hongrui Cai is a master student at the School of Data Science, USTC. He got his bachelor degree from South China University of Technology in 2019. His research interests include computer vision and computer graphics.
Zhangjin Huang received his B.S. and Ph.D. degrees in computational mathematics from USTC in 1999 and 2005, respectively. He is currently an associate professor in the School of Computer Science and Technology, USTC. His current research interests include computer graphics and image processing.
Bailin Deng is a lecturer in the School of Computer Science & Informatics at Cardiff University. He received his B.Eng. degree in computer software (2005) and his M.Sc. degree in computer science (2008) from Tsinghua University, China, and his Ph.D. degree in technical mathematics (2011) from Vienna University of Technology, Austria. His research interests include geometry processing, numerical optimization, computational design, and digital fabrication. He is a member of the IEEE.
Electronic supplementary material
Supplementary material, approximately 19.7 MB.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Guo, Y., Zhang, J., Chen, Y. et al. Real-time face view correction for front-facing cameras. Comp. Visual Media 7, 437–452 (2021). https://doi.org/10.1007/s41095-021-0215-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-021-0215-y