Replacement of Facial Parts in Images

  • Jiang Du
  • Yanjing Wu
  • Dan Song
  • Ruofeng TongEmail author
  • Min Tang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10582)


It is interesting to edit facial appearance in images to create a desirable facial shape of persons. In this paper, we propose a novel method to modify facial appearance by replacing facial parts between arbitrarily paired images. To this end, our method consists of face segmentation, face reconstruction, mesh deformation and image editing. Given one source and one target image, the target image is first segmented into the front facial region and background image. Secondly, 3D facial models and relevant scene parameters are estimated from both images. Thirdly, the target facial part is replaced with the selected source part on the 3D mesh. Then, the new replaced 3D face is rendered into a facial image. Finally, the new facial image is generated by seamlessly blending the rendered image and background image. The main advantage of this method is that we transfer facial geometric information between images using 3D model, which can deal with arbitrarily paired images with the different facial viewpoint. We present several experimental results to show the effectiveness of our method and comparison with those existing methods to demonstrate that our method is more advantageous and flexible in terms of practical applications.


Facial parts replacement Mesh deformation Face reconstruction 



The research is supported in part by NSFC (61572424) and the People Programme (Marie Curie Actions) of the European Union’s Seventh Framework Programme FP7 (2007–2013) under REA grant agreement No. 612627-“AniNex”. Min Tang is supported in part by NSFC (61572423) and Zhejiang Provincial NSFC (LZ16F020003).


  1. 1.
    Chou, J.K., Yang, C.K., Gong, S.D.: Face-off: automatic alteration of facial features. Multimedia Tools Appl. 56(3), 569–596 (2012)CrossRefGoogle Scholar
  2. 2.
    Klum, S., Han, H., Jain, A.K., Klara, B.: Sketch based face recognition: Forensic vs. Composite sketches. In: 2013 International Conference on Biometrics (ICB), pp. 1–8. IEEE, Madrid, Spain (2013)Google Scholar
  3. 3.
  4. 4.
    Blanz, V., Scherbaum, K., Vetter, T., Seidel, H.P.: Exchanging faces in images. Comput. Graph. Forum 23(3), 669–676 (2004)CrossRefGoogle Scholar
  5. 5.
    Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., Nayar, S.K.: Face Swapping: automatically replacing faces in photographs. ACM Trans. Graph. (TOG) 27(3), 39:1–39:8 (2008)CrossRefGoogle Scholar
  6. 6.
    Kemelmacher-Shlizerman, I.: Transfiguring portraits. ACM Trans. Graph. (TOG) 35(4), 94:1–94:8 (2016)CrossRefGoogle Scholar
  7. 7.
    Afifi, M., Hussain, K.F., Ibrahim, H.M., Omar, N.M., Video face replacement system using a modified Poisson blending technique. In: 2014 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), pp. 205–210. IEEE, Kuching, Malaysia (2014)Google Scholar
  8. 8.
    Nirkin, Y., Masi, I., Tran, A. T, Hassner, T., Medioni, G.: On Face Segmentation, Face Swapping, and Face Perception. arXiv preprint arXiv:1704.06729, (2017)
  9. 9.
    Liao, Q., Jin, X., Zeng, W.: Enhancing the symmetry and proportion of 3D face geometry. IEEE Trans. Visual Comput. Graph. 18(10), 1704–1716 (2012)CrossRefGoogle Scholar
  10. 10.
    Zhao, H., Jin, X., Huang, X., Chai, M., Zhou, K.: Parametric weight-change reshaping for portrait images. IEEE Comput. Graph. Appl. 36 (2016)Google Scholar
  11. 11.
    Best-Rowden, L., Han, H., Otto, C., Klare, B.F., Jain, A.K.: Unconstrained face recognition: identifying a person of interest from a media collection. IEEE Trans. Inf. Forensics Secur. 9(12), 2144–2157 (2014)CrossRefGoogle Scholar
  12. 12.
    Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., Niebner, M.: Face2Face: real-time face capture and reenactment of RGB videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2387–2395. IEEE, Las Vegas, NV, USA (2016)Google Scholar
  13. 13.
    Cao, C., Weng, Y., Lin, S., Zhou, K.: 3D shape regression for real-time facial animation. ACM Trans. Graph. (TOG) 32(4), 41:1–41:10 (2013)CrossRefzbMATHGoogle Scholar
  14. 14.
    Li, H., Yu, J., Ye, Y., Bregler, C.: Realtime facial animation with on-the-fly correctives. ACM Trans. Graph. (TOG) 32(4), 42:1–42:10 (2013)zbMATHGoogle Scholar
  15. 15.
    Paysan, P., Knothe, R., Amberg, B.: A 3D face model for pose and illumination invariant face recognition. In: 6th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS 2009), pp. 296–301. IEEE, Genova, Italy (2009)Google Scholar
  16. 16.
    Jacobson, A., Tosun, E., Sorkine, O.: Mixed finite elements for variational surface modeling. Comput. Graph. Forum 29(5), 1565–1574 (2010)CrossRefGoogle Scholar
  17. 17.
    Wang, H., Cao, J., Liu, X., Wang, J., Fan, T., Hu, J.: Least-squares images for edge-preserving smoothing. Comput. Visual Media 1(1), 27–35 (2015)CrossRefGoogle Scholar
  18. 18.
    Shao, H., Chen, S., Zhao, J., Cui, W., Yu, T.: Face recognition based on subset selection via metric learning on manifold. Front. Inf. Technol. Electron. Eng. 16(12), 1046–1058 (2015)Google Scholar
  19. 19.
    Oikawa, M.A., Dias, Z., de Rezende Rocha, A., Goldenstein, S.: Manifold learning and spectral clustering for image phylogeny forests. IEEE Trans. Inf. Forensics Secur. 11(1), 5–18 (2016)CrossRefGoogle Scholar
  20. 20.
    Blanz, V., Basso, C., Poggio, T., Vetter, T.: Reanimating faces in images and video. Comput. Graph. Forum 22(3), 641–650 (2003)CrossRefGoogle Scholar
  21. 21.
    Vlasic, D., Brand, M., Pfister, H., Popovic, J.: Face transfer with multilinear models. ACM Trans. Graph. (TOG) 24(3), 426–433 (2005)CrossRefGoogle Scholar
  22. 22.
    Cao, C., Weng, Y., Zhou, S., Tong, Y., Zhou, K.: FaceWarehouse: a 3D facial expression database for visual computing. IEEE Trans. Visual Comput. Graph. 20(3), 413–425 (2014)CrossRefGoogle Scholar
  23. 23.
    Cao, C., Wu, H., Weng, Y., Shao, T., Zhou, K.: Real-time facial animation with image-based dynamic avatars. ACM Trans. Graph. (TOG) 35(4), 1–12 (2016)CrossRefGoogle Scholar
  24. 24.
    Saito, S., Li, T., Li, H.: Real-time facial segmentation and performance capture from RGB input. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 244–261. Springer, Cham (2016). doi: 10.1007/978-3-319-46484-8_15 CrossRefGoogle Scholar
  25. 25.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 187–194. ACM Press/Addison-Wesley Publishing Co., New York, USA (1999)Google Scholar
  26. 26.
    Lin, Y., Wang, S., Lin Q., Tang, F.: Face swapping under large pose variations: a 3D model based approach. In: IEEE International Conference on Multimedia and Expo (ICME), pp. 333–338. IEEE, Melbourne, VIC, Australia (2012)Google Scholar
  27. 27.
    Song, H., Lv, J., Liu, H., Zhao, Q.: A face replacement system based on 3D face model. In: Deng, Z., Li, H. (eds.) Proceedings of the 2015 Chinese Intelligent Automation Conference. LNEE, vol. 336, pp. 237–246. Springer, Heidelberg (2015). doi: 10.1007/978-3-662-46469-4_25 CrossRefGoogle Scholar
  28. 28.
    Lin, Y., Lin, Q., Tang, F., Wang, S.: Face replacement with large-pose differences. In: 20th ACM International Conference on Multimedia, pp. 1249–1250. ACM, Nara, Japan (2012)Google Scholar
  29. 29.
    Tran, A.T., Hassner, T., Masi, I., Medioni, G.: Regressing robust and discriminative 3D morphable models with a very deep neural network. arXiv preprint arXiv:1612.04904 (2017)
  30. 30.
    Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874. IEEE, Columbus, OH, USA (2014)Google Scholar
  31. 31.
    Huber, P., Hu, G., Tena, R., Mortazavian, P., Koppen W.P., Christmas, W., Ratsch, M., Kittler, J.: A multiresolution 3D Morphable Face Model and fitting framework. In: 11th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, pp. 1–8 (2016)Google Scholar
  32. 32.
    Sorkine, O.: Least-squares rigid motion using svd. Tech. Notes 120(3), 52 (2009)Google Scholar
  33. 33.
    Takayama, K., Schmidt, R., Singh, K., Igarashi, T., Boubekeur, T., Sorkine, O.: GeoBrush: interactive mesh geometry cloning. Comput. Graph. Forum 30(2), 613–622 (2011)CrossRefGoogle Scholar
  34. 34.
    Yu, Y., Zhou, K., Xu, D., Shi, X., Bao, H., Guo, B., Shum, H.-Y.: Mesh editing with poisson-based gradient field manipulation. ACM Trans. Graph. (TOG) 23(3), 644–651 (2004)CrossRefGoogle Scholar
  35. 35.
    Schmidt, R., Singh, K.: Drag, drop, and clone: an interactive interface for surface composition. Technical Report CSRG-611, Department of Computer Science, University of Toronto (2010)Google Scholar
  36. 36.
    Perez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. (TOG) 22(3), 313–318 (2003)CrossRefGoogle Scholar
  37. 37.
    Libigl. Accessed 2016
  38. 38.
    Zhao, J., Tang, M., Tong, R.: Mesh segmentation for parallel decompression on GPU. In: Hu, S.-M., Martin, R.R. (eds.) CVM 2012. LNCS, vol. 7633, pp. 83–90. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-34263-9_11 CrossRefGoogle Scholar
  39. 39.
    Tang, X., Guo, J., Li, P., Lv, J.: A surgical simulation system for predicting facial soft tissue deformation. Comput. Visual Media 2(2), 163–171 (2016)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Jiang Du
    • 1
  • Yanjing Wu
    • 2
  • Dan Song
    • 1
  • Ruofeng Tong
    • 1
    Email author
  • Min Tang
    • 1
  1. 1.State Key Lab of CAD&CGZhejiang UniversityHangzhouChina
  2. 2.The Third Affiliated HospitalZhejiang Chinese Medicine UniversityHangzhouChina

Personalised recommendations