Face Swapping for Solving Collateral Privacy Issues in Multimedia Analytics

  • Werner BailerEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11295)


A wide range of components of multimedia analytics systems relies on visual content that is used for supervised (e.g., classification) and unsupervised (e.g., clustering) machine learning methods. This content may contain privacy sensitive information, e.g., show faces of persons. In many cases it is just an inevitable side-effect that persons appear in the content, and the application may not require identification – a situation which we call “collateral privacy issues”. We propose de-identification of faces in images by using a generative adversarial network to generate new face images, and use them to replace faces in the original images. We demonstrate that face swapping does not impact the performance of visual descriptor matching and extraction.



The research leading to these results has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no 761802, MARCONI (“Multimedia and Augmented Radio Creation: Online, iNteractive, Individual”,


  1. 1.
    Agarwal, A., Singh, R., Vatsa, M., Noore, A.: Swapped! digital face presentation attack detection via weighted local magnitude pattern. In: 2017 IEEE International Joint Conference on Biometrics (IJCB), pp. 659–665. IEEE (2017)Google Scholar
  2. 2.
    Ahonen, T., Rahtu, E., Ojansivu, V., Heikkila, J.: Recognition of blurred faces using local phase quantization. In: 19th International Conference on Pattern Recognition, ICPR 2008, pp. 1–4. IEEE (2008)Google Scholar
  3. 3.
    Badii, A., Einig, M., Piatrik, T., et al.: Overview of the mediaeval 2013 visual privacy task. In: MediaEval (2014)Google Scholar
  4. 4.
    Bergeron, C., Sidaty, N., Hamidouche, W., Boyadjis, B., Le Feuvre, J., Lim, Y.: Real-time selective encryption solution based on ROI for MPEG-a visual identity management AF. In: 2017 22nd International Conference on Digital Signal Processing (DSP), pp. 1–5, Aug 2017Google Scholar
  5. 5.
    Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., Nayar, S.K.: Face swapping: automatically replacing faces in photographs. In: ACM Transactions on Graphics (TOG), vol. 27, p. 39. ACM (2008)CrossRefGoogle Scholar
  6. 6.
    Evaluation framework for compact descriptors for video analysis - search and retrieval - version 2.0. Technical report ISO/IEC JTC1/SC29/WG11/N15729 (2015)Google Scholar
  7. 7.
    Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). Official Journal of the European Union, L119:1–88, May 2016Google Scholar
  8. 8.
    Hadid, A., Nishiyama, M., Sato, Y.: Recognition of blurred faces via facial deblurring combined with blur-tolerant descriptors. In: 2010 20th International Conference on Pattern Recognition (ICPR), pp. 1160–1163. IEEE (2010)Google Scholar
  9. 9.
    King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)Google Scholar
  10. 10.
    Lander, K., Bruce, V., Hill, H.: Evaluating the effectiveness of pixelation and blurring on masking the identity of familiar faces. Appl. Cogn. Psychol. 15(1), 101–116 (2001)CrossRefGoogle Scholar
  11. 11.
    Letournel, G., Bugeau, A., Ta, V.T., Domenger, J.P.: Face de-identification with expressions preservation. In: 2015 IEEE International Conference on Image Processing (ICIP), pp. 4366–4370. IEEE (2015)Google Scholar
  12. 12.
    Lin, J., Duan, L.-Y., Huang, Y., Luo, S., Huang, T., Gao, W.: Rate-adaptive compact fisher codes for mobile visual search. IEEE Signal Process. Lett. 21(2), 195–198 (2014)CrossRefGoogle Scholar
  13. 13.
    Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision (ICCV) (2015)Google Scholar
  14. 14.
    Lou, Y., et al.: Compact deep invariant descriptors for video retrieval. In: Data Compression Conference (DCC), pp. 420–429, April 2017Google Scholar
  15. 15.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Mahajan, S., Chen, L.J., Tsai, T.C.: SwapItUP: a face swap application for privacy protection. In: 2017 IEEE 31st International Conference on Advanced Information Networking and Applications (AINA), pp. 46–50. IEEE (2017)Google Scholar
  17. 17.
    Meng, L., Sun, Z., Collado, O.T.: Efficient approach to de-identifying faces in videos. IET Sig. Process. 11(9), 1039–1045 (2017)CrossRefGoogle Scholar
  18. 18.
    Nakashima, Y., Babaguchi, N., Fan, J.: Intended human object detection for automatically protecting privacy in mobile video surveillance. Multimed. Syst. 18(2), 157–173 (2012)CrossRefGoogle Scholar
  19. 19.
    Natsume, R., Yatagawa, T., Morishima, S.: RSGAN: face swapping and editing using face and hair representation in latent spaces. arXiv preprint arXiv:1804.03447 (2018)
  20. 20.
    Newton, E.M., Sweeney, L., Malin, B.: Preserving privacy by de-identifying face images. IEEE Trans. Knowl. Data Eng. 17(2), 232–243 (2005)CrossRefGoogle Scholar
  21. 21.
    Nirkin, Y., Masi, I., Tuan, A.T., Hassner, T., Medioni, G.: On face segmentation, face swapping, and face perception. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition, FG 2018, pp. 98–105. IEEE (2018)Google Scholar
  22. 22.
    Nishiyama, M., Takeshima, H., Shotton, J., Kozakaya, T., Yamaguchi, O.: Facial deblur inference to improve recognition of blurred faces. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 1115–1122. IEEE (2009)Google Scholar
  23. 23.
    Padilla-López, J.R., Chaaraoui, A.A., Flórez-Revuelta, F.: Visual privacy protection methods: a survey. Expert. Syst. Appl. 42(9), 4177–4195 (2015)CrossRefGoogle Scholar
  24. 24.
    Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval with large vocabularies and fast spatial matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2007)Google Scholar
  25. 25.
    Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR, abs/1511.06434 (2015)Google Scholar
  26. 26.
    Saini, M., Atrey, P.K., Mehrotra, S., Kankanhalli, M.: W3-privacy: understanding what, when, and where inference channels in multi-camera surveillance video. Multimed. Tools Appl. 68(1), 135–158 (2014)CrossRefGoogle Scholar
  27. 27.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556 (2014)Google Scholar
  28. 28.
    Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Sig. Process. Lett. 23(10), 1499–1503 (2016)CrossRefGoogle Scholar
  29. 29.
    Zhang, Y., Zheng, L., Thing, V.L.: Automated face swapping and its detection. In: 2017 IEEE 2nd International Conference on Signal and Image Processing (ICSIP), pp. 15–19. IEEE (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.JOANNEUM RESEARCH Forschungsgesellschaft mbH, DIGITAL – Institute for Information and Communication TechnologiesGrazAustria

Personalised recommendations