Occluded Face Recognition by Identity-Preserving Inpainting

  • Chenyu Li
  • Shiming GeEmail author
  • Yingying Hua
  • Haolin Liu
  • Xin Jin
Part of the Studies in Computational Intelligence book series (SCI, volume 810)


Occluded face recognition, which has an attractive application in the visual analysis field, is challenged by the missing cues due to heavy occlusions. Recently, several face inpainting methods based on generative adversarial networks (GANs) fill in the occluded parts by generating images fitting the real image distributions. They can lead to a visually natural result and satisfy human perception. However, these methods fail to capture the identity attributes, thus the inpainted faces may be recognized at a low accuracy by machine. To enable the convergence of human perception and machine perception, this paper proposes an Identity Preserving Generative Adversary Networks (IP-GANs) to jointly inpaint and recognize occluded faces. The IP-GANs consists of an inpainting network for regressing missing facial parts, a global-local discriminative network for guiding the inpainted face to the real conditional distribution, a parsing network for enhancing structure consistence and an identity network for recovering missing identity cues. Especially, the novel identity network suppresses the identity diffusion by constraining the feature consistence from the early subnetwork of a well-trained face recognition network between the inpainted face and its corresponding ground-true. In this way, it regularizes the inpaintor, enforcing the generated faces to preserve identity attributes. Experimental results prove the proposed IP-GANs capable of dealing with varieties of occlusions and producing photorealistic and identity-preserving results, promoting occluded face recognition performance.


Image inpainting Occluded face recognition Generative adversarial networks (GANs) Identity preserving 



This work is supported in part by the National Natural Science Foundation of China (61772513 & 61402463), the National Key Research and Development Plan (2016YFC0801005) and the International Cooperation Project of Institute of Information Engineering, Chinese Academy of Sciences (Y7Z0511101). Shiming Ge is also supported by Youth Innovation Promotion Association, CAS.


  1. 1.
    Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of SIGGRAPH, vol. 4, no. 9, pp. 417–424 (2005)Google Scholar
  2. 2.
    Deng, Y., Dai, Q., Zhang, Z.: Graph Laplace for occluded face completion and recognition. IEEE Trans. Image Process. Publ. IEEE Signal Process. Soc. 20(8), 2329–2338 (2011)Google Scholar
  3. 3.
    Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: International Conference on Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  4. 4.
    Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst (October 2007)Google Scholar
  5. 5.
    Ishikawa, H., Ishikawa, H., Ishikawa, H.: Globally and locally consistent image completion. ACM (2017)Google Scholar
  6. 6.
    Jia, Y., Shelhamer, E., Donahue, J., et al.: Caffe: convolutional architecture for fast feature embedding (2014). arXiv:1408.5093
  7. 7.
    Li, Y., Liu, S., Yang, J., Yang, M.H.: Generative face completion. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017Google Scholar
  8. 8.
    Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: IEEE International Conference on Computer Vision, pp. 3730–3738 (2015)Google Scholar
  9. 9.
    Lu, H., Li, B., Zhu, J., Li, Y., Li, Y., Xu, X., He, L., Li, X., Li, J., Serikawa, S.: Wound intensity correction and segmentation with convolutional neural networks. Concur. Comput. Pract. Exp. 29(6) (2016)Google Scholar
  10. 10.
    Lu, H., Li, Y., Chen, M., Kim, H., Serikawa, S.: Brain intelligence: go beyond artificial intelligence. Mob. Netw. Appl. 23(2), 368–375 (2017)CrossRefGoogle Scholar
  11. 11.
    Lu, H., Li, Y., Mu, S., Wang, D., Kim, H., Serikawa, S.: Motor anomaly detection for unmanned aerial vehicles using reinforcement learning. IEEE Internet Things J 5(4), 2315–2322 (2018)CrossRefGoogle Scholar
  12. 12.
    Oh, H.J., Lee, K.M., Sang, U.L.: Occlusion invariant face recognition using selective local non-negative matrix factorization basis images. Image Vis. Comput. 26(11), 1515–1523 (2008)CrossRefGoogle Scholar
  13. 13.
    Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: British Machine Vision Conference (2015)Google Scholar
  14. 14.
    Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2536–2544 (2016)Google Scholar
  15. 15.
    Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR (2015). arXiv:1511.06434
  16. 16.
    Serikawa, S., Lu, H.: Underwater image dehazing using joint trilateral filter. Pergamon Press, Inc. (2014)Google Scholar
  17. 17.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR (2014). arXiv:1409.1556
  18. 18.
    Smith, B.M., Zhang, L., Brandt, J., Lin, Z., Yang, J.: Exemplar-based face parsing. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3484–3491 (2013)Google Scholar
  19. 19.
    Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. Computer Science (2013)Google Scholar
  20. 20.
    Wang, C., Xu, C., Wanga, C., Tao, D.: Perceptual adversarial networks for image-to-image transformation. IEEE Trans. Image Process. 27(8), 4066–4079 (2018)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Wright, J., Ganesh, A., Zhou, Z., Wagner, A., Ma, Y.: Demo: robust face recognition via sparse representation. In: IEEE International Conference on Automatic Face & Gesture Recognition, pp. 1–2 (2009)Google Scholar
  22. 22.
    Xie, J., Xu, L., Chen, E.: Image denoising and inpainting with deep neural networks. In: International Conference on Neural Information Processing Systems, pp. 341–349 (2012)Google Scholar
  23. 23.
    Xu, X., He, L., Lu, H., Gao, L., Ji, Y.: Deep adversarial metric learning for cross-modal retrieval. In: World Wide Web-internet & Web Information Systems, pp. 1–16 (2018)Google Scholar
  24. 24.
    Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. CoRR (2018). arXiv:1801.07892
  25. 25.
    Zhang, S., He, R., Sun, Z., Tan, T.: Demeshnet: blind face inpainting for deep meshface verification. IEEE Trans. Inf. Forensics Secur. 13(3), 637–647 (2017)CrossRefGoogle Scholar
  26. 26.
    Zhang, W., Shan, S., Chen, X., Gao, W.: Local Gabor binary patterns based on Kullback–Leibler divergence for partially occluded face recognition. IEEE Signal Process. Lett. 14(11), 875–878 (2007)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Chenyu Li
    • 1
    • 2
  • Shiming Ge
    • 1
    Email author
  • Yingying Hua
    • 1
    • 2
  • Haolin Liu
    • 1
    • 2
  • Xin Jin
    • 3
  1. 1.Institute of Information EngineeringChinese Academy of SciencesBeijingChina
  2. 2.School of Cyber SecurityUniversity of Chinese Academy of SciencesBeijingChina
  3. 3.Department of Cyber SecurityBeijing Electronic Science and Technology InstituteBeijingChina

Personalised recommendations