Model and Dictionary Guided Face Inpainting in the Wild

  • Reuben A. FarrugiaEmail author
  • Christine Guillemot
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10116)


This work presents a method that can be used to inpaint occluded facial regions with unconstrained pose and orientation. This approach first warps the facial region onto a reference model to synthesize a frontal view. A modified Robust Principal Component Analysis (RPCA) approach is then used to suppress warping errors. It then uses a novel local patch-based face inpainting algorithm which hallucinates missing pixels using a dictionary of face images which are pre-aligned to the same reference model. The hallucinated region is then warped back onto the original image to restore missing pixels.

Experimental results on synthetic occlusions demonstrate that the proposed face inpainting method has the best performance achieving PSNR gains of up to 0.74 dB over the second-best method. Moreover, experiments on the COFW dataset and a number of real-world images show that the proposed method successfully restores occluded facial regions in the wild even for CCTV quality images.


Face Image Training Image Locally Linear Embedding Landmark Point Occlude Region 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Hancock, P.J., Bruce, V., Burton, A.: Recognition of unfamiliar faces. Trends Cogn. Sci. 4, 330–337 (2000)CrossRefGoogle Scholar
  2. 2.
    Terry, R.L.: How wearing eyeglasses affects facial recognition. Curr. Psychol. 12, 151–162 (1993)CrossRefGoogle Scholar
  3. 3.
    Yarmey, A.D.: Eyewitness recall and photo identification: a field experiment. Psychol. Crime Law 10, 53–68 (2004)CrossRefGoogle Scholar
  4. 4.
    Klare, B.F., Klein, B., Taborsky, E., Blanton, A., Cheney, J., Allen, K., Grother, P., Mah, A., Burge, M., Jain, A.K.: Pushing the frontiers of unconstrained face detection and recognition: IARPA Janus benchmark A. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1931–1939 (2015)Google Scholar
  5. 5.
    Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Annual Conference on Computer Graphics and Interactive Techniques, pp. 417–424 (2000)Google Scholar
  6. 6.
    Criminisi, A., Perez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13, 1200–1212 (2004)CrossRefGoogle Scholar
  7. 7.
    Liu, J., Musialski, P., Wonka, P., Ye, J.: Tensor completion for estimating missing values in visual data. IEEE Trans. Pattern Anal. Mach. Intell. 35, 208–220 (2013)CrossRefGoogle Scholar
  8. 8.
    Guillemot, C., Meur, O.L.: Image inpainting : overview and recent advances. IEEE Sig. Process. Mag. 31, 127–144 (2014)CrossRefGoogle Scholar
  9. 9.
    Mo, Z., Lewis, J., Neumann, U.: Face inpainting with local linear representations. In: British Machine Vision Conference, pp. 37.1–37.10 (2004)Google Scholar
  10. 10.
    Wang, Z.M., Tao, J.H.: Reconstruction of partially occluded face by fast recursive PCA. In: International Conference on Computational Intelligence and Security Workshops, pp. 304–307 (2007)Google Scholar
  11. 11.
    Zhou, Z., Wagner, A., Mobahi, H., Wright, J., Ma, Y.: Face recognition with contiguous occlusion using markov random fields. In: IEEE International Conference on Computer Vision, pp. 1050–1057 (2009)Google Scholar
  12. 12.
    Lin, D., Tang, X.: Quality-driven face occlusion detection and recovery. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–7 (2007)Google Scholar
  13. 13.
    Hosoi, T., Nagashima, S., Kobayashi, K., Ito, K., Aoki, T.: Restoring occluded regions using FW-PCA for face recognition. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 23–30 (2012)Google Scholar
  14. 14.
    Hwang, B.W., Lee, S.W.: Reconstruction of partially damaged face images based on a morphable face model. IEEE Trans. Pattern Anal. Mach. Intell. 25, 365–372 (2003)CrossRefGoogle Scholar
  15. 15.
    Liwicki, S., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: Euler principal component analysis. Int. J. Comput. Vis. 101, 498–518 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Min, R., Dugelay, J.L.: Inpainting of sparse occlusion in face recognition. In: IEEE International Conference on Image Processing, pp. 1425–1428 (2012)Google Scholar
  17. 17.
    Roth, S., Black, M.J.: Fields of experts. Int. J. Comput. Vis. 82, 205–229 (2009)CrossRefGoogle Scholar
  18. 18.
    Burgos-Artizzu, X.P., Zepeda, J., Clerc, F.L., Perez, P.: Pose and expression-coherent face recovery in the wild. In: IEEE International Conference on Computer Vision Workshop, pp. 877–885 (2015)Google Scholar
  19. 19.
    Hassner, T., Harel, S., Paz, E., Enbar, R.: Effective face frontalization in unconstrained images. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4295–4304 (2015)Google Scholar
  20. 20.
    Candès, E.J., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM 58, 11:1–11:37 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Yuan, Z., Xie, X., Ma, X., Lam, K.M.: Color facial image denoising based on RPCA and noisy pixel detection. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2449–2453 (2013)Google Scholar
  22. 22.
    Peng, Y., Ganesh, A., Wright, J., Xu, W., Ma, Y.: RASL: robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Trans. Pattern Anal. Mach. Intell. 34, 2233–2246 (2012)CrossRefGoogle Scholar
  23. 23.
    Wong, A., Orchard, J.: A nonlocal-means approach to exemplar-based inpainting. In: IEEE International Conference on Image Processing, pp. 2600–2603 (2008)Google Scholar
  24. 24.
    Xu, Z., Sun, J.: Image inpainting by patch propagation using patch sparsity. IEEE Trans. Image Process. 19, 1153–1165 (2010)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Studer, C., Kuppinger, P., Pope, G., Bolcskei, H.: Recovery of sparsely corrupted signals. IEEE Trans. Inf. Theor. 58, 3115–3130 (2012)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Guillemot, C., Turkan, M., Meur, O.L., Ebdelli, M.: Object removal and loss concealment using neighbor embedding methods. Sig. Process. Image Commun. 28, 1405–1419 (2013)CrossRefGoogle Scholar
  27. 27.
    Ma, X., Zhang, J., Qi, C.: Position-based face hallucination method. In: Proceedings of the IEEE International Conference on Multimedia and Expo, pp. 290–293 (2009)Google Scholar
  28. 28.
    Jiang, J., Hu, R., Wang, Z., Han, Z.: Face super-resolution via multilayer locality-constrained iterative neighbor embedding and intermediate dictionary learning. IEEE Trans. Image Process. 23, 4220–4231 (2014)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Burgos-Artizzu, X.P., Perona, P., Dollár, P.: Robust face landmark estimation under occlusion. In: IEEE International Conference on Computer Vision, pp. 1513–1520 (2013)Google Scholar
  30. 30.
    Mairal, J., Bach, F., Ponce, J., Sapiro, G.: Online dictionary learning for sparse coding. In: Annual International Conference on Machine Learning, pp. 689–696 (2009)Google Scholar
  31. 31.
    Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2879–2886 (2012)Google Scholar
  32. 32.
    Koestinger, M., Wohlhart, P., Roth, P.M., Bischof, H.: Annotated facial landmarks in the wild: a large-scale, real-world database for facial landmark localization. In: IEEE International Conference on Computer Vision, pp. 2144–2151 (2011)Google Scholar
  33. 33.
    Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2323–2326 (2000)CrossRefGoogle Scholar
  34. 34.
    Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15, 1373–1396 (2003)CrossRefzbMATHGoogle Scholar
  35. 35.
    He, X., Yan, S., Hu, Y., Niyogi, P., Zhang, H.J.: Face recognition using laplacianfaces. IEEE Trans. Pattern Anal. Mach. Intell. 27, 328–340 (2005)CrossRefGoogle Scholar
  36. 36.
    Hu, C., Chang, Y., Feris, R., Turk, M.: Manifold based analysis of facial expression. In: IEEE International Conference on Computer Vision and Pattern Recognition, p. 81 (2004)Google Scholar
  37. 37.
    Lin, M., Chen, L., Wu, Y.M.: The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. Technical report, University of Illinois (2009)Google Scholar
  38. 38.
    Phillips, P.J., Wechsler, H., Huang, J., Rauss, P.J.: The FERET database and evaluation procedure for face-recognition algorithms. Image Vis Comput. 16, 295–306 (1998)CrossRefGoogle Scholar
  39. 39.
    Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical report 07–49, University of Massachusetts, Amherst (2007)Google Scholar
  40. 40.
    Martinez, A.M., Benavente, R.: The AR Face Database. Technical report, CVC (1998)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.University of MaltaMsidaMalta
  2. 2.INRIA Rennes-Bretagne-AtlantiqueRennesFrance

Personalised recommendations