Image Inpainting Based on Local Patch Search Supported by Image Segmentation

  • Sarah Almeida Carneiro
  • Helio Pedrini
  • Silvio Jamil Ferzoli GuimarãesEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11401)


Image inpainting can be defined as a restoration process in which damaged or selected regions are repaired by taking into account the image content. In this work, we employ a local-based strategy instead of a global one to identify the best existing patch with information to replace the damaged/selected patch. In order to properly identify the most representative patches, we propose a method based on the (i) creation of a local graph using a similarity of patches in the original image and (ii) partition of the image into regions according to hierarchical image segmentation to support the local patch identification. The experimental results demonstrate that our local search outperformed the results of image inpainting in terms of both qualitative and quantitative aspects, when compared to global search of patches.


Image inpainting Hierarchical image segmentation Patch graph 


  1. 1.
    Ambrosio, L., Fusco, N., Hutchinson, J.E.: Higher integrability of the gradient and dimension of the singular set for minimisers of the Mumford-Shah functional. Calc. Var. Partial. Differ. Equ. 16(2), 187–215 (2003)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Bertalmio, M., Bertozzi, A.L., Sapiro, G.: Navier-stokes, fluid dynamics, and image and video inpainting. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1. IEEE (2001)Google Scholar
  3. 3.
    Bertalmio, M., Caselles, V., Masnou, S., Sapiro, G.: Inpainting. Technical report, Duke University (2011)Google Scholar
  4. 4.
    Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 417–424. ACM Press/Addison-Wesley Publishing Co. (2000)Google Scholar
  5. 5.
    Cousty, J., Najman, L., Kenmochi, Y., Guimarães, S.: Hierarchical segmentations with graphs: quasi-flat zones, minimum spanning trees, and saliency maps. J. Math. Imaging Vis. 60, 479–502 (2017)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)CrossRefGoogle Scholar
  7. 7.
    Defferrard, M.: Graph-based image inpainting. Technical report, École Polytechnique Fédérale de Lausanne (EFPL) (2014)Google Scholar
  8. 8.
    Efros, A., Leung, T.: Texture synthesis by non-parametric sampling. In: Seventh IEEE International Conference on Computer Vision, p. 1033. IEEE, September 1999Google Scholar
  9. 9.
    Fan, Q., Zhang, L.: A novel patch matching algorithm for exemplar-based image inpainting. Multimedia Tools Appl. 77, 1–15 (2017)Google Scholar
  10. 10.
    Guillemot, C., Le Meur, O.: Image inpainting: overview and recent advances. IEEE Signal Process. Mag. 31(1), 127–144 (2014)CrossRefGoogle Scholar
  11. 11.
    Karaca, E., Tunga, M.A.: An interpolation-based texture and pattern preserving algorithm for inpainting color image. Expert Syst. Appl. 91, 223–234 (2018)CrossRefGoogle Scholar
  12. 12.
    Masnou, S., Morel, J.-M.: Level Lines based disocclusion. In: International Conference on Image Processing, pp. 259–263. IEEE (1998)Google Scholar
  13. 13.
    Ogden, J.M., Adelson, E.H., Bergen, J.R., Burt, P.J.: Pyramid-based computer graphics. RCA Eng. 30(5), 4–15 (1985)Google Scholar
  14. 14.
    Perret, B., Cousty, J., Guimarães, S.J.F., Maia, D.S.: Evaluation of hierarchical watersheds. IEEE Trans. Image Process. 27(4), 1676–1688 (2018)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Qureshi, M.A., Deriche, M., Beghdadi, A., Amin, A.: A critical survey of state-of-the-art image inpainting quality assessment metrics. J. Vis. Commun. Image Represent. 49, 177–191 (2017)CrossRefGoogle Scholar
  16. 16.
    Roubíček, T.: Nonlinear Partial Differential Equations with Applications, vol. 153. Springer, Basel (2013). Scholar
  17. 17.
    Wang, H., Jiang, L., Liang, R., Li, X.-X.: Exemplar-based image inpainting using structure consistent patch matching. Neurocomputing 269, 90–96 (2017)CrossRefGoogle Scholar
  18. 18.
    Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, p. 3 (2017)Google Scholar
  19. 19.
    Zhao, G., Liu, J., Jiang, J., Wang, W.: A deep cascade of neural networks for image inpainting, deblurring and denoising. Multimed. Tools Appl. 77, 1–16 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Sarah Almeida Carneiro
    • 1
    • 2
  • Helio Pedrini
    • 2
  • Silvio Jamil Ferzoli Guimarães
    • 1
    Email author
  1. 1.Computer Science DepartmentPontifical Catholic University of Minas GeraisBelo HorizonteBrazil
  2. 2.Institute of ComputingUniversity of CampinasCampinasBrazil

Personalised recommendations