Learning Reaction-Diffusion Models for Image Inpainting

  • Wei Yu
  • Stefan Heber
  • Thomas Pock
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9358)


In this paper we present a trained diffusion model for image inpainting based on the structural similarity measure. The proposed diffusion model uses several parametrized linear filters and influence functions. Those parameters are learned in a loss based approach, where we first perform a greedy training before conducting a joint training to further improve the inpainting performance. We provide a detailed comparison to state-of-the-art inpainting algorithms based on the TUM-image inpainting database. The experimental results show that the proposed diffusion model is efficient and achieves superior performance. Moreover, we also demonstrate that the proposed method has a texture preserving property, that makes it stand out from previous PDE based methods.


  1. 1.
    Bertalmio, M., Vese, L., Sapiro, G., Osher, S.: Simultaneous structure and texture image inpainting. In: proceedings of CVPR, vol. 2, pp. II-707-12 (2003)Google Scholar
  2. 2.
    Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Conference on Computer graphics and interactive techniques, pp. 417–424. ACM Press/Addison-Wesley Publishing Co. (2000)Google Scholar
  3. 3.
    Bugeau, A., Bertalmio, M., Caselles, V., Sapiro, G.: A comprehensive framework for image inpainting. Image Process. 19(10), 2634–2645 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Cao, F., Gousseau, Y., Masnou, S., Pérez, P.: Geometrically guided exemplar-based inpainting. SIAM J. Img. Sci. 4(4), 1143–1179 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Caselles, V.: Exemplar-based image inpainting and applications. SIAM News 44(10), 1–3 (2011)Google Scholar
  6. 6.
    Chan, T.F., Shen, J.: Local inpainting models and TV inpainting. SIAM J. Appl. Math. 62(3), 1019–1043 (2001)MathSciNetGoogle Scholar
  7. 7.
    Chan, T.F., Shen, J.: Nontexture inpainting by curvature-driven diffusions. J. Vis. Commun. Image Represent. 12(4), 436–449 (2001)CrossRefGoogle Scholar
  8. 8.
    Chen, Y., Ranftl, R., Pock, T.: A bi-level view of inpainting-based image compression (2014). arXiv preprint arXiv:1401.4112
  9. 9.
    Chen, Y., Ranftl, R., Pock, T.: Insights into analysis operator learning: from patch-based sparse models to higher order MRFs. Image Process. 23(3), 1060–1072 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Chen, Y., Yu, W., Pock, T.: On learning optimized reaction diffusion processes for effective image restoration. In: Proceeding of CVPR (2015)Google Scholar
  11. 11.
    Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. Trans. Img. Proc. 13(9), 1200–1212 (2004)CrossRefGoogle Scholar
  12. 12.
    Efros, A., Leung, T.: Texture synthesis by non-parametric sampling. In: proceeding of ICCV, vol. 2, pp. 1033–1038 (1999)Google Scholar
  13. 13.
    Esedoglu, S., Shen, J.: Digital inpainting based on the Mumford-Shah-Euler image model. Eur. J. Appl. Math. 13(04), 353–370 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Facciolo, G., Arias, P., Caselles, V., Sapiro, G.: Exemplar-based interpolation of sparsely sampled images. In: Cremers, D., Boykov, Y., Blake, A., Schmidt, F.R. (eds.) EMMCVPR 2009. LNCS, vol. 5681, pp. 331–344. Springer, Heidelberg (2009) CrossRefGoogle Scholar
  15. 15.
    Galić, I., Weickert, J., Welk, M., Bruhn, A., Belyaev, A., Seidel, H.P.: Image compression with anisotropic diffusion. JMIV 31(2–3), 255–269 (2008)MathSciNetzbMATHGoogle Scholar
  16. 16.
    Getreuer, P.: Total variation inpainting using split bregman. Image Process. On Line 2, 147–157 (2012)CrossRefGoogle Scholar
  17. 17.
    Grossauer, H.: A combined PDE and texture synthesis approach to inpainting. In: Pajdla, T., Matas, J.G. (eds.) ECCV 2004. LNCS, vol. 3022, pp. 214–224. Springer, Heidelberg (2004) CrossRefGoogle Scholar
  18. 18.
    Hel-Or, Y., Shaked, D.: A discriminative approach for wavelet denoising. IEEE Trans. Image Process. 17(4), 443–457 (2008)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Herling, J., Broll, W.: Pixmix: A real-time approach to high-quality diminished reality. In: International Symposium on Mixed and Augmented Reality (ISMAR), pp. 141–150. IEEE (2012)Google Scholar
  20. 20.
    Jain, V., Seung, S.: Natural image denoising with convolutional networks. In: Advances in Neural Information Processing Systems, pp. 769–776 (2009)Google Scholar
  21. 21.
    Kokaram, A.C., Morris, R.D., Fitzgerald, W.J., Rayner, P.J.: Interpolation of missing data in image sequences. Image Process. 4(11), 1509–1519 (1995)CrossRefGoogle Scholar
  22. 22.
    Komodakis, N., Tziritas, G.: Image completion using efficient belief propagation via priority scheduling and dynamic pruning. Trans. Img. Proc. 16(11), 2649–2661 (2007)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Kong, X., Li, K., Yang, Q., Wenyin, L., Yang, M.H.: A new image quality metric for image auto-denoising. In: Proceeding of ICCV, pp. 2888–2895. IEEE (2013)Google Scholar
  24. 24.
    Levin, A., Zomet, A., Weiss, Y.: Learning how to inpaint from global image statistics. In: poceeding of ICCV, pp. 305–312 (2003)Google Scholar
  25. 25.
    Liu, A., Lin, W., Narwaria, M.: Image quality assessment based on gradient similarity. Image Process. 21(4), 1500–1512 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Liu, D., Sun, X., Wu, F., Li, S., Zhang, Y.Q.: Image compression with edge-based inpainting. Circuits Syst. Video Technol. 17(10), 1273–1287 (2007)CrossRefGoogle Scholar
  27. 27.
    Liu, D.C., Nocedal, J.: On the limited memory BFGS method for large scale optimization. Math. Program. 45(1–3), 503–528 (1989)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Masnou, S., Morel, J.M.: Level lines based disocclusion. In: Proceeding of ICIP, vol. 3, pp. 259–263 (1998)Google Scholar
  29. 29.
    Mumford, D., Shah, J.: Optimal approximations by piecewise smooth functions and associated variational problems. Comm. Pure Appl. Math. 42(5), 577–685 (1989)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Nikolaidis, N., Pitas, I.: Digital image processing in painting restoration and archiving. In: proceeding of ICCV, vol. 1, pp. 586–589. IEEE (2001)Google Scholar
  31. 31.
    Peter, P., Weickert, J.: Compressing images with diffusion- and exemplar-based inpainting. In: Aujol, J.-F., Nikolova, M., Papadakis, N. (eds.) SSVM 2015. LNCS, vol. 9087, pp. 154–165. Springer, Heidelberg (2015) Google Scholar
  32. 32.
    Rehman, A., Wang, Z.: Ssim-based non-local means image denoising. In: proceeding of ICIP, pp. 217–220. IEEE (2011)Google Scholar
  33. 33.
    Roth, S., Black, M.: Fields of experts: a framework for learning image priors. In: proceeding of CVPR, vol. 2, pp. 860–867 (2005)Google Scholar
  34. 34.
    Roth, S., Black, M.: Steerable random fields. In: proceeding of ICCV, pp. 1–8 (2007)Google Scholar
  35. 35.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena 60(1–4), 259–268 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Schmidt, U., Gao, Q., Roth, S.: A generative perspective on MRFs in low-level vision. In: proceeding of CVPR, pp. 1751–1758. IEEE (2010)Google Scholar
  37. 37.
    Tiefenbacher, P., Bogischef, V., Merget, D., Rigoll, G.: Subjective and objective evaluation of image inpainting quality. In: proceeding of ICIP. IEEE (2015)Google Scholar
  38. 38.
    TUM-image inpainting database.
  39. 39.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  40. 40.
    Xu, Z., Sun, J.: Image inpainting by patch propagation using patch sparsity. Image Process. 19(5), 1153–1165 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Zhu, S.C., Mumford, D.: Prior learning and gibbs reaction-diffusion. Pattern Anal. Mach. Intell. 19(11), 1236–1250 (1997)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Open Access This chapter is distributed under the terms of the Creative Commons Attribution Noncommercial License, which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Authors and Affiliations

  1. 1.Institute for Computer Graphics and VisionGraz University of TechnologyGrazAustria
  2. 2.Safety & Security DepartmentAIT Austrian Institute of TechnologyGrazAustria

Personalised recommendations