Advertisement

Generic 3D Convolutional Fusion for Image Restoration

  • Jiqing WuEmail author
  • Radu Timofte
  • Luc Van Gool
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10116)

Abstract

Also recently, exciting strides forward have been made in the area of image restoration, particularly for image denoising and single image super-resolution. Deep learning techniques contributed to this significantly. The top methods differ in their formulations and assumptions, so even if their average performance may be similar, some work better on certain image types and image regions than others. This complementarity motivated us to propose a novel 3D convolutional fusion (3DCF) method. Unlike other methods adapted to different tasks, our method uses the exact same convolutional network architecture to address both image denoising and single image super-resolution. Our 3DCF method achieves substantial improvements (0.1 dB–0.4 dB PSNR) over the state-of-the-art methods that it fuses on standard benchmarks for both tasks. At the same time, the method still is computationally efficient.

Keywords

Image Restoration Sparse Code Convolutional Neural Network Denoising Method Stochastic Gradient Descent 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgments

This work was supported by the ERC project VarCity (#273940), the ETH General Fund (OK) and by an Nvidia GPU grant.

References

  1. 1.
    Katsaggelos, A.K.: Digital Image Restoration. Springer Publishing Company, Incorporated, Heidelberg (2012)Google Scholar
  2. 2.
    Stathaki, T.: Image Fusion: Algorithms and Applications. Academic Press, Amsterdam (2011)Google Scholar
  3. 3.
    Lebrun, M., Colom, M., Buades, A., Morel, J.: Secrets of image denoising cuisine. Acta Numerica 21, 475–576 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Zontak, M., Mosseri, I., Irani, M.: Separating signal from noise using patch recurrence across scales. In: CVPR (2013)Google Scholar
  5. 5.
    Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 2, pp. 60–65. IEEE (2005)Google Scholar
  6. 6.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3d transform-domain collaborative filtering. IEEE Trans. Image Process. 16, 2080–2095 (2007)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Yu, G., Sapiro, G.: DCT image denoising: a simple and effective image denoising algorithm (2011)Google Scholar
  8. 8.
    Gu, S., Zhang, L., Zuo, W., Feng, X.: Weighted nuclear norm minimization with application to image denoising. In: CVPR (2014)Google Scholar
  9. 9.
    Yu, G., Sapiro, G., Mallat, S.: Solving inverse problems with piecewise linear estimators: from gaussian mixture models to structured sparsity. IEEE Trans. Image Process. 21(5), 2481–2499 (2012)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Zoran, D., Weiss, Y.: From learning models of natural image patches to whole image restoration. In: IEEE International Conference on Computer Vision, pp. 479–486 (2011)Google Scholar
  11. 11.
    Mairal, J., Bach, F., Ponce, J., Sapiro, G., Zisserman, A.: Non-local sparse models for image restoration. In: IEEE 12th International Conference on Computer Vision, pp. 2272–2279 (2009)Google Scholar
  12. 12.
    Burger, H., Schuler, C., Harmeling, S.: Image denoising: can plain neural networks compete with bm3d? In: IEEE Computer Vision and Pattern Recognition, pp. 2392–2399 (2012)Google Scholar
  13. 13.
    Schmidt, U., Roth, S.: Shrinkage fields for effective image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2774–2781 (2014)Google Scholar
  14. 14.
    Chen, Y., Pock, T., Ranftl, R., Bischof, H.: Revisiting loss-specific training of filter-based MRFs for image restoration. In: Weickert, J., Hein, M., Schiele, B. (eds.) GCPR 2013. LNCS, vol. 8142, pp. 271–281. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40602-7_30 CrossRefGoogle Scholar
  15. 15.
    Chen, Y., Yu, W., Pock, T.: On learning optimized reaction diffusion processes for effective image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5261–5269 (2015)Google Scholar
  16. 16.
    Jancsary, J., Nowozin, S., Rother, C.: Loss-specific training of non-parametric image restoration models: a new state of the art. In: IEEE European Conference of Computer Vision (2012)Google Scholar
  17. 17.
    Burger, H.C., Schuler, C., Harmeling, S.: Learning how to combine internal and external denoising methods. In: Weickert, J., Hein, M., Schiele, B. (eds.) GCPR 2013. LNCS, vol. 8142, pp. 121–130. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40602-7_13 CrossRefGoogle Scholar
  18. 18.
    Levin, A., Nadler, B., Durand, F., Freeman, W.T.: Patch complexity, finite pixel correlations and optimal denoising. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7576, pp. 73–86. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33715-4_6 CrossRefGoogle Scholar
  19. 19.
    Mosseri, I., Zontak, M., Irani, M.: Combining the power of internal and external denoising. In: IEEE International Conference on Computational Photography (ICCP), pp. 1–9 (2013)Google Scholar
  20. 20.
    Schmidt, U., Jancsary, J., Nowozin, S., Roth, S., Rother, C.: Cascades of regression tree fields for image restoration (2014)Google Scholar
  21. 21.
    Timofte, R.: Anchored fusion for image restoration. In: ICPR (2016)Google Scholar
  22. 22.
    Timofte, R., Rothe, R., Van Gool, L.: Seven ways to improve example-based single image super resolution. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  23. 23.
    Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  24. 24.
    Wang, Z., Liu, D., Yang, J., Han, W., Huang, T.: Deep networks for image super-resolution with sparse prior. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 370–378 (2015)Google Scholar
  25. 25.
    Agustsson, E., Timofte, R., Van Gool, L.: Regressor basis learning for anchored super-resolution. In: ICPR (2016)Google Scholar
  26. 26.
    Dai, D., Timofte, R., Van Gool, L.: Jointly optimized regressors for image super-resolution. Comput. Graph. Forum 34, 95–104 (2015)CrossRefGoogle Scholar
  27. 27.
    Timofte, R., De Smet, V., Van Gool, L.: Semantic super-resolution: when and where is it useful? Comput. Vis. Image Underst. 142, 1–12 (2016)CrossRefGoogle Scholar
  28. 28.
    Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19, 2861–2873 (2010)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Timofte, R., De Smet, V., Van Gool, L.: A+: adjusted anchored neighborhood regression for fast super-resolution. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 111–126. Springer, Cham (2015). doi: 10.1007/978-3-319-16817-3_8 Google Scholar
  30. 30.
    Timofte, R., Smet, V., Gool, L.: Anchored neighborhood regression for fast example-based super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1920–1927 (2013)Google Scholar
  31. 31.
    Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling with super-resolution forests. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3791–3799 (2015)Google Scholar
  32. 32.
    Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5197–5206. IEEE (2015)Google Scholar
  33. 33.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks (2015)Google Scholar
  34. 34.
    Ji, S., Xu, W., Yang, M., Yu, K.: 3d convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35, 221–231 (2013)CrossRefGoogle Scholar
  35. 35.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space. In: IEEE International Conference on Image Processing, ICIP 2007, vol. 1, p. I-313. IEEE (2007)Google Scholar
  36. 36.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Trans. Image Process. 16, 2080–2095 (2007)MathSciNetCrossRefGoogle Scholar
  37. 37.
    Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML 2010), pp. 807–814 (2010)Google Scholar
  38. 38.
    Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Cogn. Model. 5, 1 (1988)Google Scholar
  39. 39.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998)CrossRefGoogle Scholar
  40. 40.
    Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: Lechevallier, Y., Saporta, G. (eds.) Proceedings of COMPSTAT 2010, pp. 177–186. Springer, Heidelberg (2010)Google Scholar
  41. 41.
    Kingma, D., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  42. 42.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the 8th International Conference on Computer Vision, vol. 2, pp. 416–423 (2001)Google Scholar
  43. 43.
    Roth, S., Black, M.J.: Fields of experts. Int. J. Comput. Vis. 82, 205–229 (2009)CrossRefGoogle Scholar
  44. 44.
    Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding (2012)Google Scholar
  45. 45.
    Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Boissonnat, J.-D., Chenin, P., Cohen, A., Gout, C., Lyche, T., Mazure, M.-L., Schumaker, L. (eds.) Curves and Surfaces 2010. LNCS, vol. 6920, pp. 711–730. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-27413-8_47 CrossRefGoogle Scholar
  46. 46.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the ACM International Conference on Multimedia, pp. 675–678. ACM (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Computer Vision Laboratory, D-ITETETH ZurichZürichSwitzerland

Personalised recommendations