Advertisement

Signal, Image and Video Processing

, Volume 13, Issue 3, pp 557–565 | Cite as

An improved method for single image super-resolution based on deep learning

  • Chao XieEmail author
  • Ying Liu
  • Weili Zeng
  • Xiaobo Lu
Original Paper
  • 148 Downloads

Abstract

This paper strives for presenting an improved method for single image super-resolution based on deep learning, and therefore, a well-designed network structure is proposed by simultaneously considering the merits of convolutional sparse coding (CSC) and deep convolutional neural networks (CNN). In our model, contrary to most existing methods that directly operate on the raw input, we first perform a global decomposition on the input based on CSC for the purpose of extracting two specific components from it. Since the generated components are designed to have predefined physical meanings (i.e., residual or smooth), they can be discriminatively super-resolved according to their distinctive appearances. Specifically, a strong preference is given to the residual one as it is much more crucial to our task, while the other should just provide a quick reference. Based on this analysis, deep CNN and plain interpolation are selected to map them, respectively. In all, the proposed model integrates the above procedures into a completely end-to-end trainable deep network. Thorough experimental results demonstrate that our proposed network is able to gain considerable accuracy from this deep and delicate architecture, thereby outperforming many recently published baselines in terms of both objective evaluation and visual fidelity.

Keywords

Single image super-resolution Deep learning Convolutional sparse coding Deep convolutional neural networks 

Notes

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No. 61374194, No. 61403081), the National Key Science & Technology Pillar Program of China (No. 2014BAG01B03), the Key Research and Development Program of Jiangsu Province (No. BE2016739), and a Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions.

References

  1. 1.
    Nguyen, K., Fookes, C., Sridharan, S., Tistarelli, M., Nixon, M.: Super-resolution for biometrics: a comprehensive survey. Pattern Recogn. 78, 23–42 (2018)CrossRefGoogle Scholar
  2. 2.
    Nasrollahi, K., Escalera, S., Rasti, P., Anbarjafari, G., Baro, X., Escalante, H.J., Moeslund, T. B.: Deep learning based super-resolution for improved action recognition. In: Proceedings of 2015 International Conference on Image Processing Theory, Tools and Applications (IPTA), pp. 67–72. IEEE (2015)Google Scholar
  3. 3.
    Zeng, W., Lu, X.: A generalized DAMRF image modeling for superresolution of license plates. IEEE Trans. Intell. Transp. Syst. 13(2), 828–837 (2012)CrossRefGoogle Scholar
  4. 4.
    Park, S.C., Park, M.K., Kang, M.G.: Super-resolution image reconstruction: a technical overview. IEEE Signal Process. Mag. 20(3), 21–36 (2003)CrossRefGoogle Scholar
  5. 5.
    Nasrollahi, K., Moeslund, T.B.: Super-resolution: a comprehensive survey. Mach. Vis. Appl. 25(6), 1423–1468 (2014)CrossRefGoogle Scholar
  6. 6.
    Zhu, S., Li, Y.: Single image super-resolution under multi-frame method. In: Signal, Image and Video Processing, pp. 1–9 (2018)Google Scholar
  7. 7.
    Freeman, W.T., Pasztor, E.C., Carmichael, O.T.: Learning low-level vision. Int. J. Comput. Vis. 40(1), 25–47 (2000)CrossRefzbMATHGoogle Scholar
  8. 8.
    Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Comput. Gr. Appl. 22(2), 56–65 (2002)CrossRefGoogle Scholar
  9. 9.
    Gao, X., Zhang, K., Tao, D., Li, X.: Image super-resolution with sparse neighbor embedding. IEEE Trans. Image Process. 21(7), 3194–3205 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Chang, H., Yeung, D., Xiong, Y.: Super-resolution through neighbor embedding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. I–I. IEEE (2004)Google Scholar
  11. 11.
    Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Candes, E.J., Wakin, M.B.: An introduction to compressive sampling. IEEE Signal Process. Mag. 25(2), 21–30 (2008)CrossRefGoogle Scholar
  13. 13.
    Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Ahmed, J., Waqas, M., Ali, S., Memon, R.A., Klette, R.: Coupled dictionary learning in wavelet domain for Single-Image Super-Resolution. SIViP 12(3), 453–461 (2018)CrossRefGoogle Scholar
  15. 15.
    Chaudhry, A.M., Riaz, M.M., Ghafoor, A.: Super-resolution based on self-example learning and guided filtering. In: Signal, Image and Video Processing, pp. 1–8 (2018)Google Scholar
  16. 16.
    Rasti, P., Nasrollahi, K., Orlova, O., Tamberg, G., Ozcinar, C., Moeslund, T.B., Anbarjafari, G.: A new low-complexity patch-based image super-resolution. IET Comput. Vision 11(7), 567–576 (2017)CrossRefGoogle Scholar
  17. 17.
    Rasti, P., Nasrollahi, K., Orlova, O., Tamberg, G., Moeslund, T.B., Anbarjafari, G.: Reducible dictionaries for single image super-resolution based on patch matching and mean shifting. J. Electron. Imaging 26(2), 23024 (2017)CrossRefGoogle Scholar
  18. 18.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Proceedings of Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  19. 19.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Proceedings of International Conference on Learning Representations (2015)Google Scholar
  20. 20.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRefGoogle Scholar
  21. 21.
    Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Curves and Surfaces, pp. 711–730. Springer (2012)Google Scholar
  22. 22.
    Zeiler, M.D., Krishnan, D., Taylor, G.W., Fergus, R.: Deconvolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2528–2535. IEEE (2010)Google Scholar
  23. 23.
    Liu, D., Wang, Z., Wen, B., Yang, J., Han, W., Huang, T.S.: Robust single image super-resolution via deep networks with sparse prior. IEEE Trans. Image Process. 25(7), 3194–3207 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)CrossRefGoogle Scholar
  25. 25.
    Chen, Y., Pock, T.: Trainable nonlinear reaction diffusion: a flexible framework for fast and effective image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1256–1272 (2017)CrossRefGoogle Scholar
  26. 26.
    Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  27. 27.
    Peleg, T., Elad, M.: A statistical prediction model based on sparse representations for single image super-resolution. IEEE Trans. Image Process. 23(6), 2569–2582 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Gu, S., Zuo, W., Xie, Q., Meng, D., Feng, X., Zhang, L.: Convolutional sparse coding for image super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1823–1831 (2015)Google Scholar
  29. 29.
    Timofte, R., De Smet, V., Van Gool, L.: A + : adjusted anchored neighborhood regression for fast super-resolution. In: Proceedings of Asian Conference on Computer Vision, pp. 111–126. Springer (2014)Google Scholar
  30. 30.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)Google Scholar
  31. 31.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  32. 32.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  33. 33.
    Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. In: Proceedings of International Conference on Machine Learning, pp. 1310–1318 (2013)Google Scholar
  34. 34.
    Bevilacqua, M., Roumy, A., Guillemot, C., Morel, M.A.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings British Machine Vision Conference (2012)Google Scholar
  35. 35.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 416–423. IEEE (2001)Google Scholar
  36. 36.
    Huang, J., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015)Google Scholar
  37. 37.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceeding of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)Google Scholar
  38. 38.
    Vedaldi, A., Lenc, K.: Matconvnet: convolutional neural networks for matlab. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 689–692. ACM (2015)Google Scholar
  39. 39.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2018

Authors and Affiliations

  1. 1.College of Mechanical and Electronic EngineeringNanjing Forestry UniversityNanjingChina
  2. 2.College of Civil AviationNanjing University of Aeronautics and AstronauticsNanjingChina
  3. 3.School of AutomationSoutheast UniversityNanjingChina

Personalised recommendations