Advertisement

Multimedia Tools and Applications

, Volume 78, Issue 14, pp 19621–19640 | Cite as

Multipath feedforward network for single image super-resolution

  • Mingyu Shen
  • Pengfei Yu
  • Ronggui Wang
  • Juan YangEmail author
  • Lixia Xue
  • Min Hu
Article
  • 113 Downloads

Abstract

Single image super-resolution (SR) models which based on convolutional neural network mostly use chained stacking to build the network. It ignores the role of hierarchical features and relationship between layers, resulting in the loss of high-frequency components. To address these drawbacks, we introduce a novel multipath feedforward network (MFNet) based on staged feature fusion unit (SFF). By changing the connection between networks, MFNet strengthens the inter-layer relationship and improves the information flow in the network, thereby extracting more abundant high-frequency components. Firstly, SFF extracts and integrates hierarchical features by dense connection, which expands the information flow of the network. Afterwards, we use adaptive method to learn effective features in hierarchical features. Then, in order to strengthen relationship between layers and fully use the hierarchical features, we use multi-feedforward structure to connect each SFF, which enables multipath feature re-usage and explores more abundant high-frequency components on this basis. Finally, the image reconstruction is realized by combining the shallow features and the global residual. Extensive benchmark evaluation shows that the performance of MFNet has a significant improvement over the state-of-the-art methods.

Keywords

Super-resolution Convolutional neural network Multipath feedforward network Staged feature fusion 

Notes

Acknowledgements

We express our sincere thanks to the anonymous reviewers for their helpful comments and suggestions to raise the standard of our paper.

Funding

This study was funded by the National Natural Science Foundation of China under (grant number 61672202).

Compliance with ethical standards

Conflict of interest

The whole authors are fulltime teachers of Hefei University of Technology besides the second author Pengfei Yu, and he is the fulltime student of Hefei University of Technology. The whole authors declare that we have no conflicts of interest to this work.

References

  1. 1.
    Bevilacqua M, Roumy A, Guillemot C, Morel M-A (2012) Low-complexity single image super-resolution based on nonnegative neighbor embedding. In: British Machine Vision Conference (BMVC). 1–10Google Scholar
  2. 2.
    Chen Y, Li J, Xiao H, Jin X, Yan S, Feng J (2017) Dual path networks. In: Neural Information Processing Systems (NIPS). 4467–4475Google Scholar
  3. 3.
    Cheong JY, Park IK (2017) Deep CNN-based super-resolution using external and internal examples. IEEE Sig Proc Lett 24(8):1252–1256CrossRefGoogle Scholar
  4. 4.
    Cho D, Tai YW, KweonI (2016) Natural image matting using deep convolutional neural networks. In: European Conference on Computer Vision (ECCV). 626–643Google Scholar
  5. 5.
    Dai S, Han M, Xu W, Wu Y, Gong Y, Katsaggelos AK (2009) Softcuts: a soft edge smoothness prior for color image super-resolution. IEEE Trans Image Process 18(5):969–981MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Dong C, Chen CL, He K, Tang X (2014) Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision (ECCV). 184–199Google Scholar
  7. 7.
    Dong C, Chen CL, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307CrossRefGoogle Scholar
  8. 8.
    Dong C, Chen CL,Tang X (2016) Accelerating the super-resolution convolutional neural network. In: European Conference on Computer Vision (ECCV). 391–407Google Scholar
  9. 9.
    Gregor K, Yann L (2010) Learning fast approximations of sparse coding. In: International Conference on Machine Learning (ICML). 399–406Google Scholar
  10. 10.
    Gu S, Zuo W, Xie Q, Meng D, Feng X,Zhang L (2015) Convolutional sparse coding for image super resolution. In: IEEE Int Conf Comput Vis (ICCV). 1823–1831Google Scholar
  11. 11.
    He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: IEEE Int Conf Comput Vis (ICCV). 1026–1034Google Scholar
  12. 12.
    He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE Conf Comput Vis Patt Recog (CVPR). 770–778Google Scholar
  13. 13.
    Huang J-B, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. IEEE Conf Comput Vis Patt Recog (CVPR):5197–5206Google Scholar
  14. 14.
    Huang G, Liu Z, Weinberger KQ, Maaten L-vd (2017) Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2261–2269Google Scholar
  15. 15.
    Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. IEEE Conf Comput Vis Patt Recog (CVPR):1637–1645Google Scholar
  16. 16.
    Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: IEEE Conf Comput Vis Patt Recog (CVPR). 1646–1654Google Scholar
  17. 17.
    Lai W, Huang J, Ahuja N, Yang M (2017) Deep laplacian pyramid networks for fast and accurate super-resolution. In: IEEE Conf Comput Vis Patt Recog (CVPR). 5835–5843Google Scholar
  18. 18.
    Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, Shi W (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conf Comput Vis Patt Recog (CVPR). 105–114Google Scholar
  19. 19.
    Li X, Orchard MT (2001) New edge-directed interpolation. IEEE Trans Image Process 10(10):1521–1527CrossRefGoogle Scholar
  20. 20.
    Li S, Fan R, Lei G, Hou G (2018) A two-channel convolutional neural network for image super-resolution. Neurocomputing 275:267–277CrossRefGoogle Scholar
  21. 21.
    Mao X, Shen C, Yang Y (2016) Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Neural Information Processing Systems (NIPS). 2802–2810Google Scholar
  22. 22.
    Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision. 416–423Google Scholar
  23. 23.
    Qian N (1999) On the momentum term in gradient descent learning algorithms. Neural Netw 12(1):145–151MathSciNetCrossRefGoogle Scholar
  24. 24.
    Schulter S, Leistner C, Bischof H (2015) Fast and accurate image upscaling with super-resolution forests. In:IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3791–3799Google Scholar
  25. 25.
    Shi W, Caballero J, Huszar F, Totz J, Aitken A-P, Bishop R, Rueckert D, Wang Z (2016) Real-Time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1874–1883Google Scholar
  26. 26.
    Shi W, Jiang F, Zhao D (2017) Single image super-resolution with dilated convolution based multi-scale information learning inception module. In: IEEE International Conference on Image Processing (ICIP). 977–981Google Scholar
  27. 27.
    Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (ICLR)Google Scholar
  28. 28.
    Sun J, Sun J, Xu Z, Shum H (2008) Image super-resolution using gradient profile prior. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1–8Google Scholar
  29. 29.
    SzegedyC LW, Jia Y, Sermanet P, Reed S (2015) Going deeper with convolutions. IEEE Conf Comput Vis Patt Recog (CVPR):1–9Google Scholar
  30. 30.
    Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. In: IEEE Conf Comput Vis Patt Recog (CVPR). 2790–2798Google Scholar
  31. 31.
    Tai Y, Yang Y, Liu X, Xu C (2017) MemNet: A persistent memory network for image restoration. In: IEEE Int Conf Comput Vis (ICCV). 4549–4557Google Scholar
  32. 32.
    Tang Y, Chen H, Liu Z, Song B, Wang Q (2016) Example-based super-resolution via social images. Neurocomputing 172:38–47CrossRefGoogle Scholar
  33. 33.
    Timofte R, De V, Gool LV (2013) Anchored neighborhood regression for fast example-based super-resolution. In: Proc IEEE International Conference on Computer Vision (ICCV). 1920–1927Google Scholar
  34. 34.
    Timofte R, De V, Gool LV (2014) A+: adjusted anchored neighborhood regression for fast super-resolution. In: Proc. Asian Conference on Computer Vision (ACCV), pp 111–126Google Scholar
  35. 35.
    Wang Q, Yuan Y (2014) High quality image resizing [J]. Neurocomputing 131:348–356CrossRefGoogle Scholar
  36. 36.
    Wang Z, Yang Y, Wang Z, Chang S, Han W, Yang J, Huang T (2015) Self-tuned deep super resolution. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 1–8Google Scholar
  37. 37.
    Wang Z, Liu D, Wen B, Yang J, Han W, Huang T-S (2015) Deep networks for image super-resolution with sparse prior. In: IEEE Int Conf Comput Vis (ICCV). 370–378Google Scholar
  38. 38.
    Wang Y, Wang L, Wang H, Li P (2016) End-to-End image super resolution via deep and shallow convolutional networksarXiv preprint arXiv:1607.07680Google Scholar
  39. 39.
    Xu J, Zhao Y, Dong Y, Bai H (2017) Fast and accurate image super-resolution using a combined loss. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 1093–1099Google Scholar
  40. 40.
    Xu J, Chae Y, Stenger B (2017) BYNET-SR: Image super resolution with a bypass connection network. In: IEEE International Conference on Image Processing (ICIP). 4053–4057Google Scholar
  41. 41.
    Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Zeyde R, Elad M, Protter M (2010) On single image scale-up using sparse-representations. In: Proceedings of the 7th International Conference on Curves and Surfaces. 711–730Google Scholar
  43. 43.
    Zhang L, Wu X (2006) An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans Image Process 15(8):2226–2238CrossRefGoogle Scholar
  44. 44.
    Zhang K, Zuo W, Gu S, Zhang L (2017) Learning deep CNN denoiser prior for image restoration. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2808–2817Google Scholar
  45. 45.
    Zhang K, Zuo W, Chen Y, Meng D, Zhang L (2017) Beyond a Gaussian Denoiser: residual learning of deep CNN for image Denoising. IEEE Trans Image Process 26(7):3142–3155MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  • Mingyu Shen
    • 1
  • Pengfei Yu
    • 1
  • Ronggui Wang
    • 1
  • Juan Yang
    • 1
    Email author
  • Lixia Xue
    • 1
  • Min Hu
    • 1
  1. 1.College of Computer and InformationHefei University of TechnologyHefeiChina

Personalised recommendations