Advertisement

SRFeat: Single Image Super-Resolution with Feature Discrimination

  • Seong-Jin Park
  • Hyeongseok Son
  • Sunghyun Cho
  • Ki-Sang Hong
  • Seungyong Lee
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11220)

Abstract

Generative adversarial networks (GANs) have recently been adopted to single image super-resolution (SISR) and showed impressive results with realistically synthesized high-frequency textures. However, the results of such GAN-based approaches tend to include less meaningful high-frequency noise that is irrelevant to the input image. In this paper, we propose a novel GAN-based SISR method that overcomes the limitation and produces more realistic results by attaching an additional discriminator that works in the feature domain. Our additional discriminator encourages the generator to produce structural high-frequency features rather than noisy artifacts as it distinguishes synthetic and real images in terms of features. We also design a new generator that utilizes long-range skip connections so that information between distant layers can be transferred more effectively. Experiments show that our method achieves the state-of-the-art performance in terms of both PSNR and perceptual quality compared to recent GAN-based methods.

Keywords

Super-resolution Adversarial network High frequency features Perceptual quality 

Notes

Acknowledgements

We appreciate the constructive comments from the reviewers. This work was supported by the Ministry of Science and ICT, Korea, through IITP grant (IITP-2015-0-00174), Giga Korea grant (GK18P0300), and NRF grant (NRF-2017M3C4A7066317). It was also supported by the DGIST Start-up Fund Program (2018010071).

Supplementary material

474218_1_En_27_MOESM1_ESM.pdf (43.1 mb)
Supplementary material 1 (pdf 44170 KB)

References

  1. 1.
    Abadi, M., et al.: Tensorflow: A system for large-scale machine learning. In: Proceedings of OSDI (2016)Google Scholar
  2. 2.
    Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: Dataset and study. In: CVPR Workshops (2017)Google Scholar
  3. 3.
    Allebach, J., Wong, P.W.: Edge-directed interpolation. In: Proceedings of ICIP (1996)Google Scholar
  4. 4.
    Baker, S., Kanade, T.: Limits on super-resolution and how to break them. IEEE Trans. Pattern Anal. Mach. Intell. 24(9), 1167–1183 (2002)CrossRefGoogle Scholar
  5. 5.
    Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of BMVC (2012)Google Scholar
  6. 6.
    Bruna, J., Sprechmann, P., LeCun, Y.: Super-resolution with deep convolutional sufficient statistics. In: Proceedings of ICLR (2016)Google Scholar
  7. 7.
    Chang, H., Yeung, D.Y., Xiong, Y.: Super-resolution through neighbor embedding. In: Proceedings of CVPR (2004)Google Scholar
  8. 8.
    Dai, S., Han, M., Xu, W., Wu, Y., Gong, Y., Katsaggelos, A.K.: Softcuts: a soft edge smoothness prior for color image super-resolution. IEEE Trans. Image Process. 18(5), 969–981 (2009)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)CrossRefGoogle Scholar
  10. 10.
    Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Proceedings of ECCV (2014)Google Scholar
  11. 11.
    Duchon, C.E.: Lanczos filtering in one and two dimensions. J. Appl. Meteorol. 18(8), 1016–1022 (1979)CrossRefGoogle Scholar
  12. 12.
    Freeman, W.T., Pasztor, E.C., Carmichael, O.T.: Learning low-level vision. Int. J. Comput. Vis. 40(1), 25–47 (2000)CrossRefGoogle Scholar
  13. 13.
    Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of CVPR (2016)Google Scholar
  14. 14.
    Goodfellow, I., et al.: Generative adversarial nets. In: Proceedings of NIPS (2014)Google Scholar
  15. 15.
    Gu, S., Sang, N., Ma, F.: Fast image super resolution via local regression. In: Proceedings of ICPR (2012)Google Scholar
  16. 16.
    Gu, S., Zuo, W., Xie, Q., Meng, D., Feng, X., Zhang, L.: Convolutional sparse coding for image super-resolution. In: Proceedings of ICCV (2015)Google Scholar
  17. 17.
    Gupta, P., Srivastava, P., Bhardwaj, S., Bhateja, V.: A modified psnr metric based on hvs for quality assessment of color images. In: Proceedings of ICCIA (2011)Google Scholar
  18. 18.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Procwwdings of CVPR (2016)Google Scholar
  19. 19.
    Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Gr. 36(4), 107 (2017)CrossRefGoogle Scholar
  20. 20.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of CVPR (2017)Google Scholar
  21. 21.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of ACM MM (2014)Google Scholar
  22. 22.
    Johnson, J., Alahi, A., fei Li, F.: Perceptual losses for real-time style transfer and super-resolution. In: Proceedings of ECCV (2016)Google Scholar
  23. 23.
    Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of gans for improved quality, stability, and variation. In: Proceedings of ICLR (2018)Google Scholar
  24. 24.
    Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of CVPR (2016)Google Scholar
  25. 25.
    Kim, T., Cha, M., Kim, H., Lee, J., Kim, J.: Learning to discover cross-domain relations with generative adversarial networks. In: Proc. ICML (2017)Google Scholar
  26. 26.
    Kingma, D., Ba, J.: Adam: A method for stochastic optimization. In: Proceedings of ICLR (2015)Google Scholar
  27. 27.
    Ledig, C., Theis, L., Huszar, F., Caballero, J., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of CVPR (2017)Google Scholar
  28. 28.
    Li, J., Liang, X., Wei, Y., Xu, T., Feng, J., Yan, S.: Perceptual generative adversarial networks for small object detection. In: Proceedings of CVPR (2017)Google Scholar
  29. 29.
    Li, X., Orchard, M.T.: New edge-directed interpolation. IEEE Trans. Image Process. 10(10), 1521–1527 (2001)CrossRefGoogle Scholar
  30. 30.
    Li, Y., Liu, S., Yang, J., Yang, M.H.: Generative face completion. In: Proceedings of CVPR (2017)Google Scholar
  31. 31.
    Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: CVPR Workshops (2017)Google Scholar
  32. 32.
    Lin, G., Milan, A., Shen, C., Reid, I.: Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In: Proceedings of CVPR (2017)Google Scholar
  33. 33.
    Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of ICML (2013)Google Scholar
  34. 34.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of ICCV (2001)Google Scholar
  35. 35.
    Park, S.J., Hong, K.S., Lee, S.: Rdfnet: Rgb-d multi-level residual feature fusion for indoor semantic segmentation. In: Proceedings of ICCV (2017)Google Scholar
  36. 36.
    Perez-Pellitero, E., Salvador, J., Ruiz-Hidalgo, J., Rosenhahn, B.: Psyco: Manifold span reduction for super resolution. In: Proceedings of CVPR (2016)Google Scholar
  37. 37.
    Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: Proceedings of ICLR (2016)Google Scholar
  38. 38.
    Romano, Y., Isidoro, J., Milanfar, P.: Raisr: rapid and accurate image super resolution. IEEE Trans. Comput. Imag. 3, 110–125 (2017)MathSciNetCrossRefGoogle Scholar
  39. 39.
    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  40. 40.
    Sajjadi, M., Schölkopf, B., Hirsch, M.: Enhancenet: Single image super-resolution through automated texture synthesis. In: Proceedings of ICCV (2017)Google Scholar
  41. 41.
    Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of CVPR (2016)Google Scholar
  42. 42.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Proceedings of ICLR (2015)Google Scholar
  43. 43.
    Sun, J., Sun, J., Xu, Z., Shum, H.Y.: Gradient profile prior and its applications in image super-resolution and enhancement. IEEE Trans. Image Process. 20(6), 1529–1542 (2011)MathSciNetCrossRefGoogle Scholar
  44. 44.
    Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: Proceedings of CVPR (2017)Google Scholar
  45. 45.
    Timofte, R., De, V., Gool, L.V.: Anchored neighborhood regression for fast example-based super-resolution. In: Proceedings of ICCV (2013)Google Scholar
  46. 46.
    Timofte, R., De, V., Gool, L.V.: A+: Adjusted anchored neighborhood regression for fast super-resolution. In: Proceedings of ACCV (2014)Google Scholar
  47. 47.
    Tong, T., Li, G., Liu, X., Gao, Q.: Image super-resolution using dense skip connections. In: Proceedings of ICCV (2017)Google Scholar
  48. 48.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  49. 49.
    Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: Proceedings of ACSSC (2003)Google Scholar
  50. 50.
    Yang, C.Y., Yang, M.H.: Fast direct super-resolution by simple functions. In: Proceedings of ICCV (2013)Google Scholar
  51. 51.
    Yang, J., Wang, Z., Lin, Z., Cohen, S., Huang, T.: Coupled dictionary training for image super-resolution. IEEE Trans. Image Process. 21(8) (2012)Google Scholar
  52. 52.
    Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)MathSciNetCrossRefGoogle Scholar
  53. 53.
    Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Proceedings of Curves and Surfaces (2012)Google Scholar
  54. 54.
    Zhang, K., Gao, X., Tao, D., Li, X.: Multi-scale dictionary for single image super-resolution. In: Proceedings of CVPR (2012)Google Scholar
  55. 55.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of ICCV (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Seong-Jin Park
    • 1
  • Hyeongseok Son
    • 1
  • Sunghyun Cho
    • 2
  • Ki-Sang Hong
    • 1
  • Seungyong Lee
    • 1
  1. 1.POSTECHPohangSouth Korea
  2. 2.DGISTDaeguSouth Korea

Personalised recommendations