Advertisement

Multiple Connected Residual Network for Image Enhancement on Smartphones

  • Jie LiuEmail author
  • Cheolkon JungEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11133)

Abstract

Image enhancement on smartphones needs rapid processing speed with comparable performance. Recently, convolutional neural networks (CNNs) have achieved outstanding performance in image processing tasks such as image super-resolution and enhancement. In this paper, we propose a lightweight generator for image enhancement based on CNN to keep a balance between quality and speed, called multi-connected residual network (MCRN). The proposed network consists of one discriminator and one generator. The generator is a two-stage network: (1) The first stage extracts structural features; (2) the second stage focuses on enhancing perceptual visual quality. By utilizing the style of multiple connections, we achieve good performance in image enhancement while making our network converge fast. Experimental results demonstrate that the proposed method outperforms the state-of-the-art approaches in terms of the perceptual quality and runtime. The code is available at https://github.com/JieLiu95/MCRN.

Keywords

Image enhancement Generator Residual Network Multiple connections Perceptual quality 

Notes

Acknowledgment

This work was supported by the National Natural Science Foundation of China (No. 61271298) and the International S&T Cooperation Program of China (No. 2014DFG12780).

References

  1. 1.
    Ignatov, A., Kobyshev, N., Timofte, R., Vanhoey, K., Van Gool, L.: DSLR-quality photos on mobile devices with deep convolutional networks. In: The IEEE International Conference on Computer Vision (ICCV), pp. 3277–3285 (2017)Google Scholar
  2. 2.
    Goodfellow, I., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 27, pp. 2672–2680. Curran Associates, Inc. (2014)Google Scholar
  3. 3.
    Ignatov, A., Kobyshev, N., Timofte, R., Vanhoey, K., Gool, L.V.: WESPE: weakly supervised photo enhancer for digital cameras. CoRR abs/1709.01118 (2017)Google Scholar
  4. 4.
    Huang, G., Liu, S., van der Maaten, L., Weinberger, K.Q.: CondenseNet: an efficient densenet using learned group convolutions. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2752–2761 (2018)Google Scholar
  5. 5.
    Beghdadi, A., Negrate, A.: Contrast enhancement technique based on local detection of edges. Comput. Vis. Graph. Image Process. 46(2), 162–174 (1989)CrossRefGoogle Scholar
  6. 6.
    Arici, T., Dikbas, S., Altunbasak, Y.: A histogram modification framework and its application for image contrast enhancement. IEEE Trans. Image Process. 18(9), 1921–1935 (2009)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Reza, A.M.: Realization of the contrast limited adaptive histogram equalization (CLAHE) for real-time image enhancement. J. VLSI Signal Process. Syst. Signal Image Video Technol. 38(1), 35–44 (2004)CrossRefGoogle Scholar
  8. 8.
    Wang, C., Ye, Z.: Brightness preserving histogram equalization with maximum entropy. IEEE Trans. Consum. Electron. 51(4), 1326–1334 (2005)CrossRefGoogle Scholar
  9. 9.
    Güçlütürk, Y., Güçlü, U., van Lier, R., van Gerven, M.A.J.: Convolutional sketch inversion. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9913, pp. 810–824. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46604-0_56CrossRefGoogle Scholar
  10. 10.
    Gatys, L.A., Ecker, A.S., Bethge, M., Hertzmann, A., Shechtman, E.: Controlling perceptual factors in neural style transfer. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3985–3993 (2017)Google Scholar
  11. 11.
    Chen, Y.S., Wang, Y.C., Kao, M.H., Chuang, Y.Y.: Deep photo enhancer: unpaired learning for image enhancement from photographs with gans. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6306–6314 (2018)Google Scholar
  12. 12.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  13. 13.
    Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein GAN (2017). arXiv:1701.07875
  14. 14.
    Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10593-2_13CrossRefGoogle Scholar
  15. 15.
    Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 105–114 (2017)Google Scholar
  16. 16.
    Timofte, R., Gu, S., Wu, J., Gool, L.V., Yang, M.H., et al.: Ntire 2018 challenge on single image super-resolution: methods and results. In: CVPRW, pp. 852–863 (2018)Google Scholar
  17. 17.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, June 2016Google Scholar
  18. 18.
    Huang, G., Liu, Z., Weinberger, K.Q.: Densely connected convolutional networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261–2269 (2017)Google Scholar
  19. 19.
    Ulyanov, D., Lebedev, V., Vedaldi, A., Lempitsky, V.: Texture networks: feed-forward synthesis of textures and stylized images. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning, ICML 20116, vol. 48, pp. 1349–1357. JMLR.org (2016)Google Scholar
  20. 20.
    Liu, Y., Cohen, M., Uyttendaele, M., Rusinkiewicz, S.: Autostyle: automatic style transfer from image collections to users’ images. In: Proceedings of the 25th Eurographics Symposium on Rendering, EGSR 2014, Aire-la-Ville, Switzerland, Switzerland, pp. 21–31. Eurographics Association (2014)Google Scholar
  21. 21.
    Okura, F., Vanhoey, K., Bousseau, A., Efros, A.A., Drettakis, G.: Unifying color and texture transfer for predictive appearance manipulation. In: Proceedings of the 26th Eurographics Symposium on Rendering, EGSR 2015, Aire-la-Ville, Switzerland, Switzerland, pp. 53–63. Eurographics Association (2015)Google Scholar
  22. 22.
    Zhang, R., et al.: Real-time user-guided image colorization with learned deep priors. ACM Trans. Graph. 36(4), 119:1–119:11 (2017)Google Scholar
  23. 23.
    Monroe, W., Hawkins, R.X.D., Goodman, N.D., Potts, C.: Colors in context: a pragmatic neural model for grounded language understanding. Trans. Assoc. Comput. Linguist. 5, 325–338 (2017)CrossRefGoogle Scholar
  24. 24.
    Solli, M., Lenz, R.: Color semantics for image indexing. In: Conference on Colour in Graphics, Imaging, and Vision, vol. 2010, no. 1 (2010)Google Scholar
  25. 25.
    Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems 30, pp. 700–708. Curran Associates, Inc. (2017)Google Scholar
  26. 26.
    Isola, P., Zhu, J., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967–5976, July 2017Google Scholar
  27. 27.
    Wang, X., Gupta, A.: Generative image modeling using style and structure adversarial networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 318–335. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46493-0_20CrossRefGoogle Scholar
  28. 28.
    Huang, X., Li, Y., Poursaeed, O., Hopcroft, J., Belongie, S.: Stacked generative adversarial networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017Google Scholar
  29. 29.
    Aly, H.A., Dubois, E.: Image up-sampling using total-variation regularization with a new observation model. IEEE Trans. Image Process. 14(10), 1647–1659 (2005)CrossRefGoogle Scholar
  30. 30.
    Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: the missing ingredient for fast stylization. arXiv:1607.08022 (2016)
  31. 31.
    Goldluecke, B., Cremers, D.: An approach to vectorial total variation based on geometric measure theory. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 327–333, July 2010Google Scholar
  32. 32.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1701.07875 (2014)
  33. 33.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46475-6_43CrossRefGoogle Scholar
  34. 34.
    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. CoRR abs/1412.6980 (2014)Google Scholar
  35. 35.
    Ignatov, A., Timofte, R., et al.: PIRM challenge on perceptual image enhancement on smartphones: Report. In: European Conference on Computer Vision Workshops (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.School of Electronic EngineeringXidian UniversityXi’anChina

Personalised recommendations