Advertisement

A neural network approach to remove rain using reconstruction and feature losses

  • Kamran Javed
  • Ghulam Hussain
  • Furqan Shaukat
  • Seong Oun HwangEmail author
Green and Human Information Technology 2019
  • 25 Downloads

Abstract

Rain streaks can eclipse some information of an image taken during rainfall which can degrade the performance of a vision system. While existing rain removing methods can recover the semantic structure, they lack natural texture recovery. The aim of this work is to recover the hidden structure and texture under the rain streaks with fine details. We propose a novel generative adversarial network with two discriminators to remove rain called rain removal generative adversarial network, where a combination of reconstruction, feature and adversarial losses is used for low level, structural and natural recovery, respectively. We have found that exploiting low-level \({l_1}\) loss with high-level structural similarity loss as a reconstruction loss is quite effective in attaining visually plausible and consistent texture. Qualitative and quantitative evaluations on our synthetically created dataset and a benchmark dataset show substantial performance gain than state-of-the-art rain removing methods.

Keywords

Rain removal Generative adversarial network Structural similarity loss UNET Pix2Pix 

Notes

Acknowledgements

This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIP) (No. 2017R1A2B4001801).

References

  1. 1.
    Barnum PC, Narasimhan S, Kanade T (2010) Analysis of rain and snow in frequency space. Int J Comput Vis 86(2–3):256CrossRefGoogle Scholar
  2. 2.
    Bossu J, Hautière N, Tarel JP (2011) Rain or snow detection in image sequences through use of a histogram of orientation of streaks. Int J Comput Vis 93(3):348–367CrossRefGoogle Scholar
  3. 3.
    Chen LC, Papandreou G, Schroff F, Adam H (2017) Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587
  4. 4.
    Chen YL, Hsu CT (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: Proceedings of the IEEE international conference on computer vision, pp 1968–1975Google Scholar
  5. 5.
    Choi Y, Choi M, Kim M, Ha JW, Kim S, Choo J (2018) Stargan: unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8789–8797Google Scholar
  6. 6.
    Eigen D, Krishnan D, Fergus R (2013) Restoring an image taken through a window covered with dirt or rain. In: Proceedings of the IEEE international conference on computer vision, pp 633–640Google Scholar
  7. 7.
    Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: a deep network architecture for single-image rain removal. IEEE Tran Image Process 26(6):2944–2956MathSciNetCrossRefGoogle Scholar
  8. 8.
    Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) Removing rain from single images via a deep detail network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3855–3863Google Scholar
  9. 9.
    Garg K, Nayar SK (2004) Detection and removal of rain from videos. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, CVPR 2004, vol 1. IEEE, pp I–IGoogle Scholar
  10. 10.
    Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Adv Neural Inf Process Syst 2:2672–2680Google Scholar
  11. 11.
    Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville AC (2017) Improved training of wasserstein gans. Adv Neural Inf Process Syst 27:5767–5777Google Scholar
  12. 12.
    Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141Google Scholar
  13. 13.
    Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: IEEE CVPR, pp 5967–5976Google Scholar
  14. 14.
    Jetchev N, Bergmann U, Vollgraf R (2016) Texture synthesis with spatial generative adversarial networks. arXiv preprint arXiv:1611.08207
  15. 15.
    Kim JH, Lee C, Sim JY, Kim CS (2013) Single-image deraining using an adaptive nonlocal means filter. In: 2013 IEEE international conference on image processing. IEEE, pp 914–917Google Scholar
  16. 16.
    Kingma DP, Ba JL (2014) Adam: a method for stochastic optimization. In: Proceedings of the 3rd international conference on learning representationsGoogle Scholar
  17. 17.
    Li S, Ren W, Zhang J, Yu J, Guo X (2019) Single image rain removal via a deep decomposition-composition network. Computer Vis Image Underst 186:8–57Google Scholar
  18. 18.
    Mao X, Li Q, Xie H, Lau RY, Wang Z, Smolley SP (2017) Least squares generative adversarial networks. In: 2017 IEEE international conference on computer vision (ICCV). IEEE, pp 2813–2821Google Scholar
  19. 19.
    Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241Google Scholar
  20. 20.
    Rudin LI, Osher S, Fatemi E (1992) Nonlinear total variation based noise removal algorithms. Phys D Nonlinear Phenom 60(1–4):259–268MathSciNetCrossRefGoogle Scholar
  21. 21.
    Santhaseelan V, Asari VK (2015) Utilizing local phase information to remove rain from video. Int J Comput Vis 112(1):71–89CrossRefGoogle Scholar
  22. 22.
    Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: ICLRGoogle Scholar
  23. 23.
    Wang Y, Gong D, Yang J, Shi Q, Hengel Avd, Xie D, Zeng B (2019) An effective two-branch model-based deep network for single image deraining. arXiv preprint arXiv:1905.05404
  24. 24.
    Wang YT, Zhao XL, Jiang TX, Deng LJ, Chang Y, Huang TZ (2018) Rain streak removal for single image via kernel guided cnn. arXiv preprint arXiv:1808.08545
  25. 25.
    Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612CrossRefGoogle Scholar
  26. 26.
    Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1357–1366Google Scholar
  27. 27.
    Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative image inpainting with contextual attention. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5505–5514Google Scholar
  28. 28.
    Zhang H, Sindagi V, Patel VM (2019) Image de-raining using a conditional generative adversarial network. In: IEEE transactions on circuits and systems for video technologyGoogle Scholar
  29. 29.
    Zhang H, Xu T, Li H, Zhang S, Huang X, Wang X, Metaxas D (2017) Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In: IEEE international conference on computer vision (ICCV), pp 5907–5915Google Scholar
  30. 30.
    Zhou B, Lapedriza A, Khosla A, Oliva A, Torralba A (2017) Places: a 10 million image database for scene recognition. IEEE Trans Pattern Anal Mach Intell 40(6):1452–1464CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.Faculty of Electronics and Electrical EngineeringUniversity of Engineering and TechnologyTaxilaPakistan
  2. 2.College of Information and Communication EngineeringSungkyunkwan UniversitySuwonKorea
  3. 3.Quaid-e-Awam University of Engineering, Science and TechnologyLarkanaPakistan
  4. 4.Department of Software and Communications EngineeringHongik UniversitySejongKorea

Personalised recommendations