Advertisement

Single image rain streaks removal: a review and an exploration

  • Hong Wang
  • Qi Xie
  • Yichen Wu
  • Qian Zhao
  • Deyu MengEmail author
Original Article
  • 29 Downloads

Abstract

Recently, rain streaks removal from a single image has attracted much research attention to alleviate the degenerated performance of computer vision tasks implemented on rainy images. In this paper, we provide a thorough review for current single-image-based rain removal techniques, which can be mainly categorized into three classes: early filter-based, conventional prior-based, and recent deep learning-based approaches. Furthermore, inspired by the rationality of current deep learning-based methods and insightful characteristics underlying rain shapes, we build a specific coarse-to-fine deraining network architecture, which can finely deliver the rain structures and progressively removes rain streaks from the input image, accordingly. The superiority of the proposed network is substantiated by experiments implemented on synthetic and real rainy images both visually and quantitatively, as compared with comprehensive state-of-the-art methods along this line. Especially, it is verified that the proposed network possesses better generalization capability on real rainy images, implying its potential usefulness for this task.

Keywords

Single image deraining Conventional model Deep learning Encoder–decoder Generalization capability 

Notes

Acknowledgements

This research was supported by National Key R&D Program of China (2018YFB1004300) and China NSFC projects (61661166011, 11690011, 61603292, 61721002, U1811461). This work is also partially supported by MoE-CMCC “Artifical Intelligence” Project No. MCM20190701.

References

  1. 1.
    Farenzena M, Bazzani L, Perina A, Murino V, Cristani M (2010) Person re-identification by symmetry-driven accumulation of local features. In Proc. of the IEEE Conf. on Comput. vision and pattern recognitionGoogle Scholar
  2. 2.
    Maji S, Berg AC, Malik J (2008) Classification using intersection kernel support vector machines is efficient. In Proc. of the IEEE Conf. on Comput. vision and pattern recognition, pp 1–8Google Scholar
  3. 3.
    Junior OL, Delgado D, Gonalve V, Nunes U (2009) Trainable classifier-fusion schemes: an application to pedestrian detection. In Intelligent Transportation Syst., vol. 2Google Scholar
  4. 4.
    Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 20(11):1254–1259CrossRefGoogle Scholar
  5. 5.
    Kang L, Lin C, Fu Y (2012) Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans Image Process 24(4):1742–1755MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Li Y, Tan RT, Guo X, Lu J, Brown MS (2016) Rain streak removal using layer priors. IEEE Conf. on Comput. vision and pattern recognitionGoogle Scholar
  7. 7.
    Gu S, Meng D, Zuo W, Zhang L (2017) Joint convolutional analysis and synthesis sparse representation for single image layer separation. IEEE Int. Conf. on Comput. Vision, pp 1717–1725Google Scholar
  8. 8.
    Luo Y, Xu Y, Ji H (2015) Removing rain from a single image via discriminative sparse coding. IEEE Int. Conf. on Comput. Vision, pp 3397–3405Google Scholar
  9. 9.
    Wei W, Yi L, Xie Q, Zhao Q, Meng D, Xu Z (2017) Should We encode rain streaks in video as deterministic or stochastic? IEEE Int. Conf. on Comput. Vision, pp 2535–2544Google Scholar
  10. 10.
    Kang L, Lin C, Lin C, Lin Y (2012) Self-learning-based rain streak removal for image/video. IEEE Int Symp Circuits Syst 57(1):1871–1874Google Scholar
  11. 11.
    Chen YL, Hsu CT (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. IEEE Int. Conf. on Comput. Vision, pp 1968–1975Google Scholar
  12. 12.
    Ren D, Zuo W, Hu Q, Zhu P, Meng D (2019) Progressive image deraining networks: a better and simpler baseline. IEEE Conf. on Comput. Vision and Pattern RecognitionGoogle Scholar
  13. 13.
    He K, Sun J, Tang X (2010) Guided image filtering. European Conf. on Comput. vision, pp 1–14Google Scholar
  14. 14.
    Xu J, Zhao W, Liu P, Tang X (2012) Removing rain and snow in a single image using guided filter. IEEE Int. Conf. on Comput. Sci. and Automation Eng, pp 304–307Google Scholar
  15. 15.
    Zheng X, Liao Y, Guo W, Fu X, Ding X (2013) Single-image-based rain and snow removal using multi-guided filter. Neural Inform. Process, pp. 258–265CrossRefGoogle Scholar
  16. 16.
    Chen D, Chen C, Kang L (2014) Visual depth guided color image rain streaks removal using sparse coding. IEEE Trans Circuits Syst Video Technol 24(24):1430–1455CrossRefGoogle Scholar
  17. 17.
    Ding X, Chen L, Zheng X, Huang Y, Zeng D (2016) Single image rain and snow removal via guided L0 smoothing filter. Multimed Tools Appl 75(5):2697–2712CrossRefGoogle Scholar
  18. 18.
    Kim JH, Lee C, Sim JY, Kim (2013) Single-image deraining using an adaptive nonlocal means filter. IEEE Int. Conf. on Image Process, pp 914–917Google Scholar
  19. 19.
    Fu Y, Kang L, Lin C, Hsu CT (2013) Single-frame-based rain removal via image decomposition. IEEE Int. Conf. on Acoustics, pp 914–917Google Scholar
  20. 20.
    Wang Y, Liu S, Chen C, Zeng B (2017) A hierarchical approach for rain or snow removing in a single color image. IEEE Trans Image Process 26(8):3936–3950MathSciNetzbMATHCrossRefGoogle Scholar
  21. 21.
    Wei W, Meng D, Zhao Q, Xu Z, Wu Y (2019) Semi-supervised transfer learning for image rain removal. In Proc. of the IEEE Conf. on Comput. vision and pattern recognition, pp 3877–3886Google Scholar
  22. 22.
    Meng D, Zhao Q, Xu Z (2012) Improve robustness of sparse pca by l1-norm maximization. Pattern Recogn 45(1):487–497zbMATHCrossRefGoogle Scholar
  23. 23.
    Zhu L, Fu C, Lischinski D, Heng P (2017) Joint bi-layer optimization for single-image rain streak removal. IEEE Int. Conf. on Comput. vision, pp 2545–2553Google Scholar
  24. 24.
    Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. IEEE Conf. on Comput. vision and pattern recognition, pp 1685–1694Google Scholar
  25. 25.
    Mu P, Chen J, Liu R, Fan X, Luo Z (2018) Learning bilevel layer priors for single image rain streaks removal. IEEE Signal Process Lett 26(2):307–311CrossRefGoogle Scholar
  26. 26.
    Zhang K, Zuo W, Gu S, Zhang L (2017) Learning deep CNN denoiser prior for image restoration. In Proc. of the IEEE Conf. on computer vision and pattern recognition, pp 3929–3938Google Scholar
  27. 27.
    Xie Q, Zhou M, Zhao Q, Meng D, Zuo W, Xu Z (2019) Multispectral and hyperspectral image fusion by MS/HS fusion net. IEEE Conf. on Comput. vision and pattern recognitionGoogle Scholar
  28. 28.
    Eigen D, Krishnan D, Fergus R (2013) Restoring an image taken through a window covered with dirt or rain. IEEE Int. Conf. on comput. vision, pp 633–640Google Scholar
  29. 29.
    Qian R, Tan R, Yang W, Su J, Liu J (2018) Attentive generative adversarial network for raindrop removal from a single image. IEEE Conf. on Comput. vision and pattern recognition, pp 1–1Google Scholar
  30. 30.
    Fu X, Huang J, Ding X (2017) Clearing the skies: a deep network architecture for single-image rain streaks removal. IEEE Trans Image Process 1(1):99zbMATHGoogle Scholar
  31. 31.
    He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In Proc. of the IEEE Conf. on Comput. vision and pattern recognition, pp 770–778Google Scholar
  32. 32.
    Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisle J (2017) Removing rain from single images via a deep detail network. IEEE Conf. on Comput. vision and pattern recognition, pp 1715–1723Google Scholar
  33. 33.
    Zhang H, Sindagi V, Patel VM (2017) Image de-raining using a conditional generative adversarial network. arXiv:1701.05957
  34. 34.
    Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. IEEE Conf. on Comput. vision and pattern recognition, pp 1–10Google Scholar
  35. 35.
    Yu F, Koltun V (2016) Multi-scale context aggregation by dilated convolutions. In Int. Conf. on learning representationGoogle Scholar
  36. 36.
    Li X, Wu J, Lin Z, Liu H, Zha H (2018) Recurrent squeezeand-excitation context aggregation net for single image deraining. In European Conf. on computer vision, pp 262–277CrossRefGoogle Scholar
  37. 37.
    Li G, He X, Zhang W, Chang H, Dong L, Lin L (2018) Non-locally enhanced encoder-decoder network for single image de-raining. In 2018 ACM Multimedia Conf. on Multimedia Conf., pp 1056–1064Google Scholar
  38. 38.
    Li S, Araujo IB, Ren W, Wang Z, Tokuda EK, Junior RH et al (2019) Single image deraining: a comprehensive benchmark analysis. arXiv:1903.08558
  39. 39.
    Wang T, Yang X, Xu K, Chen S, Zhang Q, Lau RWH (2019) Spatial attentive single-image deraining with a high quality real rain dataset. IEEE Conf. on Comput. vision and pattern recognitionGoogle Scholar
  40. 40.
    Wang Y, Zhao X, Jiang T, Deng L, Chang Y, Huang T (2018) Rain streak removal for single image via kernel guided cnn. arXiv:1808.08545
  41. 41.
    Pan J, Liu S, Zhang J, Liu Y, Ren J, Li Zechao (2018) Learning dual convolutional neural networks for low-level vision. IEEE Conf. on Comput. vision and pattern recognition, pp 1–10Google Scholar
  42. 42.
    Quan H, Ghanbari M (2008) Scope of validity of PSNR in image/video quality assessment. Electron Lett 44(13):800–801CrossRefGoogle Scholar
  43. 43.
    Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612CrossRefGoogle Scholar
  44. 44.
    Li M, Xie Q, Zhao Q, Wei W, Gu S, Tao J, Meng D (2018) Video rain streak removal by multiscale convolutional sparse coding. IEEE Conf. on Comput. vision and pattern recognition, pp. 1-10Google Scholar
  45. 45.
    Fan Z, Wu H, Fu X, Huang Y, Ding X (2018) Residual guide feature fusion network for single image deraining. In ACM MultimediaGoogle Scholar
  46. 46.
    Fu X, Liang B, Huang Y, Ding X, Paisley J (2018) Lightweight pyramid networks for image deraining. arXiv:1805.06173
  47. 47.
    Lin H, Li Y, Ding X, Zeng W, Huang Y, Paisley J (2019) Rain O’er Me: synthesizing real rain to derain with data distillation. arXiv:1904.04605
  48. 48.
    Paszke A, Gross S, Chintala S, Chanan G, Yang E, De-Vito Z (2017) Automatic differentiation in pytorchGoogle Scholar
  49. 49.
    Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980
  50. 50.
    Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. IEEE Int Conf Comput Vis 2:416–423Google Scholar
  51. 51.
    Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167
  52. 52.
    Xu J, Zhao W, Liu P, Tang X (2012) An improved guidance image based method to remove rain and snow in a single image. Comput Inf Sci 5(3):49–55Google Scholar
  53. 53.
    Zhang H, Patel M (2017) Convolutional sparse and low-rank coding-based rain streak removal. IEEE Winter Conf. on applications of Comput. vision, pp 1259–1267Google Scholar
  54. 54.
    Zhou W, Bovik AC (2009) Mean squared error: love it or leave it? a new look at signal fidelity measures. IEEE Signal Process Mag 26(1):98–117CrossRefGoogle Scholar
  55. 55.
    Zhao H, Gallo O, Frosio I, Kautz J (2017) Loss functions for image restoration with neural networks. IEEE Trans Comput Imaging 3(1):47–57CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2020

Authors and Affiliations

  • Hong Wang
    • 1
  • Qi Xie
    • 1
  • Yichen Wu
    • 1
  • Qian Zhao
    • 1
  • Deyu Meng
    • 1
    • 2
    Email author
  1. 1.Institute for Information and System Sciences and Ministry of Education Key Lab of Intelligent Networks and Network SecurityXian Jiaotong UniversityXi’anPeople’s Republic of China
  2. 2.Faculty of Information TechnologyMacau University of Science and TechnologyTaipaPeople’s Republic of China

Personalised recommendations