Single image rain streak removal via layer similarity prior

Abstract

Single image rain streak removal is a significant and challenging task, which is widely applied in many artificial intelligence domains as preprocessing process. Most of existing rain streak removal works focus on designing various deraining unit (e.g., multi-stream dilation convolution) without considering the correlation between different convolution layers, which may lead to large model size. In this paper, we propose a simple and effective deep network architecture for single image rain streak removal based on deep Convolutional Neural Network (CNN). Benefit from the adjacent layers with different dilation factors have similar feature structures, we design a powerful rain streak representation network based on the Layer Similarity Prior Block (LSPB). To better cater to the property of layer similarity prior, the multi-dense-short-connection is developed and it regards every LSPB as a convolution layer, this connection style makes our overall framework to be a layer similarity prior network. To the best of our knowledge, this is the first paper to investigate the effectiveness of exploiting the layer similarity prior and the multi-dense-short-connection. Quantitative and qualitative experimental results demonstrate that the proposed method outperforms other state-of-the-art methods with the least parameters.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

References

  1. 1.

    Brewer N, Liu N (2008) Using the shape characteristics of rain to identify and remove rain from video. In: Structural, syntactic, and statistical pattern recognition, pp 451–458, https://doi.org/10.1007/978-3-540-89689-0_49, (to appear in print)

  2. 2.

    Chen L, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2018) Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI 40 (4):834–848. https://doi.org/10.1109/TPAMI.2017.2699184

    Article  Google Scholar 

  3. 3.

    Chen Y, Hsu C (2013) A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: ICCV, pp 1968–1975, https://doi.org/10.1109/ICCV.2013.247, (to appear in print)

  4. 4.

    Cui Z, Chang H, Shan S, Zhong B, Chen X (2014) Deep network cascade for image super-resolution. In: ECCV, pp 49–64, https://doi.org/10.1007/978-3-319-10602-1_4, (to appear in print)

  5. 5.

    Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. 38(2):295–307. https://doi.org/10.1109/TPAMI.2015.2439281

  6. 6.

    Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: a deep network architecture for single-image rain removal. 26(6):2944–2956. https://doi.org/10.1109/TIP.2017.2691802

  7. 7.

    Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) Removing rain from single images via a deep detail network. In: CVPR, pp 1715–1723, https://doi.org/10.1109/CVPR.2017.186, (to appear in print)

  8. 8.

    Gonzalez-Garcia A, van de Weijer J, Bengio Y (2018) Image-to-image translation for cross-domain disentanglement. In: NIPS, pp 1294–1305

  9. 9.

    Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: CVPR, pp 7132–7141, https://doi.org/10.1109/CVPR.2018.00745, (to appear in print)

  10. 10.

    Huynh-Thu Q, Ghanbari M (2008) Scope of validity of psnr in image/video quality assessment. Electron Lett 44(13):800–801. https://doi.org/10.1049/el:20080522

    Article  Google Scholar 

  11. 11.

    Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: ECCV, pp 694–711, https://doi.org/10.1007/978-3-319-46475-6_43, (to appear in print)

  12. 12.

    Kang L, Lin C, Fu Y (2012) Automatic single-image-based rain streaks removal via image decomposition. 21(4):1742–1755. https://doi.org/10.1109/TIP.2011.2179057

  13. 13.

    Kim J, Lee C, Sim J, Kim C (2013) Single-image deraining using an adaptive nonlocal means filter. In: ICIP, pp 914–917, https://doi.org/10.1109/ICIP.2013.6738189, (to appear in print)

  14. 14.

    Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: ICLR

  15. 15.

    Li G, He X, Zhang W, Chang H, Dong L, Lin L (2018) Non-locally enhanced encoder-decoder network for single image de-raining. In: ACM MM, pp 1056–1064, https://doi.org/10.1145/3240508.3240636, (to appear in print)

  16. 16.

    Li S, Araujo IB, Ren W, Wang Z, Tokuda EK, Junior RH, Cesar-Junior R, Zhang J, Guo X, Cao X (2019) Single image deraining: a comprehensive benchmark analysis. In: CVPR. computer vision foundation / IEEE, pp 3838–3847, https://doi.org/10.1109/CVPR.2019.00396, (to appear in print)

  17. 17.

    Li X, Wu J, Lin Z, Liu H, Zha H (2018) Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: ECCV, pp 262–277, https://doi.org/10.1007/978-3-030-01234-2_16, (to appear in print)

  18. 18.

    Li Y, Tan RT, Guo X, Lu J, Brown MS (2016) Rain streak removal using layer priors. In: CVPR, pp 2736–2744, https://doi.org/10.1109/CVPR.2016.299, (to appear in print)

  19. 19.

    Lin T, Dollár P, Girshick RB, He K, Hariharan B, Belongie SJ (2017) Feature pyramid networks for object detection. In: CVPR, pp 936–944, https://doi.org/10.1109/CVPR.2017.106, (to appear in print)

  20. 20.

    Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: CVPR, pp 3431–3440, https://doi.org/10.1109/CVPR.2015.7298965, (to appear in print)

  21. 21.

    Luo Y, Xu Y, Ji H (2015) Removing rain from a single image via discriminative sparse coding. In: ICCV, pp 3397–3405, https://doi.org/10.1109/ICCV.2015.388, (to appear in print)

  22. 22.

    Mittal A, Soundararajan R, Bovik AC (2013) Making a “completely blind” image quality analyzer. IEEE Signal Process Lett 20(3):209–212. https://doi.org/10.1109/LSP.2012.2227726

    Article  Google Scholar 

  23. 23.

    Santhaseelan V, Asari VK (2015) Utilizing local phase information to remove rain from video. IJCV 112(1):71–89. https://doi.org/10.1007/s11263-014-0759-8

    Article  Google Scholar 

  24. 24.

    Tripathi AK, Mukhopadhyay S (2014) Removal of rain from videos: a review. SIViP 8 (8):1421–1430. https://doi.org/10.1007/s11760-012-0373-6

    Article  Google Scholar 

  25. 25.

    Wang C, Wang H, Su Z, Yang Y (2019) Embedding non-local mean in squeeze-and-excitation network for single image deraining. In: ICMEW, pp 264–269, https://doi.org/10.1109/ICMEW.2019.00-76, (to appear in print)

  26. 26.

    Wang C, Zhang M, Pan J, Su Z (2019) Single image rain removal via densely connected contextual and semantic correlation net. JEI 28(3):033,018. https://doi.org/10.1117/1.JEI.28.3.033018

    Article  Google Scholar 

  27. 27.

    Wang C, Zhang M, Su Z, Wu Y, Yao G, Wang H (2019) Learning a multi-level guided residual network for single image deraining. Signal Process Image Commun 78:206–215. https://doi.org/10.1016/j.image.2019.07.003

    Article  Google Scholar 

  28. 28.

    Wang C, Zhang M, Su Z, Yao G, Wang Y, Sun X, Luo X (2019) From coarse to fine: a stage-wise deraining net. IEEE Access 7:84,420–84,428. https://doi.org/10.1109/ACCESS.2019.2922549

    Article  Google Scholar 

  29. 29.

    Wang X, Shrivastava A, Gupta A (2017) A-fast-rcnn: hard positive generation via adversary for object detection. In: CVPR, pp 3039–3048, https://doi.org/10.1109/CVPR.2017.324, (to appear in print)

  30. 30.

    Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. TIP 13(4):600–612. https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  31. 31.

    Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: CVPR, pp 1685–1694, https://doi.org/10.1109/CVPR.2017.183, (to appear in print)

  32. 32.

    Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. In: CVPR, pp 695–704, https://doi.org/10.1109/CVPR.2018.00079, (to appear in print)

  33. 33.

    Zhang H, Sindagi V, Patel VM (2019) Image de-raining using a conditional generative adversarial network. TCSVT. https://doi.org/10.1109/TCSVT.2019.2920407

  34. 34.

    Zhang X, Li H, Qi Y, Leow WK, Ng TK (2006) Rain removal in video by combining temporal and chromatic properties. In: ICME, pp 461–464, https://doi.org/10.1109/ICME.2006.262572, (to appear in print)

  35. 35.

    Zhang Y, Wang L, Qi J, Wang D, Feng M, Lu H (2018) Structured siamese network for real-time visual tracking. In: ECCV, pp 355–370, https://doi.org/10.1007/978-3-030-01240-3_22, (to appear in print)

Download references

Acknowledgments

This work was supported by the National Natural Science Foundation of China [grant numbers 61976041]; Major National Science and Technology Project of China [grant number 2018ZX04041001].

Author information

Affiliations

Authors

Corresponding author

Correspondence to Wanshu Fan.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Fan, W., Wu, Y. & Wang, C. Single image rain streak removal via layer similarity prior. Appl Intell (2021). https://doi.org/10.1007/s10489-020-02056-w

Download citation

Keywords

  • Deraining
  • Deep-learning
  • Layer similarity prior
  • Multi-dense-short-connection