Rain removal system for dynamic scene in diminished reality

  • Damon Shing-Min LiuEmail author
  • Yi-Jhih Chen
Original Paper


Visibility in the rain is poor. Here, we develop a rain removal system for dynamic scene in diminished reality. Our system provides a clear and comfortable view. We apply and extend chromatic prior and explore chromatic pair, dark tail removal and spatial chromatic prior. We design an optimized intuitive alignment to solve moving camera problem. We apply anti-flicker to completely remove drizzle and defocused rain. Benefit from parallel pixel-based chromatic prior, our system has high scalability and extensibility. Our experiments reference numerous videos from prior related works, involving realistic and diverse property of rain, motion of the scene and motion of the camera. The experimental results show a satisfactory rain removal quality, especially for drizzle and defocused rain.


Rain removal Diminished reality Augmented reality Drizzle removal Defocused rain removal Chromatic prior Dark tail removal Anti-flicker Homography 



This work of Professor Damon Shing-Min Liu was supported in part by the Ministry of Science and Technology of Taiwan under Grant Number NSC102-2221-E-194-048.

Compliance with ethical standards

Conflict of interest

Mr. Yi-Jhih Chen has no conflict of interest.

Supplementary material

Supplementary material 1 (avi 1522 KB)

11760_2019_1626_MOESM2_ESM.wmv (7.7 mb)
Supplementary material 2 (wmv 7864 KB)
11760_2019_1626_MOESM3_ESM.avi (1.8 mb)
Supplementary material 3 (avi 1886 KB)
11760_2019_1626_MOESM4_ESM.wmv (3.4 mb)
Supplementary material 4 (wmv 3502 KB)
11760_2019_1626_MOESM5_ESM.avi (445 kb)
Supplementary material 5 (avi 444 KB)
11760_2019_1626_MOESM6_ESM.wmv (11.4 mb)
Supplementary material 6 (wmv 11698 KB)
11760_2019_1626_MOESM7_ESM.avi (5.2 mb)
Supplementary material 7 (avi 5341 KB)


  1. 1.
    Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B.: Recent advances in augmented reality. IEEE Comput. Graph. Appl. 21(6), 34–47 (2001)CrossRefGoogle Scholar
  2. 2.
    Herling, J., Broll, W.: Advanced self-contained object removal for realizing real-time diminished reality in unconstrained environments. In: 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 207–212, (2010)Google Scholar
  3. 3.
    Leao, C.W.M., Lima, J.P., Teichrieb, V., Albuquerque, E.S., Kelner, J.: Altered reality: augmenting and diminishing reality in real time. In: IEEE Virtual Reality Conference (VR), pp. 219–220, (2011)Google Scholar
  4. 4.
    Zokai, S., Esteve, J., Genc, Y., Navab, N.: Multiview paraperspective projection model for diminished reality. In: The 2nd IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 217–226, (2003)Google Scholar
  5. 5.
    Kawai, N., Sato, T., Yokoya, N.: Diminished reality considering background structures. In: 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 259–260, (2013)Google Scholar
  6. 6.
    Korkalo, O., Aittala, M., Siltanen, S.: Light-weight marker hiding for augmented reality. In: 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 247–248, (2010)Google Scholar
  7. 7.
    Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: PatchMatch. ACM Trans. Graph. 28(3), 1 (2009)CrossRefGoogle Scholar
  8. 8.
    Bossu, J., Hautière, N., Tarel, J.-P.: Rain or snow detection in image sequences through use of a histogram of orientation of streaks. Int. J. Comput. Vis. 93(3), 348–367 (2011)CrossRefGoogle Scholar
  9. 9.
    Barnum, P., Kanade, T., Narasimhan, S.: Spatio-temporal frequency analysis for removing rain and snow from videos. In: Proceedings of the 1st International Workshop on Photometric Analysis for Computer Vision, (2007)Google Scholar
  10. 10.
    Barnum, P.C., Narasimhan, S., Kanade, T.: Analysis of rain and snow in frequency space. Int. J. Comput. Vis. 86(2–3), 256–274 (2009)Google Scholar
  11. 11.
    Zhang, X., Li, H., Qi, Y., Leow, W., Ng, T.: Rain removal in video by combining temporal and chromatic properties. In: IEEE International Conference on Multimedia and Expo, pp. 461–464, (2006)Google Scholar
  12. 12.
    Huang, D.-A., Kang, L.-W., Yang, M.-C., Lin, C.-W., Wang, Y.-C.F.: Context-aware single image rain removal. In: IEEE International Conference on Multimedia and Expo, pp. 164–169, (2012)Google Scholar
  13. 13.
    Fu, Y.-H., Kang, L.-W., Lin, C.-W., Hsu, C.-T.: Single-frame-based rain removal via image decomposition. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1453–1456, (2011)Google Scholar
  14. 14.
    Kang, L.-W., Lin, C.-W., Fu, Y.-H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–55 (2012)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Kang, L.-W., Lin, C.-W., Lin, C.-T., Lin, Y.-C.: Self-learning-based rain streak removal for image/video. In: IEEE International Symposium on Circuits and Systems, pp. 1871–1874, (2012)Google Scholar
  16. 16.
    Xu, J., Zhao, W., Liu, P., Tang, X.: Removing rain and snow in a single image using guided filter. In: IEEE International Conference on Computer Science and Automation Engineering (CSAE), pp. 304–307, (2012)Google Scholar
  17. 17.
    He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–409 (2013)CrossRefGoogle Scholar
  18. 18.
    Garg, K., Nayar, S.K.: When does a camera see rain? In: 10th IEEE International Conference on Computer Vision (ICCV’05) vol. 1, pp. 1067–1074, (2005)Google Scholar
  19. 19.
    Garg, K., Nayar, S.K.: Vision and rain. Int. J. Comput. Vis. 75(1), 3–27 (2007)CrossRefGoogle Scholar
  20. 20.
    de Charette, R., Tamburo, R., Barnum, P.C., Rowe, A., Kanade, T., Narasimhan, S.G.: Fast reactive control for illumination through rain and snow. In: IEEE International Conference on Computational Photography, pp. 1–10, (2012)Google Scholar
  21. 21.
    Qian, R., Tan, R.T., Yang, W., Su, J., Liu, J.: Attentive generative adversarial network for raindrop removal from a single image, In: IEEE Conference on Computer Vision and Pattern Recognition, (2018)Google Scholar
  22. 22.
    Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network. In: IEEE Conference on Computer Vision and Pattern Recognition, (2018)Google Scholar
  23. 23.
    Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: IEEE Conference on Computer Vision and Pattern Recognition, (2017)Google Scholar
  24. 24.
    Jarusirisawad, S., Saito, H.: Diminished reality via multiple hand-held CAMERAS. In: 1st ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC), pp. 251–258, (2007)Google Scholar
  25. 25.
    Seo, B.-K., Lee, M.-H., Park, H., Park, J.-I.: Projection-based diminished reality system. In: International Symposium on Ubiquitous Virtual Reality, pp. 25–28, (2008)Google Scholar
  26. 26.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  27. 27.
    “Speed of a Falling Raindrop.” Accessed 20 Sep 2018
  28. 28.
    Mohler, B.J., Thompson, W.B., Creem-Regehr, S.H., Pick, H.L., Warren, W.H.: Visual flow influences gait transition speed and preferred walking speed. Exp. Brain Res. 181(2), 221–8 (2007)CrossRefGoogle Scholar
  29. 29.
    Garg, K., Nayar, S.K.: Photorealistic rendering of rain streaks. In: ACM SIGGRAPH, vol. 25, no. 3, p. 996, (2006)Google Scholar
  30. 30.
    Gonda, F.: “Rain Simulator.” Accessed 20 Sep 2018
  31. 31.
    Rousseau, P., Jolivet, V., Ghazanfarpour, D.: Realistic real-time rain rendering. Comput. Graph. 30(4), 507–518 (2006)CrossRefGoogle Scholar
  32. 32.
    “Valve.” Accessed 20 Sep 2018

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2020

Authors and Affiliations

  1. 1.Computer Science DepartmentNational Chung Cheng UniversityChiayiTaiwan

Personalised recommendations