Advertisement

Sun position estimation and tracking for virtual object placement in time-lapse videos

  • 340 Accesses

  • 3 Citations

Abstract

Realistic illumination of virtual objects placed in real videos is important in terms of achieving visual coherence. We propose a novel approach for illumination estimation on time-lapse videos and seamlessly insert virtual objects in these videos in a visually consistent way. The proposed approach works for both outdoor and indoor environments where the main light source is the Sun. We first modify an existing illumination estimation method that aims to obtain sparse radiance map of the environment in order to estimate the initial Sun position. We then track the hard ground shadows on the time-lapse video by using an energy-based pixel-wise method. The proposed method aims to track the shadows by utilizing the energy values of the pixels that forms them. We tested the method on various time-lapse videos recorded in outdoor and indoor environments and obtained successful results.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

References

  1. 1.

    Panagopoulos, A., Wang, C., Samaras, D., Paragios, N.: Illumination estimation and cast shadow detection through a higher-order graphical model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11), pp. 673–680 (2011)

  2. 2.

    Lalonde, J.-F., Efros, A.A., Narasimhan, S.G.: Estimating the natural illumination conditions from a single outdoor image. Int. J. Comput. Vis. 98(2), 123–145 (2012)

  3. 3.

    Guo, R., Dai, Q., Hoiem, D.: Paired regions for shadow detection and removal. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2956–2967 (2013)

  4. 4.

    Liu, Y., Gevers, T., Li, X.: Estimation of sunlight direction using 3D object models. IEEE Trans. Image Proc. 24(3), 932–942 (2014)

  5. 5.

    Lalonde, J.-F., Matthews, I.: Lighting estimation in outdoor image collections. In: Proceedings of the International Conference on 3D Vision, vol. 1, Tokyo, Japan, pp. 131–138 (2014)

  6. 6.

    Liu, Y., Granier, X.: Online tracking of outdoor lighting variations for augmented reality with moving cameras. IEEE Trans. Vis. Comput. Graph. 18(4), 573–580 (2012)

  7. 7.

    Andersen, M.S., Jensen, T., Madsen, C.B.: Estimation of dynamic light changes in outdoor scenes without the use of calibration objects. In: Proceedings of the International Conference on Pattern Recognition, vol. 4, 91–94 (2006)

  8. 8.

    Xing, G., Liu, Y., Qin, X., Peng, Q.: On-line illumination estimation of outdoor scenes based on area selection for augmented reality. In: Proceedings of the 12th International Conference on Computer-Aided Design and Computer Graphics (CADGRAPHICS’11), pp. 439–442 (2011)

  9. 9.

    Ikeda, T., Oyamada, Y., Sugimoto, M., Saito, H.: Illumination estimation from shadow and incomplete object shape captured by an RGB-D camera. In: Proceedings of the 21st International Conference on Patten Recognition Series (ICPR’12), Tsukuba, Japan, pp. 165–169 (2012)

  10. 10.

    Gruber, L., Langlotz, T., Sen, P., Hoherer, T., Schmalstieg, D.: Efficient and robust radiance transfer for probeless photorealistic augmented reality. In: Proceedings of the IEEE Virtual Reality, pp. 15–20 (2014)

  11. 11.

    Lensing, P., Broll, W.: Instant indirect illumination for dynamic mixed reality scenes. In: Proceedings of the 11th IEEE International Symposium on Mixed and Augmented Reality (ISMAR’12), pp. 109–118 (2012)

  12. 12.

    Neverova, N., Muselet, D., Trémeau, A.: Lighting estimation in indoor environments from low-quality images. In: Proceedings of the 12th International Conference on Computer Vision (ICCV’12), vol. 2, pp. 380–389 (2012)

  13. 13.

    Yoo, J., Lee, K.: Light source estimation for realistic shadow using segmented HDR images. In: Hong, D., Jeon, S. (eds.) Proceedings of the International Symposium on Ubiquitous Virtual Reality (ISUVR’07), vol. 260 (2007)

  14. 14.

    Lopez-Moreno, J., Garces, E., Hadap, S., Reinhard, E., Gutierrez, D.: Multiple light source estimation in a single image. Comput. Graph. Forum 32(8), 170–182 (2013)

  15. 15.

    Zhao, W., Zheng, Y., Wang, L., Peng, S.: Lighting estimation of a convex Lambertian object using weighted spherical harmonic frames. Signal Image Video Proc. 9(1), 57–75 (2015)

  16. 16.

    Sunkavalli, K., Matusik, W., Pfister, H., Rusinkiewicz, S.: Factored time-lapse video. In: ACM Transactions on Graphics. Proceedings of the SIGGRAPH’07, vol. 26, no. 3, Article No. 101 (2007)

  17. 17.

    Zhang, R., Zhong, F., Lin, L., Xing, G., Peng, Q., Qin, X.: Basis image decomposition of outdoor time-lapse videos. Vis. Comput. 29(11), 1197–1210 (2013)

  18. 18.

    Lalonde, J.-F., Efros, A.A., Narasimhan, S.G.: Webcam clip art: Appearance and illuminant transfer from time-lapse sequences. In: ACM Transactions on Graphics. Proceedings of the SIGGRAPH Asia’09, vol. 28, no. 5, Article No. 131 (2009)

  19. 19.

    Chen, X., Wang, K., Jin, X.: Single image based illumination estimation for lighting virtual object in real scene. In: Proceedings of the 12th International Conference on Computer-Aided Design and Computer Graphics, pp. 450–455 (2011)

  20. 20.

    Saxena, A., Sun, M., Ng, A.: Make3D: learning 3D scene structure from a single still image. IEEE Trans. Pattern Anal. Mach. Intell. 31(5), 824–840 (2009)

  21. 21.

    Garces, E., Munoz, A., Lopez-Moreno, J., Gutierrez, D.: Intrinsic images by clustering. In: Computer Graphics Forum. Proceedings of the Eurographics Symposium on Rendering, vol. 31, no. 4, pp. 1415–1424 (2012)

  22. 22.

    Phong, B.T.: Illumination for computer generated pictures. Commun. ACM 18(6), 311–317 (1975)

  23. 23.

    Moré, J.J.: The Levenberg–Marquardt algorithm: implementation and theory. In: Watson, G.A. (ed.) Numerical Analysis, pp. 105–116. Springer, Berlin (1977)

  24. 24.

    Lalonde, J.-F., Efros, A., Narasimhan, S.: Detecting ground shadows in outdoor consumer photographs. In: Proceedings of the European Conference on Computer Vision (ECCV’10), LNCS, vol. 6312, pp. 322–335 (2010)

  25. 25.

    Avidan, S., Shamir, A.: Seam carving for content-aware image resizing. In: Proceedings of the 34th International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’07). New York, NY, USA: ACM (2007)

  26. 26.

    Unity Technologies: Unity Game Engine. http://unity3d.com/. Accessed 2016-11-12

  27. 27.

    MathWorks: MATLAB. http://www.mathworks.com/products/matlab/. Accessed 2016-11-12

  28. 28.

    Gazda, T.: Sun’s shadow time lapse. https://www.youtube.com/watch?v=3B7KLstUZbI. Accessed 2016-11-12

  29. 29.

    Davies, M.: Shadows timelapse. https://www.youtube.com/watch?v=Lvhjbrr5GI8. Accessed 2015-11-12

  30. 30.

    Bates, G.: Fast food shadows timelapse. https://www.youtube.com/watch?v=mdhS6pds8VY. Accessed 2015-11-12

  31. 31.

    Zheng, Y., Chen, X., Cheng, M.-M., Zhou, K., Hu, S.-M., Mitra, N.J.: Interactive images: Cuboid proxies for smart image manipulation. In: ACM Transactions on Graphics. Proceedings of the SIGGRAPH’12, vol. 31, no. 4, Article No. 99 (2012)

Download references

Acknowledgements

This research is supported by The Scientific and Technological Research Council of Turkey (TÜBİTAK) under Grant No. 112E110. The first author is supported by TÜBİTAK under BİDEB 2210 Graduate Scholarship. We gratefully acknowledge Gordon Bates, Matthew Davies and Tom Gazda for permitting us to use their videos.

Author information

Correspondence to Uğur Güdükbay.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 2 (mp4 64775 KB)

Supplementary material 1 (pdf 889 KB)

Supplementary material 2 (mp4 64775 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Balcı, H., Güdükbay, U. Sun position estimation and tracking for virtual object placement in time-lapse videos. SIViP 11, 817–824 (2017) doi:10.1007/s11760-016-1027-x

Download citation

Keywords

  • Sun position estimation
  • Light source estimation
  • Illumination estimation
  • Time-lapse video
  • Shadow tracking
  • Image/video editing