Advertisement

Computational Visual Media

, Volume 5, Issue 1, pp 105–115 | Cite as

ShadowGAN: Shadow synthesis for virtual objects with conditional adversarial networks

  • Shuyang ZhangEmail author
  • Runze Liang
  • Miao Wang
Open Access
Research Article
  • 70 Downloads

Abstract

We introduce ShadowGAN, a generative adversarial network (GAN) for synthesizing shadows for virtual objects inserted in images. Given a target image containing several existing objects with shadows, and an input source object with a specified insertion position, the network generates a realistic shadow for the source object. The shadow is synthesized by a generator; using the proposed local adversarial and global adversarial discriminators, the synthetic shadow’s appearance is locally realistic in shape, and globally consistent with other objects’ shadows in terms of shadow direction and area. To overcome the lack of training data, we produced training samples based on public 3D models and rendering technology. Experimental results from a user study show that the synthetic shadowed results look natural and authentic.

Keywords

shadow synthesis deep learning generative adversarial networks image synthesis 

Notes

Acknowledgements

The authors would like to thank all the reviewers. This work was supported by the National Natural Science Foundation of China (Project Nos. 61561146393 and 61521002), the China Postdoctoral Science Foundation (Project No. 2016M601032), and a Research Grant of Beijing Higher Institution Engineering Research Center.

References

  1. [1]
    Karsch, K.; Hedau, V.; Forsyth, D.; Hoiem, D. Rendering synthetic objects into legacy photographs. ACM Transactions on Graphics Vol. 30, No. 6, Article No. 157, 2011.Google Scholar
  2. [2]
    Karsch, K.; Sunkavalli, K.; Hadap, S.; Carr, N.; Jin, H.; Fonte, R.; Sittig, M.; Forsyth, D. Automatic scene inference for 3D object compositing. ACM Transactions on Graphics Vol. 33, No. 3, Article No. 32, 2014.Google Scholar
  3. [3]
    Kee, E.; O’Brien, J. F.; Farid, H. Exposing photo manipulation from shading and shadows. ACM Transactions on Graphics Vol. 33, No. 5, Article No. 165, 2014.Google Scholar
  4. [4]
    Liu, B.; Xu, K.; Martin, R. R. Static scene illumination estimation from videos with applications. Journal of Computer Science and Technology Vol. 32, No. 3, 430–442, 2017.CrossRefGoogle Scholar
  5. [5]
    Bell, S.; Bala, K.; Snavely, N. Intrinsic images in the wild. ACM Transactions on Graphics Vol. 33, No. 4, Article No. 159, 2014.Google Scholar
  6. [6]
    Bi, S.; Han, X.; Yu, Y. An L 1 image transform for edge-preserving smoothing and scene-level intrinsic decomposition. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 78, 2015.Google Scholar
  7. [7]
    Bousseau, A.; Paris, S.; Durand, F. User-assisted intrinsic images. ACM Transactions on Graphics Vol. 28, No. 5, Article No. 130, 2009.Google Scholar
  8. [8]
    Fan, Q.; Yang, J.; Hua, G.; Chen, B.; Wipf, D. Revisiting deep intrinsic image decompositions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8944–8952, 2018.Google Scholar
  9. [9]
    Panagopoulos, A.; Samaras, D.; Paragios, N. Robust shadow and illumination estimation using a mixture model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 651–658, 2009.Google Scholar
  10. [10]
    Sato, I.; Sato, Y.; Ikeuchi, K. Illumination from shadows. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 25, No. 3, 290–300, 2003.CrossRefGoogle Scholar
  11. [11]
    Khan, E. A.; Reinhard, E.; Fleming, R. W.; Bulthoff, H. H. Image-based material editing. ACM Transactions on Graphics Vol. 25, No. 3, 654–663, 2006.CrossRefGoogle Scholar
  12. [12]
    Ge, S.; Jin, X.; Ye, Q.; Luo, Z.; Li, Q. Image editing by object-aware optimal boundary searching and mixed-domain composition. Computational Visual Media Vol. 4, No. 1, 71–82, 2018.CrossRefGoogle Scholar
  13. [13]
    Kronander, J.; Banterle, F.; Gardner, A.; Miandji, E.; Unger, J. Photorealistic rendering of mixed reality scenes. Computer Graphics Forum Vol. 34, 643–665, 2015.CrossRefGoogle Scholar
  14. [14]
    Debevec, P. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In: Proceedings of the ACM SIGGRAPH 2008 Classes, Article No. 32, 2008.CrossRefGoogle Scholar
  15. [15]
    Shor, Y.; Lischinski, D. The shadow meets the mask: Pyramid-based shadow removal. Computer Graphics Forum Vol. 27, No. 2, 577–586, 2008.CrossRefGoogle Scholar
  16. [16]
    Wang, J.; Li, X.; Yang, J. Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1788–1797, 2018.Google Scholar
  17. [17]
    Qu, L.; Tian, J.; He, S.; Tang, Y.; Lau, R. W. H. DeshadowNet: A multi-context embedding deep network for shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4067–4075, 2017.Google Scholar
  18. [18]
    Xu, L.; Qi, F.; Jiang, R. Shadow removal from a single image. In: Proceedings of the 6th International Conference on Intelligent Systems Design and Applications, 1049–1054, 2006.Google Scholar
  19. [19]
    Guo, R.; Dai, Q.; Hoiem, D. Single-image shadow detection and removal using paired regions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2033–2040, 2011.Google Scholar
  20. [20]
    Zhang, L.; Zhang, Q.; Xiao, C. Shadow remover: Image shadow removal based on illumination recovering optimization. IEEE Transactions on Image Processing Vol. 24, No. 11, 4623–4636, 2015.MathSciNetCrossRefzbMATHGoogle Scholar
  21. [21]
    Ma, L.-Q.; Wang, J.; Shechtman, E.; Sunkavalli, K.; Hu, S.-M. Appearance harmonization for single image shadow removal. Computer Graphics Forum Vol. 35, No. 7, 189–197, 2016.CrossRefGoogle Scholar
  22. [22]
    Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. In: Proceedings of the Advances in Neural Information Processing Systems 27, 2672–2680, 2014.Google Scholar
  23. [23]
    Li, M.; Zuo, W.; Zhang, D. Deep identity-aware transfer of facial attributes. arXiv preprint arXiv:1610.05586, 2016.Google Scholar
  24. [24]
    Choi, Y.; Choi, M.; Kim, M.; Ha, J.-W.; Kim, S.; Choo, J. StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8789–8797, 2018.Google Scholar
  25. [25]
    Shen, W.; Liu, R. Learning residual images for face attribute manipulation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4030–4038, 2017.Google Scholar
  26. [26]
    Ledig, C.; Theis, L.; Huszar, F.; Caballero, J.; Cunningham, A.; Acosta, A.; Aitken, A.; Tejani, A.; Totz, J.; Wang, Z.; Shi, W. Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4681–4690, 2017.Google Scholar
  27. [27]
    Pathak, D.; Krahenbuhl, P.; Donahue, J.; Darrell, T.; Efros, A. A. Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2536–2544, 2016.Google Scholar
  28. [28]
    Iizuka, S.; Simo-Serra, E.; Ishikawa, H. Globally and locally consistent image completion. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 107, 2017.Google Scholar
  29. [29]
    Mirza, M.; Osindero, S. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.Google Scholar
  30. [30]
    Isola, P.; Zhu, J.-Y.; Zhou, T.; Efros, A. A. Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1125–1134, 2017.Google Scholar
  31. [31]
    Zhu, J.-Y.; Park, T.; Isola, P.; Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, 2223–2232, 2017.Google Scholar
  32. [32]
    Portenier, T.; Hu, Q.; Szabo, A.; Bigdeli, S. A.; Favaro, P.; Zwicker, M. Faceshop: Deep sketch-based face image editing. ACM Transactions on Graphics Vol. 37, No. 4, Article No. 99, 2018.Google Scholar
  33. [33]
    Wei, Z.; Sun, Y.; Lin, J.; Liu, S. Learning adaptive receptive fields for deep image parsing networks. Computational Visual Media Vol. 4, No. 3, 231–244, 2018.CrossRefGoogle Scholar
  34. [34]
    Chang, A. X.; Funkhouser, T.; Guibas, L.; Hanrahan, P.; Huang, Q.; Li, Z.; Savarese, S.; Savva, M.; Song, S.; Su, H.; Xiao, J.; Yi, L.; Yu, F. ShapeNet: An information-rich 3D model repository. arXiv preprint arXiv:1512.03012, 2015.Google Scholar
  35. [35]
    Lafortune, E. P.; Willems, Y. D. Bi-directional path tracing. In: Proceedings of the 3rd International Conference on Computational Graphics and Visualization Techniques, 145–153, 1993.Google Scholar
  36. [36]
    Shrivastava, A.; Pfister, T.; Tuzel, O.; Susskind, J.; Wang, W.; Webb, R. Learning from simulated and unsupervised images through adversarial training. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2107–2116, 2017.Google Scholar
  37. [37]
    Yu, F.; Koltun, V. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122, 2015.Google Scholar
  38. [38]
    Kingma, D. P.; Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.Google Scholar

Copyright information

© The Author(s) 2019

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit https://doi.org/creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from https://doi.org/www.springer.com/journal/41095. To submit a manuscript, please go to https://doi.org/www.editorialmanager.com/cvmj.

Authors and Affiliations

  1. 1.University of MichiganAnn ArborUSA
  2. 2.Department of Computer Science and TechnologyTsinghua UniversityBeijingChina
  3. 3.State Key Laboratory of Virtual Reality Technology and SystemsBeihang UniversityBeijingChina

Personalised recommendations