Synthesizing Virtual-Real Artworks Using Sun Orientation Estimation

  • Xin Jin
  • Xing Sun
  • Xiaokun Zhang
  • Hongbo Sun
  • Ri Xu
  • Xiaodong LiEmail author
  • Nan Sun
Part of the Studies in Computational Intelligence book series (SCI, volume 810)


The illumination effect is essential for the realistic results in images which are created by inserting virtual objects into real scene. For outdoor scenes, automatic estimation of sun orientation condition from a single outdoor image is fundamental for inserting 3D models to a single image. Traditional methods for outdoor sun orientation estimation often use handcraft illumination features or cues. These cues heavily rely on the experiences of human and pre-processing progresses using other image understanding technologies such as shadow and sky detection, geometry recovery and intrinsic image decomposition, which limit their performances. We propose an end to end way of outdoor sun orientation estimation via a novel deep convolutional neural network (DCNN), which directly outputs the orientation of the sun from an outdoor image. Our proposed SunOriNet contains a contact layer that directly contacts the intermediate feature maps to the high-level ones and learns hierarchical features automatically from a large-scale image dataset with annotated sun orientations. The experiments reveal that our DCNN can well estimate sun orientation from a single outdoor image. The estimation accuracy of our method outperforms model state-of-the-art DCNN based methods.


Sun orientation estimation Inserting 3D models Single outdoor image Augmented reality 



We thank all the reviewers and ACs. This work is partially supported by the National Natural Science Foundation of China (Grant Nos. 61772047, 61772513, 61402021), the Science and Technology Project of the State Archives Administrator (Grant No. 2015-B-10), the open funding project of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University (Grant No. BUAA-VR-16KF-09), the Fundamental Research Funds for the Central Universities (Grant No. 3122014C017), the China Postdoctoral Science Foundation (Grant No. 2015M581841), and the Postdoctoral Science Foundation of Jiangsu Province (Grant No. 1501019A).


  1. 1.
    Basri, R., Jacobs, D., Kemelmacher, I.: Photometric stereo with general, unknown lighting. Int. J. Comput. Vis. 72, 239–257 (2007)CrossRefGoogle Scholar
  2. 2.
    Debevec, P.: Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques, New York, pp. 189–198 (1998)Google Scholar
  3. 3.
    Frahm, J.M., Koeser, K., Grest, D., et al.: Markerless augmented reality with light source estimation for direct illumination. In: Proceedings of the European Conference on Visual Media Production, London, pp. 211–220 (2005)Google Scholar
  4. 4.
    Lalonde, J.F., Efros, A., Narasimhan, S.: Estimating natural illumination from a single outdoor image. In: Proceedings of the IEEE International Conference on Computer Vision, Kyoto, pp. 183–190 (2009)Google Scholar
  5. 5.
    Lalonde, J.-F., Efros, A.A., Narasimhan, S.G.: Estimating the natural illumination conditions from a single outdoor image. Int. J. Comput. Vis. (IJCV). 98(2), 123–145 (June 2012)Google Scholar
  6. 6.
    Chen, X., Jin, X., Wang, K.: Lighting virtual objects in a single image via coarse scene understanding. Sci. China Inf. Sci. (SCIS) 57(9), 1–14 (2014)CrossRefGoogle Scholar
  7. 7.
    Liu, Y., Gevers, T., Li, X.: Estimation of sunlight direction using 3D object models. IEEE Trans. Image Process. 24(3), 932–942 (March 2015)Google Scholar
  8. 8.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (NIPS), pp. 1097–1105 (2012)Google Scholar
  9. 9.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: ICCV (2015)Google Scholar
  10. 10.
    Lalonde, J.-F., Efros, A.A., Narasimhan, S.G.: Webcam clip art: appearance and illuminant transfer from time-lapse sequences. ACM Trans. Graph. (SIGGRAPH Asia 2009), 28(5) (2009)Google Scholar
  11. 11.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding (2014). arXiv:1408.5093
  12. 12.
    Zhang, J., Lalonde, J.F.: Learning high dynamic range from outdoor panoramas. In: IEEE International Conference on Computer Vision (ICCV), Venice, pp. 4529–4538 (2017)Google Scholar
  13. 13.
    Ma, W.-C., Wang, S., Brubaker, M.A., Fidler, S., Urtasun, R.: Find your way by observing the sun and other semantic cues. In: IEEE International Conference on Robotics and Automation (ICRA), Singapore, 29 May–3 June 2017Google Scholar
  14. 14.
    Hold-Geoffroy, Y., Sunkavalli, K., Hadap, S., Gambaretto, E., Lalonde, J.-F.: Deep outdoor illumination estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  15. 15.
    Lu, H., Li, Y., Mu, S., Wang, D., Kim, H., Serikawa, S.: Motor anomaly detection for unmanned aerial vehicles using reinforcement learning. IEEE Internet Things J. (2017).
  16. 16.
    Lu, H., Li, Y., Chen, M., Kim, H., Serikawa, S.: Brain intelligence: go beyond artificial intelligence. Mob. Netw. Appl. 1–8 (2017)Google Scholar
  17. 17.
    Lu, H., Li, B., Zhu, J.: Wound intensity correction and segmentation with convolutional neural networks. Concurr. Comput. Pract. Exp. (2017).
  18. 18.
    Lu, H., Li, Y., Uemura, T., Kim, H., Serikawa, S.: Low illumination underwater light field images reconstruction using deep convolutional neural networks. Futur. Gener. Comput. Syst. (2018).
  19. 19.
    Serikawa, S., Lu, H.: Underwater image dehazing using joint trilateral filter. Comput. Electr. Eng. 40(1), 41–50 (2014)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Xin Jin
    • 1
    • 2
  • Xing Sun
    • 1
  • Xiaokun Zhang
    • 1
  • Hongbo Sun
    • 1
  • Ri Xu
    • 1
  • Xiaodong Li
    • 1
    Email author
  • Nan Sun
    • 1
  1. 1.Department of Cyber SecurityBeijing Electronic Science and Technology InstituteBeijingChina
  2. 2.CETC Big Data Research Institute Co., Ltd.GuiyangChina

Personalised recommendations