Advertisement

Faces as Lighting Probes via Unsupervised Deep Highlight Extraction

  • Renjiao Yi
  • Chenyang Zhu
  • Ping Tan
  • Stephen Lin
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11213)

Abstract

We present a method for estimating detailed scene illumination using human faces in a single image. In contrast to previous works that estimate lighting in terms of low-order basis functions or distant point lights, our technique estimates illumination at a higher precision in the form of a non-parametric environment map. Based on the observation that faces can exhibit strong highlight reflections from a broad range of lighting directions, we propose a deep neural network for extracting highlights from faces, and then trace these reflections back to the scene to acquire the environment map. Since real training data for highlight extraction is very limited, we introduce an unsupervised scheme for finetuning the network on real images, based on the consistent diffuse chromaticity of a given face seen in multiple real images. In tracing the estimated highlights to the environment, we reduce the blurring effect of skin reflectance on reflected light through a deconvolution determined by prior knowledge on face material properties. Comparisons to previous techniques for highlight extraction and illumination estimation show the state-of-the-art performance of this approach on a variety of indoor and outdoor scenes.

Keywords

Illumination estimation Unsupervised learning 

Notes

Acknowledgments

This work is supported by Canada NSERC Discovery Grant 611664. Renjiao Yi is supported by scholarship from China Scholarship Council.

Supplementary material

474192_1_En_20_MOESM1_ESM.pdf (12.7 mb)
Supplementary material 1 (pdf 13050 KB)

References

  1. 1.
    Barron, J.T., Malik, J.: Shape, illumination, and reflectance from shading. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 37(8), 1670–1687 (2015)CrossRefGoogle Scholar
  2. 2.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: ACM SIGGRAPH, pp. 187–194. ACM (1999)Google Scholar
  3. 3.
    Calian, D.A., Lalonde, J.F., Gotardo, P., Simon, T., Matthews, I., Mitchell, K.: From faces to outdoor light probes. In: Computer Graphics Forum, vol. 37, pp. 51–61. Wiley Online Library (2018)Google Scholar
  4. 4.
    Engel, J., Koltun, V., Cremers, D.: Direct sparse odometry. IEEE Trans. Pattern Anal. Mach. Intell. 40, 611–625 (2017)CrossRefGoogle Scholar
  5. 5.
    Gardner, M.A., et al.: Learning to predict indoor illumination from a single image. ACM Trans. Graph. (SIGGRAPH Asia) 9(4) (2017)Google Scholar
  6. 6.
    Garrido, P., Valgaerts, L., Wu, C., Theobalt, C.: Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graph. (TOG) 32(6), 158:1–158:10 (2013)CrossRefGoogle Scholar
  7. 7.
    Guo, Y., Zhang, L., Hu, Y., He, X., Gao, J.: MS-Celeb-1M: a dataset and benchmark for large-scale face recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 87–102. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46487-9_6CrossRefGoogle Scholar
  8. 8.
    Hassner, T., Harel, S., Paz, E., Enbar, R.: Effective face frontalization in unconstrained images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4295–4304 (2015)Google Scholar
  9. 9.
    Hold-Geoffroy, Y., Sunkavalli, K., Hadap, S., Gambaretto, E., Lalonde, J.F.: Deep outdoor illumination estimation. In: IEEE International Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  10. 10.
    Kemelmacher-Shlizerman, I., Basri, R.: 3D face reconstruction from a single image using a single reference face shape. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 33(2), 394–405 (2011)CrossRefGoogle Scholar
  11. 11.
    Kim, H., Jin, H., Hadap, S., Kweon, I.: Specular reflection separation using dark channel prior. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1460–1467 (2013)Google Scholar
  12. 12.
    Knorr, S.B., Kurz, D.: Real-time illumination estimation from faces for coherent rendering. In: 2014 IEEE International Symposium on Mixed and Augmented Reality, ISMAR, pp. 113–122. IEEE (2014)Google Scholar
  13. 13.
    Lalonde, J.-F., Narasimhan, S.G., Efros, A.A.: What does the sky tell us about the camera? In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5305, pp. 354–367. Springer, Heidelberg (2008).  https://doi.org/10.1007/978-3-540-88693-8_26CrossRefGoogle Scholar
  14. 14.
    Lalonde, J.F., Narasimhan, S.G., Efros, A.A.: What do the sun and the sky tell us about the camera? Int. J. Comput. Vis. 88(1), 24–51 (2010)CrossRefGoogle Scholar
  15. 15.
    Li, C., Lin, S., Zhou, K., Ikeuchi, K.: Radiometric calibration from faces in images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3117–3126 (2017)Google Scholar
  16. 16.
    Li, C., Lin, S., Zhou, K., Ikeuchi, K.: Specular highlight removal in facial images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3107–3116 (2017)Google Scholar
  17. 17.
    Li, C., Zhou, K., Lin, S.: Intrinsic face image decomposition with human face priors. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 218–233. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10602-1_15CrossRefGoogle Scholar
  18. 18.
    Li, Y., Lin, S., Lu, H., Shum, H.Y.: Multiple-cue illumination estimation in textured scenes. In: Proceedings of International Conference on Computer Vision, pp. 1366–1373 (2003)Google Scholar
  19. 19.
    Lombardi, S., Nishino, K.: Reflectance and illumination recovery in the wild. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 38(1), 129–141 (2016)CrossRefGoogle Scholar
  20. 20.
    Lopez-Moreno, J., Hadap, S., Reinhard, E., Gutierrez, D.: Compositing images through light source detection. Comput. Graph. 34(6), 698–707 (2010)CrossRefGoogle Scholar
  21. 21.
    Lucy, L.B.: An iterative technique for the rectification of observed distributions. The Astron. J. 79, 745 (1974)CrossRefGoogle Scholar
  22. 22.
    Mallick, S.P., Zickler, T., Belhumeur, P.N., Kriegman, D.J.: Specularity removal in images and videos: a PDE approach. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 550–563. Springer, Heidelberg (2006).  https://doi.org/10.1007/11744023_43CrossRefGoogle Scholar
  23. 23.
  24. 24.
    Narihira, T., Maire, M., Yu, S.X.: Direct intrinsics: learning albedo-shading decomposition by convolutional regression. In: Proceedings of the IEEE International Conference on Computer Vision, p. 2992 (2015)Google Scholar
  25. 25.
    Nishino, K., Nayar, S.K.: Eyes for relighting. In: ACM Transactions on Graphics, TOG, vol. 23, pp. 704–711. ACM (2004)Google Scholar
  26. 26.
    Okabe, T., Sato, I., Sato, Y.: Spherical harmonics vs. haar wavelets: basis for recovering illumination from cast shadows. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 50–57 (2004)Google Scholar
  27. 27.
    Panagopoulos, A., Wang, C., Samaras, D., Paragios, N.: Illumination estimation and cast shadow detection through a higher-order graphical model. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2011)Google Scholar
  28. 28.
    Papadopoulo, T., Lourakis, M.I.A.: Estimating the Jacobian of the singular value decomposition: theory and applications. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1842, pp. 554–570. Springer, Heidelberg (2000).  https://doi.org/10.1007/3-540-45054-8_36CrossRefGoogle Scholar
  29. 29.
    Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition. In: Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, AVSS 2009, pp. 296–301. IEEE (2009)Google Scholar
  30. 30.
    Pessoa, S., Moura, G., Lima, J., Teichrieb, V., Kelner, J.: Photorealistic rendering for augmented reality: a global illumination and BRDF solution. In: 2010 IEEE Virtual Reality Conference, VR, pp. 3–10. IEEE (2010)Google Scholar
  31. 31.
    Phong, B.T.: Illumination for computer generated pictures. Commun. ACM 18(6), 311–317 (1975)CrossRefGoogle Scholar
  32. 32.
    Ramamoorthi, R., Hanrahan, P.: A signal-processing framework for inverse rendering. In: ACM SIGGRAPH, pp. 117–128. ACM (2001)Google Scholar
  33. 33.
    Rematas, K., Ritschel, T., Fritz, M., Gavves, E., Tuytelaars, T.: Deep reflectance maps. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4508–4516 (2016)Google Scholar
  34. 34.
    Richardson, E., Sela, M., Or-El, R., Kimmel, R.: Learning detailed face reconstruction from a single image. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  35. 35.
    Richardson, W.H.: Bayesian-based iterative method of image restoration. JOSA 62(1), 55–59 (1972)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Sato, I., Sato, Y., Ikeuchi, K.: Acquiring a radiance distribution to superimpose virtual objects onto a real scene. IEEE Trans. Vis. Comput. Graph. (TVCG) 5, 1–12 (1999)CrossRefGoogle Scholar
  37. 37.
    Sato, I., Sato, Y., Ikeuchi, K.: Illumination from shadows. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 25, 290–300 (2003)CrossRefGoogle Scholar
  38. 38.
    Sela, M., Richardson, E., Kimmel, R.: Unrestricted facial geometry reconstruction using image-to-image translation. In: Proceedings of International Conference on Computer Vision (2017)Google Scholar
  39. 39.
    Shafer, S.: Using color to separate reflection components. Color Res. Appl. 10(4), 210–218 (1985)CrossRefGoogle Scholar
  40. 40.
    Shen, H.L., Zheng, Z.H.: Real-time highlight removal using intensity ratio. Appl. Opt. 52(19), 4483–4493 (2013)CrossRefGoogle Scholar
  41. 41.
    Shi, J., Dong, Y., Su, H., Yu, S.X.: Learning non-Lambertian object intrinsics across ShapeNet categories. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  42. 42.
    Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., Webb, R.: Learning from simulated and unsupervised images through adversarial training. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  43. 43.
    Snap Inc.: Snapchat. https://www.snapchat.com/
  44. 44.
    Tan, P., Lin, S., Quan, L.: Separation of highlight reflections on textured surfaces. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 1855–1860 (2006)Google Scholar
  45. 45.
    Tan, P., Lin, S., Quan, L., Shum, H.Y.: Highlight removal by illumination-constrained inpainting. In: Proceedings of International Conference on Computer Vision (2003)Google Scholar
  46. 46.
    Tan, R., Ikeuchi, K.: Reflection components decomposition of textured surfaces using linear basis functions. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 125–131 (2005)Google Scholar
  47. 47.
    Tan, R.T., Nishino, K., Ikeuchi, K.: Separating reflection components based on chromaticity and noise analysis. IEEE Trans. Pattern Anal. Mach. Intell. 26(10), 1373–1379 (2004)CrossRefGoogle Scholar
  48. 48.
    Wang, Y., Samaras, D.: Estimation of multiple illuminants from a single image of arbitrary known geometry. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2352, pp. 272–288. Springer, Heidelberg (2002).  https://doi.org/10.1007/3-540-47977-5_18CrossRefGoogle Scholar
  49. 49.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  50. 50.
    Weyrich, T., et al.: Analysis of human faces using a measurement-based skin reflectance model. In: ACM Transactions on Graphics, TOG, vol. 25, pp. 1013–1024. ACM (2006)Google Scholar
  51. 51.
    Yang, F., Wang, J., Shechtman, E., Bourdev, L., Metaxas, D.: Expression flow for 3D-aware face component transfer. In: ACM Transactions on Graphics, TOG, vol. 30, p. 60. ACM (2011)CrossRefGoogle Scholar
  52. 52.
    Yang, Q., Wang, S., Ahuja, N.: Real-time specular highlight removal using bilateral filtering. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 87–100. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-15561-1_7CrossRefGoogle Scholar
  53. 53.
    Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition CVPR, pp. 2879–2886. IEEE (2012)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Renjiao Yi
    • 1
    • 2
  • Chenyang Zhu
    • 1
    • 2
  • Ping Tan
    • 1
  • Stephen Lin
    • 3
  1. 1.Simon Fraser UniversityBurnabyCanada
  2. 2.National University of Defense TechnologyChangshaChina
  3. 3.Microsoft ResearchBeijingChina

Personalised recommendations