Advertisement

Shadow and Specularity Priors for Intrinsic Light Field Decomposition

  • Anna AlperovichEmail author
  • Ole Johannsen
  • Michael Strecke
  • Bastian Goldluecke
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10746)

Abstract

In this work, we focus on the problem of intrinsic scene decomposition in light fields. Our main contribution is a novel prior to cope with cast shadows and inter-reflections. In contrast to other approaches which model inter-reflection based only on geometry, we model indirect shading by combining geometric and color information. We compute a shadow confidence measure for the light field and use it in the regularization constraints. Another contribution is an improved specularity estimation by using color information from sub-aperture views. The new priors are embedded in a recent framework to decompose the input light field into albedo, shading, and specularity. We arrive at a variational model where we regularize albedo and the two shading components on epipolar plane images, encouraging them to be consistent across all sub-aperture views. Our method is evaluated on ground truth synthetic datasets and real world light fields. We outperform both state-of-the art approaches for RGB+D images and recent methods proposed for light fields.

Notes

Acknowledgments

This work was supported by the ERC Starting Grant “Light Field Imaging and Analysis” (LIA 336978, FP7-2014).

Supplementary material

466071_1_En_26_MOESM1_ESM.zip (84.1 mb)
Supplementary material 1 (zip 86121 KB)

References

  1. 1.
    Alperovich, A., Goldluecke, B.: A variational model for intrinsic light field decomposition. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10113, pp. 66–82. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-54187-7_5 CrossRefGoogle Scholar
  2. 2.
    Barron, J.T., Malik, J.: Intrinsic scene properties from a single RGB-D image. IEEE TPAMI 38(4), 690–703 (2015)CrossRefGoogle Scholar
  3. 3.
    Barrow, H.G., Tenenbaum, J.M.: Recovering intrinsic scene characteristics from images. Comput. Vis. Syst. 23(1), 3–26 (1978)Google Scholar
  4. 4.
    Bell, S., Bala, K., Snavely, N.: Intrinsic images in the wild. ACM Trans. Graph. (SIGGRAPH) 33(4), 159:1–159:12 (2014)CrossRefGoogle Scholar
  5. 5.
    Bolles, R., Baker, H., Marimont, D.: Epipolar-plane image analysis: an approach to determining structure from motion. Int. J. Comput. Vis. 1(1), 7–55 (1987)CrossRefGoogle Scholar
  6. 6.
    Bonneel, N., Sunkavalli, K., Tompkin, J., Sun, D., Paris, S., Pfister, H.: Interactive intrinsic video editing. ACM Trans. Graph. 33(6), 197 (2014)CrossRefGoogle Scholar
  7. 7.
    Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM J. Imaging Sci. 3(3), 492–526 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 611–625. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-33783-3_44 CrossRefGoogle Scholar
  9. 9.
    Chen, Q., Koltun, V.: A simple model for intrinsic image decomposition with depth cues. In: Proceedings of the ICCV (2013)Google Scholar
  10. 10.
    Finlayson, G., Schaefer, G.: Solving for colour constancy using a constrained dichromatic reflection model. Int. J. Comput. Vis. 42(3), 127–144 (2001)CrossRefzbMATHGoogle Scholar
  11. 11.
    Finlayson, G.D., Drew, M.S., Lu, C.: Intrinsic images by entropy minimization. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3023, pp. 582–595. Springer, Heidelberg (2004).  https://doi.org/10.1007/978-3-540-24672-5_46 CrossRefGoogle Scholar
  12. 12.
    Finlayson, G.D., Hordley, S.D., Lu, C., Drew, M.S.: On the removal of shadows from images. IEEE Trans. Pattern Anal. Mach. Intell. 28(1), 59–68 (2006)CrossRefGoogle Scholar
  13. 13.
    Garces, E., Echevarria, J.I., Zhang, W., Wu, H., Zhou, K., Gutierrez, D.: Intrinsic light field images. Comput. Graph. Forum 36, 589–599 (2017)CrossRefGoogle Scholar
  14. 14.
    Garces, E., Munoz, A., Lopez-Moreno, J., Gutierrez, D.: Intrinsic images by clustering. Comput. Graph. Forum (Proc. EGSR 2012) 31(4), 1415–1424 (2012)CrossRefGoogle Scholar
  15. 15.
    Gehler, P.V., Rother, C., Kiefel, M., Zhang, L., Schölkopf, B.: Recovering intrinsic images with a global sparsity prior on reflectance. In: Proceedings of the NIPS (2011)Google Scholar
  16. 16.
    Goldluecke, B., Wanner, S.: The variational structure of disparity and regularization of 4D light fields. In: Proceedings of the CVPR (2013)Google Scholar
  17. 17.
    Gortler, S., Grzeszczuk, R., Szeliski, R., Cohen, M.: The Lumigraph. In: Proceedings of the SIGGRAPH, pp. 43–54 (1996)Google Scholar
  18. 18.
    Grosse, R., Johnson, M.K., Adelson, E.H., Freeman, W.T.: Ground truth dataset and baseline evaluations for intrinsic image algorithm. In: Proceedings of the ICCV (2009)Google Scholar
  19. 19.
    Honauer, K., Johannsen, O., Kondermann, D., Goldluecke, B.: A dataset and evaluation methodology for depth estimation on 4D light fields. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10113, pp. 19–34. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-54187-7_2 CrossRefGoogle Scholar
  20. 20.
    Jeon, J., Cho, S., Tong, X., Lee, S.: Intrinsic image decomposition using structure-texture separation and surface normals. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 218–233. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10584-0_15 Google Scholar
  21. 21.
    Kim, S., Park, K., Sohn, K., Lin, S.: Unified depth prediction and intrinsic image decomposition from a single image via joint convolutional neural fields. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 143–159. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46484-8_9 CrossRefGoogle Scholar
  22. 22.
    Lalonde, J.-F., Efros, A.A., Narasimhan, S.G.: Detecting ground shadows in outdoor consumer photographs. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6312, pp. 322–335. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-15552-9_24 CrossRefGoogle Scholar
  23. 23.
    Land, E.H., McCann, J.J.: Lightness and retinex theory. J. Opt. Soc. Am. 61(1), 1–11 (1978)CrossRefGoogle Scholar
  24. 24.
    Lee, K.J., Zhao, Q., Tong, X., Gong, M., Izadi, S., Lee, S.U., Tan, P., Lin, S.: Estimation of intrinsic image sequences from image+depth video. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 327–340. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-33783-3_24 CrossRefGoogle Scholar
  25. 25.
    Lee, T.S.: Top-down influence in early visual processing: a Bayesian perspective. Physiol. Behav. 77, 645–650 (2002)CrossRefGoogle Scholar
  26. 26.
    Levoy, M.: Light fields and computational imaging. Computer 39(8), 46–55 (2006)CrossRefGoogle Scholar
  27. 27.
    Olsson, C., Boykov, Y.: Curvature-based regularization for surface approximation. In: Proceedings of the CVPR (2012)Google Scholar
  28. 28.
    Panagopoulos, A., Wang, C., Samaras, D., Paragios, N.: Estimating shadows with the bright channel cue. In: Kutulakos, K.N. (ed.) ECCV 2010. LNCS, vol. 6554, pp. 1–12. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-35740-4_1 CrossRefGoogle Scholar
  29. 29.
    Shafer, S.: Using color to separate reflection components. Color Res. Appl. 10(4), 210–218 (1985)CrossRefGoogle Scholar
  30. 30.
    Shen, L., Tan, P., Lin, S.: Intrinsic image decomposition with non-local texture cues. In: Proceedings of the CVPR (2008)Google Scholar
  31. 31.
    Strecke, M., Alperovich, A., Goldluecke, B.: Accurate depth and normal maps from occlusion-aware focal stack symmetry. In: Proceedings of the CVPR (2017)Google Scholar
  32. 32.
    Sulc, A., Alperovich, A., Marniok, N., Goldluecke, B.: Reflection separation in light fields based on sparse coding and specular flow. In: Proceedings of the VMV (2016)Google Scholar
  33. 33.
    Tan, R.T., Ikeuchi, K.: Separating reflection components of textured surfaces using a single image. In: Proceedings of the ICCV (2003)Google Scholar
  34. 34.
    Tan, R.T., Nishino, K., Ikeuchi, K.: Color constancy through inverse-intensity chromaticity space. J. Opt. Soc. Am. A: 21(3), 321–334 (2004)CrossRefGoogle Scholar
  35. 35.
    Tao, M., Su, J.C., Wang, T.C., Malik, J., Ramamoorthi, R.: Depth estimation and specular removal for glossy surfaces using point and line consistency with light-field cameras. IEEE TPAMI 38(6), 1155–1169 (2015)CrossRefGoogle Scholar
  36. 36.
    Tappen, M.F., Freeman, W.T., Adelson, E.H.: Recovering intrinsic images from a single image. IEEE TPAMI 27(9), 1459–1472 (2005)CrossRefGoogle Scholar
  37. 37.
    Triantafyllidis, G., Dimitriou, M., Kounalakis, T., Vidakis, N.: Detection and classification of multiple objects using an RGB-D sensor and linear spatial pyramid matching. ELCVIA Electron. Lett. Comput. Vis. Image Anal. 12(2), 78–87 (2013)CrossRefGoogle Scholar
  38. 38.
    Wang, H., Xu, C., Wang, X., Zhang, Y., Peng, B.: Light field imaging based accurate image specular highlight removal. PLoS ONE 11(6), e0156173 (2016)CrossRefGoogle Scholar
  39. 39.
    Weiss, Y.: Deriving intrinsic images from image sequences. In: Proceedings of the ICCV (2001)Google Scholar
  40. 40.
    Xiao, Y., Tsougenis, E., Tang, C.K.: Shadow removal from single RGB-D images. In: Proceedings of the CVPR (2014)Google Scholar
  41. 41.
    Yang, K., Gao, S., Li, Y.: Efficient illuminant estimation for color constancy using grey pixels. In: Proceedings of the CVPR (2015)Google Scholar
  42. 42.
    Ye, G., Garces, E., Liu, Y., Dai, Q., Gutierrez, D.: Intrinsic video and applications. ACM Trans. Graph. (SIGGRAPH 2014) 33(4), 80 (2014)Google Scholar
  43. 43.
    Zeisl, B., Zach, C., Pollefeys, M.: Variational regularization and fusion of surface normal maps. In: 3DV (2014)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Anna Alperovich
    • 1
    Email author
  • Ole Johannsen
    • 1
  • Michael Strecke
    • 1
  • Bastian Goldluecke
    • 1
  1. 1.University of KonstanzKonstanzGermany

Personalised recommendations