Advertisement

Intrinsic Textures for Relightable Free-Viewpoint Video

  • James Imber
  • Jean-Yves Guillemaut
  • Adrian Hilton
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8690)

Abstract

This paper presents an approach to estimate the intrinsic texture properties (albedo, shading, normal) of scenes from multiple view acquisition under unknown illumination conditions. We introduce the concept of intrinsic textures, which are pixel-resolution surface textures representing the intrinsic appearance parameters of a scene. Unlike previous video relighting methods, the approach does not assume regions of uniform albedo, which makes it applicable to richly textured scenes. We show that intrinsic image methods can be used to refine an initial, low-frequency shading estimate based on a global lighting reconstruction from an original texture and coarse scene geometry in order to resolve the inherent global ambiguity in shading. The method is applied to relighting of free-viewpoint rendering from multiple view video capture. This demonstrates relighting with reproduction of fine surface detail. Quantitative evaluation on synthetic models with textured appearance shows accurate estimation of intrinsic surface reflectance properties.

Keywords

Free-Viewpoint Video Rendering Image-Based Rendering Relighting Intrinsic Images 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Supplementary material

978-3-319-10605-2_26_MOESM1_ESM.mp4 (15.7 mb)
Electronic Supplementary Material (MP4 16,072 KB)

References

  1. 1.
    Zitnick, C., Kang, S., Uyttendaele, M.: High-quality video view interpolation using a layered representation. ACM Transactions on Graphics 1(212), 600–608 (2004)CrossRefGoogle Scholar
  2. 2.
    Matusik, W., Buehler, C., Raskar, R., Gortler, S.J., McMillan, L.: Image-based visual hulls. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 369–374 (2000)Google Scholar
  3. 3.
    Vedula, S., Baker, S., Kanade, T.: Image-based spatio-temporal modeling and view interpolation of dynamic events. ACM Transactions on Graphics 24(2), 240–261 (2005)CrossRefGoogle Scholar
  4. 4.
    Kanade, T., Rander, P., Narayanan, P.: Virtualized reality: constructing virtual worlds from real scenes. IEEE Multimedia 4(1), 34–47 (1997)CrossRefGoogle Scholar
  5. 5.
    Guillemaut, J.Y., Hilton, A.: Joint Multi-Layer Segmentation and Reconstruction for Free-Viewpoint Video Applications. International Journal of Computer Vision 93(1), 73–100 (2010)CrossRefGoogle Scholar
  6. 6.
    Pan, C.H., Huang, S.C., Chang, Y.L., Lian, C.J., Chen, L.G.: Real-time free viewpoint rendering system for face-to-face video conference. In: Proceedings of IEEE International Conference on Consumer Electronics, pp. 1–2 (2008)Google Scholar
  7. 7.
    Debevec, P., Taylor, C., Malik, J.: Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In: Proceedings of the 23rd Annual conference on Computer Graphics and Interactive Techniques, pp. 1–10 (1996)Google Scholar
  8. 8.
    Starck, J., Hilton, A.: Surface capture for performance-based animation. IEEE Computer Graphics and Applications 27(3), 21–31 (2007)CrossRefGoogle Scholar
  9. 9.
    Li, G., Wu, C., Stoll, C., Liu, Y., Varanasi, K., Dai, Q., Theobalt, C.: Capturing Relightable Human Performances under General Uncontrolled Illumination. Computer Graphics Forum 32(2), 275–284 (2013)CrossRefGoogle Scholar
  10. 10.
    Debevec, P., Taylor, C., Malik, J.: Image-based modeling and rendering of architecture with interactive photogrammetry and view-dependent texture mapping. In: Proceedings of the 1998 IEEE International Symposium on Circuits and Systems, pp. 14–17 (1998)Google Scholar
  11. 11.
    Starck, J., Kilner, J., Hilton, A.: A Free-Viewpoint Video Renderer. Journal of Graphics, GPU, and Game Tools 14(3), 57–72 (2009)CrossRefGoogle Scholar
  12. 12.
    Land, E.H., McCann, J.J.: Lightness and retinex theory. Journal of the Optical Society of America 61(1), 1–11 (1971)CrossRefGoogle Scholar
  13. 13.
    Barrow, H., Tenenbaum, J.: Recovering intrinsic scene characteristics from images. Computer Vision Systems, 3–26 (1978)Google Scholar
  14. 14.
    Tappen, M.F., Freeman, W.T., Adelson, E.H.: Recovering intrinsic images from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(9), 1459–1472 (2005)CrossRefGoogle Scholar
  15. 15.
    Shen, L., Yeo, C.: Intrinsic images decomposition using a local and global sparse representation of reflectance. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2011)Google Scholar
  16. 16.
    Bousseau, A., Paris, S., Durand, F.: User-assisted intrinsic images. ACM Transactions on Graphics 28(5), 1 (2009)CrossRefGoogle Scholar
  17. 17.
    Barron, J., Malik, J.: Shape, albedo, and illumination from a single image of an unknown object. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 334–341 (2012)Google Scholar
  18. 18.
    Debevec, P., Hawkins, T., Tchou, C.: Acquiring the reflectance field of a human face. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 145–156 (2000)Google Scholar
  19. 19.
    Ahmed, N., Theobalt, C., Seidel, H.-P.: Spatio-temporal Reflectance Sharing for Relightable 3D Video. In: Gagalowicz, A., Philips, W. (eds.) MIRAGE 2007. LNCS, vol. 4418, pp. 47–58. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  20. 20.
    Matusik, W., Pfister, H., Ngan, A., Beardsley, P., Ziegler, R., McMillan, L.: Image-based 3D photography using opacity hulls. In: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, p. 427 (2002)Google Scholar
  21. 21.
    Einarsson, P., Chabert, C., Jones, A., Ma, W.C., Lamond, B., Hawkins, T., Bolas, M., Sylwan, S., Debevec, P.: Relighting human locomotion with flowed reflectance fields. ACM Transactions on Graphics 2006 Sketches (2006)Google Scholar
  22. 22.
    Lensch, H.P.A., Kautz, J., Goesele, M., Heidrich, W., Seidel, H.P.: Image-based reconstruction of spatial appearance and geometric detail. ACM Transactions on Graphics 22(2), 234–257 (2003)CrossRefGoogle Scholar
  23. 23.
    Ramamoorthi, R., Hanrahan, P.: On the relationship between radiance and irradiance: determining the illumination from images of a convex Lambertian object. Journal of the Optical Society of America A 18(10), 2448 (2001)CrossRefMathSciNetGoogle Scholar
  24. 24.
    Wu, C., Varanasi, K., Liu, Y., Seidel, H.P., Theobalt, C.: Shading-based dynamic shape refinement from multi-view video under general illumination. In: 2011 International Conference on Computer Vision, pp. 1108–1115 (November 2011)Google Scholar
  25. 25.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. International Journal of Computer Vision 59(2), 167–181 (2004)CrossRefGoogle Scholar
  26. 26.
    Shen, J., Yang, X., Jia, Y., Li, X.: Intrinsic images using optimization. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2011)Google Scholar
  27. 27.
    Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range images. ACM Transactions on Graphics (TOG) (2002)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • James Imber
    • 1
  • Jean-Yves Guillemaut
    • 2
  • Adrian Hilton
    • 2
  1. 1.Imagination Technologies Ltd.Kings LangleyUK
  2. 2.Centre for Vision, Speech and Signal ProcessingUniversity of SurreyGuildfordUK

Personalised recommendations