Abstract
This paper presents an approach to estimate the intrinsic texture properties (albedo, shading, normal) of scenes from multiple view acquisition under unknown illumination conditions. We introduce the concept of intrinsic textures, which are pixel-resolution surface textures representing the intrinsic appearance parameters of a scene. Unlike previous video relighting methods, the approach does not assume regions of uniform albedo, which makes it applicable to richly textured scenes. We show that intrinsic image methods can be used to refine an initial, low-frequency shading estimate based on a global lighting reconstruction from an original texture and coarse scene geometry in order to resolve the inherent global ambiguity in shading. The method is applied to relighting of free-viewpoint rendering from multiple view video capture. This demonstrates relighting with reproduction of fine surface detail. Quantitative evaluation on synthetic models with textured appearance shows accurate estimation of intrinsic surface reflectance properties.
Chapter PDF
References
Zitnick, C., Kang, S., Uyttendaele, M.: High-quality video view interpolation using a layered representation. ACM Transactions on Graphics 1(212), 600–608 (2004)
Matusik, W., Buehler, C., Raskar, R., Gortler, S.J., McMillan, L.: Image-based visual hulls. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 369–374 (2000)
Vedula, S., Baker, S., Kanade, T.: Image-based spatio-temporal modeling and view interpolation of dynamic events. ACM Transactions on Graphics 24(2), 240–261 (2005)
Kanade, T., Rander, P., Narayanan, P.: Virtualized reality: constructing virtual worlds from real scenes. IEEE Multimedia 4(1), 34–47 (1997)
Guillemaut, J.Y., Hilton, A.: Joint Multi-Layer Segmentation and Reconstruction for Free-Viewpoint Video Applications. International Journal of Computer Vision 93(1), 73–100 (2010)
Pan, C.H., Huang, S.C., Chang, Y.L., Lian, C.J., Chen, L.G.: Real-time free viewpoint rendering system for face-to-face video conference. In: Proceedings of IEEE International Conference on Consumer Electronics, pp. 1–2 (2008)
Debevec, P., Taylor, C., Malik, J.: Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In: Proceedings of the 23rd Annual conference on Computer Graphics and Interactive Techniques, pp. 1–10 (1996)
Starck, J., Hilton, A.: Surface capture for performance-based animation. IEEE Computer Graphics and Applications 27(3), 21–31 (2007)
Li, G., Wu, C., Stoll, C., Liu, Y., Varanasi, K., Dai, Q., Theobalt, C.: Capturing Relightable Human Performances under General Uncontrolled Illumination. Computer Graphics Forum 32(2), 275–284 (2013)
Debevec, P., Taylor, C., Malik, J.: Image-based modeling and rendering of architecture with interactive photogrammetry and view-dependent texture mapping. In: Proceedings of the 1998 IEEE International Symposium on Circuits and Systems, pp. 14–17 (1998)
Starck, J., Kilner, J., Hilton, A.: A Free-Viewpoint Video Renderer. Journal of Graphics, GPU, and Game Tools 14(3), 57–72 (2009)
Land, E.H., McCann, J.J.: Lightness and retinex theory. Journal of the Optical Society of America 61(1), 1–11 (1971)
Barrow, H., Tenenbaum, J.: Recovering intrinsic scene characteristics from images. Computer Vision Systems, 3–26 (1978)
Tappen, M.F., Freeman, W.T., Adelson, E.H.: Recovering intrinsic images from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(9), 1459–1472 (2005)
Shen, L., Yeo, C.: Intrinsic images decomposition using a local and global sparse representation of reflectance. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2011)
Bousseau, A., Paris, S., Durand, F.: User-assisted intrinsic images. ACM Transactions on Graphics 28(5), 1 (2009)
Barron, J., Malik, J.: Shape, albedo, and illumination from a single image of an unknown object. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 334–341 (2012)
Debevec, P., Hawkins, T., Tchou, C.: Acquiring the reflectance field of a human face. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 145–156 (2000)
Ahmed, N., Theobalt, C., Seidel, H.-P.: Spatio-temporal Reflectance Sharing for Relightable 3D Video. In: Gagalowicz, A., Philips, W. (eds.) MIRAGE 2007. LNCS, vol. 4418, pp. 47–58. Springer, Heidelberg (2007)
Matusik, W., Pfister, H., Ngan, A., Beardsley, P., Ziegler, R., McMillan, L.: Image-based 3D photography using opacity hulls. In: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, p. 427 (2002)
Einarsson, P., Chabert, C., Jones, A., Ma, W.C., Lamond, B., Hawkins, T., Bolas, M., Sylwan, S., Debevec, P.: Relighting human locomotion with flowed reflectance fields. ACM Transactions on Graphics 2006 Sketches (2006)
Lensch, H.P.A., Kautz, J., Goesele, M., Heidrich, W., Seidel, H.P.: Image-based reconstruction of spatial appearance and geometric detail. ACM Transactions on Graphics 22(2), 234–257 (2003)
Ramamoorthi, R., Hanrahan, P.: On the relationship between radiance and irradiance: determining the illumination from images of a convex Lambertian object. Journal of the Optical Society of America A 18(10), 2448 (2001)
Wu, C., Varanasi, K., Liu, Y., Seidel, H.P., Theobalt, C.: Shading-based dynamic shape refinement from multi-view video under general illumination. In: 2011 International Conference on Computer Vision, pp. 1108–1115 (November 2011)
Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. International Journal of Computer Vision 59(2), 167–181 (2004)
Shen, J., Yang, X., Jia, Y., Li, X.: Intrinsic images using optimization. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2011)
Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range images. ACM Transactions on Graphics (TOG) (2002)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
1 Electronic Supplementary Material
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Imber, J., Guillemaut, JY., Hilton, A. (2014). Intrinsic Textures for Relightable Free-Viewpoint Video. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds) Computer Vision – ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol 8690. Springer, Cham. https://doi.org/10.1007/978-3-319-10605-2_26
Download citation
DOI: https://doi.org/10.1007/978-3-319-10605-2_26
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-10604-5
Online ISBN: 978-3-319-10605-2
eBook Packages: Computer ScienceComputer Science (R0)