Skip to main content
SpringerLink
Log in

3D Shape Reconstruction from Multi-view Video Data

  • Chapter
3D Video and Its Applications

  • 1169 Accesses

Abstract

3D shape reconstruction has been a major research topic in computer vision and a variety of different algorithms have been developed. This chapter first defines a computational model for 3D video production, followed by an overview of Shape from X methods to investigate practically useful visual cues for 3D shape reconstruction from multi-view video data. Then, we categorize 3D shape reconstruction methods for 3D video production into three types and analyze their computational characteristics: (1) frame-wise 3D shape reconstruction, (2) simultaneous 3D shape and motion estimation, and (3) 3D shape sequence estimation. The computational characteristic analysis leads us to the three essential design factors for 3D shape reconstruction algorithms: (1) photo-consistency, (2) visibility evaluation, and (3) shape representation with corresponding computational model for optimization. Based on these design factors, we implemented several practical algorithms for frame-wise 3D shape reconstruction and simultaneous 3D shape and motion reconstruction. Their performance is evaluated quantitatively with synthesized data as well as qualitatively with real world multi-view video data of MAIKO dance and Yoga performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The projection in computer tomography is characterized as an integral projection where rays go through the interior area of an object and their degrees of attenuation are observed in projected data. In computer vision, most of objects in the real world are non-transparent and reflect light rays at their surfaces. Thus the projection in computer vision implies ray blocking or shadowing.

  2. 2.

    The reasons for this will be discussed later in this chapter.

  3. 3.

    An earlier version can be found in the voxel coloring study by Seitz and Dyer in 1997 [44].

References

  1. Alexa, M., Behr, J., Cohen-Or, D., Fleishman, S., Levin, D., Silva, C.T.: Point set surfaces. In: The Conference on Visualization, pp. 21–28 (2001)

    Google Scholar 

  2. Barr, A.H.: Rigid Physically Based Superquadrics, pp. 137–159. Academic Press, San Diego (1992)

    Google Scholar 

  3. Baumgart, B.G.: Geometric modeling for computer vision. Technical Report AIM-249, Artificial Intelligence Laboratory, Stanford University (1974)

    Google Scholar 

  4. Baumgart, B.G.: A polyhedron representation for computer vision. In: Proceedings of the National Computer Conference and Exposition, AFIPS’75, pp. 589–596 (1975)

    Google Scholar 

  5. Campbell, N., Vogiatzis, G., Hernández, C., Cipolla, R.: Automatic 3D object segmentation in multiple views using volumetric graph-cuts. Image Vis. Comput. 28(1), 14–25 (2010)

    Article  Google Scholar 

  6. Virtualizing Engine. Private communication with Profs. Takeo Kanade and Yaser Sheikh, Robotics Institute, Carnegie Mellon University, PA (2011)

    Google Scholar 

  7. Cremers, D., Kolev, K.: Multiview stereo and silhouette consistency via convex functionals over convex domains. IEEE Trans. Pattern Anal. Mach. Intell., 1161–1174 (2010)

    Google Scholar 

  8. Felzenszwalb, P., Huttenlocher, D.: Efficient belief propagation for early vision. Int. J. Comput. Vis. 70, 41–54 (2006)

    Article  Google Scholar 

  9. Franco, J.-S., Boyer, E.: Efficient polyhedral modeling from silhouettes. IEEE Trans. Pattern Anal. Mach. Intell. 31(3), 414–427 (2009)

    Article  Google Scholar 

  10. Fua, P., Leclerc, Y.G.: Using 3-dimensional meshes to combine image-based and geometry-based constraints. In: Proc. of European Conference on Computer Vision, pp. 281–291 (1994)

    Google Scholar 

  11. Furukawa, Y., Ponce, J.: Accurate, dense, and robust multi-view stereopsis. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)

    Google Scholar 

  12. Goldlüecke, B., Magnor, M.: Space-time isosurface evolution for temporally coherent 3D reconstruction. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 350–355 (2004)

    Google Scholar 

  13. Habbecke, M., Kobbelt, L.: A surface-growing approach to multi-view stereo reconstruction. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)

    Google Scholar 

  14. Hernandez, C., Vogiatzis, G., Cipolla, R.: Probabilistic visibility for multi-view stereo. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)

    Google Scholar 

  15. Hernandez Esteban, C., Vogiatzis, G., Cipolla, R.: Multiview photometric stereo. IEEE Trans. Pattern Anal. Mach. Intell. 30, 548–554 (2008)

    Article  Google Scholar 

  16. Horn, B.K.P., Brooks, M.J.: Shape from Shading. MIT Press, Cambridge (1989)

    Google Scholar 

  17. Hornung, A., Kobbelt, L.: Robust and efficient photo-consistency estimation for volumetric 3d reconstruction. In: Proc. of ECCV, pp. 179–190 (2006)

    Google Scholar 

  18. Ikeuchi, K.: Shape from regular patterns. Artif. Intell. 22(1), 49–75 (1984)

    Article  Google Scholar 

  19. Ikeuchi, K., Oishi, T., Takamatsu, J., Sagawa, R., Nakazawa, A., Kurazume, R., Nishino, K., Kamakura, M., Okamoto, Y.: The great Buddha project: digitally archiving, restoring, and analyzing cultural heritage objects. Int. J. Comput. Vis. 75, 189–208 (2007)

    Article  Google Scholar 

  20. Ishikawa, H.: Higher-order clique reduction in binary graph cut. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2993–3000 (2009)

    Google Scholar 

  21. Kanade, T., Rander, P., Narayanan, P.J.: Virtualized reality: constructing virtual worlds from real scenes. In: IEEE Multimedia, pp. 34–47 (1997)

    Google Scholar 

  22. Kang, S.B., Webb, J.A., Zitnick, C.L., Kanade, T.: A multibaseline stereo system with active illumination and real-time image acquisition. In: Proc. of International Conference on Computer Vision, pp. 88–93 (1995)

    Google Scholar 

  23. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: active contour models. Int. J. Comput. Vis. 1(4), 321–331 (1988)

    Article  Google Scholar 

  24. Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Symposium on Geometry Processing, pp. 61–70 (2006)

    Google Scholar 

  25. Kolmogorov, V., Zabih, R.: What energy functions can be minimizedvia graph cuts? IEEE Trans. Pattern Anal. Mach. Intell. 26, 147–159 (2004)

    Article  Google Scholar 

  26. Kutulakos, K.N., Seitz, S.M.: A theory of shape by space carving. In: Proc. of International Conference on Computer Vision, pp. 307–314 (1999)

    Chapter  Google Scholar 

  27. Laurentini, A.: The visual hull concept for silhouette-based image understanding. IEEE Trans. Pattern Anal. Mach. Intell. 16(2), 150–162 (1994)

    Article  Google Scholar 

  28. Lazebnik, S., Furukawa, Y., Ponce, J.: Projective visual hulls. Int. J. Comput. Vis. 74, 137–165 (2007)

    Article  Google Scholar 

  29. Lempitsky, V., Boykov, Y., Ivanov, D., Ivanov, D.: Oriented visibility for multiview reconstruction. In: Proc. of European Conference on Computer Vision, pp. 226–238 (2006)

    Google Scholar 

  30. Marr, D.: Vision. W. H. Freeman & Co, New York (1982)

    Google Scholar 

  31. Martin, W.N., Aggarwal, J.K.: Volumetric description of objects from multiple views. IEEE Trans. Pattern Anal. Mach. Intell. 5(2), 150–158 (1983)

    Article  Google Scholar 

  32. Matsuyama, T., Wu, X., Takai, T., Nobuhara, S.: Real-time 3D shape reconstruction, dynamic 3D mesh deformation and high fidelity visualization for 3D video. Comput. Vis. Image Underst. 96, 393–434 (2004)

    Article  Google Scholar 

  33. Miller, G., Hilton, A.: Safe hulls. In: Proc. 4th European Conference on Visual Media Production, IET (2007)

    Google Scholar 

  34. Moezzi, S., Tai, L.-C., Gerard, P.: Virtual view generation for 3D digital video. In: IEEE Multimedia, pp. 18–26 (1997)

    Google Scholar 

  35. Nagel, H.H., Enkelmann, W.: An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 8, 565–593 (1986)

    Article  Google Scholar 

  36. Nayar, S.K., Nakagawa, Y.: Shape from focus. IEEE Trans. Pattern Anal. Mach. Intell. 16, 824–831 (1994)

    Article  Google Scholar 

  37. Nayar, S.K., Watanabe, M., Noguchi, M.: Real-time focus range sensor. IEEE Trans. Pattern Anal. Mach. Intell. 18, 1186–1198 (1996)

    Article  Google Scholar 

  38. Nobuhara, S., Matsuyama, T.: Dynamic 3D shape from multi-viewpoint images using deformable mesh models. In: Proceedings of the 3rd International Symposium on Image and Signal Processing and Analysis, pp. 192–197 (2003)

    Google Scholar 

  39. Nobuhara, S., Matsuyama, T.: Heterogeneous deformation model for 3D shape and motion recovery from multi-viewpoint images. In: Proc. of International Symposium on 3D Data Processing, Visualization and Transmission, pp. 566–573 (2004)

    Chapter  Google Scholar 

  40. Nobuhara, S., Matsuyama, T.: Deformable mesh model for complex multi-object 3D motion estimation from multi-viewpoint video. In: Proc. of International Symposium on 3D Data Processing, Visualization and Transmission, pp. 264–271 (2006)

    Google Scholar 

  41. Nobuhara, S., Matsuyama, T.: A 3D deformation model for complex 3D shape and motion estimation from multi-viewpoint video. IEICE Trans. Inf. Syst. J91-D(6), 1613–1624 (2008) (in Japanese)

    Google Scholar 

  42. Nobuhara, S., Tsuda, Y., Ohama, I., Matsuyama, T.: Multi-viewpoint silhouette extraction with 3D context-aware error detection, correction, and shadow suppression. IPSJ Trans. Comput. Vis. Appl. 1, 242–259 (2009)

    Google Scholar 

  43. Okutomi, M., Kanade, T.: A multiple-baseline stereo. IEEE Trans. Pattern Anal. Mach. Intell. 15(1), 353–363 (1993)

    Article  Google Scholar 

  44. Seitz, S., Dyer, C.: Photorealistic scene reconstruction by voxel coloring. Int. J. Comput. Vis. 25(3), 151–173 (1999)

    Article  Google Scholar 

  45. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 519–528 (2006)

    Google Scholar 

  46. Sinha, S.N., Mordohai, P., Pollefeys, M.: Multi-view stereo via graph cuts on the dual of an adaptive tetrahedral mesh. In: Proc. of International Conference on Computer Vision, pp. 1–8 (2007)

    Google Scholar 

  47. Starck, J., Hilton, A.: Surface capture for performance based animation. IEEE Comput. Graph. Appl. 27(3), 21–31 (2007)

    Article  Google Scholar 

  48. Starck, J., Hilton, A., Miller, G.: Volumetric stereo with silhouette and feature constraints. In: Proc. of British Machine Vision Conference, pp. 1189–1198 (2006)

    Google Scholar 

  49. Starck, J., Maki, A., Nobuhara, S., Hilton, A., Matsuyama, T.: The multiple-camera 3-d production studio. IEEE Trans. Circuits Syst. Video Technol. 19(6), 856–869 (2009)

    Article  Google Scholar 

  50. Subbarao, M., Surya, G.: Depth from defocus: a spatial domain approach. Int. J. Comput. Vis. 13, 271–294 (1994)

    Article  Google Scholar 

  51. Szeliski, R.: Rapid octree construction from image sequences. CVGIP, Image Underst. 58(1), 23–32 (1993)

    Article  Google Scholar 

  52. Tomasi, C., Kanade, T.: Shape and motion from image streams: a factorization method—full report on the orthographic case. Int. J. Comput. Vis. 9, 137–154 (1992)

    Article  Google Scholar 

  53. Tran, S., Davis, L.: 3d surface reconstruction using graph cuts with surface constraints. In: Proc. of European Conference on Computer Vision, vol. 3952, pp. 219–231 (2006)

    Google Scholar 

  54. Tung, T., Schmitt, F.: The augmented multiresolution reeb graph approach for content-based retrieval of 3D shapes. Int. J. Shape Model. 11(1), 91–120 (2005)

    Article  Google Scholar 

  55. Vedula, S., Baker, S., Seitz, S., Kanade, T.: Shape and motion carving in 6D. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition (2000)

    Google Scholar 

  56. Vogiatzis, G., Hernandez, C., Torr, P., Cipolla, R.: Multiview stereo via volumetric graph-cuts and occlusion robust photo-consistency. IEEE Trans. Pattern Anal. Mach. Intell. 29(12), 2241–2246 (2007)

    Article  Google Scholar 

  57. Vogiatzis, G., Torr, P., Seitz, S.M., Cipolla, R.: Reconstructing relief surfaces. In: Proc. of British Machine Vision Conference, pp. 117–126 (2004)

    Google Scholar 

  58. Vogiatzis, G., Torr, P., Cipolla, R.: Multi-view stereo via volumetric graph-cuts. In: Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp. 391–398 (2005)

    Google Scholar 

  59. Wada, T., Ukida, H., Matsuyama, T.: Shape from shading with interreflections under a proximal light source: distortion-free copying of an unfolded book. Int. J. Comput. Vis. 24(2), 125–135 (1997)

    Article  Google Scholar 

  60. Wu, X.: Parallel pipeline volume intersection for real-time 3D shape reconstruction on a PC cluster. PhD thesis, Kyoto University (March 2005)

    Google Scholar 

  61. Zaharescu, A., Boyer, E., Horaud, R.: Transformesh: a topology-adaptive mesh-based approach to surface evolution. In: Proc. of Asian Conference on Computer Vision, pp. 166–175 (2007)

    Google Scholar 

  62. Zeng, G., Quan, L.: Silhouette extraction from multiple images of an unknown background. In: Proc. of Asian Conference on Computer Vision, pp. 628–633 (2004)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Graduate School of Informatics, Kyoto University, Sakyo, Kyoto, Japan

    Takashi Matsuyama, Shohei Nobuhara, Takeshi Takai & Tony Tung

Authors
  1. Takashi Matsuyama
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shohei Nobuhara
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Takeshi Takai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tony Tung
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer-Verlag London

About this chapter

Cite this chapter

Matsuyama, T., Nobuhara, S., Takai, T., Tung, T. (2012). 3D Shape Reconstruction from Multi-view Video Data. In: 3D Video and Its Applications. Springer, London. https://doi.org/10.1007/978-1-4471-4120-4_4

Download citation

  • DOI: https://doi.org/10.1007/978-1-4471-4120-4_4

  • Publisher Name: Springer, London

  • Print ISBN: 978-1-4471-4119-8

  • Online ISBN: 978-1-4471-4120-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation