Advertisement

Capture, Reconstruction, and Representation of the Visual Real World for Virtual Reality

  • Christian RichardtEmail author
  • James Tompkin
  • Gordon Wetzstein
Chapter
  • 100 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11900)

Abstract

We provide an overview of the concerns, current practice, and limitations for capturing, reconstructing, and representing the real world visually within virtual reality. Given that our goals are to capture, transmit, and depict complex real-world phenomena to humans, these challenges cover the opto-electro-mechanical, computational, informational, and perceptual fields. Practically producing a system for real-world VR capture requires navigating a complex design space and pushing the state of the art in each of these areas. As such, we outline several promising directions for future work to improve the quality and flexibility of real-world VR capture systems.

Keywords

Cameras Reconstruction Representation Virtual reality Image-based rendering Novel-view synthesis 

References

  1. 1.
    Aggarwal, R., Vohra, A., Namboodiri, A.M.: Panoramic stereo videos with a single camera. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3755–3763, June 2016.  https://doi.org/10.1109/CVPR.2016.408
  2. 2.
    Aliev, K.A., Ulyanov, D., Lempitsky, V.: Neural point-based graphics (2019). arXiv:1906.08240
  3. 3.
    Anderson, R., et al.: Jump: virtual reality video. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 35(6), 198:1–198:13 (2016).  https://doi.org/10.1145/2980179.2980257CrossRefGoogle Scholar
  4. 4.
    Bau, D., et al.: Seeing what a GAN cannot generate. In: Proceedings of the International Conference on Computer Vision (ICCV) (2019)Google Scholar
  5. 5.
    Bertel, T., Campbell, N.D.F., Richardt, C.: MegaParallax: casual 360\(^\circ \) panoramas with motion parallax. IEEE Trans. Visual Comput. Graphics 25(5), 1828–1835 (2019).  https://doi.org/10.1109/TVCG.2019.2898799CrossRefGoogle Scholar
  6. 6.
    Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74(1) (2007).  https://doi.org/10.1007/s11263-006-0002-3
  7. 7.
    Buehler, C., Bosse, M., McMillan, L., Gortler, S., Cohen, M.: Unstructured lumigraph rendering. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 425–432 (2001).  https://doi.org/10.1145/383259.383309
  8. 8.
    Bussone, W.: Linear and angular head accelerations in daily life. Ph.D. thesis, Virginia Tech (2005)Google Scholar
  9. 9.
    Cabral, B.: VR capture: designing and building an open source 3D-360 video camera. In: SIGGRAPH Asia Keynote, December 2016Google Scholar
  10. 10.
    Chai, J.X., Tong, X., Chan, S.C., Shum, H.Y.: Plenoptic sampling. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 307–318 (2000).  https://doi.org/10.1145/344779.344932
  11. 11.
    Chaurasia, G., Duchêne, S., Sorkine-Hornung, O., Drettakis, G.: Depth synthesis and local warps for plausible image-based navigation. ACM Trans. Graph. 32(3), 30:1–30:12 (2013).  https://doi.org/10.1145/2487228.2487238CrossRefGoogle Scholar
  12. 12.
    Chaurasia, G., Sorkine-Hornung, O., Drettakis, G.: Silhouette-aware warping for image-based rendering. Comput. Graph. Forum (Proc. Eurographics Symp. Rendering) 30(4), 1223–1232 (2011).  https://doi.org/10.1111/j.1467-8659.2011.01981.xCrossRefGoogle Scholar
  13. 13.
    Chen, S.E., Williams, L.: View interpolation for image synthesis. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 279–288 (1993).  https://doi.org/10.1145/166117.166153
  14. 14.
    Cohen, T.S., Welling, M.: Transformation properties of learned visual representations. In: Proceedings of the International Conference on Learning Representations (ICLR) (2015)Google Scholar
  15. 15.
    Collet, A., et al.: High-quality streamable free-viewpoint video. ACM Trans. Graph. (Proc. SIGGRAPH) 34(4), 69:1–69:13 (2015).  https://doi.org/10.1145/2766945CrossRefGoogle Scholar
  16. 16.
    Curless, B., Seitz, S., Bouguet, J.Y., Debevec, P., Levoy, M., Nayar, S.K.: 3D photography. In: SIGGRAPH Courses (2000). http://www.cs.cmu.edu/~seitz/course/3DPhoto.html
  17. 17.
    Dai, A., Nießner, M., Zollhöfer, M., Izadi, S., Theobalt, C.: BundleFusion: real-time globally consistent 3D reconstruction using on-the-fly surface reintegration. ACM Trans. Graph. 36(3), 24:1–24:18 (2017).  https://doi.org/10.1145/3054739CrossRefGoogle Scholar
  18. 18.
    Davis, A., Levoy, M., Durand, F.: Unstructured light fields. Comput. Graph. Forum (Proc. Eurographics) 31(2), 305–314 (2012).  https://doi.org/10.1111/j.1467-8659.2012.03009.xCrossRefGoogle Scholar
  19. 19.
    Debevec, P.: The light stages and their applications to photoreal digital actors. In: SIGGRAPH Asia Technical Briefs (2012)Google Scholar
  20. 20.
    Debevec, P., Bregler, C., Cohen, M.F., McMillan, L., Sillion, F., Szeliski, R.: Image-based modeling, rendering, and lighting. In: SIGGRAPH Courses (2000). https://www.pauldebevec.com/IBMR99/
  21. 21.
    Debevec, P.E., Taylor, C.J., Malik, J.: Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 11–20, August 1996.  https://doi.org/10.1145/237170.237191
  22. 22.
    Dosovitskiy, A., Springenberg, J.T., Tatarchenko, M., Brox, T.: Learning to generate chairs, tables and cars with convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 692–705 (2017).  https://doi.org/10.1109/TPAMI.2016.2567384CrossRefGoogle Scholar
  23. 23.
    DXOMARK: RED Helium 8K DxOMark sensor score: 108—a new all-time-high score! https://www.dxomark.com/red-helium-8k-dxomark-sensor-score-108-a-new-all-time-high-score2/. Accessed 30 Oct 2019
  24. 24.
    Eslami, S.M.A., et al.: Neural scene representation and rendering. Science 360(6394), 1204–1210 (2018).  https://doi.org/10.1126/science.aar6170CrossRefGoogle Scholar
  25. 25.
    Flynn, J., et al.: DeepView: view synthesis with learned gradient descent. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2367–2376, June 2019Google Scholar
  26. 26.
    Flynn, J., Neulander, I., Philbin, J., Snavely, N.: DeepStereo: learning to predict new views from the world’s imagery. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5515–5524, June 2016.  https://doi.org/10.1109/CVPR.2016.595
  27. 27.
    Fuhrmann, S., Langguth, F., Goesele, M.: MVE: a multi-view reconstruction environment. In: Proceedings of the Eurographics Workshop on Graphics and Cultural Heritage, pp. 11–18 (2014).  https://doi.org/10.2312/gch.20141299
  28. 28.
    Galliani, S., Lasinger, K., Schindler, K.: Massively parallel multiview stereopsis by surface normal diffusion. In: Proceedings of the International Conference on Computer Vision (ICCV), pp. 873–881, December 2015.  https://doi.org/10.1109/ICCV.2015.106
  29. 29.
    Garon, M., Sunkavalli, K., Hadap, S., Carr, N., Lalonde, J.F.: Fast spatially-varying indoor lighting estimation. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  30. 30.
    Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M.F.: The lumigraph. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 43–54, August 1996.  https://doi.org/10.1145/237170.237200
  31. 31.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press (2004).  https://doi.org/10.1017/CBO9780511811685
  32. 32.
    Hedman, P., Alsisan, S., Szeliski, R., Kopf, J.: Casual 3D photography. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 36(6), 234:1–234:15 (2017).  https://doi.org/10.1145/3130800.3130828CrossRefGoogle Scholar
  33. 33.
    Hedman, P., Kopf, J.: Instant 3D photography. ACM Trans. Graph. (Proc. SIGGRAPH) 37(4), 101:1–101:12 (2018).  https://doi.org/10.1145/3197517.3201384CrossRefGoogle Scholar
  34. 34.
    Hedman, P., Philip, J., Price, T., Frahm, J.M., Drettakis, G.: Deep blending for free-viewpoint image-based rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37(6), 257:1–257:15 (2018).  https://doi.org/10.1145/3272127.3275084CrossRefGoogle Scholar
  35. 35.
    Hedman, P., Ritschel, T., Drettakis, G., Brostow, G.: Scalable inside-out image-based rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 35(6), 231:1–231:11 (2016).  https://doi.org/10.1145/2980179.2982420CrossRefGoogle Scholar
  36. 36.
    Huang, J., Chen, Z., Ceylan, D., Jin, H.: 6-DOF VR videos with a single 360-camera. In: Proceedings of IEEE Virtual Reality (VR), pp. 37–44, March 2017.  https://doi.org/10.1109/VR.2017.7892229
  37. 37.
    Huang, P.H., Matzen, K., Kopf, J., Ahuja, N., Huang, J.B.: DeepMVS: learning multi-view stereopsis. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  38. 38.
    Hukkelås, H., Mester, R., Lindseth, F.: DeepPrivacy: a generative adversarial network for face anonymization. In: Bebis, G., et al. (eds.) ISVC 2019. LNCS, vol. 11844, pp. 565–578. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-33720-9_44CrossRefGoogle Scholar
  39. 39.
    Ishiguro, H., Yamamoto, M., Tsuji, S.: Omni-directional stereo. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 257–262 (1992).  https://doi.org/10.1109/34.121792CrossRefGoogle Scholar
  40. 40.
    Jahanian, A., Chai, L., Isola, P.: On the “steerability” of generative adversarial networks (2019). arXiv:1907.07171
  41. 41.
    Jancosek, M., Pajdla, T.: Multi-view reconstruction preserving weakly-supported surfaces. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3121–3128, June 2011.  https://doi.org/10.1109/CVPR.2011.5995693
  42. 42.
    Ji, D., Kwon, J., McFarland, M., Savarese, S.: Deep view morphing. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7092–7100, July 2017.  https://doi.org/10.1109/CVPR.2017.750
  43. 43.
    Kalantari, N.K., Wang, T.C., Ramamoorthi, R.: Learning-based view synthesis for light field cameras. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 35(6), 193:1–193:10 (2016).  https://doi.org/10.1145/2980179.2980251CrossRefGoogle Scholar
  44. 44.
    Keysers, C., Xiao, D.K., Földiák, P., Perrett, D.I.: The speed of sight. J. Cogn. Neurosci. 13(1), 90–101 (2001).  https://doi.org/10.1162/089892901564199CrossRefGoogle Scholar
  45. 45.
    Kim, C., Zimmer, H., Pritch, Y., Sorkine-Hornung, A., Gross, M.: Scene reconstruction from high spatio-angular resolution light fields. ACM Trans. Graph. (Proc. SIGGRAPH) 32(4), 73:1–73:12 (2013).  https://doi.org/10.1145/2461912.2461926CrossRefzbMATHGoogle Scholar
  46. 46.
    Kim, H., et al.: Deep video portraits. ACM Trans. Graph. (Proc. SIGGRAPH) 37(4), 163:1–163:14 (2018).  https://doi.org/10.1145/3197517.3201283CrossRefGoogle Scholar
  47. 47.
    Kirillov, A., He, K., Girshick, R., Rother, C., Dollár, P.: Panoptic segmentation. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  48. 48.
    Konrad, R., Dansereau, D.G., Masood, A., Wetzstein, G.: SpinVR: towards live-streaming 3D virtual reality video. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 36(6), 209:1–209:12 (2017).  https://doi.org/10.1145/3130800.3130836CrossRefGoogle Scholar
  49. 49.
    Kopf, J., et al.: Practical 3D photography. In: Proceedings of CVPR Workshops (2019)Google Scholar
  50. 50.
    Koulieris, G.A., Akşit, K., Stengel, M., Mantiuk, R.K., Mania, K., Richardt, C.: Near-eye display and tracking technologies for virtual and augmented reality. Comput. Graph. Forum 38(2), 493–519 (2019).  https://doi.org/10.1111/cgf.13654CrossRefGoogle Scholar
  51. 51.
    Kulkarni, T.D., Whitney, W., Kohli, P., Tenenbaum, J.B.: Deep convolutional inverse graphics network. In: Advances in Neural Information Processing Systems (NIPS) (2015)Google Scholar
  52. 52.
    Lanman, D., Wetzstein, G., Hirsch, M., Heidrich, W., Raskar, R.: Polarization fields: dynamic light field display using multi-layer LCDs. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 30(6), 186:1–186:10 (2011).  https://doi.org/10.1145/2070781.2024220CrossRefGoogle Scholar
  53. 53.
    Lee, J., Kim, B., Kim, K., Kim, Y., Noh, J.: Rich360: optimized spherical representation from structured panoramic camera arrays. ACM Trans. Graph. (Proc. SIGGRAPH) 35(4), 63:1–63:11 (2016).  https://doi.org/10.1145/2897824.2925983CrossRefGoogle Scholar
  54. 54.
    LeGendre, C., et al.: DeepLight: learning illumination for unconstrained mobile mixed reality. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  55. 55.
    Levoy, M., Hanrahan, P.: Light field rendering. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 31–42, August 1996.  https://doi.org/10.1145/237170.237199
  56. 56.
    Lipski, C., Linz, C., Berger, K., Sellent, A., Magnor, M.: Virtual video camera: image-based viewpoint navigation through space and time. Comput. Graph. Forum 29(8), 2555–2568 (2010).  https://doi.org/10.1111/j.1467-8659.2010.01824.xCrossRefGoogle Scholar
  57. 57.
    Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., Sheikh, Y.: Neural volumes: learning dynamic renderable volumes from images. ACM Trans. Graph. (Proc. SIGGRAPH) 38(4), 65:1–65:14 (2019).  https://doi.org/10.1145/3306346.3323020CrossRefGoogle Scholar
  58. 58.
    Luo, B., Xu, F., Richardt, C., Yong, J.H.: Parallax360: stereoscopic 360\(^\circ \) scene representation for head-motion parallax. IEEE Trans. Vis. Comput. Graph. 24(4), 1545–1553 (2018).  https://doi.org/10.1109/TVCG.2018.2794071CrossRefGoogle Scholar
  59. 59.
    Magnor, M., Grau, O., Sorkine-Hornung, O., Theobalt, C. (eds.): Digital Representations of the Real World: How to Capture, Model, and Render Visual Reality. A K Peters/CRC Press, New York (2015)zbMATHGoogle Scholar
  60. 60.
    Martin-Brualla, R., et al.: LookinGood: enhancing performance capture with real-time neural re-rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37(6), 255:1–255:14 (2018).  https://doi.org/10.1145/3272127.3275099CrossRefGoogle Scholar
  61. 61.
    Meka, A., et al.: Deep reflectance fields: high-quality facial reflectance field inference from color gradient illumination. ACM Trans. Graph. (Proc. SIGGRAPH) 38(4), 77:1–77:12 (2019).  https://doi.org/10.1145/3306346.3323027CrossRefGoogle Scholar
  62. 62.
    Meshry, M., et al.: Neural rerendering in the wild. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  63. 63.
    Mildenhall, B., et al.: Local light field fusion: practical view synthesis with prescriptive sampling guidelines. ACM Trans. Graph. (Proc. SIGGRAPH) 38(4), 29:1–29:14 (2019).  https://doi.org/10.1145/3306346.3322980CrossRefGoogle Scholar
  64. 64.
    Mori, M.: The uncanny valley. Energy 7(4), 33–35 (1970). (in Japanese)Google Scholar
  65. 65.
    Moulon, P., Monasse, P., Marlet, R.: Adaptive structure from motion with a Contrario model estimation. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012. LNCS, vol. 7727, pp. 257–270. Springer, Heidelberg (2013).  https://doi.org/10.1007/978-3-642-37447-0_20CrossRefGoogle Scholar
  66. 66.
    Mustafa, A., Volino, M., Guillemaut, J.Y., Hilton, A.: 4D temporally coherent light-field video. In: Proceedings of International Conference on 3D Vision (3DV) (2017)Google Scholar
  67. 67.
    Mustafa, A., Volino, M., Kim, H., Guillemaut, J.Y., Hilton, A.: Temporally coherent general dynamic scene reconstruction (2019). arXiv:1907.08195
  68. 68.
    Nam, G., Lee, J.H., Gutierrez, D., Kim, M.H.: Practical SVBRDF acquisition of 3D objects with unstructured flash photography. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37(6), 267:1–267:12 (2018).  https://doi.org/10.1145/3272127.3275017CrossRefGoogle Scholar
  69. 69.
    Newcombe, R.A., et al.: KinectFusion: real-time dense surface mapping and tracking. In: Proceedings of the International Symposium on Mixed and Augmented Reality (ISMAR), pp. 127–136, October 2011.  https://doi.org/10.1109/ISMAR.2011.6092378
  70. 70.
    Nguyen-Phuoc, T., Li, C., Theis, L., Richardt, C., Yang, Y.L.: HoloGAN: unsupervised learning of 3D representations from natural images. In: Proceedings of the International Conference on Computer Vision (ICCV) (2019)Google Scholar
  71. 71.
    Nießner, M., Zollhöfer, M., Izadi, S., Stamminger, M.: Real-time 3D reconstruction at scale using voxel hashing. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 32(6), 169:1–169:11 (2013).  https://doi.org/10.1145/2508363.2508374CrossRefGoogle Scholar
  72. 72.
    Niklaus, S., Mai, L., Yang, J., Liu, F.: 3D Ken Burns effect from a single image. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38(6), 184:1–184:15 (2019).  https://doi.org/10.1145/3355089.3356528CrossRefGoogle Scholar
  73. 73.
    Oculus: From the lab to the living room: the story behind Facebook’s Oculus Insight technology and a new era of consumer VR. https://tech.fb.com/the-story-behind-oculus-insight-technology/. Accessed 30 Oct 2019
  74. 74.
    Olszewski, K., Tulyakov, S., Woodford, O., Li, H., Luo, L.: Transformable bottleneck networks. In: Proceedings of the International Conference on Computer Vision (ICCV) (2019)Google Scholar
  75. 75.
    Overbeck, R.S., Erickson, D., Evangelakos, D., Pharr, M., Debevec, P.: A system for acquiring, compressing, and rendering panoramic light field stills for virtual reality. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37(6), 197:1–197:15 (2018).  https://doi.org/10.1145/3272127.3275031CrossRefGoogle Scholar
  76. 76.
    Park, E., Yang, J., Yumer, E., Ceylan, D., Berg, A.C.: Transformation-grounded image generation network for novel 3D view synthesis. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 702–711, July 2017.  https://doi.org/10.1109/CVPR.2017.82
  77. 77.
    Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: learning continuous signed distance functions for shape representation. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  78. 78.
    Parra Pozo, A., et al.: An integrated 6DoF video camera and system design. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38(6), 216:1–216:16 (2019).  https://doi.org/10.1145/3355089.3356555. https://github.com/facebook/facebook360depCrossRefGoogle Scholar
  79. 79.
    Peleg, S., Ben-Ezra, M., Pritch, Y.: Omnistereo: panoramic stereo imaging. IEEE Trans. Pattern Anal. Mach. Intell. 23(3), 279–290 (2001).  https://doi.org/10.1109/34.910880CrossRefzbMATHGoogle Scholar
  80. 80.
    Penner, E., Zhang, L.: Soft 3D reconstruction for view synthesis. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 36(6), 235:1–235:11 (2017).  https://doi.org/10.1145/3130800.3130855CrossRefGoogle Scholar
  81. 81.
    Perazzi, F., et al.: Panoramic video from unstructured camera arrays. Comput. Graph. Forum (Proc. Eurographics) 34(2), 57–68 (2015).  https://doi.org/10.1111/cgf.12541CrossRefGoogle Scholar
  82. 82.
    Prada, F., Kazhdan, M., Chuang, M., Collet, A., Hoppe, H.: Spatiotemporal atlas parameterization for evolving meshes. ACM Trans. Graph. (Proc. SIGGRAPH) 36(4), 58:1–58:12 (2017).  https://doi.org/10.1145/3072959.3073679CrossRefGoogle Scholar
  83. 83.
    Qi, M., Li, W., Yang, Z., Wang, Y., Luo, J.: Attentive relational networks for mapping images to scene graphs. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  84. 84.
    Rhodin, H., Salzmann, M., Fua, P.: Unsupervised geometry-aware representation for 3D human pose estimation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11214, pp. 765–782. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01249-6_46CrossRefGoogle Scholar
  85. 85.
    Richardt, C., Hedman, P., Overbeck, R.S., Cabral, B., Konrad, R., Sullivan, S.: Capture4VR: from VR photography to VR video. In: SIGGRAPH Courses (2019).  https://doi.org/10.1145/3305366.3328028
  86. 86.
    Richardt, C., Pritch, Y., Zimmer, H., Sorkine-Hornung, A.: Megastereo: constructing high-resolution stereo panoramas. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1256–1263, June 2013.  https://doi.org/10.1109/CVPR.2013.166
  87. 87.
    Schroers, C., Bazin, J.C., Sorkine-Hornung, A.: An omnistereoscopic video pipeline for capture and display of real-world VR. ACM Trans. Graph. 37(3), 37:1–37:13 (2018).  https://doi.org/10.1145/3225150CrossRefGoogle Scholar
  88. 88.
    Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4104–4113 (2016).  https://doi.org/10.1109/CVPR.2016.445
  89. 89.
    Schönberger, J.L., Zheng, E., Frahm, J.-M., Pollefeys, M.: Pixelwise view selection for unstructured multi-view stereo. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 501–518. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46487-9_31CrossRefGoogle Scholar
  90. 90.
    Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 519–528 (2006).  https://doi.org/10.1109/CVPR.2006.19
  91. 91.
    Serrano, A., et al.: Motion parallax for 360\(^\circ \) RGBD video. IEEE Trans. Vis. Comput. Graph. 25(5), 1817–1827 (2019).  https://doi.org/10.1109/TVCG.2019.2898757CrossRefGoogle Scholar
  92. 92.
    Shum, H., Kang, S.B.: Review of image-based rendering techniques. In: Proceedings of the SPIE Visual Communications and Image Processing, vol. 4067 (2000).  https://doi.org/10.1117/12.386541
  93. 93.
    Shum, H.Y., Chan, S.C., Kang, S.B.: Image-Based Rendering. Springer, Boston (2007).  https://doi.org/10.1007/978-0-387-32668-9CrossRefzbMATHGoogle Scholar
  94. 94.
    Sitzmann, V., Thies, J., Heide, F., Nießner, M., Wetzstein, G., Zollhöfer, M.: DeepVoxels: learning persistent 3D feature embeddings. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2437–2446 (2019)Google Scholar
  95. 95.
    Sitzmann, V., Zollhöfer, M., Wetzstein, G.: Scene representation networks: continuous 3D-structure-aware neural scene representations. In: Advances in Neural Information Processing Systems (NeurIPS) (2019)Google Scholar
  96. 96.
    Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3D. ACM Trans. Graph. (Proc. SIGGRAPH) 25(3), 835–846 (2006).  https://doi.org/10.1145/1141911.1141964CrossRefGoogle Scholar
  97. 97.
    Speciale, P., Schönberger, J.L., Kang, S.B., Sinha, S.N., Pollefeys, M.: Privacy preserving image-based localization. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  98. 98.
    Srinivasan, P.P., Tucker, R., Barron, J.T., Ramamoorthi, R., Ng, R., Snavely, N.: Pushing the boundaries of view extrapolation with multiplane images. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  99. 99.
    Sweeney, C.: Theia multiview geometry library (2016). http://theia-sfm.org
  100. 100.
    Sweeney, C., Holynski, A., Curless, B., Seitz, S.M.: Structure from motion for panorama-style videos (2019). arXiv:1906.03539
  101. 101.
    Szeliski, R.: Image alignment and stitching: a tutorial. Found. Trends Comput. Graph. Vis. 2(1), 1–104 (2006).  https://doi.org/10.1561/0600000009MathSciNetCrossRefzbMATHGoogle Scholar
  102. 102.
    Szeliski, R.: Computer Vision: Algorithms and Applications. Springer, London (2010).  https://doi.org/10.1007/978-1-84882-935-0. http://szeliski.org/Book/CrossRefzbMATHGoogle Scholar
  103. 103.
    Tarko, J., Tompkin, J., Richardt, C.: Real-time virtual object insertion for moving 360\(^\circ \) videos. In: Proceedings of the International Conference on Virtual-Reality Continuum and its Applications in Industry (VRCAI) (2019)Google Scholar
  104. 104.
    Tatarchenko, M., Dosovitskiy, A., Brox, T.: Multi-view 3D models from single images with a convolutional network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 322–337. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46478-7_20CrossRefGoogle Scholar
  105. 105.
    Thies, J., Zollhöfer, M., Nießner, M.: Deferred neural rendering: image synthesis using neural textures. ACM Trans. Graph. (Proc. SIGGRAPH) 38(4), 66:1–66:12 (2019).  https://doi.org/10.1145/3306346.3323035CrossRefGoogle Scholar
  106. 106.
    Tulsiani, S., Tucker, R., Snavely, N.: Layer-structured 3D scene inference via view synthesis. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 311–327. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01234-2_19CrossRefGoogle Scholar
  107. 107.
    Tung, H.Y.F., Cheng, R., Fragkiadaki, K.: Learning spatial common sense with geometry-aware recurrent networks. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2595–2603 (2019)Google Scholar
  108. 108.
    Valve: Index headset. www.valvesoftware.com/en/index/headset. Accessed 30 Oct 2019
  109. 109.
    Ventura, J.: Structure from motion on a sphere. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 53–68. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46487-9_4CrossRefGoogle Scholar
  110. 110.
    Wei, S.E., et al.: VR facial animation via multiview image translation. ACM Trans. Graph. (Proc. SIGGRAPH) 38(4), 67:1–67:16 (2019).  https://doi.org/10.1145/3306346.3323030CrossRefGoogle Scholar
  111. 111.
    Weissig, C., Schreer, O., Eisert, P., Kauff, P.: The ultimate immersive experience: panoramic 3D video acquisition. In: Schoeffmann, K., Merialdo, B., Hauptmann, A.G., Ngo, C.-W., Andreopoulos, Y., Breiteneder, C. (eds.) MMM 2012. LNCS, vol. 7131, pp. 671–681. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-27355-1_72CrossRefGoogle Scholar
  112. 112.
    Wetzstein, G., Lanman, D., Heidrich, W., Raskar, R.: Layered 3D: tomographic image synthesis for attenuation-based light field and high dynamic range displays. ACM Trans. Graph. (Proc. SIGGRAPH) 30(4), 95:1–95:12 (2011).  https://doi.org/10.1145/2010324.1964990CrossRefGoogle Scholar
  113. 113.
    Wetzstein, G., Lanman, D., Hirsch, M., Raskar, R.: Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting. ACM Trans. Graph. (Proc. SIGGRAPH) 31(4), 80:1–80:11 (2012).  https://doi.org/10.1145/2185520.2185576CrossRefGoogle Scholar
  114. 114.
    Whelan, T., Salas-Moreno, R.F., Glocker, B., Davison, A.J., Leutenegger, S.: ElasticFusion: real-time dense SLAM and light source estimation. Int. J. Robot. Res. 35(14), 1697–1716 (2016).  https://doi.org/10.1177/0278364916669237CrossRefGoogle Scholar
  115. 115.
    Wood, D.N., et al.: Surface light fields for 3D photography. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 287–296 (2000).  https://doi.org/10.1145/344779.344925
  116. 116.
    Worrall, D.E., Garbin, S.J., Turmukhambetov, D., Brostow, G.J.: Interpretable transformations with encoder-decoder networks. In: Proceedings of the International Conference on Computer Vision (ICCV), pp. 5737–5746 (2017).  https://doi.org/10.1109/ICCV.2017.611
  117. 117.
    Wu, C.: VisualSFM: a visual structure from motion system (2011). http://ccwu.me/vsfm/
  118. 118.
    Yang, J., Reed, S.E., Yang, M.H., Lee, H.: Weakly-supervised disentangling with recurrent transformations for 3D view synthesis. In: Advances in Neural Information Processing Systems (NIPS), pp. 1099–1107 (2015)Google Scholar
  119. 119.
    Yifan, W., Serena, F., Wu, S., Öztireli, C., Sorkine-Hornung, O.: Differentiable surface splatting for point-based geometry processing. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38(6) (2019).  https://doi.org/10.1145/3355089.3356513
  120. 120.
    Yücer, K., Sorkine-Hornung, A., Wang, O., Sorkine-Hornung, O.: Efficient 3D object segmentation from densely sampled light fields with applications to 3D reconstruction. ACM Trans. Graph. 35(3), 22:1–22:15 (2016).  https://doi.org/10.1145/2876504CrossRefGoogle Scholar
  121. 121.
    Zaragoza, J., Chin, T.J., Tran, Q.H., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving DLT. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1285–1298 (2014).  https://doi.org/10.1109/TPAMI.2013.247CrossRefGoogle Scholar
  122. 122.
    Zhang, F., Liu, F.: Parallax-tolerant image stitching. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3262–3269, June 2014.  https://doi.org/10.1109/CVPR.2014.423
  123. 123.
    Zheng, K.C., Kang, S.B., Cohen, M.F., Szeliski, R.: Layered depth panoramas. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR) (2007).  https://doi.org/10.1109/CVPR.2007.383295
  124. 124.
    Zhou, T., Tucker, R., Flynn, J., Fyffe, G., Snavely, N.: Stereo magnification: learning view synthesis using multiplane images. ACM Trans. Graph. (Proc. SIGGRAPH) 37(4), 65:1–65:12 (2018).  https://doi.org/10.1145/3197517.3201323CrossRefGoogle Scholar
  125. 125.
    Zhou, T., Tulsiani, S., Sun, W., Malik, J., Efros, A.A.: View synthesis by appearance flow. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 286–301. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46493-0_18CrossRefGoogle Scholar
  126. 126.
    Zollhöfer, M., et al.: State of the art on monocular 3D face reconstruction, tracking, and applications. Comput. Graph. Forum 37(2), 523–550 (2018).  https://doi.org/10.1111/cgf.13382CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.University of BathBathUK
  2. 2.Brown UniversityProvidenceUSA
  3. 3.Stanford UniversityStanfordUSA

Personalised recommendations