Learning to Reconstruct 3D Structure from Object Motion

  • Wentao LiuEmail author
  • Haobin Dou
  • Xihong Wu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9489)


In this paper, we propose a new approach for reconstructing 3D structure from motion parallax. Instead of obtaining 3D structure from multi-view geometry or factorization, a Deep Neural Network (DNN) based method is proposed without assuming the camera model explicitly. In the proposed method, the targets are first split into connected 3D corners, and then the DNN regressor is trained to estimate the relative 3D structure of each corner from the target rotation. Finally, a temporal integration is performed to further improve the reconstruction accuracy. The effectiveness of the method is proved by a typical experiment of the Kinetic Depth Effect (KDE) in human visual system, in which the DNN regressor reconstructs the structure of a rotating 3D bent wire. The proposed method is also applied to reconstruct another two real targets. Experimental results on both synthetic and real images show that the proposed method is accurate and effective.


3D Reconstruction Structure from Motion Deep Neural Network Kinetic Depth Effect 



The work was supported in part by the National Basic Research Program of China (2013CB329304), the “Twelfth Five-Year” National Science&Technology Support Program of China (No.2012BAI12B01), the Major Project of National Social Science Foundation of China (No.12&ZD119), the research special fund for public welfare industry of health (201202001) and National Natural Science Foundation of China (No.81170906).


  1. 1.
    Wallach, H., O’Connell, D.N.: The kinetic depth effect. J. Exp. Psychol. 45, 205–217 (1953)CrossRefGoogle Scholar
  2. 2.
    Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization method. Int. J. Comput. Vision 9, 137–154 (1992)CrossRefGoogle Scholar
  3. 3.
    Shi, J., Tomasi, C.: Good features to track. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Seattle, pp. 593–600 (1994)Google Scholar
  4. 4.
    Palmer, S.E.: Vision Science: Photons to Phenomenology. MIT Press, Cambridge (1999)Google Scholar
  5. 5.
    Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3D shape from image streams. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Hilton Head Island, pp. 690–696 (2000)Google Scholar
  6. 6.
    Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment – a modern synthesis. In: Triggs, B., Zisserman, A., Szeliski, R. (eds.) ICCV-WS 1999. LNCS, vol. 1883, p. 298. Springer, Heidelberg (2000) CrossRefGoogle Scholar
  7. 7.
    Gruber, A., Weiss, Y.: Multibody factorization with uncertainty and missing data using the EM algorithm. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, pp. 707–714 (2004)Google Scholar
  8. 8.
    Bay, H., Tuytelaars, T., Van Gool, L.: SURF: speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006, Part I. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006) CrossRefGoogle Scholar
  9. 9.
    Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3D. ACM Trans. Graph. 5, 835–846 (2006)CrossRefGoogle Scholar
  11. 11.
    Klein, G., Murray, D.: Parallel tracking and mapping for small AR workspaces. In: 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, Nara, pp. 225–234 (2007)Google Scholar
  12. 12.
    Saxena, A., Sun, M., Ng, A.Y.: Learning 3-D scene structure from a single still image. In: International Conference on Computer Vision Workshop, Rio de Janeiro, pp. 1–8 (2007)Google Scholar
  13. 13.
    Saxena, A., Schulte, J., Ng, A.Y.: Depth estimation using monocular and stereo cues. In: International Joint Conference on Artificial Intelligence, Hyderabad, pp, 2197–2203 (2007)Google Scholar
  14. 14.
    Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2, 1–127 (2009)CrossRefzbMATHGoogle Scholar
  15. 15.
    Ross, D.A., Tarlow, D., Zemel, R.S.: Learning articulated structure and motion. Int. J. Comput. Vision 88, 214–237 (2010)CrossRefGoogle Scholar
  16. 16.
    Hedau, V., Hoiem, D., Forsyth, D.: Recovering free space of indoor scenes from a single image. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Rhode Island, pp. 2807–2814 (2012)Google Scholar
  17. 17.
    Xiao, J., Russell, B.C., Torralba, A.: Localizing 3D cuboids in single-view images. In: Advances in Neural Information Processing Systems, Lake Tahoe, pp. 746–754 (2012)Google Scholar
  18. 18.
    Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Learning hierarchical features for scene labeling. Pattern Anal. Mach. Intell. 35, 1915–1929 (2013)CrossRefGoogle Scholar
  19. 19.
    Fouhey, D.F., Gupta, A., Hebert, M.: Data-driven 3D primitives for single image understanding. In: International Conference on Computer Vision, Sydney, pp. 3392–3399 (2013)Google Scholar
  20. 20.
    Tanskanen, P., Kolev, K., Meier, L., Camposeco, F., Saurer, O., Pollefeys, M.: Live metric 3D reconstruction on mobile phones. In: International Conference on Computer Vision, Sydney, pp. 65–72 (2013)Google Scholar
  21. 21.
    Li, B., Shen, C., Dai, Y., Hengel, A., He, M.: Depth and surface normal estimation from monocular images using regression on deep features and hierarchical CRFs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, pp. 1119–1127 (2015)Google Scholar
  22. 22.
    Resch, B., Lensch, H.P.A., Wang, O., Pollefeys, M., Sorkine-Hornung, A.: Scalable Structure from Motion for Densely Sampled Videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, pp. 3936–3944 (2015)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Key Lab of Machine Perception (MOE), Speech and Hearing Research CenterPeking UniversityBeijingPeople’s Republic of China

Personalised recommendations