Beyond Feature Points: Structured Prediction for Monocular Non-rigid 3D Reconstruction

  • Mathieu Salzmann
  • Raquel Urtasun
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7575)


Existing approaches to non-rigid 3D reconstruction either are specifically designed for feature point correspondences, or require a good shape initialization to exploit more complex image likelihoods. In this paper, we formulate reconstruction as inference in a graphical model, where the variables encode the rotations and translations of the facets of a surface mesh. This lets us exploit complex likelihoods even in the absence of a good initialization. In contrast to existing approaches that set the weights of the likelihood terms manually, our formulation allows us to learn them from as few as a single training example. To improve efficiency, we combine our structured prediction formalism with a gradient-based scheme. Our experiments show that our approach yields tremendous improvement over state-of-the-art gradient-based methods.


Feature Point Structure Prediction Mesh Vertex Reprojection Error Pairwise Potential 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Akhter, I., Sheikh, Y., Khan, S., Kanade, T.: Nonrigid Structure from Motion in Trajectory Space. In: NIPS (2008)Google Scholar
  2. 2.
    Batra, D., Yadollahpour, P., Guzman-Rivera, A., Shakhnarovich, G.: Diverse M-Best Solutions in Markov Random Fields. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 1–16. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  3. 3.
    Blanz, V., Vetter, T.: A Morphable Model for the Synthesis of 3D Faces. In: SIGGRAPH (1999)Google Scholar
  4. 4.
    Bregler, C., Hertzmann, A., Biermann, H.: Recovering Non-Rigid 3D Shape from Image Streams. In: CVPR (2000)Google Scholar
  5. 5.
    Brunet, F., Hartley, R., Bartoli, A., Navab, N., Malgouyres, R.: Monocular Template-Based Reconstruction of Smooth and Inextensible Surfaces. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010, Part III. LNCS, vol. 6494, pp. 52–66. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  6. 6.
    Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active Appearance Models. In: Burkhardt, H., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1407, pp. 484–498. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  7. 7.
  8. 8.
    Fayad, J., Agapito, L., Del Bue, A.: Piecewise Quadratic Reconstruction of Non-Rigid Surfaces from Monocular Sequences. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part IV. LNCS, vol. 6314, pp. 297–310. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  9. 9.
    Fromer, M., Globerson, A.: An LP view of the M best problem. In: NIPS (2009)Google Scholar
  10. 10.
    Gumerov, N., Zandifar, A., Duraiswami, R., Davis, L.S.: Structure of Applicable Surfaces from Single Views. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3023, pp. 482–496. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  11. 11.
    Hazan, T., Urtasun, R.: A primal-dual message-passing algorithm for approximated large scale structured prediction. In: NIPS (2010)Google Scholar
  12. 12.
    Hazan, T., Shashua, A.: Norm-Product Belief Propagation: Primal-Dual Message-Passing for LP-Relaxation and Approximate Inference. In: IT (2011)Google Scholar
  13. 13.
    Kass, M., Witkin, A., Terzopoulos, D.: Snakes: Active Contour Models. IJCV (1988)Google Scholar
  14. 14.
    Mcinerney, T., Terzopoulos, D.: A Finite Element Model for 3D Shape Reconstruction and Nonrigid Motion Tracking. In: ICCV (1993)Google Scholar
  15. 15.
    Metaxas, D., Terzopoulos, D.: Constrained Deformable Superquadrics and Nonrigid Motion Tracking. PAMI (1993)Google Scholar
  16. 16.
    Peng, J., Hazan, T., McAllester, D., Urtasun, R.: Convex Max-Product Algorithms for Continuous MRFs with Applications to Protein Folding. In: ICML (2011)Google Scholar
  17. 17.
    Pentland, A., Sclaroff, S.: Closed-Form Solutions for Physically Based Shape Modeling and Recognition. PAMI (1991)Google Scholar
  18. 18.
    Perriollat, M., Hartley, R., Bartoli, A.: Monocular Template-Based Reconstruction of Inextensible Surfaces. IJCV (2010)Google Scholar
  19. 19.
    Russell, C., Fayad, J., Agapito, L.: Energy Based Multiple Model Fitting for Non-Rigid Structure from Motion. In: CVPR (2011)Google Scholar
  20. 20.
    Salzmann, M., Urtasun, R., Fua, P.: Local Deformation Models for Monocular 3D Shape Recovery. In: CVPR (2008)Google Scholar
  21. 21.
    Salzmann, M., Urtasun, R.: Combining Discriminative and Generative Methods for 3D Deformable Surface and Articulated Pose Reconstruction. In: CVPR (2010)Google Scholar
  22. 22.
    Schwing, A., Hazan, T., Pollefeys, M., Urtasun, R.: Distributed Message Passing for Large Scale Graphical Models. In: CVPR (2011)Google Scholar
  23. 23.
    Shen, S., Shi, W., Liu, Y.: Monocular 3D Tracking of Inextensible Deformable Surfaces Under L2-Norm. In: ACCV (2009)Google Scholar
  24. 24.
    Taylor, J., Jepson, A.D., Kutulakos, K.N.: Non-Rigid Structure from Locally-Rigid Motion. In: CVPR (2010)Google Scholar
  25. 25.
    Torresani, L., Hertzmann, A., Bregler, C.: Nonrigid Structure-From-Motion: Estimating Shape and Motion with Hierarchical Priors. PAMI (2008)Google Scholar
  26. 26.
    Varol, A., Shaji, A., Salzmann, M., Fua, P.: Monocular 3D Reconstruction of Locally Textured Surfaces. PAMI (2011)Google Scholar
  27. 27.
    Xiao, J., Kanade, T.: Uncalibrated Perspective Reconstruction of Deformable Structures. In: ICCV (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Mathieu Salzmann
    • 1
  • Raquel Urtasun
    • 2
  1. 1.NICTAAustralia
  2. 2.TTI ChicagoUSA

Personalised recommendations