Skip to main content
SpringerLink
Log in

Recovering Pose and 3D Deformable Shape from Multi-instance Image Ensembles

  • Conference paper
  • First Online:
Computer Vision – ACCV 2016 (ACCV 2016)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 10114))

Included in the following conference series:

Abstract

In recent years, there has been a growing interest on tackling the Non-Rigid Structure from Motion problem (NRSfM), where the shape of a deformable object and the pose of a moving camera are simultaneously estimated from a monocular video sequence. Existing solutions are limited to single objects and continuous, smoothly changing sequences. In this paper we extend NRSfM to a multi-instance domain, in which the images do not need to have temporal consistency, allowing for instance, to jointly reconstruct the face of multiple persons from an unordered list of images. For this purpose, we present a new formulation of the problem based on a dual low-rank shape representation, that simultaneously captures the between- and within-individual deformations. The parameters of this model are learned using a variant of the probabilistic linear discriminant analysis that requires consecutive batches of expectation and maximization steps. The resulting approach estimates 3D deformable shape and pose of multiple instances from only 2D point observations on a collection images, without requiring pre-trained 3D data, and is shown to be robust to noisy measurements and missing points. We provide quantitative and qualitative evaluation on both synthetic and real data, and show consistent benefits compared to current state of the art.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Videos can be found on website: http://www.iri.upc.edu/people/aagudo.

References

  1. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)

    MATH  Google Scholar 

  2. Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment - a modern synthesis. Vis. Algorithms Theory Pract. 1883, 298–372 (2000)

    Article  Google Scholar 

  3. Agarwal, S., Snavely, N., Simon, I., Seitz, S.M., Szeliski, R.: Building Rome in a day. In: ICCV (2009)

    Google Scholar 

  4. Lim, J., Frahm, J., Pollefeys, M.: Online environment mapping. In: CVPR (2011)

    Google Scholar 

  5. Torresani, L., Hertzmann, A., Bregler, C.: Nonrigid structure-from-motion: estimating shape and motion with hierarchical priors. TPAMI 30, 878–892 (2008)

    Article  Google Scholar 

  6. Gotardo, P.F.U., Martinez, A.M.: Kernel non-rigid structure from motion. In: ICCV (2011)

    Google Scholar 

  7. Lee, M., Cho, J., Choi, C.H., Oh, S.: Procrustean normal distribution for non-rigid structure from motion. In: CVPR (2013)

    Google Scholar 

  8. Chhatkuli, A., Pizarro, D., Bartoli, A.: Non-rigid shape-from-motion for isometric surfaces using infinitesimal planarity. In: BMVC (2014)

    Google Scholar 

  9. Agudo, A., Moreno-Noguer, F.: Learning shape, motion and elastic models in force space. In: ICCV (2015)

    Google Scholar 

  10. Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3D shape from image streams. In: CVPR (2000)

    Google Scholar 

  11. Bartoli, A., Gay-Bellile, V., Castellani, U., Peyras, J., Olsen, S., Sayd, P.: Coarse-to-fine low-rank structure-from-motion. In: CVPR (2008)

    Google Scholar 

  12. Paladini, M., Del Bue, A., Stosic, M., Dodig, M., Xavier, J., Agapito, L.: Factorization for non-rigid and articulated structure using metric projections. In: CVPR (2009)

    Google Scholar 

  13. Dai, Y., Li, H., He, M.: A simple prior-free method for non-rigid structure from motion factorization. In: CVPR (2012)

    Google Scholar 

  14. Zhu, Y., Huang, D., De La Torre, F., Lucey, S.: Complex non-rigid motion 3D reconstruction by union of subspaces. In: CVPR (2014)

    Google Scholar 

  15. Garg, R., Roussos, A., Agapito, L.: Dense variational reconstruction of non-rigid surfaces from monocular video. In: CVPR (2013)

    Google Scholar 

  16. Paladini, M., Bartoli, A., Agapito, L.: Sequential non rigid structure from motion with the 3D implicit low rank shape model. In: ECCV (2010)

    Google Scholar 

  17. Agudo, A., Montiel, J.M.M., Agapito, L., Calvo, B.: Online dense non-rigid 3D shape and camera motion recovery. In: BMVC (2014)

    Google Scholar 

  18. Lee, M., Choi, C.H., Oh, S.: A procrustean Markov process for non-rigid structure recovery. In: CVPR (2014)

    Google Scholar 

  19. Akhter, I., Sheikh, Y., Khan, S., Kanade, T.: Non-rigid structure from motion in trajectory space. In: NIPS (2008)

    Google Scholar 

  20. Gotardo, P.F.U., Martinez, A.M.: Non-rigid structure from motion with complementary rank-3 spaces. In: CVPR (2011)

    Google Scholar 

  21. Agudo, A., Moreno-Noguer, F., Calvo, B., Montiel, J.M.M.: Sequential non-rigid structure from motion using physical priors. TPAMI 38, 979–994 (2016)

    Article  Google Scholar 

  22. Agudo, A., Agapito, L., Calvo, B., Montiel, J.M.M.: Good vibrations: a modal analysis approach for sequential non-rigid structure from motion. In: CVPR (2014)

    Google Scholar 

  23. Agudo, A., Moreno-Noguer, F.: Simultaneous pose and non-rigid shape with particle dynamics. In: CVPR (2015)

    Google Scholar 

  24. Li, P., Fu, Y., Mohammed, U., Elder, J.H., Prince, S.J.D.: Probabilistic models for inference about identity. TPAMI 34, 144–157 (2012)

    Article  Google Scholar 

  25. Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: SIGGRAPH (1999)

    Google Scholar 

  26. Agudo, A., Montiel, J.M.M., Agapito, L., Calvo, B.: Modal space: a physics-based model for sequential estimation of time-varying shape from monocular video. JMIV 57(1), 75–98 (2016)

    Article  MathSciNet  Google Scholar 

  27. Barbic, J., James, D.: Real-time subspace integration for St. Venant-Kirchhoff deformable models. TOG 24, 982–990 (2005)

    Article  Google Scholar 

  28. Agudo, A., Montiel, J.M.M., Calvo, B., Moreno-Noguer, F.: Mode-shape interpretation: re-thinking modal space for recovering deformable shapes. In: WACV (2016)

    Google Scholar 

  29. Xiao, J., Chai, J., Kanade, T.: A closed-form solution to non-rigid shape and motion. IJCV 67, 233–246 (2006)

    Article  Google Scholar 

  30. Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization approach. IJCV 9, 137–154 (1992)

    Article  Google Scholar 

  31. Del Bue, A., Llado, X., Agapito, L.: Non-rigid metric shape and motion recovery from uncalibrated images using priors. In: CVPR (2006)

    Google Scholar 

  32. Valmadre, J., Lucey, S.: General trajectory prior for non-rigid reconstruction. In: CVPR (2012)

    Google Scholar 

  33. Gotardo, P.F.U., Martinez, A.M.: Computing smooth time-trajectories for camera and deformable shape in structure from motion with occlusion. TPAMI 33, 2051–2065 (2011)

    Article  Google Scholar 

  34. Simon, T., Valmadre, J., Matthews, I., Sheikh, Y.: Separable spatiotemporal priors for convex reconstruction of time-varying 3D point clouds. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 204–219. Springer, Cham (2014). doi:10.1007/978-3-319-10578-9_14

    Google Scholar 

  35. Sigal, L., Bhatia, S., Roth, S., Black, M.J., Isard, M.: Tracking loose-limbed people. In: CVPR (2004)

    Google Scholar 

  36. Wang, J.M., Fleet, D.J., Hertzmann, A.: Gaussian process dynamical models. In: NIPS (2005)

    Google Scholar 

  37. Urtasun, R., Fleet, D., Fua, P.: 3D people tracking with Gaussian process dynamical models. In: CVPR (2006)

    Google Scholar 

  38. Fisher, R.A.: The statistical utilization of multiple measurements. Ann. Eugenics 8, 376–386 (1938)

    Article  MATH  Google Scholar 

  39. Rao, C.R.: The utilization of multiple measurements in problems of biological classification. J. R. Stat. Soc. B 10, 159–203 (1948)

    MathSciNet  MATH  Google Scholar 

  40. Prince, S.J.D., Elder, J.H.: Probabilistic linear discriminant analysis for inferences about identity. In: ICCV (2007)

    Google Scholar 

  41. Ioffe, S.: Probabilistic linear discriminant analysis. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3954, pp. 531–542. Springer, Heidelberg (2006). doi:10.1007/11744085_41

    Chapter  Google Scholar 

  42. Woodbury, M.A.: Inverting modified matrices. Statistical Research Group, Memorandum Report 42 (1950)

    Google Scholar 

  43. Akhter, I., Simon, T., Khan, S., Matthews, I., Sheikh, Y.: Bilinear spatiotemporal basis models. TOG 31, 17:1–17:12 (2012)

    Article  Google Scholar 

  44. Milborrow, S., Morkel, J., Nicolls, F.: The MUCT landmarked face database. Pattern Recognition Association of South Africa (2010)

    Google Scholar 

  45. Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. In: Burkhardt, H., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1407, pp. 484–498. Springer, Heidelberg (1998). doi:10.1007/BFb0054760

    Chapter  Google Scholar 

Download references

Acknowledgments

This work has been partially supported by the Spanish Ministry of Science and Innovation under project RobInstruct TIN2014-58178-R; by the ERA-net CHISTERA projects VISEN PCIN-2013-047 and I-DRESS PCIN-2015-147. The authors also thank Gerard Canal for fruitful discussions.

Author information

Authors and Affiliations

  1. Institut de Robòtica i Informàtica Industrial (CSIC-UPC), Barcelona, Spain

    Antonio Agudo & Francesc Moreno-Noguer

Authors
  1. Antonio Agudo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Francesc Moreno-Noguer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Antonio Agudo .

Editor information

Editors and Affiliations

  1. National Tsing Hua University, Hsinchu, Taiwan

    Shang-Hong Lai

  2. Graz University of Technology, Graz, Austria

    Vincent Lepetit

  3. Drexel University, Philadelphia, Pennsylvania, USA

    Ko Nishino

  4. The University of Tokyo, Tokyo, Japan

    Yoichi Sato

1 Electronic supplementary material

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Agudo, A., Moreno-Noguer, F. (2017). Recovering Pose and 3D Deformable Shape from Multi-instance Image Ensembles. In: Lai, SH., Lepetit, V., Nishino, K., Sato, Y. (eds) Computer Vision – ACCV 2016. ACCV 2016. Lecture Notes in Computer Science(), vol 10114. Springer, Cham. https://doi.org/10.1007/978-3-319-54190-7_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-54190-7_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-54189-1

  • Online ISBN: 978-3-319-54190-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation