Advertisement

International Journal of Computer Vision

, Volume 122, Issue 2, pp 371–387 | Cite as

Combining Local-Physical and Global-Statistical Models for Sequential Deformable Shape from Motion

Article

Abstract

In this paper, we simultaneously estimate camera pose and non-rigid 3D shape from a monocular video, using a sequential solution that combines local and global representations. We model the object as an ensemble of particles, each ruled by the linear equation of the Newton’s second law of motion. This dynamic model is incorporated into a bundle adjustment framework, in combination with simple regularization components that ensure temporal and spatial consistency. The resulting approach allows to sequentially estimate shape and camera poses, while progressively learning a global low-rank model of the shape that is fed back into the optimization scheme, introducing thus, global constraints. The overall combination of local (physical) and global (statistical) constraints yields a solution that is both efficient and robust to several artifacts such as noisy and missing data or sudden camera motions, without requiring any training data at all. Validation is done in a variety of real application domains, including articulated and non-rigid motion, both for continuous and discontinuous shapes. Our on-line methodology yields significantly more accurate reconstructions than competing sequential approaches, being even comparable to the more computationally demanding batch methods.

Keywords

Sequential non-rigid structure from motion Particle dynamics Bundle adjustment Low-rank models 

Notes

Acknowledgements

We would like to thank the anonymous reviewers for their insights and comments that have significantly contributed to improving this manuscript. This work has been partially supported by the Spanish Ministry of Science and Innovation under Project RobInstruct TIN2014-58178-R; by a scholarship FPU12/04886 from the Spanish MECD; and by the ERA-net CHISTERA Projects VISEN PCIN-2013-047 and I-DRESS PCIN-2015-147. The authors also thank Chris Russell, Lourdes Agapito and Paulo Gotardo for making their data available.

References

  1. Agudo, A., & Moreno-Noguer, F. (2015). Simultaneous pose and non-rigid shape with particle dynamics. In Conference on computer vision and pattern recognition, pp. 2179–2187Google Scholar
  2. Agudo, A., Calvo, B., Montiel, & J. M. M. (2012). Finite element based sequential Bayesian non-rigid structure from motion. In Conference on computer vision and pattern recognition, pp. 1418–1425Google Scholar
  3. Agudo, A., Agapito, L., Calvo, B., & Montiel, J. M. M. (2014a). Good vibrations: A modal analysis approach for sequential non-rigid structure from motion. In Conference on computer vision and pattern recognition, pp. 1558–1565Google Scholar
  4. Agudo, A., Montiel, J. M. M., Agapito, L., & Calvo, B. (2014b). Online dense non-rigid 3D shape and camera motion recovery. In British machine vision conference Google Scholar
  5. Agudo, A., Moreno-Noguer, F., Calvo, B., & Montiel, J. M. M. (2016). Sequential non-rigid structure from motion using physical priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(5), 979–994.CrossRefGoogle Scholar
  6. Akhter, I., Sheikh, Y., Khan, S., & Kanade, T. (2008). Non-rigid structure from motion in trajectory space. In Neural information processing systems, pp. 41–48Google Scholar
  7. Baraff, D. (1989). Analytical methods for dynamic simulation of non-penetrating rigid bodies. In Conference on computer graphics and interactive techniques, pp. 223–232Google Scholar
  8. Bartoli, A., Gay-Bellile, V., Castellani, U., Peyras, J., Olsen, S., & Sayd, P. (2008). Coarse-to-fine low-rank structure-from-motion. In Conference on computer vision and pattern recognition, pp. 1–8Google Scholar
  9. Brand, M. (2001). Morphable 3D models from video. In Conference on computer vision and pattern recognition, pp. 456–463Google Scholar
  10. Bregler, C., Hertzmann, A., & Biermann, H. (2000). Recovering non-rigid 3D shape from image streams. In Conference on computer vision and pattern recognition, pp. 690–696Google Scholar
  11. Brubaker, M., Sigal, L., & Fleet, D. (2009). Estimating contact dynamics. In International conference on computer vision, pp. 2389–2396Google Scholar
  12. Chhatkuli, A., Pizarro, D., & Bartoli, A. (2014). Non-rigid shape-from-motion for isometric surfaces using infinitesimal planarity. In British machine vision conference Google Scholar
  13. Dai, Y., Li, H., & He, M. (2012). A simple prior-free method for non-rigid structure from motion factorization. In Conference on computer vision and pattern recognition, pp. 2018–2025Google Scholar
  14. Del Bue, A., Llado, X., & Agapito, L. (2006). Non-rigid metric shape and motion recovery from uncalibrated images using priors. In Conference on computer vision and pattern recognition, pp. 1191–1198Google Scholar
  15. Fayad, J., Agapito, L., & Del Bue, A. (2010). Piecewise quadratic reconstruction of non-rigid surfaces from monocular sequences. In European conference on computer vision, pp. 297–310Google Scholar
  16. Garg, R., Roussos, A., & Agapito, L. (2013). Dense variational reconstruction of non-rigid surfaces from monocular video. In Conference on computer vision and pattern recognition, pp. 1272–1279Google Scholar
  17. Gotardo, P. F. U., & Martínez, A. M. (2011a). Kernel non-rigid structure from motion. In International conference on computer vision, pp. 802–809Google Scholar
  18. Gotardo, P. F. U., & Martínez, A. M. (2011b). Non-rigid structure from motion with complementary rank-3 spaces. In Conference on computer vision and pattern recognition, pp. 3065–3072Google Scholar
  19. Koh, W., Narain, R., & O’Brien, J. F. (2014). View-dependent adaptive cloth simulation. In ACM SIGGRAPH/Eurographics symposium on computer animation, pp. 159–166Google Scholar
  20. Lee, M., Cho, J., Choi, C. H., & Oh, S. (2013). Procrustean normal distribution for non-rigid structure from motion. In Conference on computer vision and pattern recognition, pp. 1280–1287Google Scholar
  21. Lim, J., Frahm, J., & Pollefeys, M. (2011). Online environment mapping. In Conference on computer vision and pattern recognition, pp. 3489–3496Google Scholar
  22. Ma, Y., Kosecka, J., & Sastry, S. (1999). Optimization criteria and geometric algorithms for motion and structure estimation. International Journal on Computer Vision, 44(3), 219–249.CrossRefMATHGoogle Scholar
  23. Maier-Hein, L., Groch, A., Bartoli, A., Bodenstedt, S., Boissonnat, G., Chang, P. L., et al. (2014). Comparative validation of single-shot optical techniques for laparoscopic 3D surface reconstruction. IEEE Transactions on Medical Imaging, 33(10), 1913–1930.CrossRefGoogle Scholar
  24. Marques, M., & Costeira, J. (2008). Optimal shape from estimation with missing and degenerate data. In Workshop on motion and video computing, pp. 1–6Google Scholar
  25. Metaxas, D., & Terzopoulos, D. (1993). Shape and nonrigid motion estimation through physics-based synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(6), 580–591.CrossRefGoogle Scholar
  26. Moreno-Noguer, F., & Porta, J. M. (2011). Probabilistic simultaneous pose and non-rigid shape recovery. In Conference on computer vision and pattern recognition, pp. 1289–1296Google Scholar
  27. Newcome, R., & Davison, A. J. (2010). Live dense reconstruction with a single moving camera. In Conference on computer vision and pattern recognition, pp. 1498–1505Google Scholar
  28. Paladini, M., Del Bue, A., Stosic, M., Dodig, M., Xavier, J., & Agapito, L. (2009). Factorization for non-rigid and articulated structure using metric projections. In Conference on computer vision and pattern recognition, pp. 2898–2905Google Scholar
  29. Paladini, M., Bartoli, A., & Agapito, L. (2010). Sequential non rigid structure from motion with the 3D implicit low rank shape model. In European conference on computer vision, pp. 15–28Google Scholar
  30. Park, H. S., Shiratori, T., Matthews, I., & Sheikh, Y. (2010). 3D reconstruction of a moving point from a series of 2D projections. In European conference on computer vision, pp. 158–171Google Scholar
  31. Popovic, Z., & Witkin, A. (1999). Physically based motion transformation. In Conference on computer graphics and interactive techniques, pp. 11–20Google Scholar
  32. Russell, C., Fayad, J., & Agapito, L. (2011). Energy based multiple model fitting for non-rigid structure from motion. In Conference on computer vision and pattern recognition, pp. 3009–3016Google Scholar
  33. Russell, C., Yu, R., & Agapito, L. (2014). Video pop-up: Monocular 3D reconstruction of dynamic scenes. In European conference on computer vision, pp. 583–598Google Scholar
  34. Salzmann, M., & Urtasun, R. (2011). Physically-based motion models for 3D tracking: A convex formulation. In International conference on computer vision, pp. 2064–2071Google Scholar
  35. Shaji, A., & Chandran, S. (2008). Riemannian manifold optimisation for non-rigid structure from motion. In Workshop on non-rigid shape analysis and deformable image alignment, pp. 1–6Google Scholar
  36. Tao, L., Mein, S. J., Quan, W., & Matuszewski, B. J. (2013). Recursive non-rigid structure from motion with online learned shape prior. Computer Vision and Image Understanding, 117(10), 1287–1298.CrossRefGoogle Scholar
  37. Taylor, J., Jepson, A.D., & Kutulakos, K.N. (2010). Non-rigid structure from locally-rigid motion. In Conference on computer vision and pattern recognition, pp. 2761–2768Google Scholar
  38. Tomasi, C., & Kanade, T. (1992). Shape and motion from image streams under orthography: A factorization approach. International Journal on Computer Vision, 9(2), 137–154.CrossRefGoogle Scholar
  39. Torresani, L., Hertzmann, A., & Bregler, C. (2008). Nonrigid structure-from-motion: Estimating shape and motion with hierarchical priors. Transactions on Pattern Analysis and Machine Intelligence, 30(5), 878–892.CrossRefGoogle Scholar
  40. Valmadre, J., & Lucey, S. (2012). General trajectory prior for non-rigid reconstruction. In Conference on computer vision and pattern recognition, pp. 1394–1401Google Scholar
  41. Varol, A., Salzmann, M., Tola, E., & Fua, P. (2009). Template-free monocular reconstruction of deformable surfaces. In International conference on computer vision, pp. 1811–1818Google Scholar
  42. Vondrak, M., Sigal, L., & Jenkins, O.C. (2008). Physical simulation for probabilistic motion tracking. In Conference on computer vision and pattern recognition, pp. 1–8Google Scholar
  43. Xiao, J., Chai, J., & Kanade, T. (2006). A closed-form solution to non-rigid shape and motion. International Journal on Computer Vision, 67(2), 233–246.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Institut de Robòtica i Informàtica Industrial (CSIC-UPC)BarcelonaSpain

Personalised recommendations