Incremental Non-Rigid Structure-from-Motion with Unknown Focal Length

  • Thomas ProbstEmail author
  • Danda Pani Paudel
  • Ajad Chhatkuli
  • Luc Van Gool
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11217)


The perspective camera and the isometric surface prior have recently gathered increased attention for Non-Rigid Structure-from-Motion (NRSfM). Despite the recent progress, several challenges remain, particularly the computational complexity and the unknown camera focal length. In this paper we present a method for incremental Non-Rigid Structure-from-Motion (NRSfM) with the perspective camera model and the isometric surface prior with unknown focal length. In the template-based case, we provide a method to estimate four parameters of the camera intrinsics. For the template-less scenario of NRSfM, we propose a method to upgrade reconstructions obtained for one focal length to another based on local rigidity and the so-called Maximum Depth Heuristics (MDH). On its basis we propose a method to simultaneously recover the focal length and the non-rigid shapes. We further solve the problem of incorporating a large number of points and adding more views in MDH-based NRSfM and efficiently solve them with Second-Order Cone Programming (SOCP). This does not require any shape initialization and produces results orders of times faster than many methods. We provide evaluations on standard sequences with ground-truth and qualitative reconstructions on challenging YouTube videos. These evaluations show that our method performs better in both speed and accuracy than the state of the art.



Research was funded by the EU’s Horizon 2020 programme under grant No. 645331– EurEyeCase and grant No. 687757– REPLICATE, and the Swiss Commission for Technology and Innovation (CTI, Grant No. 26253.1 PFES-ES, EXASOLVED).

Supplementary material

474201_1_En_46_MOESM1_ESM.pdf (20.1 mb)
Supplementary material 1 (pdf 20578 KB)

Supplementary material 2 (mp4 21436 KB)


  1. 1.
    Longuet-Higgins, H.: A computer algorithm for reconstructing a scene from two projections. Nature 293, 133–135 (1981)CrossRefGoogle Scholar
  2. 2.
    Nistér, D.: An efficient solution to the five-point relative pose problem. IEEE Trans. Pattern Anal. Mach. Intell. 26(6), 756–777 (2004)CrossRefGoogle Scholar
  3. 3.
    Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University, New York Press (2004). ISBN 0521540518CrossRefGoogle Scholar
  4. 4.
    Agisoft PhotoScan: Agisoft PhotoScan User Manual Professional Edition, Version 1.2 (2017)Google Scholar
  5. 5.
    Autodesk ReCap: ReCap 360 - Advanced Workflows (2015)Google Scholar
  6. 6.
    Ji, P., Li, H., Dai, Y., Reid, I.: “Maximizing Rigidity” revisited: a convex programming approach for generic 3D shape reconstruction from multiple perspective views. In: ICCV (2017)Google Scholar
  7. 7.
    Dai, Y., Li, H., He, M.: A simple prior-free method for non-rigid structure-from-motion factorization. In: CVPR (2012)Google Scholar
  8. 8.
    Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3D shape from image streams. In: CVPR (2000)Google Scholar
  9. 9.
    Torresani, L., Hertzmann, A., Bregler, C.: Nonrigid structure-from-motion: estimating shape and motion with hierarchical priors. IEEE Trans. Pattern Anal. Mach. Intell. 30(5), 878–892 (2008)CrossRefGoogle Scholar
  10. 10.
    Del Bue, A.: A factorization approach to structure from motion with shape priors. In: CVPR (2008)Google Scholar
  11. 11.
    Garg, R., Roussos, A., Agapito, L.: Dense variational reconstruction of non-rigid surfaces from monocular video. In: CVPR (2013)Google Scholar
  12. 12.
    Fayad, J., Agapito, L., Del Bue, A.: Piecewise quadratic reconstruction of non-rigid surfaces from monocular sequences. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 297–310. Springer, Heidelberg (2010). Scholar
  13. 13.
    Agudo, A., Montiel, J., Agapito, L., Calvo, B.: Online dense non-rigid 3D shape and camera motion recovery. In: BMVC (2014)Google Scholar
  14. 14.
    Taylor, J., Jepson, A.D., Kutulakos, K.N.: Non-rigid structure from locally-rigid motion. In: CVPR (2010)Google Scholar
  15. 15.
    Bartoli, A., Pizarro, D., Collins, T.: A robust analytical solution to isometric shape-from-template with focal length calibration. In: ICCV (2013)Google Scholar
  16. 16.
    Ngo, T.D., Östlund, J.O., Fua, P.: Template-based monocular 3D shape recovery using Laplacian meshes. IEEE Trans. Pattern Anal. Mach. Intell. 38(1), 172–187 (2016)CrossRefGoogle Scholar
  17. 17.
    Chhatkuli, A., Pizarro, D., Bartoli, A., Collins, T.: A stable analytical framework for isometric shape-from-template by surface integration. IEEE Trans. Pattern Anal. Mach. Intell. 39(5), 833–850 (2017)CrossRefGoogle Scholar
  18. 18.
    Bartoli, A., Collins, T.: Template-based isometric deformable 3D reconstruction with sampling-based focal length self-calibration. In: CVPR (2013)Google Scholar
  19. 19.
    Chhatkuli, A., Pizarro, D., Collins, T., Bartoli, A.: Inextensible non-rigid shape-from-motion by second-order cone programming. In: CVPR (2016)Google Scholar
  20. 20.
    Perriollat, M., Hartley, R., Bartoli, A.: Monocular template-based reconstruction of inextensible surfaces. Int. J. Comput. Vis. 95(2), 124–137 (2011)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Salzmann, M., Fua, P.: Linear local models for monocular reconstruction of deformable surfaces. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 931–944 (2011)CrossRefGoogle Scholar
  22. 22.
    Kumar, S., Dai, Y., Li, H.: Monocular dense 3D reconstruction of a complex dynamic scene from two perspective frames. In: ICCV (2017)Google Scholar
  23. 23.
    Russell, C., Yu, R., Agapito, L.: Video pop-up: monocular 3D reconstruction of dynamic scenes. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 583–598. Springer, Cham (2014). Scholar
  24. 24.
    Bartoli, A., Gérard, Y., Chadebecq, F., Collins, T., Pizarro, D.: Shape-from-template. IEEE Trans. Pattern Anal. Mach. Intell. 37(10), 2099–2118 (2015)CrossRefGoogle Scholar
  25. 25.
    Vicente, S., Agapito, L.: Soft inextensibility constraints for template-free non-rigid reconstruction. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 426–440. Springer, Heidelberg (2012). Scholar
  26. 26.
    Chhatkuli, A., Pizarro, D., Bartoli, A.: Stable template-based isometric 3D reconstruction in all imaging conditions by linear least-squares. In: CVPR (2014)Google Scholar
  27. 27.
    Parashar, S., Pizarro, D., Bartoli, A.: Isometric non-rigid shape-from-motion in linear time. In: CVPR (2016)Google Scholar
  28. 28.
    Xiao, J., Kanade, T.: Uncalibrated perspective reconstruction of deformable structures. In: Tenth IEEE International Conference on Computer Vision (ICCV 2005), vol. 1, 2, pp. 1075–1082 (2005)Google Scholar
  29. 29.
    Salzmann, M., Hartley, R., Fua, P.: Convex optimization for deformable surface 3-D tracking. In: ICCV (2007)Google Scholar
  30. 30.
    Akhter, I., Sheikh, Y., Khan, S., Kanade, T.: Trajectory space: a dual representation for nonrigid structure from motion. IEEE TPAMI 33(7), 1442–1456 (2011)CrossRefGoogle Scholar
  31. 31.
    Nistér, D.: Untwisting a projective reconstruction. Int. J. Comput. Vis. 60(2), 165–183 (2004)CrossRefGoogle Scholar
  32. 32.
    Chandraker, M., Agarwal, S., Kahl, F., Nistér, D., Kriegman, D.: Autocalibration via rank-constrained estimation of the absolute quadric. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2007, pp. 1–8. IEEE (2007)Google Scholar
  33. 33.
    Dijkstra, E.W.: A note on two problems in connexion with graphs. Numer. Math. 1(1), 269–271 (1959)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Varol, A., Salzmann, M., Fua, P., Urtasun, R.: A constrained latent variable model. In: CVPR (2012)Google Scholar
  35. 35.
    Chhatkuli, A., Pizarro, D., Bartoli, A.: Non-rigid shape-from-motion for isometric surfaces using infinitesimal planarity. In: BMVC (2014)Google Scholar
  36. 36.
    White, R., Crane, K., Forsyth, D.: Capturing and animating occluded cloth. In: SIGGRAPH (2007)Google Scholar
  37. 37.
    Sundaram, N., Brox, T., Keutzer, K.: Dense point trajectories by GPU-accelerated large displacement optical flow. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 438–451. Springer, Heidelberg (2010). Scholar
  38. 38.
    Innmann, M., Zollhöfer, M., Nießner, M., Theobalt, C., Stamminger, M.: VolumeDeform: real-time volumetric non-rigid reconstruction. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 362–379. Springer, Cham (2016). Scholar
  39. 39.
    Gotardo, P.F., Martinez, A.M.: Computing smooth time trajectories for camera and deformable shape in structure from motion with occlusion. IEEE Trans. Pattern Anal. Mach. Intell. 33(10), 2051–2065 (2011)CrossRefGoogle Scholar
  40. 40.
    Chhatkuli, A., Pizarro, D., Collins, T., Bartoli, A.: Inextensible non-rigid structure-from-motion by second-order cone programming. IEEE Trans. Pattern Anal. Mach. Intell. 1(99), PP (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Thomas Probst
    • 1
    Email author
  • Danda Pani Paudel
    • 1
  • Ajad Chhatkuli
    • 1
  • Luc Van Gool
    • 1
    • 2
  1. 1.Computer Vision Lab, ETH ZürichZürichSwitzerland
  2. 2.VISICS, ESAT/PSIKU LeuvenLeuvenBelgium

Personalised recommendations