Spatiotemporal Optimization for Rolling Shutter Camera Pose Interpolation

  • Philippe-Antoine GohardEmail author
  • Bertrand Vandeportaele
  • Michel Devy
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 983)


Rolling Shutter cameras are predominant in the tablet and smart-phone market due to their low cost and small size. However, these cameras require specific geometric models when either the camera or the scene is in motion to account for the sequential exposure of the different lines of the image. This paper proposes to improve a state-of-the-art model for RS cameras through the use of Non Uniformly Time-Sampled B-splines. This allows to interpolate the pose of the camera while taking into account the varying dynamic of its motion, using higher density of control points where needed while keeping a low number of control points where the motion is smooth. Two methods are proposed to determine adequate distributions for the control points, using either an IMU sensor or an iterative reprojection error minimization. The non-uniform camera model is integrated into a Bundle Adjustment optimization which is able to converge even from a poor initial estimate. A routine of spatiotemporal optimization is presented in order to optimize both the spatial and temporal positions of the control points. Results on synthetic and real datasets are shown to prove the concepts and future works are introduced that should lead to the integration of our model in a SLAM algorithm.


Rolling Shutter Camera geometric model Bundle adjustment Simultaneous Localization and Mapping B-splines interpolation 


  1. 1.
    Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004). ISBN 0521540518CrossRefGoogle Scholar
  2. 2.
    Davison, A.J.: Real-time simultaneous localisation and mapping with a single camera. In: 9th IEEE International Conference on Computer Vision (ICCV 2003), 14–17 October 2003, Nice, France, pp. 1403–1410 (2003)Google Scholar
  3. 3.
    Klein, G., Murray, D.: Parallel tracking and mapping for small AR workspaces. In: Proceedings Sixth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2007), Nara, Japan (2007)Google Scholar
  4. 4.
    Mur-Artal, R., Montiel, J.M.M., Tardós, J.D.: ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Trans. Rob. 31, 1147–1163 (2015)CrossRefGoogle Scholar
  5. 5.
    Lovegrove, S., Patron-Perez, A., Sibley, G.: Spline fusion: a continuous-time representation for visual-inertial fusion with application to rolling shutter cameras. In: Proceedings of the British Machine Vision Conference. BMVA Press (2013)Google Scholar
  6. 6.
    Patron-Perez, A., Lovegrove, S., Sibley, G.: A spline-based trajectory representation for sensor fusion and rolling shutter cameras. Int. J. Comput. Vis. 113, 208–219 (2015)CrossRefGoogle Scholar
  7. 7.
    Vandeportaele, B., Gohard, P.A., Devy, M., Coudrin, B.: Pose interpolation for rolling shutter cameras using non uniformly time-sampled B-splines. In: Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, VISAPP, (VISIGRAPP 2017), pp. 286–293. INSTICC, SciTePress (2017)Google Scholar
  8. 8.
    Roussillon, C., et al.: RT-SLAM: a generic and real-time visual SLAM implementation. CoRR abs/1201.5450 (2012)Google Scholar
  9. 9.
    Gonzalez, A.: Localisation par vision multi-spectrale. Application aux systèmes embarqués, theses. INSA de Toulouse (2013)Google Scholar
  10. 10.
    Engel, J., Schöps, T., Cremers, D.: LSD-SLAM: large-scale direct monocular SLAM. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8690, pp. 834–849. Springer, Cham (2014). Scholar
  11. 11.
    Mouragnon, E., Lhuillier, M., Dhome, M., Dekeyser, F., Sayd, P.: Real time localization and 3D reconstruction. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), vol. 1, pp. 363–370 (2006)Google Scholar
  12. 12.
    Strasdat, H., Montiel, J., Davison, A.J.: Real-time monocular slam: why filter? In: 2010 IEEE International Conference on Robotics and Automation (ICRA), pp. 2657–2664. IEEE (2010)Google Scholar
  13. 13.
    Klein, G., Murray, D.: Parallel tracking and mapping on a camera phone. In: Proceedings Eigth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2009), Orlando (2009)Google Scholar
  14. 14.
    Hedborg, J., Ringaby, E., Forssén, P.E., Felsberg, M.: Structure and motion estimation from rolling shutter video. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 17–23 (2011)Google Scholar
  15. 15.
    Hedborg, J., Forssén, P.E., Felsberg, M., Ringaby, E.: Rolling shutter bundle adjustment. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1434–1441 (2012)Google Scholar
  16. 16.
    Furgale, P., Barfoot, T.D., Sibley, G.: Continuous-time batch estimation using temporal basis functions. In: 2012 IEEE International Conference on Robotics and Automation (ICRA), pp. 2088–2095 (2012)Google Scholar
  17. 17.
    Kim, M.J., Kim, M.S., Shin, S.Y.: A general construction scheme for unit quaternion curves with simple high order derivatives. In: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques - SIGGRAPH 1995, pp. 369–376 (1995)Google Scholar
  18. 18.
    de Boor, C.: On calculating with B-splines. J. Approx. Theory 6, 50–62 (1972)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Yang, H., Yue, W., He, Y., Huang, H., Xia, H.: The deduction of coefficient matrix for cubic non-uniform B-spline curves. In: 2009 First International Workshop on Education Technology and Computer Science, pp. 607–609 (2009)Google Scholar
  20. 20.
    Qin, K.: General matrix representations for B-splines. Vis. Comput. 16, 177–186 (2000)CrossRefGoogle Scholar
  21. 21.
    Li, M., Kim, B., Mourikis, A.I.: Real-time motion estimation on a cellphone using inertial sensing and a rolling-shutter camera. In: Proceedings of the IEEE International Conference on Robotics and Automation, Karlsruhe, Germany, pp. 4697–4704 (2013)Google Scholar
  22. 22.
    Kuemmerle, R., Grisetti, G., Strasdat, H., Konolige, K., Burgard, W.: g2o: a general framework for graph optimization. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, pp. 3607–3613 (2011)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Philippe-Antoine Gohard
    • 1
    • 2
    Email author
  • Bertrand Vandeportaele
    • 1
  • Michel Devy
    • 1
  1. 1.LAAS-CNRS, Toulouse University, CNRS, UPSToulouseFrance
  2. 2.InnersenseRamonville-Saint-AgneFrance

Personalised recommendations