Visual Servoing via Nonlinear Predictive Control

  • Guillaume Allibert
  • Estelle Courtial
  • Francçois Chaumette
Part of the Lecture Notes in Control and Information Sciences book series (LNCIS, volume 401)


In this chapter, image-based visual servoing is addressed via nonlinear model predictive control. The visual servoing task is formulated into a nonlinear optimization problem in the image plane. The proposed approach, named visual predictive control, can easily and explicitly take into account 2D and 3D constraints. Furthermore, the image prediction over a finite prediction horizon plays a crucial role for large displacements. This image prediction is obtained thanks to a model. The choice of this model is discussed. A nonlinear global model and a local model based on the interaction matrix are considered. Advantages and drawbacks of both models are pointed out. Finally, simulations obtained with a 6 degrees of freedom (DOF) free-flying camera highlight the capabilities and the efficiency of the proposed approach by a comparison with the classical image-based visual servoing.


Visual Feature Visual Servoing Prediction Horizon Camera Frame Constraint Handling 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Allibert, G., Courtial, E., Touré, Y.: Visual predictive control. In: IFAC Workshop on Nonlinear Predictive Control for Fast Systems, Grenoble, France (2006)Google Scholar
  2. 2.
    Allibert, G., Courtial, E., Touré, Y.: Real-time visual predictive controller for image-based trajectory tracking of mobile robot. In: 17th IFAC World Congress, Seoul, Korea (2008)Google Scholar
  3. 3.
    Allibert, G., Courtial, E., Touré, Y.: Visual predictive control for manipulators with catadioptric camera. In: IEEE Int. Conf. on Robotics and Automation, Pasadena, USA (2008)Google Scholar
  4. 4.
    Chaumette, F.: Potential problems of stability and convergence in image-based and position-based visual servoing. In: Kriegman, D., Hager, G., Morse, A. (eds.) The Confluence of Vision and Control. LNCIS, vol. 237, pp. 66–78. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  5. 5.
    Chaumette, F., Hutchinson, S.: Visual servo control, part i: Basic approaches. IEEE Robotics and Automation Magazine 13(4), 82–90 (2006)CrossRefGoogle Scholar
  6. 6.
    Chaumette, F., Hutchinson, S.: Visual servo control, part ii: Advanced approaches. IEEE Robotics and Automation Magazine 14(1), 109–118 (2007)CrossRefGoogle Scholar
  7. 7.
    Chesi, G.: Visual servoing path planning via homogeneous forms and lmi optimizations. IEEE Trans. Robotics and Automation 25(2), 281–291 (2009)Google Scholar
  8. 8.
    Chong, E., Zak, S.H.: An Introduction to Optimization, 2nd edn. John Wiley & Sons Inc., Chichester (2001)zbMATHGoogle Scholar
  9. 9.
    Danès, P., Bellot, D.: Towards an lmi approach to multicriteria visual servoing in robotics. European Journal of Control 12(1), 86–110 (2006)CrossRefMathSciNetGoogle Scholar
  10. 10.
    Durola, S., Danès, P., Coutinho, D., Courdesses, M.: Rational systems and matrix inequalities to the multicriteria analysis of visual servos. In: IEEE Int. Conf. on Robotics and Automation, Kobe, Japan (2009)Google Scholar
  11. 11.
    Folio, D., Cadenat, V.: Dealing with visual features loss during a vision-based task for a mobile robot. International Journal of Optomechatronics 2(3), 185–204 (2008)CrossRefGoogle Scholar
  12. 12.
    Gangloff, J., De Mathelin, M.: Visual servoing of a 6 dof manipulator for unknown 3-d profile following. IEEE Trans. Robotics and Automation 18(4), 511–520 (2002)CrossRefGoogle Scholar
  13. 13.
    Ginhoux, R., Gangloff, J., De Mathelin, M., Soler, M., Sanchez, L.: Active filtering of physiological motion in robotized surgery using predictive control. IEEE Trans. Robotics and Automation 21(1), 67–79 (2005)Google Scholar
  14. 14.
    Hashimoto, K., Kimura, H.: LQ optimal and nonlinear approaches to visual servoing. In: Hashimoto, K. (ed.) Visual Servoing, Robotics and Automated Systems, pp. 165–198. World Scientific, Singapore (1993)Google Scholar
  15. 15.
    Hutchinson, S., Hager, G.D., Corke, P.: A tutorial on visual servo control. IEEE Trans. Robotics and Automation 12(5), 651–671 (1996)CrossRefGoogle Scholar
  16. 16.
    Kazemi, M., Gupta, K., Mehrandezh, M.: Global path planning for robust visual servoing in complex environments. In: IEEE Int. Conf. on Robotics and Automation, Kobe, Japan (2009)Google Scholar
  17. 17.
    Mahony, R., Corke, P., Chaumette, F.: Choice of image features for depth-axis control in image-based visual servo control. In: IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, Lausanne, Switzerland (2002)Google Scholar
  18. 18.
    Malis E.: Improving vision-based control using efficient second-order minimization techniques. In: IEEE Int. Conf. on Robotics and Automation, New Orleans, LA, USA (2004)Google Scholar
  19. 19.
    Mezouar, Y., Chaumette, F.: Optimal camera trajectory with image-based control. Int. Journal of Robotics Research 22(10), 781–804 (2003)CrossRefGoogle Scholar
  20. 20.
    Morari, M., Zafiriou, E.: Robust Control. Dunod (1983)Google Scholar
  21. 21.
    Nasisi, O., Carelli, R.: Adaptive servo visual robot control. Robotics and Autonomous Systems 43(1), 51–78 (2003)CrossRefGoogle Scholar
  22. 22.
    Papanikolopoulos, N., Khosla, P., Kanade, T.: Visual tracking of a moving target by a camera mounted on a robot: A combination of vision and control. IEEE Trans. Robotics and Automation 9(1), 14–35 (1993)CrossRefGoogle Scholar
  23. 23.
    Sauvée, M., Poignet, P., Dombre, E., Courtial, E.: Image based visual servoing through nonlinear model predictive control. In: 45th IEEE CDC, San Diego, CA, pp. 1776–1781 (2006)Google Scholar
  24. 24.
    Schramm, F., Morel, G.: Ensuring visibility in calibration-free path planning for image-based visual servoing. IEEE Trans. Robotics and Automation 22(4), 848–854 (2006)Google Scholar

Copyright information

© Springer London 2010

Authors and Affiliations

  • Guillaume Allibert
    • 1
  • Estelle Courtial
    • 1
  • Francçois Chaumette
    • 2
  1. 1.Institut PRISMEPolytech’OrleansOrleansFrance
  2. 2.INRIARennesFrance

Personalised recommendations