Journal of Intelligent & Robotic Systems

, Volume 95, Issue 2, pp 731–743 | Cite as

Nonlinear Model Predictive Visual Path Following Control to Autonomous Mobile Robots

  • Tiago T. RibeiroEmail author
  • André G. S. Conceição


This paper proposes a novel approach to the visual path following problem based on Nonlinear Model Predictive Control. Simplified visual features are extracted from the path to be followed. Then, aiming to calculate the control actions directly from the image plane, a regulatory model is obtained in the optimal control problem scope. For this purpose a Serret-Frenet system is placed in the center of camera’s field of view and the optimal control actions generate velocity references to an inner loop embedded in the robot. Stability issues are handled through a classical method and a new approach based on constraints relaxation is proposed in order to guarantee feasibility. Experimental results with a nonholonomic platform illustrate the performance of the proposed control scheme.


Path-following Visual control Nonlinear model predictive control Autonomous robots 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


Supplementary material

(MP4 32.3 MB)

(MP4 31.6 MB)

(MP4 31.4 MB)

(MP4 15.9 MB)

(MP4 15.8 MB)

(MP4 15.8 MB)

(MP4 4.14 MB)

(MP4 6.26 MB)

(MP4 9.20 MB)


  1. 1.
    Efraim, H., Arogeti, S., Shapiro, A., Weiss, G.: Vision based output feedback control of micro aerial vehicles in indoor environments. J. Intell. Robot. Syst. 87(1), 169–186 (2017)CrossRefGoogle Scholar
  2. 2.
    Kucukyildiz, G., Ocak, H., Karakaya, S., Sayli, O.: Design and implementation of a multi sensor based brain computer interface for a robotic wheelchair. J. Intell. Robot. Syst. 87(2), 247–263 (2017)CrossRefGoogle Scholar
  3. 3.
    Ji, P., Song, A., Xiong, P., Yi, P., Xu, X., Li, H.: Egocentric-vision based hand posture control system for reconnaissance robots. J. Intell. Robot. Syst. 87(3), 583–599 (2017)CrossRefGoogle Scholar
  4. 4.
    Chaumette, F., Hutchinson, S.: Visual servo control. i. Basic approaches. IEEE Robot. Autom. Mag. 13(4), 82–90 (2006)CrossRefGoogle Scholar
  5. 5.
    Corke, P.: Robotics, Vision and Control: Fundamental Algorithms in MATLAB. Springer Tracts in Advanced Robotics. Springer, Berlin (2011)CrossRefzbMATHGoogle Scholar
  6. 6.
    Zhao, Y.-M., Xie, W.-F., Liu, S., Wang, T.: Neural network-based image moments for robotic visual servoing. J. Intell. Robot. Syst. 78(2), 239–256 (2015)CrossRefGoogle Scholar
  7. 7.
    Araar, O., Aouf, N., Vitanov, I.: Vision based autonomous landing of multirotor uav on moving platform. J. Intell. Robot. Syst. 85(2), 369–384 (2017)CrossRefGoogle Scholar
  8. 8.
    Kanellakis, C., Nikolakopoulos, G.: Survey on computer vision for uavs: current developments and trends. J. Intell. Robot. Syst. 87(1), 141–168 (2017)CrossRefGoogle Scholar
  9. 9.
    Frezza, R., Soatto, S., Picci, G.: Visual path following by recursive spline updating. In: Proceedings of the 36th IEEE Conference on Decision and Control, 1997, vol. 2, pp. 1130–1134 (1997)Google Scholar
  10. 10.
    Diosi, A., Remazeilles, A., Segvic, S., Chaumette, F.: Outdoor visual path following experiments. In: 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4265–4270 (2007)Google Scholar
  11. 11.
    Delfín, J., Becerra, H.M., Arechavaleta, G.: Visual path following using a sequence of target images and smooth robot velocities for humanoid navigation. In: 2014 14th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pp. 354–359 (2014)Google Scholar
  12. 12.
    Bertozzi, M., Broggi, A., Fascioli, A.: Vision-based intelligent vehicles: state of the art and perspectives. Robot. Auton. Syst. 32(1), 1–16 (2000)CrossRefzbMATHGoogle Scholar
  13. 13.
    Cherubini, A., Chaumette, F., Oriolo, G.: An Image-Based visual servoing scheme for following paths with nonholonomic mobile robots. In: International Conference on Control, Automation, Robotics and Vision, ICARCV 2008, pp. 108–113, Hanoi, Vietnam, France (2008)Google Scholar
  14. 14.
    de Lima, D.A., Victorino, A.C.: A visual servoing approach for road lane following with obstacle avoidance. In: 2014 IEEE 17th International Conference on Intelligent Transportation Systems (ITSC), pp. 412–417 (2014)Google Scholar
  15. 15.
    Sabatta, D.: A vision-based error metric for path following control. 2014 PRASA, RobMech and AfLaT International Joint Symposium (PRASA/RobMech/AfLaT 2014) (2014)Google Scholar
  16. 16.
    Mehrez, M.W., Mann, G.K.I., Gosine, R.G.: An optimization based approach for relative localization and relative tracking control in multi-robot systems. J. Intell. Robot. Syst. 85(2), 385–408 (2017)CrossRefGoogle Scholar
  17. 17.
    Rybus, T., Seweryn, K., Sasiadek, J.Z.: Control system for free-floating space manipulator based on nonlinear model predictive control (nmpc). J. Intell. Robot. Syst. 85(3), 491–509 (2017)CrossRefGoogle Scholar
  18. 18.
    Cao, G., Lai, E.M.-K., Alam, F.: Gaussian process model predictive control of an unmanned quadrotor. J. Intell. Robot. Syst. 88, 147–162 (2017)Google Scholar
  19. 19.
    Masri, M.A., Dbeis, S., Saba, M.A.: Autolanding a power-off uav using on-line optimization and slip maneuvers. J. Intell. Robot. Syst. 86(2), 255–276 (2017)CrossRefGoogle Scholar
  20. 20.
    Li, Z., Yang, C., Su, C.Y., Deng, J., Zhang, W.: Vision-based model predictive control for steering of a nonholonomic mobile robot. IEEE Trans. Control Syst. Technol. 24(2), 553–564 (2016)Google Scholar
  21. 21.
    Faulwasser, T., Findeisen, R.: Nonlinear model predictive control for constrained output path following. IEEE Trans. Autom. Control 61(4), 1026–1039 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Płaskonka, J.: Different kinematic path following controllers for a wheeled mobile robot of (2,0) type. J. Intell. Robot. Syst. 77(3), 481–498 (2015)CrossRefGoogle Scholar
  23. 23.
    Coulaud, J.B., Campion, G., Bastin, G., De Wan, M.: Stability analysis of a vision-based control design for an autonomous mobile robot. IEEE Trans. Robot. 22(5), 1062–1069 (2006)CrossRefGoogle Scholar
  24. 24.
    Mayne, D.Q., Rawlings, J.B., Rao, C.V., Scokaert, P.O.M.: Constrained model predictive control: stability and optimality. Automatica 36(6), 789–814 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Grüne, L., Pannek, J.: Nonlinear Model Predictive Control: Theory and Algorithms. Communications and Control Engineering. 1st edn. Springer, Berlin (2011)CrossRefzbMATHGoogle Scholar
  26. 26.
    Scokaert, P.O.M., Mayne, D.Q., Rawlings, J.B.: Suboptimal model predictive control (feasibility implies stability). IEEE Trans. Autom. Control 44(3), 648–654 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    de Oliveira, N., Biegler, L.T.: Constraint handing and stability properties of model-predictive control. AIChE J. 40(7), 1138–1155 (1994)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Microsoft. MIcrosoft LifeCam HD-3000 Product Guide. Accessed 18 Aug 2017
  29. 29.
    Park, E.J.: Exploring LEGO Mindstorms EV3: Tools and Techniques for Building and Programming Robots. 1st edn. Wiley, New York (2014)Google Scholar
  30. 30.
    Spellucci, P.: An sqp method for general nonlinear programs using only equality constrained subproblems. In: Mathematical Programming, vol. 82, pp. 413–448 (1998)Google Scholar

Copyright information

© Springer Nature B.V. 2018

Authors and Affiliations

  1. 1.LaR - Robotics Laboratory, Department of Electrical EngineeringFederal University of BahiaSalvadorBrazil

Personalised recommendations