Data-Based Approximate Policy Iteration for Optimal Course-Keeping Control of Marine Surface Vessels

  • Yuming Bai
  • Yifan LiuEmail author
  • Qihe Shan
  • Tieshan Li
  • Yuzhen Lu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11555)


In this paper, the data-based approximate policy iteration (API) method is used for optimal course-keeping control with unknown ship model. When we deal with the nonlinear optimal control problem, the Hamilton Jacobi Bellman (HJB) equation, which is difficult to be solved analytically, needs to be tackled. Furthermore, because of numerous parameters to be determined and unknown nonlinear terms, it is usually difficult to establish the accurate mathematical model for ships. In order to overcome these difficulties, the API method, which can solve the problem of model-free system, is introduced for optimal course-keeping control of marine surface vessels. And the asymptotic stability of the closed-loop system can be guaranteed via Lyapunov analysis. Finally, a numerical example is provided to demonstrate the effectiveness of the control scheme.


Optimal course-keeping control Unknown ship model Neural network Approximate policy iteration 



This work was supported in part by the National Natural Science Foundation of China (Grant Nos. 61751202, 61751205, 61572540, 61803064); the Natural Science Foundation of Liaoning Province (20180550082, 20170540093); the Science & Technology Innovation Founds of Dalian (2018J11Y022); the Fundamental Research Funds for the Central Universities (3132018306); and the Fundamental Research Funds of Dalian Maritime University (3132018157).


  1. 1.
    Huang, Z., Zhuo, Y., Li, T.: Optimal tracking control of the ship course system with partially-unknown dynamics. In: 4th International Conference on Information, Cybernetics and Computational Social Systems (ICCSS), pp. 631–635. IEEE (2017)Google Scholar
  2. 2.
    Du, J., Guo, C., Yu, S., Zhang, X.: Adaptive autopilot design of time-varying uncertain ships with completely unknown control coefficient. IEEE J. Ocean. Eng. 32(2), 346–352 (2007)Google Scholar
  3. 3.
    Wang, Y., Li, T., Philip Chen, C.L.: Adaptive terminal sliding mode control for formations of underactuated vessels. In: Sun, F., Liu, H., Hu, D. (eds.) ICCSIP 2016. CCIS, vol. 710, pp. 27–35. Springer, Singapore (2017). Scholar
  4. 4.
    Chen, M., Ge, S.S., How, B.V.E., Choo, Y.S.: Robust adaptive position mooring control for marine vessels. IEEE Trans. Control Syst. Technol. 21(2), 395–409 (2017)Google Scholar
  5. 5.
    Wang, N., Er, M.J.: Direct adaptive fuzzy tracking control of marine vehicles with fully unknown parametric dynamics and uncertainties. IEEE Trans. Control Syst. Technol. 24(5), 1845–1852 (2016)Google Scholar
  6. 6.
    Liu, S., Fang, L., Ge, Y.-M., Fu, H.-X.: GA-PID adaptive control research for ship course-keeping system. J. Syst. Simul. 19(16), 3783–3786 (2007)Google Scholar
  7. 7.
    Zhang, K., Li, T., Li, Z., Philip Chen, C.L.: Neural network based dynamic surface second order sliding mode control for AUVs. In: Sun, F., Liu, H., Hu, D. (eds.) ICCSIP 2016. CCIS, vol. 710, pp. 417–424. Springer, Singapore (2017). Scholar
  8. 8.
    Liu, C., Li, T., Chen, C.L.P.: Leader-following consensus control for multiple marine vessels based on dynamic surface control and neural network. In: 4th International Conference on Information, Cybernetics and Computational Social Systems (ICCSS), pp. 160–165. IEEE (2017)Google Scholar
  9. 9.
    Lewis, F.L., Vrabie, D., Syrmos, V.L.: Optimal Control. Wiley, Hoboken (2012)Google Scholar
  10. 10.
    Peng, Z., Wang, J., Wang, D.: Distributed maneuvering of autonomous surface vehicles based on neurodynamic optimization and fuzzy approximation. IEEE Trans. Control Syst. Technol. 26(3), 1083–1090 (2018)Google Scholar
  11. 11.
    Ejaz, M., Chen, M.: Optimal backstepping control for a ship using firefly optimization algorithm and disturbance observer. Trans. Inst. Meas. Control 40(6), 1983–1998 (2018)Google Scholar
  12. 12.
    Zhang, J., Sun, T., Liu, Z.: Robust model predictive control for path-following of underactuated surface vessels with roll constraints. Ocean Eng. 143, 125–132 (2017)Google Scholar
  13. 13.
    Khaled, N., Chalhoub, N.G.: A dynamic model and a robust controller for a fully-actuated marine surface vessel. J. Vib. Control 17(6), 801–812 (2011)Google Scholar
  14. 14.
    Yu, R., Zhu, Q., Xia, G., Liu, Z.: Sliding mode tracking control of an underactuated surface vessel. IET Control Theory Appl. 6(3), 461–466 (2012)Google Scholar
  15. 15.
    Zhang, Y., Li, S., Liu, X.: Adaptive near-optimal control of uncertain systems with application to underactuated surface vessels. IEEE Trans. Control Syst. Technol. 26(4), 1204–1218 (2018)Google Scholar
  16. 16.
    Zhang, H., Zhang, X., Luo, Y., Yang, J.: An overview of research on adaptive dynamic programming. Acta Automatica Sin. 39(4), 303–311 (2013)Google Scholar
  17. 17.
    Murray, J.J., Cox, C.J., Lendaris, G.G., Saeks, R.: Adaptive dynamic programming. IEEE Trans. Syst. Man Cybern. Part C-Appl. Rev. 32(2), 140–153 (2002)Google Scholar
  18. 18.
    Liu, D., Wei, Q.: Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems. IEEE Trans. Neural Netw. Learn. Syst. 25(3), 621–634 (2014)Google Scholar
  19. 19.
    Sun, H., Hou, Z.: A data-driven adaptive iterative learning predictive control for a class of discrete-time nonlinear systems. In: Proceedings of the 29th Chinese Control Conference (CCC), pp. 5871–5876. IEEE (2010)Google Scholar
  20. 20.
    Luo, B., Wu, H., Huang, T., Liu, D.: Data-based approximate policy iteration for affine nonlinear continuous-time optimal control design. Automatica 50(12), 3281–3290 (2014)Google Scholar
  21. 21.
    Beard, R.W., Saridis, G.N., Wen, J.T.: Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation. Automatica 33(12), 2159–2177 (1997)Google Scholar
  22. 22.
    Peng, Z., Wang, J.: Output-feedback path-following control of autonomous underwater vehicles based on an extended state observer and projection neural networks. IEEE Trans. Syst. Man Cybern. Syst. 48(4), 535–544 (2018)Google Scholar
  23. 23.
    Wang, D., Mu, C., Liu, D.: Data-driven nonlinear near-optimal regulation based on iterative neural dynamic programming. Acta Automatica Sin. 43(3), 366–375 (2017)Google Scholar
  24. 24.
    Fossen, T.I.: Guidance and Control of Ocean Vehicles. Wiley, New York (1994)Google Scholar
  25. 25.
    Abkowitz, M.A.: Lectures on Ship Hydrodynamics–Steering and Manoeuvrability (1964)Google Scholar
  26. 26.
    Kensaku, N., Kenshi, T., Keinosuke, H., Susumu, H.: On the steering qualities of ships. Int. Shipbuilding Prog. 4(35), 354–370 (1957)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Yuming Bai
    • 1
  • Yifan Liu
    • 1
    Email author
  • Qihe Shan
    • 1
  • Tieshan Li
    • 1
  • Yuzhen Lu
    • 2
  1. 1.Navigation CollegeDalian Maritime UniversityDalianChina
  2. 2.School of ScienceDalian Maritime UniversityDalianChina

Personalised recommendations