Adaptive Near-Optimal Control with Full-State Feedback

  • Yinyan Zhang
  • Shuai LiEmail author
  • Xuefeng Zhou
Part of the Studies in Systems, Decision and Control book series (SSDC, volume 265)


In this chapter, a unified online adaptive near-optimal control framework is presented for linear and nonlinear systems with parameter uncertainty. Under this framework, auxiliary systems converging to the unknown dynamics are constructed to approximate and compensate the parameter uncertainty. With the aid of the auxiliary system, future outputs of the controlled system are predicted recursively. By utilizing a predictive time-scale approximation technique, the nonlinear dynamic programming problem for optimal control is significantly simplified and decoupled from the parameter learning dynamics: the finite-horizon integral type objective function is simplified into a quadratic one relative to the control action and there is no need to solve time-consuming Hamilton equations. Theoretical analysis shows that closed-loop systems are asymptotically stable. It is also proved that the presented adaptive near-optimal control law is asymptotically optimal. The efficacy of the presented framework and the theoretical results are validated by an application to underactuated surface vessels.


  1. 1.
    Lewis, F.L., Vrabie, D., Syrmos, V.: Optimal Control, 3rd edn. Wiley, New York (2012)zbMATHCrossRefGoogle Scholar
  2. 2.
    Kim, E.K., Mwasilu, F., Choi, H.H., Jung, J.W.: An observer-based optimal voltage control scheme for three-phase UPS systems. IEEE Trans. Ind. Electron. 62(4), 2073–2081 (2015)CrossRefGoogle Scholar
  3. 3.
    Moghadasi, S., Kamalasadan, S.: Optimal fast control and scheduling of power distribution system using integrated receding horizon control and convex conic programming. IEEE Trans. Ind. Appl. 52(3), 2596–2606 (2016)CrossRefGoogle Scholar
  4. 4.
    Dierks, T., Brenner, B., Jagannathan, S.: Neural network-based optimal control of mobile robot formations with reduced information exchange. IEEE Trans. Control Syst. Technol. 21(4), 1407–1415 (2013)CrossRefGoogle Scholar
  5. 5.
    Trelat, E.: Optimal control and applications to aerospace: some results and challenges. J. Optim. Theory Appl. 254(3), 713–758 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Chen, W.H., Ballance, D.J., Gawthrop, P.J.: Optimal control of nonlinear systems: a predictive control approach. Automatica 39(4), 633–641 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Abu-Khalaf, M., Lewis, F.L.: Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach. Automatica 41(5), 779–791 (2005)MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Wang, Z., Liu, X., Liu, K., Li, S., Wang, H.: Backstepping-based Lyapunov function construction using approximate dynamic programming and sum of square techniques. IEEE Trans. Cybern., in pressGoogle Scholar
  9. 9.
    Sassano, M., Astolfi, A.: Dynamic approximate solutions of the HJ inequality and of the HJB equation for input-affine nonlinear systems. IEEE Trans. Autom. Control 57(10), 2490–2503 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Vamvoudakis, K.G., Lewis, F.L.: Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 46(5), 878–888 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    Tao, G.: Multivariable adaptive control: a survey. Automatica 50(11), 2737–2764 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Na, J., Chen, Q., Ren, X., Guo, Y.: Adaptive prescribed performance motion control of servo mechanisms with friction compensation. IEEE Trans. Control Syst. Technol. 61(1), 486–494 (2014)Google Scholar
  13. 13.
    Li, G., Na, J., Stoten, D.P., Ren, X.: Adaptive neural network feedforward control for dynamically substructured systems. IEEE Trans. Control Syst. Technol. 22(3), 944–954 (2014)CrossRefGoogle Scholar
  14. 14.
    Modares, H., Lewis, F.L., Naghibi-Sistani, M.B.: Adaptive optimal control of unknown constrained-input systems using policy iteration and neural networks. IEEE Trans. Neural Netw. Learn. Syst. 24(10), 1513–1525 (2013)CrossRefGoogle Scholar
  15. 15.
    Wang, F.Y., Zhang, H., Liu, D.: Adaptive dynamic programming: an introduction. IEEE Control Syst. Mag. 4(2), 39–47 (2009)Google Scholar
  16. 16.
    Liu, D., Wei, Q.: Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems. IEEE Trans. Neural Netw. Learn. Syst. 25(3), 621–633 (2014)CrossRefGoogle Scholar
  17. 17.
    Heydari, A., Balakrishnan, S.N.: Finite-horizon control-constrained nonlinear optimal control using single network adaptive critics. IEEE Trans. Neural Netw. Learn. Syst. 24(1), 145–157 (2013)CrossRefGoogle Scholar
  18. 18.
    Liu, Y.J., Tang, L., Tong, S., Chen, C.L.P., Li, D.J.: Reinforcement learning design-based adaptive tracking control with less learning parameters for nonlinear discrete-time MIMO systems. IEEE Trans. Neural Netw. Learn. Syst. 26(1), 165–176 (2015)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Wang, D., Liu, D., Li, H.: Policy iteration algorithm for online design of robust control for a class of continuous-time nonlinear systems. IEEE Trans. Autom. Sci. Eng. 11(2), 627–632 (2014)CrossRefGoogle Scholar
  20. 20.
    Zhang, H., Wei, Q., Luo, Y.: A novel infinite-time optimal tracking control scheme for a class of discrete-time nonlinear systems via the greedy HDP iteration algorithm. IEEE Trans. Syst. Man Cybern. Part B Cybern. 38(4), 937–942 (2008)Google Scholar
  21. 21.
    Lv, Y., Na, J., Yang, Q., Wu, X., Guo, Y.: Online adaptive optimal control for continuous-time nonlinear systems with completely unknown dynamics. Int. J. Control 89(1), 99–112 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  22. 22.
    Mayne, D.Q., Michalska, H.: Receding horizon control of nonlinear systems. IEEE Trans. Autom. Control 35(7), 814–824 (1990)MathSciNetzbMATHCrossRefGoogle Scholar
  23. 23.
    Li, S., Chen, S., Liu, B., Li, Y., Liang, Y.: Decentralized kinematic control of a class of collaborative redundant manipulators via recurrent neural networks. Neurocomputing 91, 1–10 (2012)CrossRefGoogle Scholar
  24. 24.
    Li, S., Cui, H., Li, Y., Liu, B., Lou, Y.: Decentralized control of collaborative redundant manipulators with partial command coverage via locally connected recurrent neural networks. Neural Comput. Appl. 23(3), 1051–1060 (2013)CrossRefGoogle Scholar
  25. 25.
    Jin, L., Zhang, Y., Li, S., Zhang, Y.: Modified ZNN for time-varying quadratic programming with inherent tolerance to noises and its application to kinematic redundancy resolution of robot manipulators. IEEE Trans. Ind. Electron. 63(11), 6978–6988 (2016)CrossRefGoogle Scholar
  26. 26.
    Li, S., He, J., Li, Y., Rafique, M.U.: Distributed recurrent neural networks for cooperative control of manipulators: a game-theoretic perspective. IEEE Trans. Neural Netw. Learn. Syst. 28(2), 415–426 (2017)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Jin, L., Li, S., La, H.M., Luo, X.: Manipulability optimization of redundant manipulators using dynamic neural networks. IEEE Trans. Ind. Electron. 64(6), 4710–4720 (2017)CrossRefGoogle Scholar
  28. 28.
    Li, Y., Li, S., Hannaford, B.: A novel recurrent neural network for improving redundant manipulator motion planning completeness. In: ICRA, pp. 2956–2961 (2018)Google Scholar
  29. 29.
    Zhang, Y., Li, S.: A neural controller for image-based visual servoing of manipulators with physical constraints. IEEE Trans. Neural Netw. Learn. Syst. 29(11), 5419–5429 (2018)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Li, S., Zhou, M., Luo, X.: Modified primal-dual neural networks for motion control of redundant manipulators with dynamic rejection of harmonic noises. IEEE Trans. Neural Netw. Learn. Syst. 29(10), 4791–4801 (2018)MathSciNetCrossRefGoogle Scholar
  31. 31.
    Li, S., Wang, H., Rafique, M.U.: A novel recurrent neural network for manipulator control with improved noise tolerance. IEEE Trans. Neural Netw. Learn. Syst. 29(5), 1908–1918 (2018)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Jin, L., Li, S., Luo, X., Li, Y., Qin, B.: Neural dynamics for cooperative control of redundant robot manipulators. IEEE Trans. Ind. Inform. 14(9), 3812–3821 (2018)CrossRefGoogle Scholar
  33. 33.
    Li, J., Zhang, Y., Li, S., Mao, M.: New discretization-formula-based zeroing dynamics for real-time tracking control of serial and parallel manipulators. IEEE Trans. Ind. Inform. 14(8), 3416–3425 (2018)CrossRefGoogle Scholar
  34. 34.
    Chen, D., Zhang, Y., Li, S.: Tracking control of robot manipulators with unknown models: a Jacobian-matrix-adaption method. IEEE Trans. Ind. Inform. 14(7), 3044–3053 (2018)CrossRefGoogle Scholar
  35. 35.
    Zhang, Y., Li, S., Gui, J., Luo, X.: Velocity-level control with compliance to acceleration-level constraints: a novel scheme for manipulator redundancy resolution. IEEE Trans. Ind. Inform. 14(3), 921–930 (2018)CrossRefGoogle Scholar
  36. 36.
    Xiao, L., Liao, B., Li, S., Zhang, Z., Ding, L., Jin, L.: Design and analysis of FTZNN applied to the real-time solution of a nonstationary Lyapunov equation and tracking control of a wheeled mobile manipulator. IEEE Trans. Ind. Inform. 14(1), 98–105 (2018)CrossRefGoogle Scholar
  37. 37.
    Zhang, Y., Chen, S., Li, S., Zhang, Z.: Adaptive projection neural network for kinematic control of redundant manipulators with unknown physical parameters. IEEE Trans. Ind. Electron. 65(6), 4909–4920 (2018)CrossRefGoogle Scholar
  38. 38.
    Zhang, Z., Lin, Y., Li, S., Li, Y., Yu, Z., Luo, Y.: Tricriteria optimization-coordination motion of dual-redundant-robot manipulators for complex path planning. IEEE Trans. Control Syst. Technol. 26(4), 1345–1357 (2018)CrossRefGoogle Scholar
  39. 39.
    Jin, L., Li, S., Hu, B., Yi, C.: Dynamic neural networks aided distributed cooperative control of manipulators capable of different performance indices. Neurocomputing 291, 50–58 (2018)CrossRefGoogle Scholar
  40. 40.
    Jin, L., Li, S., Yu, J., He, J.: Robot manipulator control using neural networks: a survey. Neurocomputing 285, 23–34 (2018)CrossRefGoogle Scholar
  41. 41.
    Chen, D., Zhang, Y., Li, S.: Zeroing neural-dynamics approach and its robust and rapid solution for parallel robot manipulators against superposition of multiple disturbances. Neurocomputing 275, 845–858 (2018)CrossRefGoogle Scholar
  42. 42.
    Li, S., Shao, Z., Guan, Y.: A dynamic neural network approach for efficient control of manipulators. IEEE Trans. Syst. Man Cybern. Syst. 49(5), 932–941 (2019)CrossRefGoogle Scholar
  43. 43.
    Zhang, Y., Li, S., Zhou, X.: Recurrent-neural-network-based velocity-level redundancy resolution for manipulators subject to a joint acceleration limit. IEEE Trans. Ind. Electron. 66(5), 3573–3582 (2019)CrossRefGoogle Scholar
  44. 44.
    Zhang, Z., Chen, S., Li, S.: Compatible convex-nonconvex constrained QP-based dual neural networks for motion planning of redundant robot manipulators. IEEE Trans. Control Syst. Technol. 27(3), 1250–1258 (2019)CrossRefGoogle Scholar
  45. 45.
    Xu, Z., Li, S., Zhou, X., Yan, W., Cheng, T., Huang, D.: Dynamic neural networks based kinematic control for redundant manipulators with model uncertainties. Neurocomputing 329, 255–266 (2019)CrossRefGoogle Scholar
  46. 46.
    Li, S., Zhang, Y., Jin, L.: Kinematic control of redundant manipulators using neural networks. IEEE Trans. Neural Netw. Learn. Syst., in pressGoogle Scholar
  47. 47.
    Zhang, Y., Wang, J., Xia, Y.: A dual neural network for redundancy resolution of kinematically redundant manipulators subject to joint limits and joint velocity limits. IEEE Trans. Neural Netw. 14(3), 658–667 (2003)CrossRefGoogle Scholar
  48. 48.
    Zhang, Z., Li, Z., Zhang, Y., Luo, Y., Li, Y.: Neural-dynamic-method-based dual-arm CMG scheme with time-varying constraints applied to humanoid robots. IEEE Trans. Neural Netw. Learn. Syst. 26(12), 3251–3262 (2015)MathSciNetCrossRefGoogle Scholar
  49. 49.
    Zhang, Y., Li, W., Zhang, Z.: Physical-limits-constrained minimum velocity norm coordinating scheme for wheeled mobile redundant manipulators. Robotica 33(6), 1325–1350 (2015)CrossRefGoogle Scholar
  50. 50.
    Mohammed, A.M., Li, S.: Dynamic neural networks for kinematic redundancy resolution of parallel Stewart platforms. IEEE Trans. Cybern. 46(7), 1538–1550 (2016)CrossRefGoogle Scholar
  51. 51.
    Jin, L., Zhang, Y.: G2-type SRMPC scheme for synchronous manipulation of two redundant robot arms. IEEE Trans. Cybern. 45(2), 153–164 (2015)MathSciNetCrossRefGoogle Scholar
  52. 52.
    Jin, L., Li, S., Xiao, L., Lu, R., Liao, B.: Cooperative motion generation in a distributed network of redundant robot manipulators with noises. IEEE Trans. Syst. Man Cybern. Syst. 48(10), 1715–1724 (2018)CrossRefGoogle Scholar
  53. 53.
    Jin, L., Li, S.: Distributed task allocation of multiple robots: a control perspective. IEEE Trans. Syst. Man Cybern. Syst. 48(5), 693–701 (2018)CrossRefGoogle Scholar
  54. 54.
    Liu, Y.K., Zhang, Y.M.: Model-based predictive control of weld penetration in gas tungsten arc welding. IEEE Trans. Control Syst. Technol. 22(3), 955–966 (2014)MathSciNetCrossRefGoogle Scholar
  55. 55.
    Akter, M.P., Mekhilef, S., Tan, N.M.L., Akagi, H.: Modified model predictive control of a bidirectional AC-DC converter based on Lyapunov function for energy storage systems. IEEE Trans. Ind. Electron. 63(2), 704–715 (2016)CrossRefGoogle Scholar
  56. 56.
    Mayne, D.Q.: Model predictive control: recent developments and future promise. Automatica 50(12), 2967–2986 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  57. 57.
    Pradhan, S.K., Subudhi, B.: Nonlinear adaptive model predictive controller for a flexible manipulator: an experimental study. IEEE Trans. Control Syst. Technol. 22(5), 1754–1768 (2014)CrossRefGoogle Scholar
  58. 58.
    Zhang, Y., Li, S.: Predictive suboptimal consensus of multiagent systems with nonlinear dynamics. IEEE Trans. Syst. Man Cybern. Syst. 47(7), 1701–1711 (2017)MathSciNetCrossRefGoogle Scholar
  59. 59.
    Jiang, Z.: Global tracking control of underactuated ships by Lyapunov’s direct method. Automatica 38(2), 301–309 (2002)zbMATHCrossRefGoogle Scholar
  60. 60.
    Behal, A., Dawson, D.M., Dixon, W.E., Fang, Y.: Tracking and regulation control of an underactuated surface vessel with nonintegrable dynamics. IEEE Trans. Autom. Control 47(3), 495–500 (2002)MathSciNetzbMATHCrossRefGoogle Scholar
  61. 61.
    Yan, Z., Wang, J.: Model predictive control for tracking of underactuated vessels based on recurrent neural networks. IEEE J. Ocean. Eng. 37(4), 717–726 (2012)CrossRefGoogle Scholar
  62. 62.
    Yu, R., Zhu, Q., Xia, G., Liu, Z.: Sliding mode tracking control of an underactuated surface vessel. IET Control Theory Appl. 6(3), 461–466 (2012)MathSciNetCrossRefGoogle Scholar
  63. 63.
    Ashrafiuon, H., Muske, K.R., McNinch, L.C., Soltan, R.A.: Sliding-mode tracking control of surface vessels. IEEE Trans. Ind. Electron. 55(11), 4004–4012 (2008)CrossRefGoogle Scholar
  64. 64.
    Elmokadem, T., Zribi, M., Youcef-Toumi, K.: Trajectory tracking sliding mode control of underactuated AUVs. Nonlinear Dyn. 84(2), 1079–1091 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  65. 65.
    Hu, C., Wang, R., Yan, F., Chen, N.: Robust composite nonlinear feedback path-following control for underactuated surface vessels with desired-heading amendment. IEEE Trans. Ind. Electron. 63(10), 6386–6394 (2016)CrossRefGoogle Scholar
  66. 66.
    Pan, C.-Z., Lai, X.-Z., Yang, S.X., Wu, M.: A biologically inspired approach to tracking control of underactuated surface vessels subject to unknown dynamics. Expert Syst. Appl. 42(4), 2153–2161 (2015)CrossRefGoogle Scholar
  67. 67.
    Zhang, Y., Chen, D., Jin, L., Zhang, Y., Yin, Y.: GD-aided IOL (input-output linearisation) controller for handling affine-form nonlinear system with loose condition on relative degree. Int. J. Control 89(4), 757–769 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  68. 68.
    Isidori, A.: Nonlinear Control Systems: An Introduction, 3rd edn. Springer, New York (1995)zbMATHCrossRefGoogle Scholar
  69. 69.
    Sun, W., Tang, S., Gao, H., Zhao, J.: Two time-scale tracking control of nonholonomic wheeled mobile robots. IEEE Trans. Control Syst. Technol. 24(6), 2059–2069 (2016)CrossRefGoogle Scholar
  70. 70.
    Mehrabian, A.R., Khorasani, K.: Distributed formation recovery control of heterogeneous multiagent Euler-Lagrange systems subject to network switching and diagnostic imperfections. IEEE Trans. Control Syst. Technol. 24(6), 2158–2166 (2016)CrossRefGoogle Scholar
  71. 71.
    Chen, C.T.: Linear System Theory and Design. Oxford University Press, New York (1999)Google Scholar
  72. 72.
    Yang, C., Li, Z., Cui, R., Xu, B.: Neural network-based motion control of underactuated wheeled inverted pendulum models. IEEE Trans. Neural Netw. Learn. Syst 25(11), 2004–2016 (2014)CrossRefGoogle Scholar
  73. 73.
    Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)zbMATHCrossRefGoogle Scholar
  74. 74.
    Liao, B., Zhang, Y., Jin, L.: Taylor \(O(h^3)\) discretization of ZNN models for dynamic equality-constrained quadratic programming with application to manipulators. IEEE Trans. Neural Netw. Learn. Syst. 27(2), 225–237 (2016)MathSciNetCrossRefGoogle Scholar
  75. 75.
    Khalil, H.K.: Nonlinear Systems, 3rd edn. Prentice-Hall, Upper Saddle River (2002)zbMATHGoogle Scholar
  76. 76.
    Kamalapurkar, R., Reish, B., Chowdhary, G., Dixon, W.E.: Concurrent learning for parameter estimation using dynamic state-derivative estimators. IEEE Trans. Autom. Control 62(7), 3594–3601 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
  77. 77.
    Kersting, S., Buss, M.: Direct and indirect model reference adaptive control for multivariable piecewise affine systems. IEEE Trans. Autom. Control 62(11), 5634–5649 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
  78. 78.
    Royden, H.L.: Real Analysis. Macmillan, New York (1988)zbMATHGoogle Scholar
  79. 79.
    Liu, Z., Li, C., Xu, W.: Hybrid control of biped robots in the double-support phase \(H_\infty \) approach and fuzzy neural networks. IEE Proc. Control Theory Appl. 150(4), 347–354 (2003)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.College of Cyber SecurityJinan UniversityGuangzhouChina
  2. 2.School of Information Science and EngineeringLanzhou UniversityLanzhouChina
  3. 3.Guangdong Institute of Intelligent ManufacturingGuangdong Academy of ScienceGuangzhouChina

Personalised recommendations