Nonlinear Dynamics

, Volume 93, Issue 4, pp 2089–2103 | Cite as

Adaptive dynamic programming-based stabilization of nonlinear systems with unknown actuator saturation

  • Bo ZhaoEmail author
  • Lihao Jia
  • Hongbing Xia
  • Yuanchun Li
Original Paper


This paper addresses the stabilizing control problem for nonlinear systems subject to unknown actuator saturation by using adaptive dynamic programming algorithm. The control strategy is composed of an online nominal optimal control and a neural network (NN)-based feed-forward saturation compensator. For nominal systems without actuator saturation, a critic NN is established to deal with the Hamilton–Jacobi–Bellman equation. Thus, the online approximate nominal optimal control policy can be obtained without action NN. Then, the unknown actuator saturation, which is considered as saturation nonlinearity by simple transformation, is compensated by employing a NN-based feed-forward control loop. The stability of the closed-loop nonlinear system is analyzed to be ultimately uniformly bounded via Lyapunov’s direct method. Finally, the effectiveness of the presented control method is demonstrated by two simulation examples.


Adaptive dynamic programming Unknown actuator saturation Continuous-time nonlinear systems Stabilizing control Neural networks 



This work was supported in part by the National Natural Science Foundation of China under Grants 61603387, 61533017, 61374051 and 61502494, and in part by the Early Career Development Award of SKLMCCS under Grant 20180201.


  1. 1.
    Bellman, R.E.: Dynamic Programming. Princeton University Press, New Jersey (1957)zbMATHGoogle Scholar
  2. 2.
    Werbos, P.J.: Approximate dynamic programming for real time control and neural modeling. In: White, D.A., Sofge, D.A. (eds.) Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches. Van Nostrand Reinhold, New York (1992)Google Scholar
  3. 3.
    Liu, D., Wei, Q., Wang, D., Yang, X., Li, H.: Adaptive dynamic programming with applications in optimal control. Springer, Cham (2017)CrossRefzbMATHGoogle Scholar
  4. 4.
    Ge, S., Hang, C., Zhang, T.: Adaptive neural network control of nonlinear systems by state and output feedback. IEEE Trans. Syst. Man Cybern. Part B (Cybernetics) 29(6), 818–828 (1999)CrossRefGoogle Scholar
  5. 5.
    Zhao, B., Li, Y.: Local joint information based active fault tolerant control for reconfigurable manipulator. Nonlinear Dyn. 77(3), 859–876 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Zhao, B., Li, Y., Liu, D.: Self-tuned local feedback gain based decentralized fault tolerant control for a class of large-scale nonlinear systems. Neurocomputing 235, 147–156 (2017)CrossRefGoogle Scholar
  7. 7.
    Wang, F.Y., Zhang, H., Liu, D.: Adaptive dynamic programming: an introduction. IEEE Comput. Intell. Mag. 4(2), 39–47 (2009)CrossRefGoogle Scholar
  8. 8.
    Al-Tamimi, A., Lewis, F.L., Abu-Khalaf, M.: Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof. IEEE Trans. Syst. Man Cybern. Part B (Cybernetics) 38(4), 943–949 (2008)CrossRefGoogle Scholar
  9. 9.
    Prokhorov, D.V., Wunsch, D.C.: Adaptive critic designs. IEEE Trans. Neural Netw. 8(5), 997–1007 (1997)CrossRefGoogle Scholar
  10. 10.
    Bertsekas, D.P., Tsitsiklis, J.N.: Neuro-dynamic programming: an overview. In: Proceedings of the 34th IEEE Conference on Decision and Control, pp. 560–564 (1995)Google Scholar
  11. 11.
    Si, J., Wang, Y.T.: On-line learning control by association and reinforcement. IEEE Trans. Neural Netw. 12, 264–276 (2001)CrossRefGoogle Scholar
  12. 12.
    Li, D.J., Tang, L., Liu, Y.J.: Adaptive intelligence learning for nonlinear chaotic systems. Nonlinear Dyn. 73(4), 2103–2109 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: a survey. J. Artif. Intell. Res. 4, 237–285 (1996)CrossRefGoogle Scholar
  14. 14.
    Liu, D., Wang, D., Li, H.: Decentralized stabilization for a class of continuous-time nonlinear interconnected systems using online learning optimal control approach. IEEE Trans. Neural Netw. Learn. Syst. 25(2), 418–428 (2014)CrossRefGoogle Scholar
  15. 15.
    Wang, D., He, H., Zhao, B., Liu, D.: Adaptive near-optimal controllers for nonlinear decentralised feedback stabilisation problems. IET Control Theory Appl. 11(6), 799–806 (2017)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Mehraeen, S., Jagannathan, S.: Decentralized optimal control of a class of interconnected nonlinear discrete-time systems by using online Hamilton–Jacobi–Bellman formulation. IEEE Trans. Neural Netw. 22(11), 1757–1769 (2011)CrossRefGoogle Scholar
  17. 17.
    Lin, Q., Wei, Q., Zhao, B.: Optimal control for discrete-time systems with actuator saturation. Optim. Contr. Appl. Met. 38, 1071–1080 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Gao, W., Jiang, Y., Jiang, Z.P., Chai, T.: Output-feedback adaptive optimal control of interconnected systems based on robust adaptive dynamic programming. Automatica 72, 37–45 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Fan, Q., Yang, G.: Adaptive actor-critic design-based integral sliding-mode control for partially unknown nonlinear systems with input disturbances. IEEE Trans. Neural Netw. Learn. Syst. 27(1), 165–177 (2016)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Zhao, B., Liu, D., Yang, X., Li, Y.: Observer-critic structure-based adaptive dynamic programming for decentralised tracking control of unknown large-scale nonlinear systems. Int. J. Syst. Sci. 48(9), 1978–1989 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Zhao, B., Li, Y.: Model-free adaptive dynamic programming based near-optimal decentralized tracking Control of reconfigurable manipulators. Int. J. Control Autom. Syst. 16(2), 478–490 (2018)CrossRefGoogle Scholar
  22. 22.
    Zhang, H., Song, R., Wei, Q., Zhang, T.: Optimal tracking control for a class of nonlinear discrete-time systems with time delays based on heuristic dynamic programming. IEEE Trans. Neural Netw. 22(12), 1851–1862 (2011)CrossRefGoogle Scholar
  23. 23.
    Xie, C., Yang, G.: Approximate guaranteed cost fault-tolerant control of unknown nonlinear systems with time-varying actuator faults. Nonlinear Dyn. 83(1–2), 269–282 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Zhao, B., Liu, D., Li, Y.: Online fault compensation control based on policy iteration algorithm for a class of affine non-linear systems with actuator failures. IET Control Theory Appl. 10(15), 1816–1823 (2016)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Fu, Y., Fu, J., Chai, T.: Robust adaptive dynamic programming of two-player zero-sum games for continuous-time linear systems. IEEE Trans. Neural Netw. Learn. Syst. 26(12), 3314–3319 (2015)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Wang, D., Mu, C., He, H., Liu, D.: Event-driven adaptive robust control of nonlinear systems with uncertainties through NDP strategy. IEEE Trans. Syst. Man Cybern. Syst. 47(7), 1358–1370 (2017)CrossRefGoogle Scholar
  27. 27.
    Sun, L., Huo, W., Jiao, Z.: Adaptive backstepping control of spacecraft rendezvous and proximity operations with input saturation and full-state constraint. IEEE Trans. Industr. Electron. 64(1), 480–492 (2017)CrossRefGoogle Scholar
  28. 28.
    Wang, F., Hua, C., Zong, Q.: Attitude control of reusable launch vehicle in reentry phase with input constraint via robust adaptive backstepping control. Int. J. Aadpt. Control. 29(10), 1308–1327 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Mendoza, M., Zavala-Rło, A., Santibnez, V., Reyes, F.: Output-feedback proportional–integral-derivative-type control with simple tuning for the global regulation of robot manipulators with input constraints. IET Control Theory Appl. 9(14), 2097–2106 (2015)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Chen, X., Jia, Y., Matsuno, F.: Tracking control for differential-drive mobile robots with diamond-shaped input constraints. IEEE Trans. Control Syst. Technol. 22(5), 1999–2006 (2014)CrossRefGoogle Scholar
  31. 31.
    Kiefer, T., Graichen, K., Kugi, A.: Trajectory tracking of a 3-DOF laboratory helicopter under input and state constraints. IEEE Trans. Control Syst. Technol. 18(4), 944–952 (2010)CrossRefGoogle Scholar
  32. 32.
    Zhai, D., Xia, Y.: Adaptive control for teleoperation system with varying time delays and input saturation constraints. IEEE Trans. Industr. Electron. 63(11), 6921–6929 (2016)CrossRefGoogle Scholar
  33. 33.
    Zhao, F., Ge, S., Tu, F., Qin, Y., Dong, M.: Adaptive neural network control for active suspension system with actuator saturation. IET Control Theory Appl. 10(14), 1696–1705 (2016)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Sun, J., Liu, C.: Finite-horizon differential games for missile–target interception system using adaptive dynamic programming with input constraints. Int. J. Syst. Sci. 49(2), 1–20 (2018)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Liu, Y., Tong, S., Li, D., Gao, Y.: Fuzzy adaptive control with state observer for a class of nonlinear discrete-time systems with input constraint. IEEE Trans. Fuzzy Syst. 24(5), 1147–1158 (2016)CrossRefGoogle Scholar
  36. 36.
    Chen, M., Ge, S., Ren, B.: Adaptive tracking control of uncertain MIMO nonlinear systems with input constraints. Automatica 47(3), 452–465 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Song, G., Li, T., Hu, K., Zheng, B.: Observer-based quantized control of nonlinear systems with input saturation. Nonlinear Dyn. 86(2), 1157–1169 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    He, P., Jagannathan, S.: Reinforcement learning-based output feedback control of nonlinear systems with input constraints. IEEE Trans. Syst. Man Cybern. Part B (Cybernetics) 35(1), 150–154 (2005)CrossRefGoogle Scholar
  39. 39.
    Abu-Khalaf, M., Lewis, F.L., Huang, J.: Neurodynamic programming and zero-sum games for constrained control systems. IEEE Trans. Neural Netw. 19(7), 1243–1252 (2008)CrossRefGoogle Scholar
  40. 40.
    Heydari, A., Balakrishnan, S.N.: Finite-horizon control-constrained nonlinear optimal control using single network adaptive critics. IEEE Trans. Neural Netw. Learn. Syst. 24(1), 145–157 (2013)CrossRefGoogle Scholar
  41. 41.
    Zhang, H., Qin, C., Luo, Y.: Neural-network-based constrained optimal control scheme for discrete-time switched nonlinear system using dual heuristic programming. IEEE Trans. Autom. Sci. Eng. 11(3), 839–849 (2014)CrossRefGoogle Scholar
  42. 42.
    Dong, L., Zhong, X., Sun, C., He, H.: Event-triggered adaptive dynamic programming for continuous-time systems with control constraints. IEEE Trans. Neural Netw. Learn. 28(8), 1941–1952 (2017)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Modares, H., Lewis, F.L., Naghibi-Sistani, M.B.: Adaptive optimal control of unknown constrained-input systems using policy iteration and neural networks. IEEE Trans. Neural Netw. Learn. Syst. 24(10), 1513–1525 (2013)CrossRefGoogle Scholar
  44. 44.
    Xu, H., Zhao, Q., Jagannathan, S.: Finite-horizon near-optimal output feedback neural network control of quantized nonlinear discrete-time systems with input constraint. IEEE Trans. Neural Netw. Learn. Syst. 26(8), 1776–1788 (2015)MathSciNetCrossRefGoogle Scholar
  45. 45.
    Song, R., Zhang, H., Luo, Y., Wei, Q.: Optimal control laws for time-delay systems with saturating actuators based on heuristic dynamic programming. Neurocomputing 73(16), 3020–3027 (2010)CrossRefGoogle Scholar
  46. 46.
    Zhang, H., Luo, Y., Liu, D.: Neural-network-based near-optimal control for a class of discrete-time affine nonlinear systems with control constraints. IEEE Trans. Neural Netw. 20(9), 1490–1503 (2009)CrossRefGoogle Scholar
  47. 47.
    Liu, D., Yang, X., Wang, D., Wei, Q.: Reinforcement-learning-based robust controller design for continuous-time uncertain nonlinear systems subject to input constraints. IEEE Trans. Cybern. 45(7), 1372–1385 (2015)CrossRefGoogle Scholar
  48. 48.
    Liu, D., Wang, D., Yang, X.: An iterative adaptive dynamic programming algorithm for optimal control of unknown discrete-time nonlinear systems with constrained inputs. Inform Sci. 220, 331–342 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  49. 49.
    Yang, X., Liu, D., Ma, H., Xu, Y.: Online approximate solution of HJI equation for unknown constrained-input nonlinear continuous-time systems. Inform Sci. 328, 435–454 (2016)CrossRefzbMATHGoogle Scholar
  50. 50.
    Yang, X., Liu, D., Luo, B., Li, C.: Data-based robust adaptive control for a class of unknown nonlinear constrained-input systems via integral reinforcement learning. Inform Sci. 369, 731–747 (2016)CrossRefGoogle Scholar
  51. 51.
    Pomprapa, A., Leonhardt, S., Misgeld, B.J.E.: Optimal learning control of oxygen saturation using a policy iteration algorithm and a proof-of-concept in an interconnecting three-tank system. Control Eng. Pract. 59, 194–203 (2017)CrossRefGoogle Scholar
  52. 52.
    Yang, X., Liu, D., Wang, D.: Reinforcement learning for adaptive optimal control of unknown continuous-time nonlinear systems with input constraints. Int. J. Contr. 87(3), 553–566 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  53. 53.
    Song, R., Lewis, F.L., Wei, Q., Zhang, H.: Off-policy actor-critic structure for optimal control of unknown systems with disturbances. IEEE Trans. Cybern. 46(5), 1041–1050 (2016)CrossRefGoogle Scholar
  54. 54.
    Wu, H.N., Li, M.M., Guo, L.: Finite-horizon approximate optimal guaranteed cost control of uncertain nonlinear systems with application to mars entry guidance. IEEE Trans. Neural Netw. Learn. Syst. 26(7), 1456–1467 (2015)MathSciNetCrossRefGoogle Scholar
  55. 55.
    Terra, M.H., Cerri, J.P., Ishihara, J.Y.: Optimal robust linear quadratic regulator for systems subject to uncertainties. IEEE Trans. Automat. Control 59(9), 2586–2591 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  56. 56.
    Cui, X., Zhang, H., Luo, Y., Jiang, H.: Adaptive dynamic programming for \(H_\infty \) tracking design of uncertain nonlinear systems with disturbances and input constraints. Int. J. Adapt. Control. 31(11), 1567–1583 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  57. 57.
    Zhao, B., Liu, D., Li, Y.: Observer based adaptive dynamic programming for fault tolerant control of a class of nonlinear systems. Inform Sci. 384, 21–33 (2017)CrossRefGoogle Scholar
  58. 58.
    Liu, D., Wei, Q.: Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems. IEEE Trans. Neural Netw. Learn. Syst. 25(3), 621–634 (2014)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V., part of Springer Nature 2018

Authors and Affiliations

  1. 1.The State Key Laboratory of Management and Control for Complex Systems, Institute of AutomationChinese Academy of SciencesBeijingChina
  2. 2.Research Center for Brain-inspired Intelligence, Institute of AutomationChinese Academy of SciencesBeijingChina
  3. 3.Department of Control Science and EngineeringChangchun University of TechnologyChangchunChina

Personalised recommendations