An Improved Adaptive Optimal Regulation Framework with Robust Control Synthesis

  • Ding WangEmail author
  • Chaoxu Mu
Part of the Studies in Systems, Decision and Control book series (SSDC, volume 167)


In this chapter, we focus on developing adaptive optimal regulators for a class of continuous-time nonlinear dynamical systems through an improved neural learning mechanism. The main objective lies in that establishing an additional stabilizing term to reinforce the traditional training process of the critic neural network, so that to reduce the requirement with respect to the initial stabilizing control, and therefore, bring in an obvious convenience to the adaptive-critic-based learning control implementation. It is exhibited that by employing the novel updating rule, the adaptive optimal control law can be obtained with an excellent approximation property. The closed-loop system is constructed and its stability issue is handled by considering the improved learning criterion. After that, we apply the adaptation-oriented approximate optimal control strategy to perform robust stabilization when including complex nonlinearity and uncertainty. By considering the dynamical uncertainties, it is proven that the developed near-optimal control law can achieve uniform ultimate boundedness of the closed-loop state vector, thereby guaranteeing a certain extent of robustness for the uncertain nonlinear plant. Simulation for a classical nonlinear system and experiment on an overhead crane are conducted to verify the efficient performance of the present design methods, especially the major role that the stabilizing term performed.


  1. 1.
    Bian, T., Jiang, Y., Jiang, Z.P.: Decentralized adaptive optimal control of large-scale systems with application to power systems. IEEE Trans. Industr. Electron. 62(4), 2439–2447 (2015)CrossRefGoogle Scholar
  2. 2.
    Chen, M., Tao, G.: Adaptive fault-tolerant control of uncertain nonlinear large-scale systems with unknown dead zone. IEEE Trans. Cybern. 46(8), 1851–1862 (2016)CrossRefGoogle Scholar
  3. 3.
    Dierks, T., Jagannathan, S.: Optimal control of affine nonlinear continuous-time systems. In: Proceedings of the American Control Conference, pp. 1568–1573 (2010)Google Scholar
  4. 4.
    Dierks, T., Thumati, B.T., Jagannathan, S.: Optimal control of unknown affine nonlinear discrete-time systems using offline-trained neural networks with proof of convergence. Neural Netw. 22(5–6), 851–860 (2009)CrossRefGoogle Scholar
  5. 5.
    Gao, W., Jiang, Z.P.: Adaptive dynamic programming and adaptive optimal output regulation of linear systems. IEEE Trans. Autom. Control 61(12), 4164–4169 (2016)MathSciNetCrossRefGoogle Scholar
  6. 6.
    He, W., Dong, Y., Sun, C.: Adaptive neural impedance control of a robotic manipulator with input saturation. IEEE Trans. Syst. Man Cybern. Syst. 46(3), 334–344 (2016)CrossRefGoogle Scholar
  7. 7.
    Heydari, A., Balakrishnan, S.N.: Finite-horizon control-constrained nonlinear optimal control using single network adaptive critics. IEEE Trans. Neural Netw. Learn. Syst. 24(1), 145–157 (2013)CrossRefGoogle Scholar
  8. 8.
    Jiang, Y., Jiang, Z.P.: Robust adaptive dynamic programming and feedback stabilization of nonlinear systems. IEEE Trans. Neural Netw. Learn. Syst. 25(5), 882–893 (2014)CrossRefGoogle Scholar
  9. 9.
    Jiang, Y., Jiang, Z.P.: Global adaptive dynamic programming for continuous-time nonlinear systems. IEEE Trans. Autom. Control 60(11), 2917–2929 (2015)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Lewis, F.L., Liu, D.: Reinforcement Learning and Approximate Dynamic Programming for Feedback Control. Wiley, New Jersey (2013)Google Scholar
  11. 11.
    Lewis, F.L., Jagannathan, S., Yesildirek, A.: Neural Network Control of Robot Manipulators and Nonlinear Systems. Taylor & Francis, London (1998)Google Scholar
  12. 12.
    Li, C., Gao, J., Yi, J., Zhang, G.: Analysis and design of functionally weighted single-input-rule-modules connected fuzzy inference systems. IEEE Trans. Fuzzy Syst. 26(1), 56–71 (2018)CrossRefGoogle Scholar
  13. 13.
    Liu, D., Wei, Q., Wang, D., Yang, X., Li, H.: Adaptive Dynamic Programming with Applications in Optimal Control. Springer, London (2017)CrossRefGoogle Scholar
  14. 14.
    Liu, Y.J., Tong, S., Chen, C.L.P., Li, D.J.: Neural controller design-based adaptive control for nonlinear MIMO systems with unknown hysteresis inputs. IEEE Trans. Cybern. 46(1), 9–19 (2016)CrossRefGoogle Scholar
  15. 15.
    Luo, B., Wu, H.N., Huang, T., Liu, D.: Reinforcement learning solution for HJB equation arising in constrained optimal control problem. Neural Netw. 71, 150–158 (2015)CrossRefGoogle Scholar
  16. 16.
    Modares, H., Lewis, F.L.: Linear quadratic tracking control of partially-unknown continuous-time systems using reinforcement learning. IEEE Trans. Autom. Control 59(11), 3051–3056 (2014)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Mu, C., Wang, D.: Neural-network-based adaptive guaranteed cost control of nonlinear dynamical systems with matched uncertainties. Neurocomputing 245, 46–54 (2017)CrossRefGoogle Scholar
  18. 18.
    Mu, C., Sun, C., Song, A., Yu, H.: Iterative GDHP-based approximate optimal tracking control for a class of discrete-time nonlinear systems. Neurocomputing 214, 775–784 (2016)CrossRefGoogle Scholar
  19. 19.
    Mu, C., Ni, Z., Sun, C., He, H.: Air-breathing hypersonic vehicle tracking control based on adaptive dynamic programming. IEEE Trans. Neural Netw. Learn. Syst. 28(3), 584–598 (2017)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Mu, C., Wang, D., He, H.: Novel iterative neural dynamic programming for data-based approximate optimal control design. Automatica 81, 240–252 (2017)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Na, J., Herrmann, G.: Online adaptive approximate optimal tracking control with simplified dual approximation structure for continuoustime unknown nonlinear systems. IEEE/CAA J. Autom. Sin. 1(4), 412–422 (2014)CrossRefGoogle Scholar
  22. 22.
    Nodland, D., Zargarzadeh, H., Jagannathan, S.: Neural network-based optimal adaptive output feedback control of a helicopter UAV. IEEE Trans. Neural Netw. Learn. Syst. 24(7), 1061–1073 (2013)CrossRefGoogle Scholar
  23. 23.
    Qin, C., Zhang, H., Wang, Y., Luo, Y.: Neural network-based online \(H_{\infty }\) control for discrete-time affine nonlinear system using adaptive dynamic programming. Neurocomputing 198, 91–99 (2016)CrossRefGoogle Scholar
  24. 24.
    Song, R., Wei, Q., Sun, Q.: Nearly finite-horizon optimal control for a class of nonaffine time-delay nonlinear systems based on adaptive dynamic programming. Neurocomputing 156, 166–175 (2015)CrossRefGoogle Scholar
  25. 25.
    Vamvoudakis, K.G., Lewis, F.L.: Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 46(5), 878–888 (2010)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Vamvoudakis, K.G., Antsaklis, P.J., Dixon, W.E., Hespanha, J.P., Lewis, F.L., Modares, H., Kiumarsi, B.: Autonomy and machine intelligence in complex systems: a tutorial. In: Proceedings of the 2015 American Control Conference, pp. 5062–5079 (2015)Google Scholar
  27. 27.
    Vamvoudakis, K.G., Modares, H., Kiumarsi, B., Lewis, F.L.: Game theory-based control system algorithms with real-time reinforcement learning: how to solve multiplayer games online. IEEE Control Syst. Mag. 37(1), 33–52 (2017)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Wang, D.: Adaptation-oriented near-optimal control and robust synthesis of an overhead crane system. In: Proceedings of 24th International Conference on Neural Information Processing, Part VI, Guangzhou, China, November 2017, pp. 42–50 (2017)Google Scholar
  29. 29.
    Wang, D., Mu, C.: Developing nonlinear adaptive optimal regulators through an improved neural learning mechanism. Sci. China Inf. Sci. 60(5), 058201:1–058201:3 (2017)Google Scholar
  30. 30.
    Wang, D., Mu, C.: A novel neural optimal control framework with nonlinear dynamics: closed-loop stability and simulation verification. Neurocomputing 266, 353–360 (2017)CrossRefGoogle Scholar
  31. 31.
    Wang, D., Liu, D., Zhang, Q., Zhao, D.: Data-based adaptive critic designs for nonlinear robust optimal control with uncertain dynamics. IEEE Trans. Syst. Man Cybern. Syst. 46(11), 1544–1555 (2016)CrossRefGoogle Scholar
  32. 32.
    Wang, D., He, H., Liu, D.: Adaptive critic nonlinear robust control: a survey. IEEE Trans. Cybern. 47(10), 3429–3451 (2017)CrossRefGoogle Scholar
  33. 33.
    Wang, D., He, H., Mu, C., Liu, D.: Intelligent critic control with disturbance attenuation for affine dynamics including an application to a micro-grid system. IEEE Trans. Industr. Electron. 64(6), 4935–4944 (2017)CrossRefGoogle Scholar
  34. 34.
    Wang, D., Mu, C., Liu, D.: Data-driven nonlinear near-optimal regulation based on iterative neural dynamic programming. Acta Autom. Sin. 43(3), 366–375 (2017)zbMATHGoogle Scholar
  35. 35.
    Wang, T., Tong, S.: Observer-based output-feedback asynchronous control for switched fuzzy systems. IEEE Trans. Cybern. 47(9), 2579–2591 (2016)CrossRefGoogle Scholar
  36. 36.
    Wang, Y., Cheng, L., Hou, Z.G., Yu, J., Tan, M.: Optimal formation of multi-robot systems based on a recurrent neural network. IEEE Trans. Neural Netw. Learn. Syst. 27(2), 322–333 (2016)MathSciNetCrossRefGoogle Scholar
  37. 37.
    Werbos, P.J.: Approximate dynamic programming for real-time control and neural modeling. Handbook of Intelligent Control: Neural, Fuzzy, and Adaptive Approaches, pp. 493–526 (1992)Google Scholar
  38. 38.
    Xu, B.: Robust adaptive neural control of flexible hypersonic flight vehicle with dead-zone input nonlinearity. Nonlinear Dyn. 80(3), 1509–1520 (2015)MathSciNetCrossRefGoogle Scholar
  39. 39.
    Yang, X., Liu, D., Wei, Q., Wang, D.: Guaranteed cost neural tracking control for a class of uncertain nonlinear systems using adaptive dynamic programming. Neurocomputing 198, 80–90 (2016)CrossRefGoogle Scholar
  40. 40.
    Zhang, H., Cui, L., Luo, Y.: Near-optimal control for nonzero-sum differential games of continuous-time nonlinear systems using single-network ADP. IEEE Trans. Cybern. 43(1), 206–216 (2013)CrossRefGoogle Scholar
  41. 41.
    Zhang, H., Zhang, X., Luo, Y., Yang, J.: An overview of research on adaptive dynamic programming. Acta Autom. Sin. 39(4), 303–311 (2013)CrossRefGoogle Scholar
  42. 42.
    Zhang, Q., Zhao, D., Wang, D.: Event-based robust control for uncertain nonlinear systems using adaptive dynamic programming. IEEE Trans. Neural Netw. Learn. Syst. 29(1), 37–50 (2018)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Zhao, D., Xia, Z., Wang, D.: Model-free optimal control for affine nonlinear systems with convergence analysis. IEEE Trans. Autom. Sci. Eng. 12(4), 1461–1468 (2015)CrossRefGoogle Scholar
  44. 44.
    Zhao, Q., Xu, H., Jagannathan, S.: Near optimal output feedback control of nonlinear discrete-time systems based on reinforcement neural network learning. IEEE/CAA J. Autom. Sin. 1(4), 372–384 (2014)CrossRefGoogle Scholar
  45. 45.
    Zhong, X., He, H., Zhang, H., Wang, Z.: A neural network based online learning and control approach for Markov jump systems. Neurocomputing 149, 116–123 (2015)CrossRefGoogle Scholar
  46. 46.
    Zhong, X., Ni, Z., He, H.: A theoretical foundation of goal representation heuristic dynamic programming. IEEE Trans. Neural Netw. Learn. Syst. 27(12), 2513–2525 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.The State Key Laboratory of Management and Control for Complex SystemsInstitute of Automation, Chinese Academy of SciencesBeijingChina
  2. 2.School of Electrical and Information EngineeringTianjin UniversityTianjinChina

Personalised recommendations