Abstract
In this chapter, optimal control problems of continuous-time affine nonlinear systems are studied using adaptive dynamic programming (ADP) approach. First, an identifier–critic architecture based on ADP methods is presented to derive the approximate optimal control for partially unknown continuous-time nonlinear systems. Based on the ADP approach developed in this chapter, the identifier neural network (NN) and the critic NN are tuned simultaneously. Meanwhile, using recorded and instantaneous data simultaneously for the adaptation of the critic NN, the restrictive persistence of excitation condition is relaxed. Second, an ADP algorithm is developed to obtain the optimal control for continuous-time nonlinear systems with control constraints . By using the present algorithm, a single critic NN is utilized to derive the optimal control. Moreover, unlike in the case of policy iteration, where an initial stabilizing control is indispensable, there is no special requirement imposed on the initial control law.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abu-Khalaf M, Lewis FL (2005) Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach. Automatica 41(5):779–791
Beard RW, Saridis GN, Wen JT (1997) Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation. Automatica 33(12):2159–2177
Bhasin S, Kamalapurkar R, Johnson M, Vamvoudakis KG, Lewis FL, Dixon WE (2013) A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems. Automatica 49(1):82–92
Bryson AE, Ho YC (1975) Applied optimal control: optimization, estimation and control. CRC Press, Boca Raton
Chowdhary G (2010) Concurrent learning for convergence in adaptive control without persistency of excitation. Ph.D. Thesis, Georgia Institute of Technology, USA
Chowdhary G, Johnson E (2011) A singular value maximizing data recording algorithm for concurrent learning. In: Proceedings of the American control conference. pp 3547–3552
Dierks T, Jagannathan S (2010) Optimal control of affine nonlinear continuous-time systems. In: Proceedings of the American control conference. pp 1568–1573
Haykin S (2009) Neural networks and learning machines, 3rd edn. Prentice-Hall, Upper Saddle River
Horn RA, Johnson CR (2012) Matrix analysis. Cambridge University Press, New York
Hornik K, Stinchcombe M, White H (1990) Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural Netw 3(5):551–560
Khalil HK (2001) Nonlinear systems. Prentice-Hall, Upper Saddle River
LaSalle JP, Lefschetz S (1967) Stability by Liapunov’s direct method with applications. Academic Press, New York
Lewis FL, Jagannathan S, Yesildirak A (1999) Neural network control of robot manipulators and nonlinear systems. Taylor & Francis, London
Lewis FL, Vrabie D, Syrmos VL (2012) Optimal control. Wiley, Hoboken
Lewis FL, Vrabie D, Vamvoudakis KG (2012) Reinforcement learning and feedback control: using natural decision methods to design optimal adaptive controllers. IEEE Control Syst Mag 32(6):76–105
Lewis FL, Yesildirek A, Liu K (1996) Multilayer neural-net robot controller with guaranteed tracking performance. IEEE Trans Neural Netw 7(2):388–399
Li H, Liu D (2012) Optimal control for discrete-time affine non-linear systems using general value iteration. IET Control Theory Appl 6(18):2725–2736
Liu D, Wang D, Wang FY, Li H, Yang X (2014) Neural-network-based online HJB solution for optimal robust guaranteed cost control of continuous-time uncertain nonlinear systems. IEEE Trans Cybern 44(12):2834–2847
Liu D, Wang D, Yang X (2013) An iterative adaptive dynamic programming algorithm for optimal control of unknown discrete-time nonlinear systems with constrained inputs. Inf Sci 220:331–342
Liu D, Wei Q (2013) Finite-approximation-error-based optimal control approach for discrete-time nonlinear systems. IEEE Trans Cybern 43(2):779–789
Lyshevski SE (1998) Optimal control of nonlinear continuous-time systems: design of bounded controllers via generalized nonquadratic functionals. In: Proceedings of the American Control Conference. pp 205–209
Michel AN, Hou L, Liu D (2015) Stability of dynamical systems: on the role of monotonic and non-monotonic Lyapunov functions. Birkhäuser, Boston
Modares H, Lewis FL (2014) Optimal tracking control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning. Automatica 50(7):1780–1792
Modares H, Lewis F, Naghibi-Sistani MB (2014) Online solution of nonquadratic two-player zero-sum games arising in the \(H_\infty \) control of constrained input systems. Int J Adapt Control Signal Process 28(3–5):232–254
Modares H, Lewis FL, Naghibi-Sistani MB (2014) Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems. Automatica 50(1):193–202
Nodland D, Zargarzadeh H, Jagannathan S (2013) Neural network-based optimal adaptive output feedback control of a helicopter UAV. IEEE Trans Neural Netw Learn Syst 24(7):1061–1073
Padhi R, Unnikrishnan N, Wang X, Balakrishnan S (2006) A single network adaptive critic (SNAC) architecture for optimal control synthesis for a class of nonlinear systems. Neural Netw 19(10):1648–1660
Rudin W (1976) Principles of mathematical analysis. McGraw-Hill, New York
Vamvoudakis KG, Lewis FL (2010) Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 46(5):878–888
Wang D, Liu D, Wei Q, Zhao D, Jin N (2012) Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming. Automatica 48(8):1825–1832
Yang X, Liu D, Huang Y (2013) Neural-network-based online optimal control for uncertain non-linear continuous-time systems with control constraints. IET Control Theory Appl 7(17):2037–2047
Yang X, Liu D, Wang D (2014) Reinforcement learning for adaptive optimal control of unknown continuous-time nonlinear systems with input constraints. Int J Control 87(3):553–566
Yang X, Liu D, Wang D, Wei Q (2014) Discrete-time online learning control for a class of unknown nonaffine nonlinear systems using reinforcement learning. Neural Netw 55:30–41
Yang X, Liu D, Wei Q (2014) Online approximate optimal control for affine non-linear systems with unknown internal dynamics using adaptive dynamic programming. IET Control Theory Appl 8(16):1676–1688
Yu W (2009) Recent advances in intelligent control systems. Springer, London
Zhang H, Cui L, Luo Y (2013) Near-optimal control for nonzero-sum differential games of continuous-time nonlinear systems using single-network ADP. IEEE Trans Cybern 43(1):206–216
Zhang H, Cui L, Zhang X, Luo Y (2011) Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method. IEEE Trans Neural Netw 22(12):2226–2236
Zhang H, Liu D, Luo Y, Wang D (2013) Adaptive dynamic programming for control: algorithms and stability. Springer, London
Zhang H, Qin C, Luo Y (2014) Neural-network-based constrained optimal control scheme for discrete-time switched nonlinear system using dual heuristic programming. IEEE Trans Autom Sci Eng 11(3):839–849
Zhong X, He H, Zhang H, Wang Z (2014) Optimal control for unknown discrete-time nonlinear markov jump systems using adaptive dynamic programming. IEEE Trans Neural Netw Learn Syst 25(12):2141–2155
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this chapter
Cite this chapter
Liu, D., Wei, Q., Wang, D., Yang, X., Li, H. (2017). Online Optimal Control of Continuous-Time Affine Nonlinear Systems. In: Adaptive Dynamic Programming with Applications in Optimal Control. Advances in Industrial Control. Springer, Cham. https://doi.org/10.1007/978-3-319-50815-3_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-50815-3_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-50813-9
Online ISBN: 978-3-319-50815-3
eBook Packages: EngineeringEngineering (R0)