Skip to main content
SpringerLink
Log in

Online Optimal Control of Continuous-Time Affine Nonlinear Systems

  • Chapter
  • First Online:
Book cover Adaptive Dynamic Programming with Applications in Optimal Control

Part of the book series: Advances in Industrial Control ((AIC))

  • 3915 Accesses

Abstract

In this chapter, optimal control problems of continuous-time affine nonlinear systems are studied using adaptive dynamic programming (ADP) approach. First, an identifier–critic architecture based on ADP methods is presented to derive the approximate optimal control for partially unknown continuous-time nonlinear systems. Based on the ADP approach developed in this chapter, the identifier neural network (NN) and the critic NN are tuned simultaneously. Meanwhile, using recorded and instantaneous data simultaneously for the adaptation of the critic NN, the restrictive persistence of excitation condition is relaxed. Second, an ADP algorithm is developed to obtain the optimal control for continuous-time nonlinear systems with control constraints . By using the present algorithm, a single critic NN is utilized to derive the optimal control. Moreover, unlike in the case of policy iteration, where an initial stabilizing control is indispensable, there is no special requirement imposed on the initial control law.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abu-Khalaf M, Lewis FL (2005) Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach. Automatica 41(5):779–791

    Article  MathSciNet  MATH  Google Scholar 

  2. Beard RW, Saridis GN, Wen JT (1997) Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation. Automatica 33(12):2159–2177

    Article  MathSciNet  MATH  Google Scholar 

  3. Bhasin S, Kamalapurkar R, Johnson M, Vamvoudakis KG, Lewis FL, Dixon WE (2013) A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems. Automatica 49(1):82–92

    Article  MathSciNet  MATH  Google Scholar 

  4. Bryson AE, Ho YC (1975) Applied optimal control: optimization, estimation and control. CRC Press, Boca Raton

    Google Scholar 

  5. Chowdhary G (2010) Concurrent learning for convergence in adaptive control without persistency of excitation. Ph.D. Thesis, Georgia Institute of Technology, USA

    Google Scholar 

  6. Chowdhary G, Johnson E (2011) A singular value maximizing data recording algorithm for concurrent learning. In: Proceedings of the American control conference. pp 3547–3552

    Google Scholar 

  7. Dierks T, Jagannathan S (2010) Optimal control of affine nonlinear continuous-time systems. In: Proceedings of the American control conference. pp 1568–1573

    Google Scholar 

  8. Haykin S (2009) Neural networks and learning machines, 3rd edn. Prentice-Hall, Upper Saddle River

    Google Scholar 

  9. Horn RA, Johnson CR (2012) Matrix analysis. Cambridge University Press, New York

    Book  Google Scholar 

  10. Hornik K, Stinchcombe M, White H (1990) Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural Netw 3(5):551–560

    Article  Google Scholar 

  11. Khalil HK (2001) Nonlinear systems. Prentice-Hall, Upper Saddle River

    Google Scholar 

  12. LaSalle JP, Lefschetz S (1967) Stability by Liapunov’s direct method with applications. Academic Press, New York

    Google Scholar 

  13. Lewis FL, Jagannathan S, Yesildirak A (1999) Neural network control of robot manipulators and nonlinear systems. Taylor & Francis, London

    Google Scholar 

  14. Lewis FL, Vrabie D, Syrmos VL (2012) Optimal control. Wiley, Hoboken

    Book  MATH  Google Scholar 

  15. Lewis FL, Vrabie D, Vamvoudakis KG (2012) Reinforcement learning and feedback control: using natural decision methods to design optimal adaptive controllers. IEEE Control Syst Mag 32(6):76–105

    Article  MathSciNet  Google Scholar 

  16. Lewis FL, Yesildirek A, Liu K (1996) Multilayer neural-net robot controller with guaranteed tracking performance. IEEE Trans Neural Netw 7(2):388–399

    Article  Google Scholar 

  17. Li H, Liu D (2012) Optimal control for discrete-time affine non-linear systems using general value iteration. IET Control Theory Appl 6(18):2725–2736

    Article  MathSciNet  Google Scholar 

  18. Liu D, Wang D, Wang FY, Li H, Yang X (2014) Neural-network-based online HJB solution for optimal robust guaranteed cost control of continuous-time uncertain nonlinear systems. IEEE Trans Cybern 44(12):2834–2847

    Article  Google Scholar 

  19. Liu D, Wang D, Yang X (2013) An iterative adaptive dynamic programming algorithm for optimal control of unknown discrete-time nonlinear systems with constrained inputs. Inf Sci 220:331–342

    Article  MathSciNet  MATH  Google Scholar 

  20. Liu D, Wei Q (2013) Finite-approximation-error-based optimal control approach for discrete-time nonlinear systems. IEEE Trans Cybern 43(2):779–789

    Article  Google Scholar 

  21. Lyshevski SE (1998) Optimal control of nonlinear continuous-time systems: design of bounded controllers via generalized nonquadratic functionals. In: Proceedings of the American Control Conference. pp 205–209

    Google Scholar 

  22. Michel AN, Hou L, Liu D (2015) Stability of dynamical systems: on the role of monotonic and non-monotonic Lyapunov functions. Birkhäuser, Boston

    Book  MATH  Google Scholar 

  23. Modares H, Lewis FL (2014) Optimal tracking control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning. Automatica 50(7):1780–1792

    Article  MathSciNet  MATH  Google Scholar 

  24. Modares H, Lewis F, Naghibi-Sistani MB (2014) Online solution of nonquadratic two-player zero-sum games arising in the \(H_\infty \) control of constrained input systems. Int J Adapt Control Signal Process 28(3–5):232–254

    Article  MathSciNet  MATH  Google Scholar 

  25. Modares H, Lewis FL, Naghibi-Sistani MB (2014) Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems. Automatica 50(1):193–202

    Article  MathSciNet  MATH  Google Scholar 

  26. Nodland D, Zargarzadeh H, Jagannathan S (2013) Neural network-based optimal adaptive output feedback control of a helicopter UAV. IEEE Trans Neural Netw Learn Syst 24(7):1061–1073

    Article  Google Scholar 

  27. Padhi R, Unnikrishnan N, Wang X, Balakrishnan S (2006) A single network adaptive critic (SNAC) architecture for optimal control synthesis for a class of nonlinear systems. Neural Netw 19(10):1648–1660

    Article  MATH  Google Scholar 

  28. Rudin W (1976) Principles of mathematical analysis. McGraw-Hill, New York

    MATH  Google Scholar 

  29. Vamvoudakis KG, Lewis FL (2010) Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 46(5):878–888

    Article  MathSciNet  MATH  Google Scholar 

  30. Wang D, Liu D, Wei Q, Zhao D, Jin N (2012) Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming. Automatica 48(8):1825–1832

    Article  MathSciNet  MATH  Google Scholar 

  31. Yang X, Liu D, Huang Y (2013) Neural-network-based online optimal control for uncertain non-linear continuous-time systems with control constraints. IET Control Theory Appl 7(17):2037–2047

    Article  MathSciNet  Google Scholar 

  32. Yang X, Liu D, Wang D (2014) Reinforcement learning for adaptive optimal control of unknown continuous-time nonlinear systems with input constraints. Int J Control 87(3):553–566

    Article  MathSciNet  MATH  Google Scholar 

  33. Yang X, Liu D, Wang D, Wei Q (2014) Discrete-time online learning control for a class of unknown nonaffine nonlinear systems using reinforcement learning. Neural Netw 55:30–41

    Article  MATH  Google Scholar 

  34. Yang X, Liu D, Wei Q (2014) Online approximate optimal control for affine non-linear systems with unknown internal dynamics using adaptive dynamic programming. IET Control Theory Appl 8(16):1676–1688

    Article  MathSciNet  Google Scholar 

  35. Yu W (2009) Recent advances in intelligent control systems. Springer, London

    Book  MATH  Google Scholar 

  36. Zhang H, Cui L, Luo Y (2013) Near-optimal control for nonzero-sum differential games of continuous-time nonlinear systems using single-network ADP. IEEE Trans Cybern 43(1):206–216

    Article  Google Scholar 

  37. Zhang H, Cui L, Zhang X, Luo Y (2011) Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method. IEEE Trans Neural Netw 22(12):2226–2236

    Article  Google Scholar 

  38. Zhang H, Liu D, Luo Y, Wang D (2013) Adaptive dynamic programming for control: algorithms and stability. Springer, London

    Book  MATH  Google Scholar 

  39. Zhang H, Qin C, Luo Y (2014) Neural-network-based constrained optimal control scheme for discrete-time switched nonlinear system using dual heuristic programming. IEEE Trans Autom Sci Eng 11(3):839–849

    Article  MathSciNet  Google Scholar 

  40. Zhong X, He H, Zhang H, Wang Z (2014) Optimal control for unknown discrete-time nonlinear markov jump systems using adaptive dynamic programming. IEEE Trans Neural Netw Learn Syst 25(12):2141–2155

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Derong Liu, Qinglai Wei & Ding Wang

  2. Tianjin University, Tianjin, China

    Xiong Yang

  3. Tencent Inc., Shenzhen, China

    Hongliang Li

Authors
  1. Derong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qinglai Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ding Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hongliang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Derong Liu .

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Liu, D., Wei, Q., Wang, D., Yang, X., Li, H. (2017). Online Optimal Control of Continuous-Time Affine Nonlinear Systems. In: Adaptive Dynamic Programming with Applications in Optimal Control. Advances in Industrial Control. Springer, Cham. https://doi.org/10.1007/978-3-319-50815-3_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-50815-3_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-50813-9

  • Online ISBN: 978-3-319-50815-3

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation