Skip to main content
SpringerLink
Log in

Dynamic Programming Viscosity Solution Approach and Its Applications to Optimal Control Problems

  • Chapter
  • First Online:
Mathematics Applied to Engineering, Modelling, and Social Issues

Part of the book series: Studies in Systems, Decision and Control ((SSDC,volume 200))

Abstract

This chapter is concerned with optimal control problems of dynamical systems described by partial differential equations (PDEs). Firstly, using the Dubovitskii-Milyutin approach, we obtain the necessary condition of optimality, i.e., the Pontryagin maximum principle for optimal control problem of an age-structured population dynamics for spread of universally fatal diseases. Secondly, for an optimal birth control problem of a McKendrick type age-structured population dynamics, we establish the optimal feedback control laws by the dynamic programming viscosity solution (DPVS) approach. Finally, for a well-adapted upwind finite-difference numerical scheme for the HJB equation arising in optimal control, we prove its convergence and show that the solution from this finite-difference scheme converges to the value function of the associated optimal control problem.

This work was supported in part by the National Natural Science Foundation of China under Grant No. 11471036.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Sussmann, H.J., Willems, J.C.: 300 years of optimal control: from the brachystochrone to the maximum principle. IEEE Control Syst. 17, 32–44 (1997)

    Google Scholar 

  2. Ho, Y.C.: On centralized optimal control. IEEE Trans. Autom. Control 60, 537–538 (2005)

    MathSciNet  MATH  Google Scholar 

  3. Fattorini, H.O.: Infinite-Dimensional Optimization and Control Theory. Encyclopedia of Mathematics and Its Applications. Cambridge University Press, Cambridge (1999)

    Google Scholar 

  4. Guo, B.Z., Chai, S.G.: Control Theory of Infinite-Dimensional Systems. Science Press, Beijing (2012). (in Chinese)

    Google Scholar 

  5. Rubio, J.E.: Control and Optimization: The Linear Treatment of Nonlinear Problems. Manchester University Press, Manchester (1986)

    MATH  Google Scholar 

  6. Bryson Jr., A.E.: Optimal Control—1950 to 1985. IEEE Control Syst. 13, 26–33 (1996)

    Google Scholar 

  7. von Stryk, O., Bulirsch, R.: Direct and indirect methods for trajectory optimization. Ann. Oper. Res. 37, 357–373 (1992)

    MathSciNet  MATH  Google Scholar 

  8. Teo, K.L., Goh, C.J., Wong, K.H.: A Unified Computational Approach to Optimal Control Problems. Longman Scientific & Technical, New York (1991)

    MATH  Google Scholar 

  9. Rehbock, V., Wang, S., Teo, K.L.: Computing optimal control with hyperbolic partial differential equation. J. Aust. Math. Soc. Ser. B 40, 266–287 (1998)

    MathSciNet  MATH  Google Scholar 

  10. Teo, K.L., Jennings, L.S., Lee, H.W.J., Rehbock, V.: The control parametrization enhancing transform for constrained optimal control problems. J. Aust. Math. Soc. Ser. B 40, 314–335 (1999)

    MATH  Google Scholar 

  11. Jennings, L.S., Teo, K.L., Goh, C.J.: MISER3.2—Optimal Control Software: Theory and User Manual. Department of Mathematics, The University of Western Australia, Australia (1997)

    Google Scholar 

  12. Rehbock, V., Teo, K.L., Jennings, L.S.: Suboptimal feedback control for a class of nonlinear systems using spline interpolation. Discrete Continuous Dyn. Syst. 1, 223–236 (1995)

    MathSciNet  MATH  Google Scholar 

  13. Augustin, D., Maurer, H.: Computational sensitivity analysis for state constrained optimal control problems. Ann. Oper. Res. 101, 75–99 (2001)

    MathSciNet  MATH  Google Scholar 

  14. Liu, Y., Ito, S., Lee, H.W.J., Teo, K.L.: Semi-infinite programming approach to continuously-constrained linear-quadratic optimal control problems. J. Optim. Theory Appl. 108, 617–632 (2001)

    MathSciNet  MATH  Google Scholar 

  15. Malanowski, K., Maurer, H., Pickenhain, S.: Second-order sufficient conditions for state-constrained optimal control problems. J. Optim. Theory Appl. 123, 595–617 (2004)

    MathSciNet  MATH  Google Scholar 

  16. Maurer, H., Oberle, H.J.: Second order sufficient conditions for optimal control problems with free final time: the Riccati approach. SIAM J. Control Optim. 41, 380–403 (2002)

    MathSciNet  MATH  Google Scholar 

  17. Maurer, H., Osmolovskii, N.P.: Second order sufficient conditions for time-optimal bang-bang control. SIAM J. Control Optim. 42, 2239–2263 (2004)

    MathSciNet  MATH  Google Scholar 

  18. Crandall, M.G.: Viscosity solutions: a primer. In: Capuzzo-Dolcetta, I., Lions, P.L. (eds.) Viscosity Solutions and Applications. Lecture Notes in Mathematics, vol. 1660, pp. 1–43. Springer, Berlin (1997)

    Google Scholar 

  19. Bardi, M., Capuzzo-Dolcetta, I.: Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations. Maurizio Falcone and Pierpaolo Soravia. Birkhäuser, Boston (1997)

    MATH  Google Scholar 

  20. Barron, E.N.: Application of viscosity solutions of infinite-dimensional Hamilton-Jacobi-Bellman equations to some problems in distributed optimal control. J. Optim. Theory Appl. 64, 245–268 (1990)

    MathSciNet  MATH  Google Scholar 

  21. Cannarsa, P., Gozzi, F., Soner, H.M.: A dynamic programming approach to nonlinear boundary control problems of parabolic type. J. Funct. Anal. 117, 25–61 (1993)

    MathSciNet  MATH  Google Scholar 

  22. Gozzi, F.S., Sritharan, S., Świȩch, A.: Viscosity solutions of dynamic-programming equations for the optimal control of the two-dimensional Navier-Stokes equations. Arch. Ration. Mech. Anal. 163, 295–327 (2002)

    MathSciNet  MATH  Google Scholar 

  23. Kocan, M., Soravia, P.: A viscosity approach to infinite-dimensional Hamilton-Jacobi equations arising in optimal control with state constraints. SIAM J. Control Optim. 36, 1348–1375 (1998)

    MathSciNet  MATH  Google Scholar 

  24. Yong, J.M., Zhou, X.Y.: Stochastic Controls: Hamiltonian Systems and HJB Equations. In: Applications of Mathematics (New York), vol. 43. Springer, New York (1999)

    Google Scholar 

  25. Fleming, W.H., Soner, H.M.: Controlled Markov Processes and Viscosity Solutions. In: Applications of Mathematics, vol. 25. Springer, New York (1993)

    Google Scholar 

  26. Sun, B., Guo, B.Z.: Convergence of an upwind finite-difference scheme for Hamilton-Jacobi-Bellman equation in optimal control. IEEE Trans. Autom. Control. 60(11), 3012–3017 (2015)

    MathSciNet  MATH  Google Scholar 

  27. Guo, B.Z., Sun, B.: Dynamic programming approach to the numerical solution of optimal control with paradigm by a mathematical model for drug therapies of HIV/AIDS. Optim. Eng. 15(1), 119–136 (2014)

    MathSciNet  MATH  Google Scholar 

  28. Guo, B.Z., Sun, B.: A new algorithm for finding numerical solutions of optimal feedback control. IMA J. Math. Control Inf. 26(1), 95–104 (2009)

    MathSciNet  MATH  Google Scholar 

  29. Guo, B.Z., Sun, B.: Numerical solution to the optimal feedback control of continuous casting process. J. Global Optim. 39(2), 171–195 (2007)

    MathSciNet  MATH  Google Scholar 

  30. Guo, B.Z., Sun, B.: Numerical solution to the optimal birth feedback control of a population dynamics: viscosity solution approach. Optimal Control Appl. Methods 26(5), 229–254 (2005)

    MathSciNet  Google Scholar 

  31. Huang, C.S., Wang, S., Teo, K.L.: On application of an alternating direction method to Hamilton-Jacobi-Bellman equations. J. Comput. Appl. Math. 166, 153–166 (2004)

    MathSciNet  MATH  Google Scholar 

  32. Wang, S., Jennings, L.S., Teo, K.L.: Numerical solution of Hamilton-Jacobi-Bellman equations by an upwind finite volume method. J. Global Optim. 27, 177–192 (2003)

    MathSciNet  MATH  Google Scholar 

  33. Wang, S., Gao, F., Teo, K.L.: An upwind finite-difference method for the approximation of viscosity solutions to Hamilton-Jacobi-Bellman equations. IMA J. Math. Control Inf. 17, 167–178 (2000)

    MathSciNet  MATH  Google Scholar 

  34. Sun, B., Wu, M.X.: Optimal control of age-structured population dynamics for spread of universally fatal diseases. Appl. Anal. 92, 901–921 (2013)

    MathSciNet  MATH  Google Scholar 

  35. Sun, B.: Optimal control of age-structured population dynamics for spread of universally fatal diseases II. Appl. Anal. 93, 1730–1744 (2014)

    MathSciNet  MATH  Google Scholar 

  36. Lin, F., Muthuraman, K., Lawley, M.: An optimal control theory approach to non-pharmaceutical interventions. BMC Infect. Dis. 10, 32 (2010)

    Google Scholar 

  37. Weisstein, E.W.: SIR Model. MathWorld—A Wolfram Web Resource. http://mathworld.wolfram.com/SIRModel.html

  38. Brauer, F.: Models for the spread of universally fatal diseases. J. Math. Biol. 28, 451–462 (1990)

    MathSciNet  MATH  Google Scholar 

  39. Hethcote, H.W.: The mathematics of infectious diseases. SIAM Rev. 42, 599–653 (2000)

    MathSciNet  MATH  Google Scholar 

  40. Iannelli, M.: Mathematical modelling of epidemics. In: Summer School on Mathematical Models in Life Science: Theory and Simulation, Dobbiaco (2005)

    Google Scholar 

  41. Iannelli, M., Martcheva, M.: Homogeneous dynamical systems and the age-structured SIR model with proportionate mixing incidence. Evolution Equations: Applications to Physics, Industry, Life Sciences and Economics. Progress Nonlinear Differential Equations and Applications, vol. 55, pp. 227–251. Birkhäuser, Basel (2003)

    Google Scholar 

  42. Rowthorn, R.E., Laxminarayan, R., Gilligan, C.A.: Optimal control of epidemics in metapopulations. J. R. Soc. Interface 6, 1135–1144 (2009)

    Google Scholar 

  43. Li, M.Y., Wang, L.C.: Global stability in some SEIR epidemic models. Mathematical Approaches for Emerging and Reemerging Infectious Diseases: Models, Methods, and Theory. IMA Volumes in Mathematics and its Applications, vol. 126, pp. 295–311. Springer, New York (2002)

    Google Scholar 

  44. Behncke, H.: Optimal control of deterministic epidemic. Optim. Control Appl. Meth. 21, 269–285 (2000)

    MathSciNet  MATH  Google Scholar 

  45. Iacoviello, D., Liuzzi, G.: Optimal control for SIR epidemic model: a two treatments strategy. In: The 16th Mediterranean Conference on Control and Automation, pp. 842–847. Congress Centre, Ajaccio (2008)

    Google Scholar 

  46. Clancy, D.: Optimal intervention for epidemic models with general infection and removal rate functions. J. Math. Biol. 39, 309–331 (1999)

    MathSciNet  MATH  Google Scholar 

  47. Morton, R., Wickwire, K.H.: On the optimal control of a deterministic epidemic. Adv. Appl. Probab. 6, 622–635 (1974)

    MathSciNet  MATH  Google Scholar 

  48. Sethi, S.P.: Optimal quarantine programmes for controlling an epidemic spread. J. Oper. Res. Soc. 29, 265–268 (1978)

    MathSciNet  MATH  Google Scholar 

  49. Sethi, S.P., Staats, P.W.: Optimal control of some simple deterministic epidemic models. J. Oper. Res. Soc. 29, 129–136 (1978)

    MathSciNet  MATH  Google Scholar 

  50. Wickwire, K.: Optimal immunization rules for an epidemic with recovery. Adv. Appl. Probab. 9, 222–223 (1977)

    Google Scholar 

  51. Guo, B.Z.: A controlled age-structured population model for the spread of universally fatal diseases. J. Syst. Sci. Math. Sci. 15, 269–278 (1995)

    MathSciNet  MATH  Google Scholar 

  52. Jacob, B., Omrane, A.: Optimal control for age-structured population dynamics of incomplete data. J. Math. Anal. Appl. 370, 42–48 (2010)

    MathSciNet  MATH  Google Scholar 

  53. Chan, W.L., Guo, B.Z.: Optimal birth control of population dynamics. J. Math. Anal. Appl. 144, 532–552 (1989)

    MathSciNet  MATH  Google Scholar 

  54. Chan, W.L., Guo, B.Z.: Optimal birth control of population dynamics. II. Problems with free final time, phase constraints, and mini-max costs. J. Math. Anal. Appl. 146, 523–539 (1990)

    MathSciNet  MATH  Google Scholar 

  55. Sun, B.: Maximum principle for optimal boundary control of the Kuramoto-Sivashinsky equation. J. Franklin Inst. 347, 467–482 (2010)

    MathSciNet  MATH  Google Scholar 

  56. Dubovitskii, A.Y., Milyutin, A.A.: Extremum problems in the presence of constraints. Zh. Vychisl. Mat. i Mat. Fiz. 5, 395–453 (1965) (in Russian). English Translation. USSR Comput. Math. Math. Phys. 5, 1–80 (1965)

    Google Scholar 

  57. Girsanov, I.V.: Lectures on Mathematical Theory of Extremum Problems. Economics and Mathematical Systems. Springer, Berlin (1972)

    MATH  Google Scholar 

  58. Barbu, V., Iannelli, M.: Optimal control of population dynamics. J. Optim. Theory Appl. 102, 1–14 (1999)

    MathSciNet  MATH  Google Scholar 

  59. Aniţa, S.: Optimal harvesting for a nonlinear age-dependent population dynamics. J. Math. Anal. Appl. 226, 6–22 (1998)

    MathSciNet  MATH  Google Scholar 

  60. Aniţa, S., Iannelli, M., Kim, M.-Y., Park, E.-J.: Optimal harvesting for periodic age-dependent population dynamics. SIAM J. Appl. Math. 58, 1648–1666 (1998)

    MathSciNet  MATH  Google Scholar 

  61. Song, J., Yu, J.Y.: Population System Control. Springer, Berlin (1988)

    MATH  Google Scholar 

  62. Department of Demographic Census in National Bureau of Statistics of China, China Population Statistics Yearbook, Scientific and Technical Documents Publishing House, Beijing (1990) (in Chinese)

    Google Scholar 

  63. Guo, B.Z., Yao, C.Z.: New results on the exponential stability of non-stationary population dynamics. Acta Math. Sci. (English Ed.) 16, 330–337 (1996)

    MathSciNet  MATH  Google Scholar 

  64. Rudin, W.: Functional Analysis, 2nd edn. McGraw-Hill Inc., New York (1991)

    MATH  Google Scholar 

  65. Yung, S.P.: Multiprocess optimal control problem on Hilbert space and related Hamilton-Jacobi equation. Nonlinear Anal. 29, 1319–1342 (1997)

    MathSciNet  MATH  Google Scholar 

  66. Gozzi, F., Świȩch, A., Zhou, X.Y.: A corrected proof of the stochastic verification theorem within the framework of viscosity solutions. SIAM J. Control Optim. 43, 2009–2019 (2005)

    MathSciNet  MATH  Google Scholar 

  67. Zhou, X.Y.: Verification theorems within the framework of viscosity solutions. J. Math. Anal. Appl. 177, 208–225 (1993)

    MathSciNet  MATH  Google Scholar 

  68. Li, R.H., Feng, G.C.: Numerical Solution of Differential Equations. Higher Education Press, Shanghai (1980). (in Chinese)

    Google Scholar 

  69. Crandall, M.G., Lions, P.L.: Viscosity solutions of Hamilton-Jacobi equations. Trans. Am. Math. Soc. 277, 1–42 (1983)

    MathSciNet  MATH  Google Scholar 

  70. Crandall, M.G., Lions, P.L.: Two approximations of solutions of Hamilton-Jacobi equations. Math. Comp. 43, 1–19 (1984)

    MathSciNet  MATH  Google Scholar 

  71. Falcone, M., Lanucara, P., Seghini, A.: A splitting algorithm for Hamilton-Jacobi-Bellman equations. Appl. Numer. Math. 15, 207–218 (1994)

    MathSciNet  MATH  Google Scholar 

  72. McEneaney, W.M.: Max-Plus Methods for Nonlinear Control and Estimation. Birkhäuser, Boston (2006)

    MATH  Google Scholar 

  73. McEneaney, W.M.: Convergence rate for a curse-of-dimensionality-free method for Hamilton-Jacobi-Bellman PDEs represented as maxima of quadratic forms. SIAM J. Control Optim. 48, 2651–2685 (2009)

    MathSciNet  MATH  Google Scholar 

  74. Horowitz, M.B., Damle, A., Burdick, J.W.: Linear Hamilton Jacobi Bellman equations in high dimensions. In: The 53rd IEEE Conference on Decision and Control, pp. 5880–5887. IEEE Press, Los Angeles (2014)

    Google Scholar 

  75. Gombao, S.: Approximation of optimal controls for semilinear parabolic PDE by solving Hamilton-Jacobi-Bellman equations. In: Gilliam, D.S., Rosenthal, J. (eds.) Electronic Proceedings of 15th International Symposium on the Mathematical Theory of Networks and Systems, University of Notre Dame, South Bend (2002)

    Google Scholar 

  76. Guo, B.Z., Wu, T.T.: Approximation of optimal feedback control: a dynamic programming approach. J. Global Optim. 46, 395–422 (2010)

    MathSciNet  MATH  Google Scholar 

  77. Kunisch, K., Xie, L.: POD-based feedback control of the Burgers equation by solving the evolutionary HJB equation. Comput. Math. Appl. 49, 1113–1126 (2005)

    MathSciNet  MATH  Google Scholar 

  78. Lantoine, G., Epenoy, R.: Quadratically constrained linear-quadratic regulator approach for finite-thrust orbital rendezvous. J. Guid. Control Dyn. 35, 1787–1797 (2012)

    Google Scholar 

  79. Tang, G.Y., Wang, H.H.: Successive approximation approach of optimal control for nonlinear discrete-time system. Int. J. Syst. Sci. 36, 153–161 (2005)

    MathSciNet  MATH  Google Scholar 

  80. Bokanowski, O., Megdich, N., Zidani, H.: Convergence of a non-monotone scheme for Hamilton-Jacobi-Bellman equations with discontinuous initial data. Numer. Math. 115, 1–44 (2010)

    MathSciNet  MATH  Google Scholar 

  81. Jensen, M., Smears, I.: On the convergence of finite element methods for Hamilton-Jacobi-Bellman equations. SIAM J. Numer. Anal. 51, 137–162 (2013)

    MathSciNet  MATH  Google Scholar 

  82. Souganidis, P.E.: Approximation schemes for viscosity solution of Hamilton-Jacobi equations. J. Differ. Equ. 59, 1–43 (1985)

    MathSciNet  MATH  Google Scholar 

  83. Barles, G., Souganidis, P.E.: Convergence of approximation schemes for fully nonlinear second order equations. Asymptotic Anal. 4, 271–283 (1991)

    MathSciNet  MATH  Google Scholar 

  84. Strikwerda, J.C.: Finite difference methods for the Stokes and Navier-Stokes equations. SIAM. J. Sci. Stat. Comput. 5, 56–68 (1984)

    MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Mathematics and Statistics, Beijing Institute of Technology, Beijing, 100081, China

    Bing Sun, Zhen-Zhen Tao & Yang-Yang Wang

  2. Beijing Key Laboratory on MCAACI, Beijing Institute of Technology, Beijing, 100081, China

    Bing Sun

Authors
  1. Bing Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhen-Zhen Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yang-Yang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bing Sun .

Editor information

Editors and Affiliations

  1. Department of Mathematics, University College London, London, UK

    Frank T. Smith

  2. Department of Mathematics, Gauhati University, Guwahati, Assam, India

    Hemen Dutta

  3. Department of Mathematics, Center for Mathematics of Uncertainty, Creighton University, Omaha, NE, USA

    John N. Mordeson

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Sun, B., Tao, ZZ., Wang, YY. (2019). Dynamic Programming Viscosity Solution Approach and Its Applications to Optimal Control Problems. In: Smith, F.T., Dutta, H., Mordeson, J.N. (eds) Mathematics Applied to Engineering, Modelling, and Social Issues. Studies in Systems, Decision and Control, vol 200. Springer, Cham. https://doi.org/10.1007/978-3-030-12232-4_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-12232-4_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-12231-7

  • Online ISBN: 978-3-030-12232-4

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation