Advertisement

Linear Programming Formulation of Long-Run Average Optimal Control Problem

  • Vivek S. Borkar
  • Vladimir GaitsgoryEmail author
Article

Abstract

We formulate and study the infinite-dimensional linear programming problem associated with the deterministic long-run average cost control problem. Along with its dual, it allows one to characterize the optimal value of this control problem. The novelty of our approach is that we focus on the general case wherein the optimal value may depend on the initial condition of the system.

Keywords

Long-run average optimal control Linear programming Duality Infinite horizon Vanishing discount limits 

Mathematics Subject Classification

34E15 34C29 93C70 

Notes

Acknowledgements

The work on this paper was initiated, while V.S. Borkar was visiting the Department of Mathematics at Macquarie University. The research was supported in part by a J. C. Bose Fellowship from the Government of India and in part by the Australian Research Council Discovery Grant DP150100618.

References

  1. 1.
    Bhatt, A.G., Borkar, V.S.: Occupation measures for controlled Markov processes: characterization and optimality. Ann. Probab. 24(3), 1531–1562 (1996)MathSciNetzbMATHGoogle Scholar
  2. 2.
    Buckdahn, R., Goreac, D., Quincampoix, M.: Stochastic optimal control and linear programming approach. Appl. Math. Optim. 63(2), 257–276 (2011)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Fleming, W.H., Vermes, D.: Convex duality approach to the optimal control of diffusions. SIAM J. Control Optim. 27(5), 1136–1155 (1989)MathSciNetzbMATHGoogle Scholar
  4. 4.
    Stockbridge, R.H.: Time-average control of a martingale problem: a linear programming formulation. Ann. Probab. 18, 206–217 (1990)MathSciNetzbMATHGoogle Scholar
  5. 5.
    Borkar, V.S.: A convex analytic approach to Markov decision processes. Probab. Theory Relat. Fields 78, 583–602 (1988)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Hernandez-Lerma, O., Lasserre, J.B.: The linear programming approach. In: Feinberg, E.A., Shwartz, A. (eds.) Handbook of Markov Decision Processes: Methods and Applications, pp. 377–407. Kluwer, New York (2002)Google Scholar
  7. 7.
    Hordijk, A., Kallenberg, L.C.M.: Linear programming and Markov decision chains. Manag. Sci. 25(4), 352–362 (1979)MathSciNetzbMATHGoogle Scholar
  8. 8.
    Hordijk, A., Kallenberg, L.C.M.: Constrained undiscounted stochastic dynamic programming. Math. Oper. Res. 9(2), 276–289 (1984)MathSciNetzbMATHGoogle Scholar
  9. 9.
    Klabjan, D., Adelman, D.: An infinite-dimensional linear programming algorithm for deterministic semi-Markov decision processes on Borel spaces. Math. Oper. Res. 32(3), 528–550 (2007)MathSciNetzbMATHGoogle Scholar
  10. 10.
    Goreac, D., Serea, O.-S.: Linearization techniques for \(L^{\infty }\)—control problems and dynamic programming principles in classical and \(L^{\infty }\) control problems. ESAIM Control Optim. Calc. Var. 18(3), 836–855 (2012)MathSciNetzbMATHGoogle Scholar
  11. 11.
    Hernandez-Hernandez, D., Hernandez-Lerma, O., Taksar, M.: The linear programming approach to deterministic optimal control problems. Appl. Math. 24(1), 17–33 (1996)MathSciNetzbMATHGoogle Scholar
  12. 12.
    Lasserre, J.B., Henrion, D., Prieur, C., Trélat, E.: Nonlinear optimal control via occupation measures and LMI-relaxations. SIAM J. Control Optim. 47, 1643–1666 (2008)MathSciNetzbMATHGoogle Scholar
  13. 13.
    Vinter, R.: Convex duality and nonlinear optimal control. SIAM J. Control Optim. 31(2), 518–538 (1993)MathSciNetzbMATHGoogle Scholar
  14. 14.
    Finlay, L., Gaitsgory, V., Lebedev, I.: Duality in linear programming problems related to long run average problems of optimal control. SIAM J. Control Optim. 47(4), 1667–1700 (2008)MathSciNetzbMATHGoogle Scholar
  15. 15.
    Gaitsgory, V.: On representation of the limit occupational measures set of a control systems with applications to singularly perturbed control systems. SIAM J. Optim. 43(1), 325–340 (2004)MathSciNetzbMATHGoogle Scholar
  16. 16.
    Gaitsgory, V., Quincampoix, M.: Linear programming approach to deterministic infinite horizon optimal control problems with discounting. SIAM J. Control Optim. 48(4), 2480–2512 (2009)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Gaitsgory, V., Quincampoix, M.: On sets of occupational measures generated by a deterministic control system on an infinite time horizon. Nonlinear Anal. Ser. A Theory Methods Appl. 88, 27–41 (2013)MathSciNetzbMATHGoogle Scholar
  18. 18.
    Gaitsgory, V., Rossomakhine, S.: On near optimal control of systems with slow observables. SIAM J. Control Optim. 55(3), 1398–1428 (2017)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Quincampoix, M., Serea, O.: The problem of optimal control with reflection studied through a linear optimization problem stated on occupational measures. Nonlinear Anal. 72(6), 2803–2815 (2010)MathSciNetzbMATHGoogle Scholar
  20. 20.
    Bardi, M., Capuzzo-Dolcetta, I.: Optimal Control and Viscosity Solutions of Hamilton–Jacobi–Bellman Equations. Birkhauser, Boston (1997)zbMATHGoogle Scholar
  21. 21.
    Carlson, D.A., Haurie, A.B., Leizarowicz, A.: Infinite Horizon Optimal Control. Deterministic and Stochastic Processes. Springer, Berlin (1991)Google Scholar
  22. 22.
    Zaslavski, A.: Stability of the Turnpike Phenomenon in Discrete-Time Optimal Control Problems. Springer, New York (2014)zbMATHGoogle Scholar
  23. 23.
    Zaslavski, A.: Turnpike Phenomenon and Infinite Horizon Optimal Control. Springer, New York (2014)zbMATHGoogle Scholar
  24. 24.
    Arapostathis, A., Borkar, V.S., Ghosh, M.K.: Ergodic Control of Diffusion Processes. Cambridge University Press, Cambridge (2012)zbMATHGoogle Scholar
  25. 25.
    Borkar, V.S., Gaitsgory, V.: Singular perturbation in ergodic control of diffusion. SIAM J. Control Optim. 46(5), 1562–1577 (2007)MathSciNetzbMATHGoogle Scholar
  26. 26.
    Kurtz, T.G., Stockbridge, R.H.: Existence of Markov controls and characterization of optimal Markov controls. SIAM J. Control Optim. 36(2), 609–653 (1998)MathSciNetzbMATHGoogle Scholar
  27. 27.
    Rubio, J.E.: Control and Optimization. The Linear Treatment of Nonlinear Problems. Manchester University Press, Manchester (1986)zbMATHGoogle Scholar
  28. 28.
    Arisawa, M.: Ergodic problem for the Hamilton–Jacobi–Bellman Equation. I. Existence of the ergodic attractor. Ann. Inst. Henri Poincare, Analyse Non Lineaire 14(4), 415–438 (1997)MathSciNetzbMATHGoogle Scholar
  29. 29.
    Arisawa, M.: Ergodic problem for the Hamilton–Jacobi–Bellman equation. II. Existence of the ergodic attractor. Ann. Inst. Henri Poincare, Analyse Non Lineaire 15(1), 1–24 (1998)MathSciNetzbMATHGoogle Scholar
  30. 30.
    Arisawa, M., Lions, P.-L.: On ergodic stochastic control. Commun. Partial Differ. Equ. 23(11), 2187–2217 (1998)MathSciNetzbMATHGoogle Scholar
  31. 31.
    Grüne, L.: Asymptotic controllability and exponential stabilization of nonlinear control systems at singular points. SIAM J. Control Optim. 36(5), 1495–1503 (1998)MathSciNetzbMATHGoogle Scholar
  32. 32.
    Grüne, L.: On the relation between discounted and average optimal value functions. J. Differ. Equ. 148, 65–69 (1998)MathSciNetzbMATHGoogle Scholar
  33. 33.
    Lehrer, E., Sorin, S.: A uniform Tauberian theorem in dynamic programming. Math. Oper. Res. 17(2), 303–307 (1992)MathSciNetzbMATHGoogle Scholar
  34. 34.
    M, Oliu-Barton, Vigeral, G.: A uniform Tauberian theorem in optimal control. In: Cardaliaguet, P., Grossman, R. (eds.) Annals of International Society of Dynamic Games, vol. 12, pp. 199–215. Birkhauser/Springer, New York (2013)Google Scholar
  35. 35.
    Quincampoix, M., Renault, J.: On existence of a limit value in some non-expansive optimal control problems. SIAM J. Control Optim. 49(5), 2118–2132 (2012)zbMATHGoogle Scholar
  36. 36.
    Buckdahn, R., Quincampoix, M., Renault, J.: On representation formulas for long run averaging optimal control problem. J. Differ. Equ. 259(11), 5554–5581 (2015)MathSciNetzbMATHGoogle Scholar
  37. 37.
    Anderson, E.J.: A review of duality theory for linear programming over topological vector spaces. J. Math. Anal. Appl. 97(2), 380–392 (1983)MathSciNetzbMATHGoogle Scholar
  38. 38.
    Anderson, E.J., Nash, P.: Linear Programming in Infinite-Dimensional Spaces. Wiley, Chichester (1987)zbMATHGoogle Scholar
  39. 39.
    Aubin, J.-P.: Viability Theory. Birkhauser, Basel (1991)zbMATHGoogle Scholar
  40. 40.
    Ash, R.: Measure, Integration and Functional Analysis. Academic Press, Cambridge (1972)zbMATHGoogle Scholar
  41. 41.
    Evans, L.G., Gariepy, R.F.: Measure Theory and Fine Properties of Functions. CRC Press, Boca Raton (1992)zbMATHGoogle Scholar
  42. 42.
    Czarnecki, M.-O., Rifford, L.: Approximation and regularization of Lipschitz functions: convergence of the gradients. Trans. Am. Math. Soc. 358, 4467–4520 (2006)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Electrical EngineeringIndian Institute of Technology BombayPowai, MumbaiIndia
  2. 2.Department of Mathematics and StatisticsMacquarie UniversitySydneyAustralia

Personalised recommendations