Advertisement

Linear Programming Formulation of Long-Run Average Optimal Control Problem

  • Vivek S. Borkar
  • Vladimir GaitsgoryEmail author
Article

Abstract

We formulate and study the infinite-dimensional linear programming problem associated with the deterministic long-run average cost control problem. Along with its dual, it allows one to characterize the optimal value of this control problem. The novelty of our approach is that we focus on the general case wherein the optimal value may depend on the initial condition of the system.

Keywords

Long-run average optimal control Linear programming Duality Infinite horizon Vanishing discount limits 

Mathematics Subject Classification

34E15 34C29 93C70 

Notes

Acknowledgements

The work on this paper was initiated, while V.S. Borkar was visiting the Department of Mathematics at Macquarie University. The research was supported in part by a J. C. Bose Fellowship from the Government of India and in part by the Australian Research Council Discovery Grant DP150100618.

References

  1. 1.
    Bhatt, A.G., Borkar, V.S.: Occupation measures for controlled Markov processes: characterization and optimality. Ann. Probab. 24(3), 1531–1562 (1996)MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Buckdahn, R., Goreac, D., Quincampoix, M.: Stochastic optimal control and linear programming approach. Appl. Math. Optim. 63(2), 257–276 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Fleming, W.H., Vermes, D.: Convex duality approach to the optimal control of diffusions. SIAM J. Control Optim. 27(5), 1136–1155 (1989)MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Stockbridge, R.H.: Time-average control of a martingale problem: a linear programming formulation. Ann. Probab. 18, 206–217 (1990)MathSciNetzbMATHCrossRefGoogle Scholar
  5. 5.
    Borkar, V.S.: A convex analytic approach to Markov decision processes. Probab. Theory Relat. Fields 78, 583–602 (1988)MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Hernandez-Lerma, O., Lasserre, J.B.: The linear programming approach. In: Feinberg, E.A., Shwartz, A. (eds.) Handbook of Markov Decision Processes: Methods and Applications, pp. 377–407. Kluwer, New York (2002)CrossRefGoogle Scholar
  7. 7.
    Hordijk, A., Kallenberg, L.C.M.: Linear programming and Markov decision chains. Manag. Sci. 25(4), 352–362 (1979)MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Hordijk, A., Kallenberg, L.C.M.: Constrained undiscounted stochastic dynamic programming. Math. Oper. Res. 9(2), 276–289 (1984)MathSciNetzbMATHCrossRefGoogle Scholar
  9. 9.
    Klabjan, D., Adelman, D.: An infinite-dimensional linear programming algorithm for deterministic semi-Markov decision processes on Borel spaces. Math. Oper. Res. 32(3), 528–550 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Goreac, D., Serea, O.-S.: Linearization techniques for \(L^{\infty }\)—control problems and dynamic programming principles in classical and \(L^{\infty }\) control problems. ESAIM Control Optim. Calc. Var. 18(3), 836–855 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    Hernandez-Hernandez, D., Hernandez-Lerma, O., Taksar, M.: The linear programming approach to deterministic optimal control problems. Appl. Math. 24(1), 17–33 (1996)MathSciNetzbMATHGoogle Scholar
  12. 12.
    Lasserre, J.B., Henrion, D., Prieur, C., Trélat, E.: Nonlinear optimal control via occupation measures and LMI-relaxations. SIAM J. Control Optim. 47, 1643–1666 (2008)MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Vinter, R.: Convex duality and nonlinear optimal control. SIAM J. Control Optim. 31(2), 518–538 (1993)MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    Finlay, L., Gaitsgory, V., Lebedev, I.: Duality in linear programming problems related to long run average problems of optimal control. SIAM J. Control Optim. 47(4), 1667–1700 (2008)MathSciNetzbMATHCrossRefGoogle Scholar
  15. 15.
    Gaitsgory, V.: On representation of the limit occupational measures set of a control systems with applications to singularly perturbed control systems. SIAM J. Optim. 43(1), 325–340 (2004)MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    Gaitsgory, V., Quincampoix, M.: Linear programming approach to deterministic infinite horizon optimal control problems with discounting. SIAM J. Control Optim. 48(4), 2480–2512 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  17. 17.
    Gaitsgory, V., Quincampoix, M.: On sets of occupational measures generated by a deterministic control system on an infinite time horizon. Nonlinear Anal. Ser. A Theory Methods Appl. 88, 27–41 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  18. 18.
    Gaitsgory, V., Rossomakhine, S.: On near optimal control of systems with slow observables. SIAM J. Control Optim. 55(3), 1398–1428 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
  19. 19.
    Quincampoix, M., Serea, O.: The problem of optimal control with reflection studied through a linear optimization problem stated on occupational measures. Nonlinear Anal. 72(6), 2803–2815 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  20. 20.
    Bardi, M., Capuzzo-Dolcetta, I.: Optimal Control and Viscosity Solutions of Hamilton–Jacobi–Bellman Equations. Birkhauser, Boston (1997)zbMATHCrossRefGoogle Scholar
  21. 21.
    Carlson, D.A., Haurie, A.B., Leizarowicz, A.: Infinite Horizon Optimal Control. Deterministic and Stochastic Processes. Springer, Berlin (1991)CrossRefGoogle Scholar
  22. 22.
    Zaslavski, A.: Stability of the Turnpike Phenomenon in Discrete-Time Optimal Control Problems. Springer, New York (2014)zbMATHCrossRefGoogle Scholar
  23. 23.
    Zaslavski, A.: Turnpike Phenomenon and Infinite Horizon Optimal Control. Springer, New York (2014)zbMATHCrossRefGoogle Scholar
  24. 24.
    Arapostathis, A., Borkar, V.S., Ghosh, M.K.: Ergodic Control of Diffusion Processes. Cambridge University Press, Cambridge (2012)zbMATHGoogle Scholar
  25. 25.
    Borkar, V.S., Gaitsgory, V.: Singular perturbation in ergodic control of diffusion. SIAM J. Control Optim. 46(5), 1562–1577 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  26. 26.
    Kurtz, T.G., Stockbridge, R.H.: Existence of Markov controls and characterization of optimal Markov controls. SIAM J. Control Optim. 36(2), 609–653 (1998)MathSciNetzbMATHCrossRefGoogle Scholar
  27. 27.
    Rubio, J.E.: Control and Optimization. The Linear Treatment of Nonlinear Problems. Manchester University Press, Manchester (1986)zbMATHGoogle Scholar
  28. 28.
    Arisawa, M.: Ergodic problem for the Hamilton–Jacobi–Bellman Equation. I. Existence of the ergodic attractor. Ann. Inst. Henri Poincare, Analyse Non Lineaire 14(4), 415–438 (1997)MathSciNetzbMATHCrossRefGoogle Scholar
  29. 29.
    Arisawa, M.: Ergodic problem for the Hamilton–Jacobi–Bellman equation. II. Existence of the ergodic attractor. Ann. Inst. Henri Poincare, Analyse Non Lineaire 15(1), 1–24 (1998)MathSciNetzbMATHCrossRefGoogle Scholar
  30. 30.
    Arisawa, M., Lions, P.-L.: On ergodic stochastic control. Commun. Partial Differ. Equ. 23(11), 2187–2217 (1998)MathSciNetzbMATHCrossRefGoogle Scholar
  31. 31.
    Grüne, L.: Asymptotic controllability and exponential stabilization of nonlinear control systems at singular points. SIAM J. Control Optim. 36(5), 1495–1503 (1998)MathSciNetzbMATHCrossRefGoogle Scholar
  32. 32.
    Grüne, L.: On the relation between discounted and average optimal value functions. J. Differ. Equ. 148, 65–69 (1998)MathSciNetzbMATHCrossRefGoogle Scholar
  33. 33.
    Lehrer, E., Sorin, S.: A uniform Tauberian theorem in dynamic programming. Math. Oper. Res. 17(2), 303–307 (1992)MathSciNetzbMATHCrossRefGoogle Scholar
  34. 34.
    M, Oliu-Barton, Vigeral, G.: A uniform Tauberian theorem in optimal control. In: Cardaliaguet, P., Grossman, R. (eds.) Annals of International Society of Dynamic Games, vol. 12, pp. 199–215. Birkhauser/Springer, New York (2013)Google Scholar
  35. 35.
    Quincampoix, M., Renault, J.: On existence of a limit value in some non-expansive optimal control problems. SIAM J. Control Optim. 49(5), 2118–2132 (2012)zbMATHCrossRefGoogle Scholar
  36. 36.
    Buckdahn, R., Quincampoix, M., Renault, J.: On representation formulas for long run averaging optimal control problem. J. Differ. Equ. 259(11), 5554–5581 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  37. 37.
    Anderson, E.J.: A review of duality theory for linear programming over topological vector spaces. J. Math. Anal. Appl. 97(2), 380–392 (1983)MathSciNetzbMATHCrossRefGoogle Scholar
  38. 38.
    Anderson, E.J., Nash, P.: Linear Programming in Infinite-Dimensional Spaces. Wiley, Chichester (1987)zbMATHGoogle Scholar
  39. 39.
    Aubin, J.-P.: Viability Theory. Birkhauser, Basel (1991)zbMATHGoogle Scholar
  40. 40.
    Ash, R.: Measure, Integration and Functional Analysis. Academic Press, Cambridge (1972)zbMATHGoogle Scholar
  41. 41.
    Evans, L.G., Gariepy, R.F.: Measure Theory and Fine Properties of Functions. CRC Press, Boca Raton (1992)zbMATHGoogle Scholar
  42. 42.
    Czarnecki, M.-O., Rifford, L.: Approximation and regularization of Lipschitz functions: convergence of the gradients. Trans. Am. Math. Soc. 358, 4467–4520 (2006)MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Electrical EngineeringIndian Institute of Technology BombayPowai, MumbaiIndia
  2. 2.Department of Mathematics and StatisticsMacquarie UniversitySydneyAustralia

Personalised recommendations