Skip to main content
SpringerLink
Log in

Linear Programming Formulation of Long-Run Average Optimal Control Problem

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

We formulate and study the infinite-dimensional linear programming problem associated with the deterministic long-run average cost control problem. Along with its dual, it allows one to characterize the optimal value of this control problem. The novelty of our approach is that we focus on the general case wherein the optimal value may depend on the initial condition of the system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Infinite time horizon optimal control problems have been traditionally studied with the help of other (not LP-related) techniques; see, e.g., monographs [20,21,22,23] and references therein.

  2. Extensions of these results to degenerate diffusions appear in [1]; see also [24].

  3. In (23) and everywhere in the paper, \(\limsup _{T\rightarrow \infty }\Gamma _T(y_0)\) and \(\limsup _{\lambda \rightarrow 0}\Theta ^{\lambda }(y_0)\) are understood in the Kuratowski sense. Namely, \(\gamma \in \limsup _{T\rightarrow \infty }\Gamma _T(y_0)\) if and only if there exist sequences, \( \ T_l>0, \ \gamma _l\in \Gamma _{T_l}(y_0),\ l=1,2,\ldots , \) such that \(T_l\rightarrow \infty \) and \( \gamma _l\rightarrow \gamma . \) Similarly, \(\gamma \in \limsup _{\lambda \rightarrow 0}\Theta ^{\lambda }(y_0)\) if and only if there exist sequences, \( \ \lambda _l>0, \ \gamma _l\in \Theta ^{\lambda _l}(y_0),\ l=1,2,\ldots , \) such that \(\lambda _l\rightarrow 0 \) and \(\gamma _l\rightarrow \gamma . \)

  4. Note that \(\Omega (y_0)\ne \emptyset \) if \(\Gamma _\mathrm{per}\ne \emptyset \); see Proposition 3.2.

  5. In [36], the functions in \(\mathcal {H}\) were assumed to be just continuous (not Lipschitz continuous). However, if \(v^*(\cdot ) \) is Lipschitz continuous, then representation (37) is valid with \(\mathcal {H}\) consisting of Lipschitz continuous functions.

References

  1. Bhatt, A.G., Borkar, V.S.: Occupation measures for controlled Markov processes: characterization and optimality. Ann. Probab. 24(3), 1531–1562 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  2. Buckdahn, R., Goreac, D., Quincampoix, M.: Stochastic optimal control and linear programming approach. Appl. Math. Optim. 63(2), 257–276 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  3. Fleming, W.H., Vermes, D.: Convex duality approach to the optimal control of diffusions. SIAM J. Control Optim. 27(5), 1136–1155 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  4. Stockbridge, R.H.: Time-average control of a martingale problem: a linear programming formulation. Ann. Probab. 18, 206–217 (1990)

    Article  MathSciNet  MATH  Google Scholar 

  5. Borkar, V.S.: A convex analytic approach to Markov decision processes. Probab. Theory Relat. Fields 78, 583–602 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  6. Hernandez-Lerma, O., Lasserre, J.B.: The linear programming approach. In: Feinberg, E.A., Shwartz, A. (eds.) Handbook of Markov Decision Processes: Methods and Applications, pp. 377–407. Kluwer, New York (2002)

    Chapter  Google Scholar 

  7. Hordijk, A., Kallenberg, L.C.M.: Linear programming and Markov decision chains. Manag. Sci. 25(4), 352–362 (1979)

    Article  MathSciNet  MATH  Google Scholar 

  8. Hordijk, A., Kallenberg, L.C.M.: Constrained undiscounted stochastic dynamic programming. Math. Oper. Res. 9(2), 276–289 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  9. Klabjan, D., Adelman, D.: An infinite-dimensional linear programming algorithm for deterministic semi-Markov decision processes on Borel spaces. Math. Oper. Res. 32(3), 528–550 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  10. Goreac, D., Serea, O.-S.: Linearization techniques for \(L^{\infty }\)—control problems and dynamic programming principles in classical and \(L^{\infty }\) control problems. ESAIM Control Optim. Calc. Var. 18(3), 836–855 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  11. Hernandez-Hernandez, D., Hernandez-Lerma, O., Taksar, M.: The linear programming approach to deterministic optimal control problems. Appl. Math. 24(1), 17–33 (1996)

    MathSciNet  MATH  Google Scholar 

  12. Lasserre, J.B., Henrion, D., Prieur, C., Trélat, E.: Nonlinear optimal control via occupation measures and LMI-relaxations. SIAM J. Control Optim. 47, 1643–1666 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  13. Vinter, R.: Convex duality and nonlinear optimal control. SIAM J. Control Optim. 31(2), 518–538 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  14. Finlay, L., Gaitsgory, V., Lebedev, I.: Duality in linear programming problems related to long run average problems of optimal control. SIAM J. Control Optim. 47(4), 1667–1700 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  15. Gaitsgory, V.: On representation of the limit occupational measures set of a control systems with applications to singularly perturbed control systems. SIAM J. Optim. 43(1), 325–340 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  16. Gaitsgory, V., Quincampoix, M.: Linear programming approach to deterministic infinite horizon optimal control problems with discounting. SIAM J. Control Optim. 48(4), 2480–2512 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  17. Gaitsgory, V., Quincampoix, M.: On sets of occupational measures generated by a deterministic control system on an infinite time horizon. Nonlinear Anal. Ser. A Theory Methods Appl. 88, 27–41 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  18. Gaitsgory, V., Rossomakhine, S.: On near optimal control of systems with slow observables. SIAM J. Control Optim. 55(3), 1398–1428 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  19. Quincampoix, M., Serea, O.: The problem of optimal control with reflection studied through a linear optimization problem stated on occupational measures. Nonlinear Anal. 72(6), 2803–2815 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  20. Bardi, M., Capuzzo-Dolcetta, I.: Optimal Control and Viscosity Solutions of Hamilton–Jacobi–Bellman Equations. Birkhauser, Boston (1997)

    Book  MATH  Google Scholar 

  21. Carlson, D.A., Haurie, A.B., Leizarowicz, A.: Infinite Horizon Optimal Control. Deterministic and Stochastic Processes. Springer, Berlin (1991)

    Book  Google Scholar 

  22. Zaslavski, A.: Stability of the Turnpike Phenomenon in Discrete-Time Optimal Control Problems. Springer, New York (2014)

    Book  MATH  Google Scholar 

  23. Zaslavski, A.: Turnpike Phenomenon and Infinite Horizon Optimal Control. Springer, New York (2014)

    Book  MATH  Google Scholar 

  24. Arapostathis, A., Borkar, V.S., Ghosh, M.K.: Ergodic Control of Diffusion Processes. Cambridge University Press, Cambridge (2012)

    MATH  Google Scholar 

  25. Borkar, V.S., Gaitsgory, V.: Singular perturbation in ergodic control of diffusion. SIAM J. Control Optim. 46(5), 1562–1577 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  26. Kurtz, T.G., Stockbridge, R.H.: Existence of Markov controls and characterization of optimal Markov controls. SIAM J. Control Optim. 36(2), 609–653 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  27. Rubio, J.E.: Control and Optimization. The Linear Treatment of Nonlinear Problems. Manchester University Press, Manchester (1986)

    MATH  Google Scholar 

  28. Arisawa, M.: Ergodic problem for the Hamilton–Jacobi–Bellman Equation. I. Existence of the ergodic attractor. Ann. Inst. Henri Poincare, Analyse Non Lineaire 14(4), 415–438 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  29. Arisawa, M.: Ergodic problem for the Hamilton–Jacobi–Bellman equation. II. Existence of the ergodic attractor. Ann. Inst. Henri Poincare, Analyse Non Lineaire 15(1), 1–24 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  30. Arisawa, M., Lions, P.-L.: On ergodic stochastic control. Commun. Partial Differ. Equ. 23(11), 2187–2217 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  31. Grüne, L.: Asymptotic controllability and exponential stabilization of nonlinear control systems at singular points. SIAM J. Control Optim. 36(5), 1495–1503 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  32. Grüne, L.: On the relation between discounted and average optimal value functions. J. Differ. Equ. 148, 65–69 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  33. Lehrer, E., Sorin, S.: A uniform Tauberian theorem in dynamic programming. Math. Oper. Res. 17(2), 303–307 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  34. M, Oliu-Barton, Vigeral, G.: A uniform Tauberian theorem in optimal control. In: Cardaliaguet, P., Grossman, R. (eds.) Annals of International Society of Dynamic Games, vol. 12, pp. 199–215. Birkhauser/Springer, New York (2013)

    Google Scholar 

  35. Quincampoix, M., Renault, J.: On existence of a limit value in some non-expansive optimal control problems. SIAM J. Control Optim. 49(5), 2118–2132 (2012)

    Article  MATH  Google Scholar 

  36. Buckdahn, R., Quincampoix, M., Renault, J.: On representation formulas for long run averaging optimal control problem. J. Differ. Equ. 259(11), 5554–5581 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  37. Anderson, E.J.: A review of duality theory for linear programming over topological vector spaces. J. Math. Anal. Appl. 97(2), 380–392 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  38. Anderson, E.J., Nash, P.: Linear Programming in Infinite-Dimensional Spaces. Wiley, Chichester (1987)

    MATH  Google Scholar 

  39. Aubin, J.-P.: Viability Theory. Birkhauser, Basel (1991)

    MATH  Google Scholar 

  40. Ash, R.: Measure, Integration and Functional Analysis. Academic Press, Cambridge (1972)

    MATH  Google Scholar 

  41. Evans, L.G., Gariepy, R.F.: Measure Theory and Fine Properties of Functions. CRC Press, Boca Raton (1992)

    MATH  Google Scholar 

  42. Czarnecki, M.-O., Rifford, L.: Approximation and regularization of Lipschitz functions: convergence of the gradients. Trans. Am. Math. Soc. 358, 4467–4520 (2006)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The work on this paper was initiated, while V.S. Borkar was visiting the Department of Mathematics at Macquarie University. The research was supported in part by a J. C. Bose Fellowship from the Government of India and in part by the Australian Research Council Discovery Grant DP150100618.

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, Indian Institute of Technology Bombay, Powai, Mumbai, 400076, India

    Vivek S. Borkar

  2. Department of Mathematics and Statistics, Macquarie University, Sydney, NSW, 2109, Australia

    Vladimir Gaitsgory

Authors
  1. Vivek S. Borkar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vladimir Gaitsgory
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vladimir Gaitsgory.

Additional information

Communicated by Lars Grüne.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Borkar, V.S., Gaitsgory, V. Linear Programming Formulation of Long-Run Average Optimal Control Problem. J Optim Theory Appl 181, 101–125 (2019). https://doi.org/10.1007/s10957-018-1432-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-018-1432-0

Keywords

Mathematics Subject Classification

Search

Navigation