Skip to main content
SpringerLink
Log in

Markov decision processes with constraints

  • Session 12 Stochastic Control
  • Conference paper
  • First Online:
Book cover Analysis and Optimization of Systems

Part of the book series: Lecture Notes in Control and Information Sciences ((LNCIS,volume 63))

  • 139 Accesses

Abstract

This article addresses the Markov decision problem with long-run average reward V u when there is a global constraint to be satisfied: I u ≤α, where I u is also a long-run average. Using Lagrange multiplier techniques, existence of an optimal stationary policy is proven. Unlike the unconstrained theory, optimal stationary policies are in general randomized. Structural properties of an optimal policy are determined and the corresponding dynamic programming equations are derived. Finally, conditions are given for the existence of an optimal pure policy and an optimal “almost” bang-bang policy.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Dynkin, E.B. and Yushkevich, A.A., "Controlled Markov Processes," Springer-Verlag, Berlin, 1979.

    Google Scholar 

  2. Lazar, A., "Optimal Flow Control of a Class of Queueing Networks in Equilibrium," IEEE AC-28, November 1983.

    Google Scholar 

  3. Derman, C., "Finite State Markovian Decision Processes," Academic Press, New York, 1970.

    Google Scholar 

  4. Robin, M. "On Optimal Stochastic Control with Constraints," Game Theory and Related Topics, North-Holland, 1979.

    Google Scholar 

  5. Frid, E.B., "On Optimal Strategies in Control Problems with Constraints," Theory of Prob. Appl., Vol. XVIII, No. 1, 1972.

    Google Scholar 

  6. Ross, Sheldon, "Applied Probability Models with Optimization Applications," Holden-Day, San Francisco, 1970.

    Google Scholar 

  7. Kemeny, J. and Snell, J., "Finite Markov Chains," D. Van Nostrand Company, New York, 1960.

    Google Scholar 

  8. Kato, T., "A Short Introduction to Perturbation Theory for Linear Operators," Springer-Verlag, New York, 1982.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Michigan, USA

    Keith W. Ross

  2. Inria, France

    Keith W. Ross

Authors
  1. Keith W. Ross
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

A. Bensoussan J. L. Lions

Rights and permissions

Reprints and permissions

Copyright information

© 1984 Springer-Verlag

About this paper

Cite this paper

Ross, K.W. (1984). Markov decision processes with constraints. In: Bensoussan, A., Lions, J.L. (eds) Analysis and Optimization of Systems. Lecture Notes in Control and Information Sciences, vol 63. Springer, Berlin, Heidelberg. https://doi.org/10.1007/BFb0006280

Download citation

  • DOI: https://doi.org/10.1007/BFb0006280

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-13552-4

  • Online ISBN: 978-3-540-39010-7

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation