Skip to main content
SpringerLink
Log in

Concurrent MDPs with Finite Markovian Policies

  • Conference paper
  • First Online:
Measurement, Modelling and Evaluation of Computing Systems (MMB 2020)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 12040))

  • 636 Accesses

Abstract

The recently defined class of Concurrent Markov Decision Processes (CMDPs) allows one to describe scenario based uncertainty in sequential decision problems like scheduling or admission problems. The resulting optimization problem of computing an optimal policy is NP-hard. This paper introduces a new class of policies for CMDPs on infinite horizons. A mixed integer linear program and an efficient approximation algorithm based on policy iteration are defined for the computation of optimal polices. The proposed approximation algorithm also improves the available approximate value iteration algorithm for the finite horizon case.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bertsimas, D., Mišić, V.V.: Robust product line design. Oper. Res. 65(1), 19–37 (2017)

    Article  MathSciNet  Google Scholar 

  2. Bertsimas, D., Silberholz, J., Trikalinos, T.: Optimal healthcare decision making under multiple mathematical models: application in prostate cancer screening. Health Care Manag. Sci. 21(1), 105–118 (2016)

    Article  Google Scholar 

  3. Bertsimas, D., Sim, M.: The price of robustness. Oper. Res. 52(1), 35–53 (2004)

    Article  MathSciNet  Google Scholar 

  4. Buchholz, P.: Markov decision processes with uncertain parameters. http://ls4-www.cs.tu-dortmund.de/download/buchholz/CMDP/CMDP_Description

  5. Buchholz, P., Scheftelowitsch, D.: Computation of weighted sums of rewards for concurrent MDPs. Math. Methods Oper. Res. 89(1), 1–42 (2019)

    Article  MathSciNet  Google Scholar 

  6. Buchholz, P., Scheftelowitsch, D.: Light robustness in the optimization of Markov decision processes with uncertain parameters. Comput. Oper. Res. 108, 69–81 (2019)

    Article  MathSciNet  Google Scholar 

  7. Goyal, V., Grand-Clement, J.: Robust Markov decision process: Beyond rectangularity. CoRR, abs/1811.00215 (2019)

    Google Scholar 

  8. Hager, W.W.: Updating the inverse of a matrix. SIAM Rev. 31(2), 221–239 (1989)

    Article  MathSciNet  Google Scholar 

  9. Iyengar, G.N.: Robust dynamic programming. Math. Oper. Res. 30(2), 257–280 (2005)

    Article  MathSciNet  Google Scholar 

  10. Jünger, M., et al. (eds.): 50 Years of Integer Programming 1958–2008 - From the Early Years to the State-of-the-Art. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-540-68279-0

    Book  Google Scholar 

  11. Mannor, S., Mebel, O., Xu, H.: Robust MDPs with k-rectangular uncertainty. Math. Oper. Res. 41(4), 1484–1509 (2016)

    Article  MathSciNet  Google Scholar 

  12. Nilim, A., Ghaoui, L.E.: Robust control of Markov decision processes with uncertain transition matrices. Oper. Res. 53(5), 780–798 (2005)

    Article  MathSciNet  Google Scholar 

  13. Puterman, M.L.: Markov Decision Processes. Wiley, New York (2005)

    MATH  Google Scholar 

  14. Rockafellar, R.T., Wets, R.J.: Scenarios and policy aggregation in optimization under uncertainty. Math. Oper. Res. 16(1), 119–147 (1991)

    Article  MathSciNet  Google Scholar 

  15. Satia, J.K., Lave, R.E.: Markovian decision processes with uncertain transition probabilities. Oper. Res. 21(3), 728–740 (1973)

    Article  MathSciNet  Google Scholar 

  16. Scheftelowitsch, D.: Markov decision processes with uncertain parameters. Ph.D. thesis, Technical University of Dortmund, Germany (2018)

    Google Scholar 

  17. Serfozo, R.F.: An equivalence between continuous and discrete time Markov decision processes. Oper. Res. 27(3), 616–620 (1979)

    Article  MathSciNet  Google Scholar 

  18. Steimle, L.N.: Stochastic Dynamic Optimization Under Ambiguity. Ph.D. thesis, Industrial and Operations Engineering in the University of Michigan (2019)

    Google Scholar 

  19. Steimle, L.N., Ahluwalia, V., Kamdar, C., Denton, B.T.: Decomposition methods for multi-model Markov decision processes. Technical report, Optimization-online (2018)

    Google Scholar 

  20. Steimle, L.N., Kaufman, D.L., Denton, B.T.: Multi-model Markov decision processes. Technical report, Optimization-online (2018)

    Google Scholar 

  21. White, C.C., Eldeib, H.K.: Markov decision processes with imprecise transition probabilities. Oper. Res. 42(4), 739–749 (1994)

    Article  MathSciNet  Google Scholar 

  22. Wiesemann, W., Kuhn, D., Rustem, B.: Robust Markov decision processes. Math. Oper. Res. 38(1), 153–183 (2013)

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Informatik IV, TU Dortmund, 44221, Dortmund, Germany

    Peter Buchholz & Dimitri Scheftelowitsch

Authors
  1. Peter Buchholz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dimitri Scheftelowitsch
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peter Buchholz .

Editor information

Editors and Affiliations

  1. Saarland University, Saarbrücken, Germany

    Holger Hermanns

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Buchholz, P., Scheftelowitsch, D. (2020). Concurrent MDPs with Finite Markovian Policies. In: Hermanns, H. (eds) Measurement, Modelling and Evaluation of Computing Systems. MMB 2020. Lecture Notes in Computer Science(), vol 12040. Springer, Cham. https://doi.org/10.1007/978-3-030-43024-5_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-43024-5_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-43023-8

  • Online ISBN: 978-3-030-43024-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation