Skip to main content
SpringerLink
Log in

Efficient Strategy Iteration for Mean Payoff in Markov Decision Processes

  • Conference paper
  • First Online:
Automated Technology for Verification and Analysis (ATVA 2017)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 10482))

Abstract

Markov decision processes (MDPs) are standard models for probabilistic systems with non-deterministic behaviours. Mean payoff (or long-run average reward) provides a mathematically elegant formalism to express performance related properties. Strategy iteration is one of the solution techniques applicable in this context. While in many other contexts it is the technique of choice due to advantages over e.g. value iteration, such as precision or possibility of domain-knowledge-aware initialization, it is rarely used for MDPs, since there it scales worse than value iteration. We provide several techniques that speed up strategy iteration by orders of magnitude for many MDPs, eliminating the performance disadvantage while preserving all its advantages.

This work is partially supported by the German Research Foundation (DFG) project “Verified Model Checkers” and the Czech Science Foundation grant No. 15-17564S.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The usual procedure of achieving this in general is to replace \( Act \) by \(S\times Act \) and adapting \(\mathsf {Av}\), \(\varDelta \), and \(r\) appropriately.

  2. 2.

    Some authors deliberately exclude so called “trivial” or “transient” SCCs, which are single states without a self-loop.

  3. 3.

    The \(\liminf \) is used since the limit may not exist in general for an arbitrary strategy.

  4. 4.

    Note that the procedure found in [34, Sect. 9.2.1] differs from our Algorithm in Line 6. There, the bias is improved over all available actions instead of the gain-optimal ones, which is erroneous. The proofs provided in the corresponding chapter actually prove the correctness of the algorithm as presented here.

  5. 5.

    On crafted models with less than 10 states we observed numerical errors leading to non-convergence and condition numbers of up to \(10^5\).

  6. 6.

    Restricting a general MDP to a MEC results in a “communicating” MDP.

  7. 7.

    We will go into detail why we do not deal with bias later on.

  8. 8.

    Accessible at https://www7.in.tum.de/~meggendo/artifacts/2017/atva_si.txt.

  9. 9.

    http://ojalgo.org/.

References

  1. Abate, A., Češka, M., Kwiatkowska, M.: Approximate policy iteration for Markov Decision Processes via quantitative adaptive aggregations. In: Artho, C., Legay, A., Peled, D. (eds.) ATVA 2016. LNCS, vol. 9938, pp. 13–31. Springer, Cham (2016). doi:10.1007/978-3-319-46520-3_2

    Chapter  Google Scholar 

  2. Ashok, P., Chatterjee, K., Daca, P., Křetínský, J., Meggendorfer, T.: Value iteration for long-run average reward in Markov Decision Processes. In: CAV (2017). To appear

    Google Scholar 

  3. Baier, C., Katoen, J.-P.: Principles of Model Checking (2008)

    Google Scholar 

  4. Baier, C., Klein, J., Leuschner, L., Parker, D., Wunderlich, S.: Ensuring the reliability of your model checker: Interval iteration for Markov Decision Processes. In: CAV (2017). To appear

    Google Scholar 

  5. Bertsekas, D.P.: Approximate policy iteration: a survey and some new methods. J. Control Theor. Appl. 9(3), 310–335 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  6. Björklund, H., Vorobyov, S.G.: A combinatorial strongly subexponential strategy improvement algorithm for mean payoff games. DAM 155(2), 210–229 (2007)

    MathSciNet  MATH  Google Scholar 

  7. Brázdil, T., Chatterjee, K., Chmelík, M., Forejt, V., Křetínský, J., Kwiatkowska, M., Parker, D., Ujma, M.: Verification of Markov Decision Processes using learning algorithms. In: Cassez, F., Raskin, J.-F. (eds.) ATVA 2014. LNCS, vol. 8837, pp. 98–114. Springer, Cham (2014). doi:10.1007/978-3-319-11936-6_8

    Google Scholar 

  8. Brázdil, T., Chatterjee, K., Forejt, V., Kučera, A.: MultiGain: a controller synthesis tool for MDPs with multiple mean-payoff objectives. In: Baier, C., Tinelli, C. (eds.) TACAS 2015. LNCS, vol. 9035, pp. 181–187. Springer, Heidelberg (2015). doi:10.1007/978-3-662-46681-0_12

    Google Scholar 

  9. Brim, L., Chaloupka, J.: Using strategy improvement to stay alive. IJCSIS 23(3), 585–608 (2012)

    MathSciNet  MATH  Google Scholar 

  10. Chatterjee, K., Henzinger, T.: Value iteration. 25 Years of Model Checking, pp. 107–138 (2008)

    Google Scholar 

  11. Condon, A.: On algorithms for simple stochastic games. In: Advances in Computational Complexity Theory, pp. 51–72 (1990)

    Google Scholar 

  12. Courcoubetis, C., Yannakakis, M.: The complexity of probabilistic verification. J. ACM 42(4), 857–907 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  13. de Alfaro, L.: Formal verification of probabilistic systems. Ph.D thesis (1997)

    Google Scholar 

  14. Duflot, M., Fribourg, L., Picaronny, C.: Randomized dining philosophers without fairness assumption. Distrib. Comput. 17(1), 65–76 (2004)

    Article  Google Scholar 

  15. Fearnley, J.: Exponential lower bounds for policy iteration. In: Abramsky, S., Gavoille, C., Kirchner, C., Meyer auf der Heide, F., Spirakis, P.G. (eds.) ICALP 2010. LNCS, vol. 6199, pp. 551–562. Springer, Heidelberg (2010). doi:10.1007/978-3-642-14162-1_46

    Chapter  Google Scholar 

  16. Fearnley, J.: Strategy iteration algorithms for games and Markov Decision Processes. Ph.D thesis, University of Warwick (2010)

    Google Scholar 

  17. Fearnley, J.: Efficient parallel strategy improvement for parity games. In: CAV (2017). To appear

    Google Scholar 

  18. Feng, L., Kwiatkowska, M., Parker, D.: Automated learning of probabilistic assumptions for compositional reasoning. In: Giannakopoulou, D., Orejas, F. (eds.) FASE 2011. LNCS, vol. 6603, pp. 2–17. Springer, Heidelberg (2011). doi:10.1007/978-3-642-19811-3_2

    Chapter  Google Scholar 

  19. Filar, J., Vrieze, K.: Competitive Markov Decision Processes. Springer, New York (1997)

    MATH  Google Scholar 

  20. Frausto-Solis, J., Santiago, E., Mora-Vargas, J.: Cosine policy iteration for solving infinite-horizon Markov Decision Processes. In: Aguirre, A.H., Borja, R.M., Garciá, C.A.R. (eds.) MICAI 2009. LNCS, vol. 5845, pp. 75–86. Springer, Heidelberg (2009). doi:10.1007/978-3-642-05258-3_7

    Chapter  Google Scholar 

  21. Friedmann, O.: An exponential lower bound for the parity game strategy improvement algorithm as we know it. In: LICS, pp. 145–156 (2009)

    Google Scholar 

  22. Gawlitza, T.M., Schwarz, M.D., Seidl, H.: Parametric strategy iteration. arXiv preprint arXiv:1406.5457 (2014)

  23. Haddad, S., Monmege, B.: Reachability in MDPs: refining convergence of value iteration. In: Ouaknine, J., Potapov, I., Worrell, J. (eds.) RP 2014. LNCS, vol. 8762, pp. 125–137. Springer, Cham (2014). doi:10.1007/978-3-319-11439-2_10

    Google Scholar 

  24. Hahn, E.M., Schewe, S., Turrini, A., Zhang, L.: Synthesising strategy improvement and recursive algorithms for solving 2.5 player parity games. In: Bouajjani, A., Monniaux, D. (eds.) VMCAI 2017. LNCS, vol. 10145, pp. 266–287. Springer, Cham (2017). doi:10.1007/978-3-319-52234-0_15

    Chapter  Google Scholar 

  25. Hansen, K.A., Ibsen-Jensen, R., Miltersen, P.B.: The complexity of solving reachability games using value and strategy iteration. Theor. Comput. Syst. 55(2), 380–403 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  26. Hansen, T.D., Miltersen, P.B., Zwick, U.: Strategy iteration is strongly polynomial for 2-player turn-based stochastic games with a constant discount factor. J. ACM 60(1), 1:1–1:16 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  27. Hordijk, A., Puterman, M.L.: On the convergence of policy iteration in finite state undiscounted Markov Decision Processes: the unichain case. MMOR 12(1), 163–176 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  28. Howard, R.A.: Dynamic Programming and Markov Processes (1960)

    Google Scholar 

  29. Komuravelli, A., Păsăreanu, C.S., Clarke, E.M.: Assume-guarantee abstraction refinement for probabilistic systems. In: Madhusudan, P., Seshia, S.A. (eds.) CAV 2012. LNCS, vol. 7358, pp. 310–326. Springer, Heidelberg (2012). doi:10.1007/978-3-642-31424-7_25

    Chapter  Google Scholar 

  30. Křetínský, J., Meggendorfer, T.: Efficient strategy iteration for mean payoff in Markov Decision Processes. Technical report abs/1707.01859. arXiv.org (2017)

  31. Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 585–591. Springer, Heidelberg (2011). doi:10.1007/978-3-642-22110-1_47

    Chapter  Google Scholar 

  32. Kwiatkowska, M., Norman, G., Parker, D., Vigliotti, M.G.: Probabilistic mobile ambients. Theoret. Comput. Sci. 410(12–13), 1272–1303 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  33. Luttenberger, M.: Strategy iteration using non-deterministic strategies for solving parity games. CoRR, abs/0806.2923 (2008)

    Google Scholar 

  34. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley (2014)

    Google Scholar 

  35. Schewe, S.: An optimal strategy improvement algorithm for solving parity and payoff games. In: Kaminski, M., Martini, S. (eds.) CSL 2008. LNCS, vol. 5213, pp. 369–384. Springer, Heidelberg (2008). doi:10.1007/978-3-540-87531-4_27

    Chapter  Google Scholar 

  36. Shlakhter, O., Lee, C.-G.: Accelerated modified policy iteration algorithms for Markov Decision Processes. MMOR 78(1), 61–76 (2013)

    MathSciNet  MATH  Google Scholar 

  37. Tarjan, R.: Depth-first search and linear graph algorithms. SICOMP 1(2), 146–160 (1972)

    Article  MathSciNet  MATH  Google Scholar 

  38. Vöge, J., Jurdziński, M.: A discrete strategy improvement algorithm for solving parity games. In: Emerson, E.A., Sistla, A.P. (eds.) CAV 2000. LNCS, vol. 1855, pp. 202–215. Springer, Heidelberg (2000). doi:10.1007/10722167_18

    Chapter  Google Scholar 

  39. Ye, Y.: The simplex and policy-iteration methods are strongly polynomial for the Markov decision problem with a fixed discount rate. MMOR 36(4), 593–603 (2011)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

We thank the anonymous reviewers for their insightful comments and valuable suggestions. In particular, a considerable improvement to the BSCC compression approach of Sect. 4.2 has been proposed.

Author information

Authors and Affiliations

  1. Technical University of Munich, Munich, Germany

    Jan Křetínský & Tobias Meggendorfer

Authors
  1. Jan Křetínský
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tobias Meggendorfer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tobias Meggendorfer .

Editor information

Editors and Affiliations

  1. Indian Institute of Science, Bangalore, India

    Deepak D'Souza

  2. Chennai Mathematical Institute, Kelambakkam, India

    K. Narayan Kumar

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Křetínský, J., Meggendorfer, T. (2017). Efficient Strategy Iteration for Mean Payoff in Markov Decision Processes. In: D'Souza, D., Narayan Kumar, K. (eds) Automated Technology for Verification and Analysis. ATVA 2017. Lecture Notes in Computer Science(), vol 10482. Springer, Cham. https://doi.org/10.1007/978-3-319-68167-2_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-68167-2_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-68166-5

  • Online ISBN: 978-3-319-68167-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation