Playing Optimally on Timed Automata with Random Delays

  • Nathalie Bertrand
  • Sven Schewe
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7595)


We marry continuous time Markov decision processes (CTMDPs) with stochastic timed automata into a model with joint expressive power. This extension is very natural, as the two original models already share exponentially distributed sojourn times in locations. It enriches CTMDPs with timing constraints, or symmetrically, stochastic timed automata with one conscious player. Our model maintains the existence of optimal control known for CTMDPs. This also holds for a richer model with two players, which extends continuous time Markov games. But we have to sacrifice the existence of simple schedulers: polyhedral regions are insufficient to obtain optimal control even in the single-player case.


Sojourn Time Markov Decision Process Random Delay Time Automaton Nondeterministic Choice 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alur, R., Dill, D.L.: A Theory of Timed Automata. Theoretical Computer Science 126(2), 183–235 (1994)MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Asarin, E., Maler, O., Pnueli, A., Sifakis, J.: Controller Synthesis for Timed Automata. In: Proc. of SCC 1998, pp. 469–474. Elsevier (1998)Google Scholar
  3. 3.
    Baier, C., Bertrand, N., Bouyer, P., Brihaye, T., Größer, M.: Probabilistic and Topological Semantics for Timed Automata. In: Arvind, V., Prasad, S. (eds.) FSTTCS 2007. LNCS, vol. 4855, pp. 179–191. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  4. 4.
    Baier, C., Bertrand, N., Bouyer, P., Brihaye, Th., Größer, M.: Almost-Sure Model Checking of Infinite Paths in One-Clock Timed Automata. In: Proc. of LICS 2008, pp. 217–226. IEEE (2008)Google Scholar
  5. 5.
    Baier, C., Hermanns, H., Katoen, J.-P., Haverkort, B.R.: Efficient Computation of Time-Bounded Reachability Probabilities in Uniform Continuous-Time Markov Decision Processes. Theoretical Computer Science 345(1), 2–26 (2005)MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Bouyer, P., Brihaye, Th., Jurdziński, M., Menet, Q.: Almost-Sure Model-Checking of Reactive Timed Automata. In: Proc. of QEST 2012. IEEE (to appear, 2012)Google Scholar
  7. 7.
    Bouyer, P., Forejt, V.: Reachability in Stochastic Timed Games. In: Albers, S., Marchetti-Spaccamela, A., Matias, Y., Nikoletseas, S., Thomas, W. (eds.) ICALP 2009, Part II. LNCS, vol. 5556, pp. 103–114. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  8. 8.
    Brázdil, T., Forejt, V., Krcál, J., Kretínský, J., Kucera, A.: Continuous-Time Stochastic Games with Time-Bounded Reachability. In: Proc. of FSTTCS 2009. LIPIcs, vol. 4, pp. 61–72. Schloss Dagstuhl - Leibniz-Zentrum für Informatik (2009)Google Scholar
  9. 9.
    Brázdil, T., Krčál, J., Křetínský, J., Kučera, A., Řehák, V.: Stochastic Real-Time Games with Qualitative Timed Automata Objectives. In: Gastin, P., Laroussinie, F. (eds.) CONCUR 2010. LNCS, vol. 6269, pp. 207–221. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  10. 10.
    Chen, T., Han, T., Katoen, J.-P., Mereacre, A.: Model Checking of Continuous-Time Markov Chains Against Timed Automata Specifications. Logical Methods in Computer Science 7(1:12), 1–34 (2011)MathSciNetGoogle Scholar
  11. 11.
    Chen, T., Han, T., Katoen, J.-P., Mereacre, A.: Observing Continuous-Time MDPs by 1-Clock Timed Automata. In: Delzanno, G., Potapov, I. (eds.) RP 2011. LNCS, vol. 6945, pp. 2–25. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  12. 12.
    Chen, T., Han, T., Katoen, J.-P., Mereacre, A.: Reachability Probabilities in Markovian Timed Automata. In: Proc. of CDC-ECC 2011, pp. 7075–7080. IEEE (2011)Google Scholar
  13. 13.
    Fearnley, J., Rabe, M.N., Schewe, S., Zhang, L.: Efficient Approximation of Optimal Control for Continuous-Time Markov Games. In: Proc. of FSTTCS 2011. LIPIcs, vol. 13, pp. 399–410. Schloss Dagstuhl - Leibniz-Zentrum für Informatik (2011)Google Scholar
  14. 14.
    Kwiatkowska, M.Z., Norman, G., Segala, R., Sproston, J.: Automatic Verification of Real-Time Systems with Discrete Probability Distributions. Theoretical Computer Science 282(1), 101–150 (2002)MathSciNetzbMATHCrossRefGoogle Scholar
  15. 15.
    Neuhäußer, M.R., Zhang, L.: Time-Bounded Reachability Probabilities in Continuous-Time Markov Decision Processes. In: Proc. of QEST 2010, pp. 209–218. IEEE (2010)Google Scholar
  16. 16.
    Rabe, M.N., Schewe, S.: Finite Optimal Control for Time-Bounded Reachability in CTMDPs and Continuous-Time Markov Games. Acta Informatica 48(5-6), 291–315 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  17. 17.
    Wolovick, N., Johr, S.: A Characterization of Meaningful Schedulers for Continuous-Time Markov Decision Processes. In: Asarin, E., Bouyer, P. (eds.) FORMATS 2006. LNCS, vol. 4202, pp. 352–367. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  18. 18.
    Zhang, L., Neuhäußer, M.R.: Model Checking Interactive Markov Chains. In: Esparza, J., Majumdar, R. (eds.) TACAS 2010. LNCS, vol. 6015, pp. 53–68. Springer, Heidelberg (2010)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Nathalie Bertrand
    • 1
    • 2
  • Sven Schewe
    • 2
  1. 1.Inria Rennes Bretagne AtlantiqueFrance
  2. 2.University of LiverpoolUK

Personalised recommendations