Interactive Markov chains (IMCs) constitute a powerful sto- chastic model that extends both continuous-time Markov chains and labelled transition systems. IMCs enable a wide range of modelling and analysis techniques and serve as a semantic model for many industrial and scientific formalisms, such as AADL, GSPNs and many more. Applications cover various engineering contexts ranging from industrial system-on-chip manufacturing to satellite designs. We present a survey of the state-of-the-art in modelling and analysis of IMCs.

We cover a set of techniques that can be utilised for compositional modelling, state space generation and reduction, and model checking. The significance of the presented material and corresponding tools is highlighted through multiple case studies.


Interactive Transition Goal State Sojourn Time Label Transition System Markovian State 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    de Alfaro, L.: Formal Verification of Probabilistic Systems. Ph.D. thesis, Stanford University (1997)Google Scholar
  2. 2.
    de Alfaro, L.: How to specify and verify the long-run average behavior of probabilistic systems. In: Proceedings of the 13th Annual IEEE Symposium on Logic in Computer Science (LICS), pp. 454–465. IEEE (1998)Google Scholar
  3. 3.
    de Alfaro, L.: Computing minimum and maximum reachability times in probabilistic systems. In: Baeten, J.C.M., Mauw, S. (eds.) CONCUR 1999. LNCS, vol. 1664, pp. 66–81. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  4. 4.
    Ash, R., Doléans-Dade, C.: Probability & Measure Theory. Academic Press (2000)Google Scholar
  5. 5.
    Baier, C., Haverkort, B.R., Hermanns, H., Katoen, J.P.: Model-checking algorithms for continuous-time Markov chains. IEEE Transactions on Software Engineering 29(6), 524–541 (2003)CrossRefzbMATHGoogle Scholar
  6. 6.
    Baier, C., Hermanns, H., Katoen, J.P., Haverkort, B.R.: Efficient computation of time-bounded reachability probabilities in uniform continuous-time Markov decision processes. Theoretical Computer Science 345(1), 2–26 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Baier, C., Katoen, J.P.: Principles of model checking, vol. 950. MIT Press (2008)Google Scholar
  8. 8.
    Bertsekas, D.P., Tsitsiklis, J.N.: An analysis of stochastic shortest path problems. Mathematics of Operations Research 16(3), 580–595 (1991)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Bianco, A., de Alfaro, L.: Model checking of probabalistic and nondeterministic systems. In: Thiagarajan, P.S. (ed.) FSTTCS 1995. LNCS, vol. 1026, pp. 499–513. Springer, Heidelberg (1995)CrossRefGoogle Scholar
  10. 10.
    Böde, E., Herbstritt, M., Hermanns, H., Johr, S., Peikenkamp, T., Pulungan, R., Rakow, J., Wimmer, R., Becker, B.: Compositional dependability evaluation for STATEMATE. IEEE Transactions on Software Engineering 35(2), 274–292 (2009)CrossRefGoogle Scholar
  11. 11.
    Boudali, H., Dugan, J.B.: A Bayesian network reliability modeling and analysis framework. IEEE Transactions on Reliability 55, 86–97 (2005)CrossRefGoogle Scholar
  12. 12.
    Boudali, H., Crouzen, P., Stoelinga, M.: Dynamic fault tree analysis using input/output interactive Markov chains. In: Proceedings of the 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pp. 708–717 (2007)Google Scholar
  13. 13.
    Bozzano, M., Cimatti, A., Katoen, J.P., Nguyen, V.Y., Noll, T., Roveri, M.: Safety, dependability and performance analysis of extended AADL models. The Computer Journal 54(5), 754–775 (2011)CrossRefGoogle Scholar
  14. 14.
    Bravetti, M., Hermanns, H., Katoen, J.P.: YMCA: Why Markov chain algebra? Electronic Notes in Theoretical Computer Science (ENTCS) 162, 107–112 (2006)CrossRefzbMATHGoogle Scholar
  15. 15.
    Brázdil, T., Hermanns, H., Krcál, J., Kretínský, J., Rehák, V.: Verification of open interactive Markov chains. In: IARCS Annual Conference on Foundations of Software Technology and Theoretical Computer Science (FSTTCS), pp. 474–485 (2012)Google Scholar
  16. 16.
    Buchholz, P., Hahn, E.M., Hermanns, H., Zhang, L.: Model checking algorithms for CTMDPs. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 225–242. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  17. 17.
    Chatterjee, K., Henzinger, M.: Faster and dynamic algorithms for maximal end-component decomposition and related graph problems in probabilistic verification. In: Proceedings of the Twenty-second Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pp. 1318–1336. SIAM (2011)Google Scholar
  18. 18.
    Coste, N., Garavel, H., Hermanns, H., Lang, F., Mateescu, R., Serwe, W.: Ten Years of Performance Evaluation for Concurrent Systems Using CADP. In: Margaria, T., Steffen, B. (eds.) ISoLA 2010, Part II. LNCS, vol. 6416, pp. 128–142. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  19. 19.
    Coste, N., Hermanns, H., Lantreibecq, E., Serwe, W.: Towards performance prediction of compositional models in industrial GALS designs. In: Bouajjani, A., Maler, O. (eds.) CAV 2009. LNCS, vol. 5643, pp. 204–218. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  20. 20.
    Crouzen, P., Hermanns, H.: Aggregation ordering for massively compositional models. In: Proceedings of the10th International Conference onApplication of Concurrency to System Design (ACSD), pp. 171–180. IEEE (June 2010)Google Scholar
  21. 21.
    Crouzen, P., Hermanns, H., Zhang, L.: On the minimisation of acyclic models. In: van Breugel, F., Chechik, M. (eds.) CONCUR 2008. LNCS, vol. 5201, pp. 295–309. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  22. 22.
    Eisentraut, C., Hermanns, H., Zhang, L.: Concurrency and composition in a stochastic world. In: Gastin, P., Laroussinie, F. (eds.) CONCUR 2010. LNCS, vol. 6269, pp. 21–39. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  23. 23.
    Eisentraut, C., Hermanns, H., Zhang, L.: On probabilistic automata in continuous time. In: Proceedings of the 25th Annual IEEE Symposium on Logic in Computer Science (LICS), pp. 342–351. IEEE (2010)Google Scholar
  24. 24.
    Esteve, M.A., Katoen, J.P., Nguyen, V.Y., Postma, B., Yushtein, Y.: Formal correctness, safety, dependability, and performance analysis of a satellite. In: Proceedings of the 34th International Conference on Software Engineering (ICSE), pp. 1022–1031. IEEE (2012)Google Scholar
  25. 25.
    Giacalone, A., Jou, C., Smolka, S.A.: Algebraic reasoning for probabilistic concurrent systems. In: Proceedings of the IFIP TC2 Working Conference on Programming Concepts and Methods, pp. 443–458 (1990)Google Scholar
  26. 26.
    Guck, D.: Quantitative Analysis of Markov Automata. Master’s thesis, RWTH Aachen University (2012)Google Scholar
  27. 27.
    Guck, D., Han, T., Katoen, J.-P., Neuhäußer, M.R.: Quantitative timed analysis of interactive Markov chains. In: Goodloe, A.E., Person, S. (eds.) NFM 2012. LNCS, vol. 7226, pp. 8–23. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  28. 28.
    Guck, D., Hatefi, H., Hermanns, H., Katoen, J.-P., Timmer, M.: Modelling, reduction and analysis of Markov automata. In: Joshi, K., Siegle, M., Stoelinga, M., D’Argenio, P.R. (eds.) QEST 2013. LNCS, vol. 8054, pp. 55–71. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  29. 29.
    Han, T., Katoen, J.-P., Mereacre, A.: Compositional modeling and minimization of time-inhomogeneous Markov chains. In: Egerstedt, M., Mishra, B. (eds.) HSCC 2008. LNCS, vol. 4981, pp. 244–258. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  30. 30.
    Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Formal Aspects of Computing 6(5), 512–535 (1994)CrossRefzbMATHGoogle Scholar
  31. 31.
    Harel, D., Lachover, H., Naamad, A., Pnueli, A., Politi, M., Sherman, R., Shtull-Trauring, A., Trakhtenbrot, M.B.: STATEMATE: A working environment for the development of complex reactive systems. IEEE Transactions on Software Engineering 16(4), 403–414 (1990)CrossRefGoogle Scholar
  32. 32.
    Hatefi, H., Hermanns, H.: Model checking algorithms for Markov automata. Electronic Communications of the ECEASST 53 (2012)Google Scholar
  33. 33.
    Hatefi, H., Hermanns, H.: Improving time bounded reachability computations in interactive markov chains. In: FSEN, pp. 250–266 (2013)Google Scholar
  34. 34.
    Haverkort, B.R., Kuntz, M., Remke, A., Roolvink, S., Stoelinga, M.: Evaluating repair strategies for a water-treatment facility using Arcade. In: Proceedings of the 40th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pp. 419–424 (2010)Google Scholar
  35. 35.
    Hermanns, H.: Interactive Markov Chains. Springer, Berlin (2002)Google Scholar
  36. 36.
    Hermanns, H., Katoen, J.-P.: The how and why of interactive Markov chains. In: de Boer, F.S., Bonsangue, M.M., Hallerstede, S., Leuschel, M. (eds.) FMCO 2009. LNCS, vol. 6286, pp. 311–338. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  37. 37.
    Hermanns, H., Johr, S.: Uniformity by construction in the analysis of nondeterministic stochastic systems. In: Proceedings of the 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pp. 718–728 (2007)Google Scholar
  38. 38.
    Hermanns, H., Johr, S.: May we reach it? Or must we? In what time? With what probability? In: MMB, pp. 125–140. VDE Verlag (2008)Google Scholar
  39. 39.
    Hermanns, H., Katoen, J.P., Meyer-Kayser, J., Siegle, M.: ETMCC: Model checking performability properties of Markov chains. In: Proceedings of the 33rd International Conference on Dependable Systems and Networks (DSN). IEEE Computer Society (2003)Google Scholar
  40. 40.
    Johr, S.: Model checking compositional Markov systems. Ph.D. thesis, Saarland University (2008)Google Scholar
  41. 41.
    Kanellakis, P.C., Smolka, S.A.: CCS expressions, finite state processes, and three problems of equivalence. Information and Computation 86(1), 43–68 (1990)MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Katoen, J.P., Zapreev, I.S., Hahn, E.M., Hermanns, H., Jansen, D.N.: The Ins and Outs of the probabilistic model checker MRMC. In: Proceedings of the 6th International Conference on the Quantitative Evaluation of Systems (QEST), pp. 167–176. IEEE (2009)Google Scholar
  43. 43.
    Kwiatkowska, M., Norman, G., Parker, D.: PRISM: Probabilistic symbolic model checker. In: Field, T., Harrison, P.G., Bradley, J., Harder, U. (eds.) TOOLS 2002. LNCS, vol. 2324, pp. 200–204. Springer, Heidelberg (2002)Google Scholar
  44. 44.
    Larsen, K.G., Skou, A.: Bisimulation through probabilistic testing (preliminary report). In: Proceedings of the 16th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL), pp. 344–352. ACM (1989)Google Scholar
  45. 45.
    Larsen, K.G., Skou, A.: Bisimulation through probabilistic testing. Information and Computation 94, 1–28 (1991)Google Scholar
  46. 46.
    López, G.G.I., Hermanns, H., Katoen, J.-P.: Beyond memoryless distributions: Model checking semi-Markov chains. In: de Luca, L., Gilmore, S. (eds.) PROBMIV 2001, PAPM-PROBMIV 2001, and PAPM 2001. LNCS, vol. 2165, pp. 57–70. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  47. 47.
    López, N., Núñez, M.: An overview of probabilistic process algebras and their equivalences. In: Baier, C., Haverkort, B.R., Hermanns, H., Katoen, J.-P., Siegle, M. (eds.) Validation of Stochastic Systems. LNCS, vol. 2925, pp. 89–123. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  48. 48.
    Markovski, J.: Towards supervisory control of interactive Markov chains: Controllability. In: 11th International Conference on Application of Concurrency to System Design (ACSD), pp. 108–117 (2011)Google Scholar
  49. 49.
    Markovski, J.: Towards supervisory control of interactive Markov chains: Plant minimization. In: 9th IEEE International Conference on Control and Automation (ICCA), pp. 1195–1200 (2011)Google Scholar
  50. 50.
    Neuhäußer, M.R.: Model checking nondeterministic and randomly timed systems. Ph.D. thesis, RWTH Aachen University (2010)Google Scholar
  51. 51.
    Neuhäußer, M.R., Katoen, J.-P.: Bisimulation and logical preservation for continuous-time Markov decision processes. In: Caires, L., Vasconcelos, V.T. (eds.) CONCUR 2007. LNCS, vol. 4703, pp. 412–427. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  52. 52.
    Pulungan, R.: Reduction of Acyclic Phase-Type Representations. Ph.D. thesis, Universität des Saarlandes, Saarbruecken, Germany (2009)Google Scholar
  53. 53.
    Puterman, M.L.: Markov decision processes: discrete stochastic dynamic programming, vol. 414. John Wiley & Sons (2009)Google Scholar
  54. 54.
    Schrijver, A.: Theory of linear and integer programming. John Wiley & Sons (1998)Google Scholar
  55. 55.
    Segala, R., Lynch, N.: Probabilistic simulations for probabilistic processes. Nordic Journal of Computing 2, 250–273 (1995)MathSciNetzbMATHGoogle Scholar
  56. 56.
    Sen, K., Viswanathan, M., Agha, G.: VESTA: A statistical model-checker and analyzer for probabilistic systems. In: Proceedings of the 2nd International Conference on the Quantitative Evaluation of Systems (QEST), pp. 251–252. IEEE (2005)Google Scholar
  57. 57.
    Timmer, M., Katoen, J.-P., van de Pol, J., Stoelinga, M.I.A.: Efficient modelling and generation of markov automata. In: Koutny, M., Ulidowski, I. (eds.) CONCUR 2012. LNCS, vol. 7454, pp. 364–379. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  58. 58.
    Younes, H.L.S.: Ymer: A statistical model checker. In: Etessami, K., Rajamani, S.K. (eds.) CAV 2005. LNCS, vol. 3576, pp. 429–433. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  59. 59.
    Zhang, L., Neuhäußer, M.R.: Model checking interactive markov chains. In: Esparza, J., Majumdar, R. (eds.) TACAS 2010. LNCS, vol. 6015, pp. 53–68. Springer, Heidelberg (2010)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Florian Arnold
    • 1
  • Daniel Gebler
    • 2
  • Dennis Guck
    • 1
  • Hassan Hatefi
    • 3
  1. 1.Formal Methods & Tools Group, Department of Computer ScienceUniversity of TwenteEnschedeThe Netherlands
  2. 2.Department of Computer ScienceVU University AmsterdamAmsterdamThe Netherlands
  3. 3.Department of Computer ScienceSaarland UniversitySaarbrückenGermany

Personalised recommendations