Abstract
Mathematically, stochastic hybrid systems can be described by Markov processes with discrete and continuous behaviours. This chapter provides in an incremental way the necessary mathematical background for understanding such complex systems. The chapter starts with discrete and continuous-time Markov chains. In this book, a Markov chain is understood as a stochastic process with the Markov property (or memoryless property, i.e. its future evolution depends only on the current state) defined on a discrete (finite or countable) state space. The Markov property is illustrated in the Chapman-Kolmogorov equation satisfied by the transition probabilities. Usually, in the continuous-time Markov case, the evolution of these transition probabilities is described by forward/backward Kolmogorov equations. These equations are expressed in terms of the stochastic matrix (called also infinitesimal generator) associated to the Markov chain. This is the matrix of transition rates and the practical use of Markov chains resides in our ability of handling it.
The second part of this chapter is dedicated to Markov processes defined on continuous state spaces that can be equipped with additional structures as sigma-algebras, metrics, topologies, norms or others like specific algebraic structures (lattice, vector space). Characterisations of such processes are usually given using functional analysis operators like: the operator semigroup/resolvent, the infinitesimal generator or other operators that can be associated to a Markov process. At an abstract level, these operators describe the evolution the evolution of transition probabilities for a Markov process. Chapman-Kolmogorov equation, Kolmogorov equations and other properties of these operators (e.g. Dynkin formula, martingale problem) represent the natural tools for studying such Markov processes.
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
An erratum to this chapter can be found at http://dx.doi.org/10.1007/978-1-4471-2795-6_12
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
In the mathematical field of topology, a homeomorphism or topological isomorphism is a continuous function between topological spaces that has a continuous inverse function.
- 2.
I.e., it has a countable dense subset.
References
Arnold, L.: Stochastic Differential Equations: Theory and Applications. Wiley-Interscience, New York (1974)
Beurling, A., Deny, J.: Dirichlet spaces. Proc. Natl. Acad. Sci. USA 45, 208–215 (1959)
Blumenthal, R.M., Getoor, R.K.: Markov Processes and Potential Theory. Academic Press, New York (1968)
Bouleau, N., Lepingle, D.: Numerical Methods for Stochastic Processes. Wiley Series in Probability and Mathematical Statistics (1994)
Davis, M.H.A.: Markov Models and Optimization. Chapman & Hall, London (1993)
Ethier, S.N., Kurtz, T.G.: Markov Processes: Characterization and Convergence. Wiley, New York (1986)
Fukushima, M.: Dirichlet Forms and Markov Processes. North-Holland, Amsterdam (1980)
Gihman, I.I.: On the theory of differential equations of random processes. Ukr. Math. J. 2(4), 37–63 (1950)
Itô, K.: Differential equations determining a Markov process. J. Pan-Jpn. Math. Coll. 1077 (1942) in Japanese; in English Kiyosi Ito Selected Papers. Springer, Berlin (1986)
Kurtz, T.G.: Equivalence of stochastic equations and martingale problems. In: Crisan, D. (ed.) Stochastic Analysis 2010, pp. 113–130. Springer, Berlin (2011)
Ma, M., Rockner, M.: The Theory of (Non-Symmetric) Dirichlet Forms and Markov Processes. Springer, Berlin (1990)
Meyer, P.A.: Processus de Markov. Lecture Notes in Mathematics, vol. 26. Springer, Berlin (1967)
Sharpe, M.: General Theory of Markov Processes. Academic Press, San Diego (1988)
Stroock, D.W., Varadhan, S.R.S.: Multidimensional Diffusion Processes. Springer, Berlin (1979)
Tabuada, P., Pappas, G.J., Lima, P.: Compositional abstractions of hybrid control systems. Discrete Event Dyn. Syst. Theory Appl. 14, 203–238 (2004)
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag London Limited
About this chapter
Cite this chapter
Bujorianu, L.M. (2012). Markov Models. In: Stochastic Reachability Analysis of Hybrid Systems. Communications and Control Engineering. Springer, London. https://doi.org/10.1007/978-1-4471-2795-6_2
Download citation
DOI: https://doi.org/10.1007/978-1-4471-2795-6_2
Published:
Publisher Name: Springer, London
Print ISBN: 978-1-4471-2794-9
Online ISBN: 978-1-4471-2795-6
eBook Packages: EngineeringEngineering (R0)