Abstract
In spite of their simplicity, Markov processes with countable state space (and discrete time) are interesting mathematical objects with which a variety of real-world phenomena can be modeled. We give an introduction to the basic concepts (Markov property, transition matrix, recurrence, transience, invariant distribution) and then study certain examples in more detail. For example, we show how to compute numerically very precisely, the expected number of returns to the origin of simple random walk on multidimensional integer lattices.
The connection with discrete potential theory will be investigated later, in Chapter 19. Some readers might prefer to skip the somewhat technical construction of general Markov processes in Section 17.1.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Breiman L (1968) Probability. Addison-Wesley, Reading
Brémaud P (1999) Markov chains. Texts in applied mathematics, vol 31. Springer, New York. Gibbs fields, Monte Carlo simulation, and queues
Chung KL, Fuchs WHJ (1951) On the distribution of values of sums of random variables. Memoirs of the American Mathematical Society, vol 6
Chung KL (1960) Markov chains with stationary transition probabilities. Die Grundlehren der mathematischen Wissenschaften, vol 104. Springer, Berlin
Dudley RM (2002) Real analysis and probability. Cambridge studies in advanced mathematics, vol 74. Cambridge University Press, Cambridge. Revised reprint of the 1989 original
Durrett R (2010) Probability: theory and examples, 4th edn. Cambridge series in statistical and probabilistic mathematics. Cambridge University Press, Cambridge
Feller W (1968) An introduction to probability theory and its applications, vol I, 3rd edn. Wiley, New York
Georgii H-O (2012) Stochastics: introduction to probability theory and statistics. de Gruyter Lehrbuch, 2nd edn. Walter de Gruyter, Berlin
Gibbs AL, Su FE (2002) On choosing and bounding probability metrics. Int Stat Rev 70(3):419–435
Glasser ML, Zucker IJ (1977) Extended Watson integrals for the cubic lattices. Proc Natl Acad Sci USA 74(5):1800–1801
Grimmett GR, Stirzaker DR (2001) Probability and random processes, 3rd edn. Oxford University Press, New York
Häggström O (2002) Finite Markov chains and algorithmic applications. London Mathematical Society student texts, vol 52. Cambridge University Press, Cambridge
Joyce GS (2003) Singular behaviour of the lattice Green function for the d-dimensional hypercubic lattice. J Phys A 36(4):911–921
Kallenberg O (2002) Foundations of modern probability, 2nd edn. Probability and its applications. Springer, New York
Kantorovič LV, Rubinšteĭn GŠ (1958) On a space of completely additive functions. Vestn Leningr Univ 13(7):52–59
Kemeny JG, Snell JL (1976) Finite Markov chains. Undergraduate texts in mathematics. Springer, New York. Reprinting of the 1960 original
Klenke A, Mattner L (2010) Stochastic ordering of classical discrete distributions. Adv Appl Probab 42(2):392–410
Meyn SP, Tweedie RL (1993) Markov chains and stochastic stability. Communications and control engineering series. Springer, London
Müller A, Stoyan D (2002) Comparison methods for stochastic models and risks. Wiley series in probability and statistics. Wiley, Chichester
Norris JR (1998) Markov chains. Cambridge series in statistical and probabilistic mathematics. Cambridge University Press, Cambridge. Reprint of the 1997 edition
Nummelin E (1984) General irreducible Markov chains and nonnegative operators. Cambridge tracts in mathematics, vol 83. Cambridge University Press, Cambridge
Pólya G (1921) Über eine Aufgabe der Wahrscheinlichkeitsrechnung betreffend die Irrfahrt im Straßennetz. Math Ann 84:149–160
Revuz D (1984) Markov chains, 2nd edn. North-Holland mathematical library, vol 11. North-Holland, Amsterdam
Seneta E (2006) Non-negative matrices and Markov chains. Springer series in statistics. Springer, New York. Revised reprint of the second (1981) edition
Spitzer F (1976) Principles of random walks, 2nd edn. Graduate texts in mathematics, vol 34. Springer, New York
Strassen V (1965) The existence of probability measures with given marginals. Ann Math Stat 36:423–439
Watson GN (1939) Three triple integrals. Q J Math, Oxf Ser 10:266–276
Wright S (1931) Evolution in Mendelian populations. Genetics 16:97–159
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer-Verlag London
About this chapter
Cite this chapter
Klenke, A. (2014). Markov Chains. In: Probability Theory. Universitext. Springer, London. https://doi.org/10.1007/978-1-4471-5361-0_17
Download citation
DOI: https://doi.org/10.1007/978-1-4471-5361-0_17
Publisher Name: Springer, London
Print ISBN: 978-1-4471-5360-3
Online ISBN: 978-1-4471-5361-0
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)