Markov Processes and Discrete-Time Chains
Markov property and transition kernels; finite-dimensional distributions and existence; space and time homogeneity; strong Markov property and excursions; invariant distributions and stationarity; recurrence and transience; ergodic behavior of irreducible chains; mean recurrence times
Unable to display preview. Download preview PDF.
- Markov chains in discrete time and with finitely many states were introduced by Markov (1906), who proved the first ergodic theorem, assuming the transition probabilities to be strictly positive. Kolmogorov (1936a-b) extended the theory to countable state spaces and arbitrary transition probabilities. In particular, he noted the decomposition of the state space into irreducible sets, classified the states with respect to recurrence and periodicity, and described the asymptotic behavior of the n-step transition probabilities. Kolmogorov’s original proofs were analytic. The more intuitive coupling approach was introduced by Doeblin (1938), long before the strong Markov property had been formalized.Google Scholar
- Bachelier had noted the connection between random walks and diffusions, which inspired Kolmogorov (1931a) to give a precise definition of Markov processes in continuous time. His treatment is purely analytic, with the distribution specified by a family of transition kernels satisfying the Chapman-Kolmogorov relation, previously noted in special cases by Chapman (1928) and Smoluchovsky.Google Scholar
- Kolmogorov (1931a) makes no reference to sample paths. The transition to probabilistic methods began with the work of Lévy (1934–35) and Doeblin (1938). Though the strong Markov property was used informally by those authors (and indeed already by Bachelier (1900, 1901)), the result was first stated and proved in a special case by Doob (1945). General filtrations were introduced in Markov process theory by Blumenthal (1957). The modern setup, with a canonical process X defined on the path space Ω, equipped with a filtration F, a family of shift operators θt, and a collection of probability measures P x, was developed systematically by Dynkin (1961,1965). A weaker form of Theorem 8.23 appears in Blumenthal and Getoor (1968), and the present version is from Kallenberg (1987, 1998).Google Scholar
- Elementary introductions to Markov processes appear in many textbooks, such as Rogers and Williams (2000a) and Chung (1982). More detailed or advanced accounts are given by Dynkin (1965), Blumenthal and Getoor (1968), Ethier and Kurtz (1986), Dellacherie and Meyer (1975–87), and Sharpe (1988). Feller (1968) gives a masterly introduction to Markov chains, later imitated by many authors. More detailed accounts of the discrete-time theory appear in Kemeny et al. (1966) and Freedman (1971a). The coupling method fell into oblivion after Doeblin’s untimely death in 1940 but has recently enjoyed a revival, meticulously documented by Lindvall (1992) and Thorisson (2000).Google Scholar