Recurrence and Transience of Markov Chains

Abstract

The general problem of convergence in distribution of Markov chains with a countable state space is investigated in this chapter. For the GI/GI/1 queue in Chapter 2, the convergence in distribution of the Markov chain (W _n ) has been obtained by using an explicit representation of the random variable W _n as a functional of the random walk associated with the interarrival times and the service times. Markov chains describing the behavior of most queueing systems cannot, in general, be represented in such a simple way. In this chapter, simple criteria are given to determine whether a given Markov chain is ergodic or transient. The main results are Theorem 8.6 for ergodicity and Theorem 8.10 for transience. In practice, these results can be used in many applications. These criteria can be seen as an extension, in a probabilistic setting, of a classical stability result of ordinary differential equations due to Lyapunov^[Lia07] in 1892. In a stochastic context, the first results of this type are apparently due to Khasminskii for diffusions (see Khasminskii [26]). The stability criterion by Lyapunov is recalled at the end of the chapter (see Hirsch and Smale [27] for a detailed presentation of these questions). In Chapter 9, an important scaling method is introduced that will give a more precise picture of the relation between the stability of ordinary differential equations and the stability properties of Markov