Continuous-Time Markov Chains

Abstract

A continuous-time Markov chain (CTMC) is a discrete-time Markov chain with the modification that, instead of spending one time unit in a state, it remains in a state for an exponentially distributed time whose rate depends on the state. The methodology of CTMCs is based on properties of renewal and Poisson processes as well as discrete-time chains. CTMCs are natural candidates for modeling systems in real time such as production and inventory systems, computer and telecommunications networks, and miscellaneous input-output systems. Many continuous-time processes have discretetime analogues; for instance, birth-death and Brownian motion processes are continuous-time analogues of discrete-time random walks. One’s choice of a continuous- or discrete-time model for a system typically depends on how realistic it is, its ease in addressing the issues at hand, or in computing quantities of interest.