Markov Chains

Abstract

This chapter explores the properties of a broadly applicable probability model called a Markov chain, named after Russian mathematician A. A. Markov (1856–1922). Markov observed that many real-world phenomena can be modeled as a sequence of “transitions” from one “state” to another, with each transition having some associated uncertainty. For example, a taxi driver might “transition” between several towns (or zones within a large city); each time he drops off a passenger, he can’t be certain where his next fare will want to go. Similarly, a gambler might think of her winnings as transitioning from one “state”—really, a dollar amount—to another; with each round of the game she plays, she cannot be certain whether that dollar amount will go up or down (though, obviously, she hopes it goes up!). The same could be said for modeling the daily closing prices of a stock: each new day, there is uncertainty about whether that stock will “transition” to a higher or lower value, and this uncertainty could be modeled using the tools of probability.