Markov decision processes

The finite-state, finite-action Markov decision process is a particularly simple and relatively tractable model of sequential decision making under uncertainty. It has been applied in such diverse fields as health care, highway maintenance, inventory, ma-chine maintenance, cash-flow management, and regulation of water reservoir capacity (Derman, 1970; Hernandez-Lermer, 1989; Ross, 1970; White, 1969). Here we present a definition of a Markov decision process and illustrate it with an example, followed by a discussion of the various solution procedures for several different types of Markov decision processes, all of which are based on dynamic programming (Bertsekas, 1987; Howard, 1971; Puterman, 1994; Sennott, 1999).

PROBLEM FORMULATION

Let k ∈ {0, 1,..., K − 1} represent the kth stage or decision epoch, that is, when the kth decision must be selected; K < ∞ represents the planning horizon of the Markov decision process. Let s _k be the state of the system to be con-trolled at stage k....