On Markov decision models with an absorbing set

Abstract

We study a countable state and action Markov decision process with bounded rewards occurring up to the entrance into an absorbing set. Two optimality criteria are considered, the classical total reward criterion and a target-level criterion. For all discount factors smaller than a critical one, the standard results in dynamic programming (optimality equation, optimality of a decision rule, value iteration) are shown to hold. The value iteration is combined with an extrapolation giving upper and lower bounds to the value function at each step of iteration. The asymptotic behavior of the extrapolation method as well as the characterizations of the critical discount factor are based on the Perron-Frobenius theory for nonlinear operators. The special case of a Markov decision model with a random horizon is studied in detail. Finally, as a byproduct, an efficient computation of the mean entrance time of a Markov chain into an absorbing set is obtained.