Fluid Approximations to Markov Decision Processes with Local Transitions

Abstract

In this chapter we indicate that when it is scaled properly, an absorbing Markov decision process (MDP) model with an expected total undiscounted cost can be approximated by two of its deterministic and continuous analogues, namely, the standard fluid model and the refined fluid model, at least when the scaling parameter $n = 1,2,\ldots$ grows large. We obtain the level of accuracy of such fluid approximations by showing that the absolute difference between the objective function of the scaled MDP model and the one of the (standard and refined) fluid model goes to zero as fast as $\frac{1} {n}.$ Under some extra conditions, we obtain that given a particular type of policy solving the refined fluid model, it can be translated into a policy, which is nearly optimal for the scaled MDP model.