Monotone Optimal Policies for Left-Skip-Free Markov Decision Processes

Abstract

In a previous paper (Stidham and Weber [9]), we considered a variety of models for optimal control of the service rate in a queueing system, in which the objective is to minimize the limiting average expected cost per unit time. By standard techniques, we showed how to convert such a problem into an equivalent problem in which the objective is to minimize the expected total (undiscounted) cost until the first entrance into state zero. Under weak assumptions on the one-stage (service plus holding) costs and transition probabilities, we showed that an optimal policy is monotonic, that is, a larger service rate is used in larger states. In contrast to previous models in the literature on control of queues, we assumed that the holding cost was nondecreasing, but not necessarily convex, in the state. A common assumption in all the models was that services take place one at a time, so that the state transitions are skip-free to the left: a one-step transition from state i to a state j < i − 1 is impossible. Many queueing models have this property, including all birth—death models, as well as a variety of M/GI/1-type models, including models with batch arrivals, phase-type service times, and LCFS-PR queue discipline.