Controlled Markov Chains


In Chap. 7 we considered Markov chains as a means to model stochastic DES for which explicit closed-form solutions can be obtained. Then, in Chap. 8, we saw how special classes of Markov chains (mostly, birth—death chains) can be used to model queueing systems. We pointed out, however, that queueing theory is largely “descriptive” in nature; that is, its main objective is to evaluate the behavior of queueing systems operating under a particular set of rules. On the other hand, we are often interested in “prescriptive” techniques, based on which we can make decisions regarding the “best” way to operate a system and ultimately control its performance. In this chapter, we describe some such techniques for Markov chains. Our main objective is to introduce the framework known as Markov Decision Theory, and to present some key results and techniques which can be used to control DES modeled as Markov chains. At the heart of these techniques is dynamic programming, which has played a critical role in both deterministic and stochastic control theory since the 1960s. The material in this chapter is more advanced than that of previous ones, it involves some results that were published in the research literature fairly recently, and it demands slightly higher mathematical sophistication. The results, however, should be quite gratifying for the reader, as they lead to the solution of some basic problems from everyday life experience, or related to the basic operation of some very familiar DES whose predominant characteristic is resource contention. The last part of this chapter is devoted to some such problems pertaining to queueing systems.


Optimal Policy Queue Length Admission Control Markov Decision Process Dynamic Program Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Selected References

  1. ∎ Bertsekas, D.P., Dynamic Programming: Deterministic and Stochastic Models, Prentice-Hall, Englewood Cliffs, NJ, 1987.MATHGoogle Scholar
  2. ∎ Bertsekas, D.P., Dynamic Programming and Optimal Control, Vols. 1 and 2, Athena Scientific, Belmont, MA, 1995.MATHGoogle Scholar
  3. ∎ Cao,X., Stochastic Learning and Optimization - A Sensitivity-Based Approach, Springer, New York, 2007.MATHGoogle Scholar
  4. ∎ Hajek, B., “Optimal Control of Two Interacting Service Stations,” IEEE Transactions on Automatic Control, Vol. AC-29, pp. 491–499, 1984.CrossRefMathSciNetGoogle Scholar
  5. ∎ Kumar, P.R., and P. Varaiya, Stochastic Systems: Estimation, Identification, and Adaptive Control, Prentice-Hall, Englewood Cliffs, NJ, 1986.MATHGoogle Scholar
  6. ∎ Marshall, A.W., and I. Olkin, Inequalities: Theory of Majoration and its Applica-tions, Academic Press, New York, 1979.Google Scholar
  7. ∎ Pepyne, D.L., and C.G. Cassandras, “Optimal Dispatching Control for Elevator Systems During Uppeak Traffic,” IEEE Transactions on Control Systems Technology, Vol. 5, No. 6, pp. 629–643, 1997.CrossRefGoogle Scholar
  8. ∎ Ross, S.M., Stochastic Processes, Wiley, New York, 1983a.MATHGoogle Scholar
  9. ∎ Ross, S.M., Introduction to Stochastic Dynamic Programming, Academic Press, New York, 1983b.MATHGoogle Scholar
  10. ∎ Stidham, S., Jr., “Optimal Control of Admission to a Queueing System,” IEEE Transactions on Automatic Control, Vol. AC-30, pp. 705–713, 1985.CrossRefMathSciNetGoogle Scholar
  11. ∎ Trivedi, K.S., Probability and Statistics with Reliability, Queuing and Computer Science Applications, Prentice-Hall, Englewood Cliffs, NJ, 1982.Google Scholar
  12. ∎ Walrand, J., An Introduction to Queueing Networks, Prentice-Hall, Englewood Cliffs, NJ, 1988.MATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Personalised recommendations