Markov Chains

Part of the The Kluwer International Series on Discrete Event Dynamic Systems book series (DEDS, volume 11)


In the previous chapter, we presented the Generalized Semi-Markov Process (GSMP) framework as a means of modeling stochastic DES. By allowing event clocks to tick at varying speeds, we also provided an extension to the basic GSMP. In addition, we introduced the Poisson process as a basic building block for a class of stochastic DES which possess the Markov (memoryless) property. Thus, we obtained the class of stochastic processes known as Markov chains, which we will study in some detail in this chapter. It should be pointed out that the analysis of Markov chains provides a rich framework for studying many DES of practical interest, ranging from gambling and the stock market to the design of “high-tech” computer systems and communication networks.


Markov Chain Markov Chain Model Transition Probability Matrix State Transition Diagram Recurrent State 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Selected References

  1. Asmussen, S., Applied Probability and Queues, Wiley, New York, 1987.zbMATHGoogle Scholar
  2. Bertsekas, D.P., Dynamic Programming: Deterministic and Stochastic Models, Prentice-Hall, Englewood Cliffs, NJ, 1987.zbMATHGoogle Scholar
  3. Bertsekas, D.P., Dynamic Programming and Optimal Control, Volumes 1 and 2, Athena Scientific, Belmont, MA, 1995.Google Scholar
  4. Cinlar, E., Introduction to Stochastic Processes, Prentice-Hall, Englewood Cliffs, NJ, 1975.zbMATHGoogle Scholar
  5. Clarke, A.B., and R.L. Disney, Probability and Random Processes, Wiley, New York, 1985.zbMATHGoogle Scholar
  6. Gallager, R.G., Discrete Stochastic Processes, Kluwer Academic Publishers, Boston, 1996.CrossRefGoogle Scholar
  7. eeeeeeeeeeeeeHoel, P.G., S.C. Port, and C.J. Stone, Introduction to Stochastic Processes, Houghton Mifïiin, Boston, 1972.zbMATHGoogle Scholar
  8. Kleinrock, L., Queueing Systems, Volume I: Theory, Wiley, New York, 1975.Google Scholar
  9. Parzen, E., Stochastic Processes, Holden-Day, San Francisco, 1962.zbMATHGoogle Scholar
  10. Trivedi, K.S Probability and Statistics with Reliability Queuing and Computer Science ApplicationsPrentice-Hall, Englewood Cliffs, NJ, 1982 Google Scholar

Copyright information

© Springer Science+Business Media New York 1999

Authors and Affiliations

  1. 1.Boston UniversityUSA
  2. 2.The University of MichiganUSA

Personalised recommendations