Skip to main content
SpringerLink
Log in

Iterative Methods for Finding the Stationary Vector for Markov Chains

  • Conference paper
Linear Algebra, Markov Chains, and Queueing Models

Part of the book series: The IMA Volumes in Mathematics and its Applications ((IMA,volume 48))

Abstract

This overview concerns methods for estimating the steady-state vector of an ergodic Markov chain. The problem can be cast as an ordinary eigenvalue problem, but since the eigenvalue is known, it can equally well be studied as a nullspace problem or as a linear system. We discuss iterative methods for each of these three formulations. Many of the applications, such as queuing modeling, have special structure that can be exploited computationally, and we give special emphasis to three ideas for exploiting this structure: decomposability, separability, and multilevel aggregation. Such ideas result in a large number of diverse algorithms, many of which are poorly understood.

This work was completed while the author was in residence at the Institute for Mathematics and Its Applications, University of Minnesota, supported by the General Research Board of the University of Maryland, College Park and by the IMA.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Å. Björck and Tommy Elfving. Accelerated projection methods for computing pseudoinverse solutions of systems of linear equations. BIT, 19:145–163, 1979.

    Article  MathSciNet  MATH  Google Scholar 

  2. Randy Bramley and Ahmed Sameh. Row projection methods for large nonsymmetric linear systems. Technical Report CSRD 957, Center for Supercomputing Research and Development, University of Illinois, Urbana, 1990.

    Google Scholar 

  3. W. Briggs. A Multigrid Tutorial. SIAM, Philadelphia, Pennsylvania, 1987.

    MATH  Google Scholar 

  4. W.-L. Cao and W. J. Stewart. Iterative aggregation/disaggregation techniques for nearly uncoupled Markov chains. Journal of the ACM, 32:702–719, 1985.

    Article  MathSciNet  MATH  Google Scholar 

  5. R. H. Chan. Iterative methods for overflow queueing models I. Numerische Mathematik, 51:143–180, 1987.

    Article  MathSciNet  MATH  Google Scholar 

  6. R. H. Chan. Iterative methods for overflow queueing models II. Numerische Mathematik, 54:57–78, 1988.

    Article  MathSciNet  MATH  Google Scholar 

  7. F. Chatelin. Iterative aggregation/disaggregation methods. In G. Iazeolla, P. J. Courtois, and A. Hordijk, editors, Mathematical Computer Performance and Reliability, pages 199–207, New York, 1984. Elsevier Science Publishers.

    Google Scholar 

  8. F. Chatelin and W. L. Miranker. Acceleration by aggregation of successive approximation methods. Linear Algebra and Its Applications, 43:17–47, 1982.

    Article  MathSciNet  MATH  Google Scholar 

  9. F. Chatelin and W. L. Miranker. Aggregation/disaggregation for eigenvalue problems. SIAM Journal on Numerical Analysis, 21:567–582, 1984.

    Article  MathSciNet  MATH  Google Scholar 

  10. P.-J. Courtois. Error analysis in nearly-completely decomposable stochastic systems. Econometrica, 43:691–709, 1975.

    Article  MathSciNet  MATH  Google Scholar 

  11. P.-J. Courtois. Decomposability. Academic Press, New York, 1977.

    MATH  Google Scholar 

  12. T. Elfving. Block-iterative methods for consistent and inconsistent linear equations. Numerische Mathematik, 35:1–12, 1980.

    Article  MathSciNet  MATH  Google Scholar 

  13. M. Haviv. Aggregration/disaggregation methods for computing the stationary distribution of a Markov chain. SIAM Journal on Numerical Analysis, 24:952–966, 1987.

    Article  MathSciNet  MATH  Google Scholar 

  14. A. S. Householder. The Theory of Matrices in Numerical Analysis. Dover Publishing, New York, 1964. Originally published by Ginn Blaisdell.

    MATH  Google Scholar 

  15. A. Jennings and D. R. L. Orr. Applications of the simultaneous iteration method to undamped vibration problems. International Journal for Numerical Methods in Engineering, 3:13–24, 1971.

    Article  MATH  Google Scholar 

  16. S. Kaczmarz. Ängenäherte Auflösung von Systemen linearer Gleichungen. Bull Internat. Acad. Polon. Sci. Cl. A., pages 355–356, 1937. Cited in [14].

    Google Scholar 

  17. L. Kaufman. Matrix methods for queuing problems. SIAM J. on Scientific and Statistical Computing, 4:525–552, 1983.

    Article  MathSciNet  MATH  Google Scholar 

  18. John G. Kemeny and J. Laurie Snell. Finite Markov Chains. D. Van Nostrand Co. Inc., New York, 1960.

    MATH  Google Scholar 

  19. L. Kleinrock. Queueing Systems. Volume I: Theory. John Wiley & Sons, New York, 1975.

    Google Scholar 

  20. D. F. McAllister, G. W. Stewart, and W.J. Stewart. On a Rayleigh-Ritz refinement technique for nearly uncoupled stochastic matrices. Linear Algebra and Its Applications, 60:1–25, 1984.

    Article  MathSciNet  MATH  Google Scholar 

  21. D. Mitra and P. Tsoucas. Convergence of relaxations for numerical solutions of stochastic problems. In G. Iazeolla, P.-J. Courtois, and O. J. Boxma, editors, Second International Workshop on Applied Mathematics and Performance/Reliability Models of Computer/Communication Sytems, pages 119–133, Amsterdam, 1987. North Holland.

    Google Scholar 

  22. M. Neumann and R. J. Plemmons. Convergent nonnegative matrices and iterative methods for consistent linear systems. Numerische Mathematik, 31:265–279, 1978.

    Article  MathSciNet  MATH  Google Scholar 

  23. M. Neumann and R. J. Plemmons. Generalized inverse-positivity and splittings of M-matrices. Linear Algebra and Its Applications, 23:21–35, 1979.

    Article  MathSciNet  MATH  Google Scholar 

  24. Pil Seong Park. Iterative solution of sparse singular systems of equations arising from queuing networks. Technical Report UMIACS-TR-91–83, CS-TR-2690, University of Maryland, College Park, 1991.

    Google Scholar 

  25. B. N. Parlett. The Symmetric Eigenvalue Problem. Prentice-Hall, Englewood Cliffs, New Jersey, 1980.

    MATH  Google Scholar 

  26. Bernard Philippe, Youcef Saad, and William J. Stewart. Numerical methods in Markov chain modelling. Operations Research, to appear, 1992.

    Google Scholar 

  27. Y. Saad. Chebyshev acceleration techniques for solving nonsymmetric eigenvalue problems. Mathematics of Computation, 42:567–588, 1984.

    Article  MathSciNet  MATH  Google Scholar 

  28. Y. Saad. Projection methods for the numerical solution of Markov chains. In W. J. Stewart, editor, Numerical Solution of Markov Chains, pages 455–472, Amsterdam, 1991. North Holland.

    Google Scholar 

  29. Youcef Saad. Chebyshev acceleration techniques for solving nonsymmetric eigenvalue problems. Mathematics of Computation, 42:567–588, 1984.

    Article  MathSciNet  MATH  Google Scholar 

  30. P. J. Schweitzer. A survey of aggregation-disaggregation in large Markov chains. In W. J. Stewart, editor, Numerical Methods for Markov Chains, pages 63–88, Amsterdam, 1991. North Holland.

    Google Scholar 

  31. P. J. Schweitzer and K. W. Kindle. An iterative aggregation-disaggregation algorithm for solving linear equations. Applied Mathematics and Computation, 18:313–353, 1986.

    Article  MathSciNet  MATH  Google Scholar 

  32. H.A. Simon and A. Ando. Aggregation of variables in dynamic systems. Econometrica, 29:111–138, 1961.

    Article  MATH  Google Scholar 

  33. G. W. Stewart. Simultaneous iteration for computing invariant subspaces of non-Hermitian matrices. Numerische Mathematik, 25:123–136, 1976.

    Article  MATH  Google Scholar 

  34. G. W. Stewart. SRRIT — a FORTRAN subroutine to calculate the dominant invariant subspaces of a real matrix. Technical Report TRR-514, University of Maryland, Department of Computer Science, 1978.

    Google Scholar 

  35. G. W. Stewart. Computable error bounds for aggregated Markov chains. Journal of the ACM, 30:271–285, 1983.

    Article  MATH  Google Scholar 

  36. G. W. Stewart. On the structure of nearly uncoupled Markov chains. In G. Iazeolla, P. J. Courtois, and A. Hordijk, editors, Mathematical Computer Performance and Reliability, pages 287–302, New York, 1984. Elsevier.

    Google Scholar 

  37. G. W. Stewart, W. J. Stewart, and D. F. McAllister. A two-stage iteration for solving nearly uncoupled Markov chains. Technical Report TR-1384, Department of Computer Science, University of Maryland, 1984.

    Google Scholar 

  38. Xiaobai Sun. A unified analysis of numerical methods for solving nearly uncoupled markov chains. Technical report, Ph.D. thesis, Computer Science Department, University of Maryland, College Park, 1991.

    Google Scholar 

  39. Y. Takahashi. Some Problems for Applications of Markov Chains. PhD thesis, Tokyo Institute of Technology, 1972. Cited in [41].

    Google Scholar 

  40. Y. Takahashi. A sequencing model with an application to speed class sequencing in air traffic control. J. Operations Research Society of Japan, 17:1–28, 1974. Cited in [41].

    MATH  Google Scholar 

  41. Y Takahashi. Weak D-Markov chain and its application to a queuing network. In G. Iazeolla, P. J. Courtois, and A. Hordijk, editors, Mathematical Computer Performance and Reliability, pages 153–166, New York, 1984. Elsevier.

    Google Scholar 

  42. K. S. Trivedi. Probability and Statistics with Reliability, Queuing, and Computer Science Applications. Prentice-Hall, Englewook Cliffs, NJ, 1982.

    Google Scholar 

  43. R. S. Varga. Matrix Iterative Analysis. Prentice-Hall, Englewood Cliffs, New Jersey, 1962.

    Google Scholar 

  44. D. M. Young. Iterative Solution of Large Linear Systems. Academic Press, New York, 1971.

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science Department and Institute for Advanced Computer Studies, University of Maryland, College Park, Maryland, 20742, USA

    Dianne P. O’leary

Authors
  1. Dianne P. O’leary
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Mathematics Department, North Carolina State University, 27695-8205, Raleigh, NC, USA

    Carl D. Meyer

  2. Department of Mathematics and Computer Science, Wake Forest University, 27109, Winston-Salem, NC, USA

    Robert J. Plemmons

Rights and permissions

Reprints and permissions

Copyright information

© 1993 Springer-Verlag New York, Inc.

About this paper

Cite this paper

O’leary, D.P. (1993). Iterative Methods for Finding the Stationary Vector for Markov Chains. In: Meyer, C.D., Plemmons, R.J. (eds) Linear Algebra, Markov Chains, and Queueing Models. The IMA Volumes in Mathematics and its Applications, vol 48. Springer, New York, NY. https://doi.org/10.1007/978-1-4613-8351-2_9

Download citation

  • DOI: https://doi.org/10.1007/978-1-4613-8351-2_9

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4613-8353-6

  • Online ISBN: 978-1-4613-8351-2

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation