# Numerical Methods for Computing Stationary Distributions of Finite Irreducible Markov Chains

• William J. Stewart
Chapter
Part of the International Series in Operations Research & Management Science book series (ISOR, volume 24)

## Abstract

In this chapter our attention will be devoted to computational methods for computing stationary distributions of finite irreducible Markov chains. We let q ij denote the rate at which an n-state Markov chain moves from state i to state j. The n × n matrix Q whose off-diagonal elements are q ij and whose i th diagonal element is given by $$- \sum\limits_{j = 1}^n , j \ne i$$ q ij is called the infinitesimal generator of the Markov chain. It may be shown that the stationary probability vector π, a row vector whose k-th element denotes the stationary probability of being in state k, can be obtained by solving the homogeneous system of equations πQ = 0. Alternatively, the problem may be formulated as an eigenvalue problem πP = π, where P = QΔt+I is the stochastic matrix of transition probabilities, (Δt must be chosen sufficiently small so that the probability of two or more transitions occurring in time Δt is small, i.e., of order o(t)). Mathematically, the problem is therefore quite simple. Unfortunately, problems arise from the computational point of view because of the large number of states which many systems may occupy. As indicated in Chapters 1 and 2, it is not uncommon for thousands of states to be generated even for simple applications.

## Keywords

Markov Chain Iterative Method Diagonal Block Infinitesimal Generator Computational Probability
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

## References

1. [Aho et al., 1974]
Aho, A., Hoperoft, J., and Ullman, J. (1974). The Design and Analysis of Computer Algorithms. Addison-Wesley, Reading, Mass.Google Scholar
2. [Alexsson, 1985]
Alexsson, 0. (1985). A survey of preconditioned iterative methods for linear systems of algebraic equations. BIT, 25: 166–187.Google Scholar
3. [Arnoldi, 1951]
Arnoldi, W. (1951). The principle of minimized iteration in the solution of the matrix eigenvalue problem. Quart. Appl. Math, 9: 17–29.Google Scholar
4. [Balsamo and Pandolfi, 1988]
Balsamo, S. and Pandolfi, B. (1988). Bounded aggregation in Markovian networks. In Computer Performance and Reliability, pages 73–92. North Holland, Amsterdam.Google Scholar
5. [Barker and Plemmons, 1986]
Barker, G. and Plemmons, R. (1986). Convergent iterations for computing stationary distributions of Markov chains. SIAM J. Alg. Disc. Meth, 7: 390–398.
6. [Berman and Plemmons, 1979]
Berman, A. and Plemmons, R. (1979). Non-negative Matrices in the Mathematical Sciences. Academic Press, New York.Google Scholar
7. [Cao and Stewart, 1985]
Cao, W. and Stewart, W. (1985). Iterative aggregation/disaggregation techniques for nearly uncoupled Markov chains. J. Assoc. Comp. Mach, 32 3: 702–719.
8. [Chatelin, 1984a]
Chatelin, F. (1984a). Iterative aggregation/disaggregation methods. In Iazeolla, G., Courtois, P., and Hordijk, A., editors, Mathematical Computer Performance and Reliability, pages 199–207. North Holland, Amsterdam.Google Scholar
9. [Chatelin, 1984b]
Chatelin, F. (1984b). Spectral Approximation of Linear Operators. Academic Press, New York.Google Scholar
10. [Chatelin and Miranker, 1982]
Chatelin, F. and Miranker, W. (1982). Acceleration by aggregation of successive approximation methods. Linear Algebra Appl, 43: 17–47.
11. [Courtois, 1977]
Courtois, P. (1977). Decomposability; Queueing and Computer System Applications. Academic Press, Orlando, Florida.Google Scholar
12. [Courtois and Semal, 1984]
Courtois, P. and Semal, P. (1984). Bounds for the positive eigenvectors of nonnegative matrices and their approximation by decomposition. J. Assoc. Comp. Mach., 31: 804–825.
13. [Courtois and Semal, 1986]
Courtois, P. and Semal, P. (1986). Block iterative algorithms for stochastic matrices. Linear Algebra Appl, 76: 59–70.
14. [Dayar and Stewart, 1996]
Dayar, T. and Stewart, W. (1996). On the effects of using the Grassmann-Taksar-Heyman method in iterative aggregationdisaggregation. SIAM Journal on Scientific Computing, 17: 1–17.
15. [Dayar and Stewart, 1997]
Dayar, T. and Stewart, W. (1997). Quasilumpability, lower bounding coupling matrices and nearly completely decomposable Markov chains. SIAM Journal on Matrix Analysis and Applications, 18: 482–498.
16. [Feinberg and Chiu, 1987]
Feinberg, B. and Chiu, S. (1987). A method to calculate steady state distributions of large Markov chains by aggregating states. Operations Research, 35: 282–290.
17. [Funderlic and Meyer, 1986]
Funderlic, R. and Meyer, C. (1986). Sensitivity of the stationary distribution vector for an ergodic Markov chain. Linear Algebra Appl, 76: 1–17.
18. [Funderlic and Plemmons, 1984]
Funderlic, R. and Plemmons, R. (1984). A combined direct-iterative method for certain M-matrix linear systems. SIAM J. Alg. Disc. Meth, 5 (1): 32–42.
19. [Funderlic and Plemmons, 1986]
Funderlic, R. and Plemmons, R. (1986). Updating LU factorizations for computing stationary distributions. SIAM J. Alg. Disc. Meth, 7: 30–42.
20. [Golub and Meyer, 1986]
Golub, G. and Meyer, C. (1986). Using the QR factorization and group inversion to compute, differentiate and estimate the sensitivity of stationary distributions for Markov chains. SIAM J. Alg. Disc. Meth, 7: 273–281.Google Scholar
21. [Grassmann et al., 1985]
Grassmann, W., Taksar, M., and Heyman, D. (1985). Regenerative analysis and steady state distributions for Markov chains. Operations Research, 33: 1107–1116.
22. [Harrod and Plemmons, 1984]
Harrod, W. and Plemmons, R. (1984). Comparisons of some direct methods for computing stationary distributions of Markov chains. SIAM J. Sci. Comput, 5: 453–469.Google Scholar
23. [Haviv, 1987]
Haviv, M. (1987). Aggregation/disagregation methods for computing the stationary distribution of a Markov chain. SIAM J. Numer. Anal, 24: 952–966.
24. [Haviv, 1989]
Haviv, M. (1989). More on a Rayleigh-Ritz refinement technique for nearly uncoupled stochastic matrices. SIAM J. Matrix Anal. Appl, 10: 287–293.
25. [Heyman, 1987]
Heyman, D. (1987). Further comparisons of direct methods for computing stationary distributions of Markov chains. SIAM J. Alg. Disc. Meth, 8: 226–232.Google Scholar
26. [Hoperoft and Tarjan, 1973]
Hoperoft, J. and Tarjan, R. (1973). Efficient algorithms for graph manipulation. CA CM, 16 (6): 372–378.Google Scholar
27. [Jennings and Stewart, 1975]
Jennings, A. and Stewart, W. (1975). Simultaneous iteration for partial eigensolution of real matrices. J. IMA, 15: 351–361.Google Scholar
28. [Jennings and Stewart, 1981]
Jennings, A. and Stewart, W. (1981). A simultaneous iteration algorithm for real matrices. ACM Trans. of Math. Software, 7: 184–198.
29. [Kaufman, 1983]
Kaufman, L. (1983). Matrix methods for queueing problems. SIAM J. Sci. Comput, 4: 525–552.
30. [Kim and Smith, 1991]
Kim, D. and Smith, R. (1991). An exact aggregation algorithm for mandatory set decomposable Markov chains. In Stewart, W. J., editor, Numerical Solution of Markov Chains. Marcel Dekker, New York, NY.Google Scholar
31. [Koury et al., 1984]
Koury, R., McAllister, D., and Stewart, W. (1984). Iterative methods for computing stationary distributions of nearly completely decomposable Markov chains. SIAM J. Alg. Disc. Math, 5 (2): 164–186.
32. [Krieger, 1995]
Krieger, U. (1995). Numerical solution of large finite Markov chains by algebraic multigrid techniques. In Stewart, W. J., editor, Computations with Markov Chains. Kluwer Academic Publishers, Boston.Google Scholar
33. [Leutenegger and Horton, 1995]
Leutenegger, S. and Horton, G. (1995). On the utility of the multi-level algorithm for the solution of nearly completely decomposable Markov chains. In Stewart, W. J., editor, Computations with Markov Chains. Kluwer Academic Publishers, Boston.Google Scholar
34. [Lubachevsky and Mitra, 1986]
Lubachevsky, B. and Mitra, D. (1986). A chaotic asynchronous algorithm for computing the fixed point of a non-negative matrix of unit spectral radius. J. Asoc. Comput. Mach, 33: 130–150.
35. [McAllister et al., 1984]
McAllister, D., Stewart, G., and Stewart, W. (1984). On a Raleigh-Ritz refinement technique for nearly uncoupled stochastic matrices. Linear Alg. Applications, 60: 1–25.
36. [Meyer, 1989]
Meyer, C. (1989). Stochastic complementation, uncoupling Markov chains and the theory of nearly reducible systems. SIAM Rev, 31: 240–272.
37. [Mitra and Tsoucas, 1988]
Mitra, D. and Tsoucas, P. (1988). Relaxations for the numerical solutions of some stochastic problems. Stochastic Models, 4 (3): 387–419.
38. Philippe et al., 1992] Philippe, B., Saad, Y., and Stewart, W. (1992). Numerical methods in Markov chain modelling. Operations Research,40(6):11561179.Google Scholar
39. [Rieders, 1995]
Rieders, M. (1995). State space decomposition for large Markov chains. In Stewart, W. J., editor, Computations with Markov Chains. Kluwer Academic Publishers, Boston.Google Scholar
Saad, Y. (1980). Variations on Arnoldi’s method for computing eigenelements of large unsymmetric matrices. Lin. Alg. Appl, 34: 269–295.
Saad, Y. (1981). Krylov subspace methods for solving large un-symmetric linear systems. Math. Comp, 37: 105–126.
Saad, Y. (1982). Projection methods for solving large sparse eigen-value problems. In Kagstrom, B. and Ruhe, A., editors, Matrix Pencils, Proceedings, Pitea Haysbad, pages 121–144. University of Umea, Sweden, Springer Verlag, Berlin.
Saad, Y. (1984). Chebyshev acceleration techniques for solving non-symmetric eigenvalue problems. Mathematics of Computation, 42: 567588.Google Scholar
Saad, Y. (1995a). Iterative Methods for Sparse Linear Systems. PWS Publishing Company, Boston, MA.Google Scholar
Saad, Y. (1995b). Preconditioned Krylov subspace methods for the numerical solution of Markov chains. In Stewart, W. J., editor, Computations with Markov Chains. Kluwer Academic Publishers, Boston.Google Scholar
Saad, Y. and Schultz, M. (1986). GMRES: A generalized minimal residual algorithm for solving non-symmetric linear systems. SIAM J. Sci. Stat. Comput, 7: 856–869.
47. [Schweitzer, 1983]
Schweitzer, P. (1983). Aggregation methods for large Markov chains. In Models of Computer Communication Systems, pages 225234. University of Pisa, Italy. International Workshop on Applied Mathematics and Performance Reliability.Google Scholar
48. [Schweitzer, 1986]
Schweitzer, P. (1986). Perturbation series expansions for nearly completely decomposable Markov chains. In Boxma, O., Cohen, J., and Tijms, H., editors, Teletraffic Analysis and Computer Performance Evaluation, pages 319–328. Elsevier North-Holland, Amsterdam.Google Scholar
49. [Schweitzer and Kindle, 1986]
Schweitzer, P. and Kindle, K. (1986). An iterative aggregation-disaggregation algorithm for solving linear systems. Applied Math. and Comp, 18: 313–353.
50. [Sheskin, 1985]
Sheskin, T. (1985). A Markov chain partitioning algorithm for computing steady state probabilities. Operations Research, 33: 228–235.
51. [Simon and Ando, 1961]
Simon, H. and Ando, A. (1961). Aggregation of variables in dynamic systems. Econometrica, 29: 111–138.
52. [Stewart, 1976]
Stewart, G. (1976). Simultaneous iteration for computing in- variant subspaces of non-Hermitian matrices. Numer. Mat, 25: 123–136.
53. [Stewart, 1983]
Stewart, G. (1983). Computable error bounds for aggregated Markov chains. J. Assoc. Comp. Mach., 30: 271–285.
54. [Stewart, 1991]
Stewart, G. (1991). On the sensitivity of nearly uncoupled Markov chains. In Stewart, W. J., editor, Numerical Solution of Markov Chains. Marcel Dekker, New York, NY.Google Scholar
55. [Stewart et al., 1993]
Stewart, G., Stewart, W., and McAllister, D. (1993). A two stage iteration for solving nearly uncoupled Markov chains. In Recent Advances in Iterative Methods, volume 60 of IMA Volumes in Mathematics and its Applications, pages 201–216. Springer Verlag, New York.Google Scholar
56. [Stewart, 1978]
Stewart, W. (1978). A comparison of numerical techniques in Markov modelling. Comm. ACM, 21: 144–151.
57. Stewart, 1994] Stewart, W. (1994). An Introduction to the Numerical Solution of Markov Chains. Princeton University Press„ New Jersey.Google Scholar
58. [Stewart and Jennings, 1981]
Stewart, W. and Jennings, A. (1981). A simultaneous iteration algorithm for real matrices. ACM Trans. Math. Software, 7: 184–198.
59. [Stewart and Wu, 1992]
Stewart, W. and Wu, W. (1992). Numerical experiments with iteration and aggregation for Markov chains. ORSA Journal on Computing, 4: 336–350.
60. [Sumita and Rieders, 1991]
Sumita, U. and Rieders, M. (1991). A comparison of the replacement process with aggregation-disaggregation. In Stewart, W. J., editor, Numerical Solution of Markov Chains. Marcel Dekker, New York, NY.Google Scholar
61. [Takahashi, 1975]
Takahashi, Y. (1975). A lumping method for numerical calculation of stationary distributions of Markov chains. Technical report, Department of Information Sciences, Tokyo Institute of Technology, Tokyo, Japan.Google Scholar
62. [Tarjan, 1972]
Tarjan, R. (1972). Depth first search and linear graph algorithms. SIAM J. Comput., 1 (2): 146–160.
63. [Vantilborgh, 1985]
Vantilborgh, H. (1985). Agreggation with an error of o(e2). JACM, 32: 161–190.
64. [Varga, 1962]
Varga, R. (1962). Matrix Iterative Analysis. Prentice Hall, Englewood Cliffs, NJ.Google Scholar
65. Wallace, 1973] Wallace, V. (1973). Towards an algebraic theory of Markovian networks. In Proc. Symp. Computer-Communication Networks and Teletraffic. Polytechnic Press, New York.Google Scholar
66. Wallace and Rosenberg, 1966a] Wallace, V. and Rosenberg, R. (1966a). Markovian models and numerical analysis of computer system behavior. In Proc. AFIPS Spring Joint Computer Conference. AFIPS Press, New Jersey.Google Scholar
67. [Wallace and Rosenberg, 1966b]
Wallace, V. and Rosenberg, R. (1966b). RQA1: The recursive queue analyzer. Technical Report 2, Systems Engineering Laboratory, University of Michigan, Ann Arbor, Michigan.Google Scholar
68. [Young, 1971]
Young, D. (1971). Iterative solution of large linear systems. Academic Press, New York.Google Scholar
69. [Zarling, 1976]
Zarling, R. (1976). Numerical solutions of nearly completely decomposable queueing networks. PhD thesis, Department of Computer Science, University of North Carolina.Google Scholar