Computational Probability pp 43-79 | Cite as

# Transient Solutions for Markov Chains

## Abstract

Much of the theory developed for solving Markov chain models is devoted to obtaining steady state measures, that is, measures for which the observation interval (0, *t*) is “sufficiently large” (*t* → ∞). These measures are indeed approximations of the behavior of the system for a finite, but long, time interval, where long means with respect to the interval of time between occurrences of events in the system. However, an increasing number of applications requires the calculation of measures during a relatively “short” period of time. These are the so-called *transient measures*. In these cases the steady state measures are not good approximations for the transient, and one has to resort to different techniques to obtain the desired quantities.

## Keywords

Markov Chain Krylov Subspace Transition Probability Matrix Computational Probability Transient Solution## Preview

Unable to display preview. Download preview PDF.

## References

- [Abate and Whitt, 1989]Abate, J. and Whitt, W. (1989). Calculating time-dependent performance measures for the M/M/1 queue.
*IEEE Trans. on Communications*,37(10):891–904.Google Scholar - [Abdallah and Marie, 1993]Abdallah, H. and Marie, R. (1993). The uniformized power method for transient solutions of Markov processes.
*Computers E4 Operations Research*, 20 (5): 515–526.CrossRefGoogle Scholar - [Ajmone Marsan and Chiola, 1987]Ajmone Marsan, M. and Chiola, G. (1987). On Petri nets with deterministic and exponentially distributed firing times. In Rozenberg, G., editor,
*Lecture Notes in Computer Science 266: Advances in Petri Nets 1987*, pages 132–145. Springer-Verlag.Google Scholar - [Bobbio and Telek, 1993]Bobbio, A. and Telek, M. (1993). Task completion time. In
*Proceedings of the 2nd Int’l Workshop on Performability Modeling of Computer and Communication Systems*, Le Mont Saint-Michel, France.Google Scholar - [Bobbio and Trivedi, 1986]Bobbio, A. and Trivedi, K. (1986). An aggregation technique for the transient analysis of stiff Markov chains.
*IEEE Trans. on Computers*, C-35(9): 803–814.Google Scholar - [Carmo et al., 1998]Carmo, R., de Carvalho, L., de Souza e Silva, E., Diniz, M., and Muntz, R. (1998). Performance/availability modeling with the TANGRAM-II modeling environment.
*Performance Evaluation*, 33: 45–65.CrossRefGoogle Scholar - [Carmo et al., 1996]Carmo, R., de Souza e Silva, E., and Marie, R. (1996). Efficient solutions for an approximation technique for the transient analysis of Markovian models. Technical report, IRISA, no. 3055, Campus universitaire de Beaulieu.Google Scholar
- [Carrasco and Calderón, 1995]Carrasco, J. and Calderón, A. (1995). Regenerative randomization: Theory and application examples. In
*Proc. Performance ‘85 and 1995 ACM SIGMETRICS Conf*., pages 241–262.Google Scholar - [Çinlar, 1975]Çinlar, E. (1975).
*Introduction to Stochastic Processes*. Prentice-Hall.Google Scholar - [Choi et al., 1994]Choi, H., Kulkarni, V., and Trivedi, K. (1994). Markov regenerative stochastic Petri nets.
*Performance Evaluation*, 20: 337–357.CrossRefGoogle Scholar - [Choudhury et al., 1993]Choudhury, G., Lucantoni, D., and Whitt, W. (1993). Multi-dimensional transform inversion with applications to the transient M/G/1 queue. Technical report, ATamp;T Bell Labs, ATamp;T Bell Labs, Holmdel.Google Scholar
- [Ciardo et al., 1993]Ciardo, G., Blakemore, A., Chimento, P., Muppala, J., and Trivedi, K. (1993). Automated generation and analysis of Markov reward models using stochastic reward nets. In Meyer, C. and Plemmons, R., editors,
*Linear Algebra*,*Markov Chains*,*and Queueing Models*,pages 145191. Springer-Verlag.Google Scholar - [Ciardo et al., 1989]Ciardo, G., Muppala, J., and Trivedi, K. (1989). SPNP: stochastic Petri net package. In
*Proceedings of the Third International Workshop on Petri-nets and Performance Models*, pages 142–151.Google Scholar - [de Souza e Silva and Gail, 1986]de Souza e Silva, E. and Gail, H. (1986). Calculating cumulative operational time distributions of repairable computer systems.
*IEEE Trans. on Computers*, C-35(4): 322–332.Google Scholar - [de Souza e Silva and Gail, 1989]de Souza e Silva, E. and Gail, H. (1989). Calculating availability and performability measures of repairable computer systems using randomization.
*Journal of the ACM*, 36 (1): 171–193.CrossRefGoogle Scholar - [de Souza e Silva and Gail, 1990]de Souza e Silva, E. and Gail, H. (1990). Analyzing scheduled maintenance policies for repairable computer systems.
*IEEE Trans. on Computers*, 39 (11): 1309–1324.CrossRefGoogle Scholar - [de Souza e Silva and Gail, 1992]de Souza e Silva, E. and Gail, H. (1992). Per-formability analysis of computer systems: from model specification to solution.
*Performance Evaluation*, 14: 157–196.CrossRefGoogle Scholar - [de Souza e Silva and Gail, 1996]de Souza e Silva, E. and Gail, H. (1996). The uniformization method in performability analysis. Technical report, IBM Research Report RC 20383, Yorktown Heights, N. Y.Google Scholar
- [de Souza e Silva and Gail, 1998]de Souza e Silva, E. and Gail, H. (1998). An algorithm to calculate transient distributions of cumulative rate and impulse based reward.
*Communications in Statistics–Stochastic Models*, 14 (3): 509–536.CrossRefGoogle Scholar - [de Souza e Silva et al., 1995a]de Souza e Silva, E., Gail, H., and Campos, R. V. (1995a). Calculating transient distributions of cumulative reward. In
*Proc. Performance ‘85 and 1995 ACM SIGMETRICS Conf*., pages 231–240.Google Scholar - [de Souza e Silva et al., 1995b]de Souza e Silva, E., Gail, H., and Muntz, R. (1995b). Efficient solutions for a class of non-Markovian models. In Stewart, W., editor,
*Computations with Markov Chains*,pages 483–506. Kluwer Academic Publishers.Google Scholar - [de Souza e Silva et al., 1995c]de Souza e Silva, E., Gail, H., and Muntz, R. (1995c). Polling systems with server timeouts and their application to token passing networks.
*IEEE Trans. on Networking*, 3 (5): 560–575.CrossRefGoogle Scholar - [de Souza e Silva et al., 1998]de Souza e Silva, E., Gail, H., and Muntz, R. (1998). Gated time-limited polling systems. In Hasegawa, T., Takagi, H., and Takahashi, Y., editors,
*Performance and Management of Complex Ciommunication Networks*, pages 253–274. Chapman amp; Hall.Google Scholar - [Donatiello and Grassi, 1991]Donatiello, L. and Grassi, V. (1991). On evaluating the cumulative performance distribution of fault-tolerant computer systems.
*IEEE Trans. on Computers*, 40 (11): 1301–1307.CrossRefGoogle Scholar - [Golub and van Loan, 1989]Golub, G. and van Loan, C. (1989).
*Matrix Computations*. Johns Hopkins University Press, second edition.Google Scholar - [Goyal et al., 1986]Goyal, A., Carter, W., de Souza e Silva, E., Lavenberg, S., and Trivedi, K. (1986). The system availability estimator. In
*Proceedings of FTCS-16*, pages 84–89.Google Scholar - [Grassmann, 1977a]Grassmann, W. (1977a). Transient solutions in Markovian queueing systems.
*Computers ê4 Operations Research*, 4: 47–53.CrossRefGoogle Scholar - [Grassmann, 1977b]Grassmann, W. (1977b). Transient solutions in Markovian queues.
*European Journal of Operational Research*, 1: 396–402.CrossRefGoogle Scholar - [Grassmann, 1982]Grassmann, W. (1982). The GI/PH/1 queue: a method to find the transition matrix.
*INFOR*, 20 (2): 144–156.Google Scholar - [Grassmann, 1987]Grassmann, W. (1987). Means and variances of time averages in Markovian environments.
*European Journal of Operational Research*, 31: 132–139.CrossRefGoogle Scholar - [Grassmann, 1991]Grassmann, W. (1991). Finding transient solutions in Markovian event systems through randomization. In Stewart, W. J., editor,
*Numerical Solution of Markov Chains*, pages 357–371. Marcel Dekker, Inc.Google Scholar - [Grassmann, 1993]Grassmann, W. (1993). Means and variances in Markov reward systems. In Meyer, C. and Plemmons, R., editors,
*Linear Algebra, Markov Chains, and Queueing Models*, pages 193–204. Springer-Verlag.Google Scholar - [Heidelberger and Goyal, 1988]Heidelberger, P. and Goyal, A. (1988). Sensitivity analysis of continuous time Markov chains using uniformization. In fazeolla, G., Courtois, P., and Boxma, O., editors,
*Computer Performance and Reliability*,pages 93–104. North-Holland.Google Scholar - [Horn and Johnson, 1985]Horn, R. and Johnson, C. (1985).
*Matrix Analysis*. Cambridge Univ. Press.Google Scholar - [Islam and Ammar, 1991]Islam, S. and Ammar, H. (1991). Performability analysis of distributed real-time systems.
*IEEE Trans. on Computers*, 40 (11): 1239–1251.CrossRefGoogle Scholar - [Iyer et al., 1986]Iyer, B., Donatiello, L., and Heidelberger, P. (1986). Analysis of performability for stochastic models of fault-tolerant systems.
*IEEE Trans. on Computers*, C-35(10): 902–907.Google Scholar - [Jensen, 1953]Jensen, A. (1953). Markoff chains as an aid in the study of Markoff processes.
*Skandinaysk Aktuarietidskrift*, 36: 87–91.Google Scholar - [Johnson Jr. and Malek, 1988]Johnson Jr., A. and Malek, M. (1988). Survey of software tools for evaluating reliability, availability and serviceability.
*ACM Computing Surveys*, 20: 227–271.CrossRefGoogle Scholar - [Kulkarni et al., 1986]Kulkarni, V., Nicola, V., Smith, R., and Trivedi, K. (1986). Numerical evaluation of performability and job completion time in repairable fault-tolerant systems. In
*Proceedings of FTCS-16*, pages 252–257.Google Scholar - [Leguesdron et al., 1993]Leguesdron, P., Pellaumail, J., Rubino, G., and Sericola, B. (1993). Transient analysis of the M/M/1 queue. Technical report, IRISA, no. 720, Campus universitaire de Beaulieu.Google Scholar
- [Lindemann, 1991]Lindemann, C. (1991). An improved numerical algorithm for calculating steady-state solutions of deterministic and stochastic Petri net models. In
*Proceedings of the 4th International Workshop on Petri-nets and Performance Models*, pages 176–184.Google Scholar - [Lindemann, 1992]Lindemann, C. (1992). DSPNexpress: A software package for the efficient solution of deterministic and stochastic Petri nets. In
*Proceedings of the Sixth International Conference on Modelling Techniques and Tools for Computer Systems Performance Evaluation*, pages 15–29, Edinburgh, Great Britain.Google Scholar - [Lopez-Benitez and Trivedi, 1993]Lopez-Benitez, N. and Trivedi, K. (1993). Multiprocessor performability analysis.
*IEEE Trans. on Reliability*, 42 (4): 579–587.CrossRefGoogle Scholar - [Lucantoni et al., 1994]Lucantoni, D., Choudhury, G., and Whitt, W. (1994). The transient BMAP/G/1 queue.
*Communications in Statistics - Stochastic Models*, 10 (1).Google Scholar - [Matsunawa, 1985]Matsunawa, T. (1985). The exact and approximate distributions of linear combinations of selected order statistics from a uniform distribution.
*Ann. Inst. Statist. Math*., 37: 1–16.CrossRefGoogle Scholar - [Meyer, 1980]Meyer, J. (1980). On evaluating the performability of degradable computing systems.
*IEEE Trans. on Computers*, C-29(8): 720–731.Google Scholar - [Meyer, 1982]Meyer, J. (1982). Closed-form solutions of performability.
*IEEE Trans. on Computers*, C-31(7): 648–657.Google Scholar - [Meyer, 1992]Meyer, J. (1992). Performability: A retrospective and some pointers to the future.
*Performance Evaluation*, 14: 139–156.CrossRefGoogle Scholar - [Meyer, 1995]Meyer, J. (1995). Performability evaluation: Where it is and what lies ahead. In
*IPDS’95*, pages 334–343.Google Scholar - [Meyer et al., 1980]Meyer, J., Furchtgott, D., and Wu, L. (1980). Performability evaluation of the SIFT computer.
*IEEE Trans. on Computers*, C29 (6): 501–509.CrossRefGoogle Scholar - [Moler and van Loan, 1978]Moler, C. and van Loan, C. (1978). Nineteen dubious ways to compute the exponential of a matrix.
*SIAM Review*, 20 (4): 801–836.CrossRefGoogle Scholar - [Nabli and Sericola, 1996]Nabli, H. and Sericola, B. (1996). Performability analysis: a new algorithm.
*IEEE Trans. on Computers*, 45 (4): 491–494.CrossRefGoogle Scholar - [Pattipati et al., 1993]Pattipati, K., Li, Y., and Blom, H. (1993). A unified framework for the performability evaluation of fault-tolerant computer systems.
*IEEE Trans. on Computers*, 42 (3): 312–326.CrossRefGoogle Scholar - [Pattipati and Shah, 1990]Pattipati, K. and Shah, S. (1990). On the computational aspects of performability models of fault-tolerant computer systems.
*IEEE Trans. on Computers*, 39 (6): 832–836.CrossRefGoogle Scholar - [Philippe and Sidge, 1995]Philippe, B. and Sidge, R. (1995). Transient solutions of Markov processes by Krylov subspaces. In Stewart, W., editor,
*Computations with Markov Chains*,pages 95–119. Kluwer Academic Publishers.Google Scholar - [Prodromides and Sanders, 1993]Prodromides, K. and Sanders, W. (1993). Performability evaluation of CSMA/CD and CSMA/CR protocols under transient fault conditions.
*IEEE Trans. on Reliability*, 42 (1): 116–127.CrossRefGoogle Scholar - [Puri, 1971]Puri, P. (1971). A method for studying the integral functionals of stochastic processes with applications: I. Markov chain case.
*Journal of Applied Probability*, 8 (10): 331–343.CrossRefGoogle Scholar - [Qureshi and Sanders, 1994]Qureshi, M. and Sanders, W. (1994). Reward model solution methods with impulse and rate rewards: An algorithm and numerical results.
*Performance Evaluation*, 20 (4): 413–436.CrossRefGoogle Scholar - [Reibman et al., 1989]Reibman, A., Trivedi, K., Kumar, S., and Ciardo, G. (1989). Analysis of stiff Markov chains.
*ORSA Journal on Computing*, 1 (2): 126–133.CrossRefGoogle Scholar - [Ren and Kobayashi, 1995]Ren, Q. and Kobayashi, H. (1995). Transient solutions for the buffer behavior in statistical multiplexing.
*Performance Evaluation*, 23: 65–87.CrossRefGoogle Scholar - [Ross, 1983]Ross, S. (1983).
*Stochastic Processes*. John Wiley amp; Sons.Google Scholar - [Ross, 1987]Ross, S. (1987). Approximating transition probabilities and mean occupation times in continuous-time Markov chains.
*Probability in the Engineering and Informational Sciences*, 1: 251–264.CrossRefGoogle Scholar - [Rubino and Sericola, 1993]Rubino, G. and Sericola, B. (1993). Interval availability distribution computation. In
*Proceedings of FTCS-23*, pages 48–55.Google Scholar - [Saad, 1992]Saad, Y. (1992). Analysis of some Krylov subspace approximations to the matrix exponential operator.
*SIAM Journal on Numerical Analysis*, 29 (1): 208–227.CrossRefGoogle Scholar - [Saad, 1995]Saad, Y. (1995).
*Iterative Methods for Sparse Linear Systems*. PWS Publishing Company.Google Scholar - [Sanders et al., 1995]Sanders, W., II, W. O., Qureshi, M., and Widjanarko, F. (1995). The UltraSAN modeling environment.
*Performance Evaluation*, 24: 89–115.CrossRefGoogle Scholar - [Sericola, 1999]Sericola, B. (1999). Availability analysis and stationary regime detection of Markov processes.
*IEEE Trans. on Computers*,*(to appear)*.Google Scholar - [Stewart, 1991]Stewart, W. (1991). MARCA Markov chain analyzer, a software package for Markov chains. In
*Numerical Solution of Markov Chains*, pages 37–61. Marcel Dekker, Inc.Google Scholar - [Stewart, 1994]Stewart, W. (1994).
*Introduction to the Numerical Solution of Markov Chains*. Princeton University Press.Google Scholar - [Tai and Meyer, 1996]Tai, A. and Meyer, J. (1996). Performability management in distributed database systems: an adaptive concurrency control protocol. In
*Proceedings of the 4th Int’l Workshop on Modeling*,*Analysis*,*and Simulation of Computer and Telecommunication Systems*, pages 212–216, San Jose, California.Google Scholar - [Tanaka et al., 1995]Tanaka, T., Hashida, O., and Takahashi, Y. (1995). Transient analysis of fluid model for ATM statistical multiplexer.
*Performance Evaluation*, 23: 145–162.CrossRefGoogle Scholar - [Trivedi, 1982]Trivedi, K. (1982).
*Probability and Statistics with Reliability*,*Queueing and Computer Science Applications*. Prentice-Hall.Google Scholar - [van As, 1986]van As, H. (1986). Transient analysis of Markovian queueing systems and its application to congestion control modeling.
*IEEE Journal on Selected Areas in Communications*, SAC-4(6): 891–904.Google Scholar - [van Dijk, 1990]van Dijk, N. (1990). The importance of bias-terms for error bounds and comparison results. In
*The First International Conference on the Numerical Solution of Markov Chains*, pages 640–663.Google Scholar - [van Dijk, 1992a]van Dijk, N. (1992a). Approximate uniformization for continuous-time Markov chains with an application to performability analysis.
*Stochastic Processes and Their Applications*, 40 (2): 339–357.CrossRefGoogle Scholar - [van Dijk, 1992b]van Dijk, N. (1992b). Uniformization for nonhomogeneous Markov chains.
*Operations Research Letters*, 12: 283–291.CrossRefGoogle Scholar - [van Moorsel and Sanders, 1994]van Moorsel, A. and Sanders, W. (1994). Adaptive uniformization.
*Communications in Statistics–Stochastic Models*, 10 (3): 619–648.CrossRefGoogle Scholar - [Weisberg, 1971]Weisberg, H. (1971). The distribution of linear combinations of order statistics from the uniform distribution.
*Annals Math. Stat*., 42: 704709.Google Scholar - [Yoon and Shanthikumar, 1989]Yoon, B. and Shanthikumar, J. (1989). Bounds and approximations for the transient behavior of continuous time Markov chains.
*Probability in the Engineering and Informational Sciences*, 3: 175–198.CrossRefGoogle Scholar