Approximation of the exit probability of a stable Markov modulated constrained random walk


Let X be the constrained random walk on \({\mathbb {Z}}_+^2\) having increments (1, 0), \((-\,1,1)\), \((0,-\,1)\) with jump probabilities \(\lambda (M_k)\), \(\mu _1(M_k)\), and \(\mu _2(M_k)\) where M is an irreducible aperiodic finite state Markov chain. The process X represents the lengths of two tandem queues with arrival rate \(\lambda (M_k)\), and service rates \(\mu _1(M_k)\), and \(\mu _2(M_k)\); the process M represents the random environment within which the system operates. We assume that the average arrival rate with respect to the stationary measure of M is less than the average service rates, i.e., X is assumed stable. Let \(\tau _n\) be the first time when the sum of the components of X equals n for the first time. Let Y be the random walk on \({{\mathbb {Z}}} \times {{\mathbb {Z}}}_+\) having increments \((-\,1,0)\), (1, 1), \((0,-\,1)\) with probabilities \(\lambda (M_k)\), \(\mu _1(M_k)\), and \(\mu _2(M_k)\). Supposing that the queues share a joint buffer of size n, \(p_n =P_{(x_n,m)}(\tau _n < \tau _0)\) is the probability that this buffer overflows during a busy cycle of the system. To the best of our knowledge, the only methods currently available for the approximation of \(p_n\) are classical large deviations analysis giving the exponential decay rate of \(p_n\) and rare event simulation. Let \(\tau \) be the first time the components of Y are equal. For \(x \in {{\mathbb {R}}}_+^2\), \(x(1) + x(2) < 1\), \(x(1) > 0\), and \(x_n = \lfloor nx \rfloor \), we show that \(P_{(n-x_n(1),x_n(2),m)}( \tau < \infty )\) approximates \(P_{(x_n,m)}(\tau _n < \tau _0)\) with exponentially vanishing relative error as \(n\rightarrow \infty \). For the analysis we define a characteristic matrix in terms of the jump probabilities of (XM). The 0-level set of the characteristic polynomial of this matrix defines the characteristic surface; conjugate points on this surface and the associated eigenvectors of the characteristic matrix are used to define (sub/super) harmonic functions which play a fundamental role both in our analysis and the computation/approximation of \(P_{(y,m)}(\tau < \infty )\).

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6


  1. 1.

    Note that \(\log ([(\beta ,\alpha ), y ]) = \langle (\log (\beta ),\log (\alpha )), ((y(1)-y(2)),y(2)) \rangle \) where \(\langle \cdot ,\cdot \rangle \) denotes the standard inner product in \({{\mathbb {R}}}^2\); the notation \([(\beta ,\alpha ), y ]\) was chosen with this connection to the inner product in mind. It also leads to much easier to read formulas because multiindex variables will be substituted for \(\beta \) and \(\alpha \), see, e.g., display (92).


  1. Berman, A., & Plemmons, R. J. (1994). Nonnegative matrices in the mathematical sciences (Vol. 9). Philadelphia: SIAM.

    Google Scholar 

  2. Blanchet, J. (2013). Optimal sampling of overflow paths in Jackson networks. Mathematics of Operations Research, 38(4), 698–719.

    Google Scholar 

  3. Blanchet, J. H., Leder, K., & Glynn, P. W. (2008). Efficient simulation of light-tailed sums: An old folk song sung to a faster new tune. In Monte Carlo and Quasi-Monte Carlo Methods (Vol. 2009, pp. 227–248).

  4. Borovkov, A. A., & Mogul’skii, A. A. (2001). Large deviations for Markov chains in the positive quadrant. Russian Mathematical Surveys, 56(5), 803–916.

    Google Scholar 

  5. Comets, F., Delarue, F., & Schott, R. (2009). Large deviations analysis for distributed algorithms in an ergodic Markovian environment. Applied Mathematics and Optimization, 60(3), 341–396.

    Google Scholar 

  6. De Boer, P.-T. (2006). Analysis of state-independent importance-sampling measures for the two-node tandem queue. ACM Transactions on Modeling and Computer Simulation (TOMACS), 16(3), 225–250.

    Google Scholar 

  7. De Boer, P.-T., Kroese, D. P., & Rubenstein, R. Y. (2004). A fast cross-entropy method for estimating buffer overflows in queueing networks. Management Science, 50(7), 883–895.

    Google Scholar 

  8. Dean, T., & Dupuis, P. (2009). Splitting for rare event simulation: A large deviation approach to design and analysis. Stochastic Processes and Their applications, 119(2), 562–587.

    Google Scholar 

  9. Dean, T., & Dupuis, P. (2011). The design and analysis of a generalized RESTART/DPR algorithm for rare event simulation. Annals of Operations Research, 189(1), 63–102.

    Google Scholar 

  10. Dupuis, P., Leder, K., & Wang, H. (2007). Importance sampling for sums of random variables with regularly varying tails. ACM Transactions on Modeling and Computer Simulation (TOMACS), 17(3), 14.

    Google Scholar 

  11. Dupuis, P., Sezer, A. D., & Wang, H. (2007). Dynamic importance sampling for queueing networks. Annals of Applied Probability, 17(4), 1306–1346.

    Google Scholar 

  12. Dupuis, P., & Wang, H. (2004). Importance sampling, large deviations and differential games. Stochastics : An International Journal of Probability and Stochastic Processes, 76(6), 481–508.

    Google Scholar 

  13. Dupuis, P., & Wang, H. (2009). Importance sampling for Jackson networks. Queueing Systems, 62(1–2), 113–157.

    Google Scholar 

  14. Durrett, R. (2010). Probability: Theory and examples (4th ed.). Cambridge: Cambridge University Press.

    Google Scholar 

  15. Flajolet, P. (1986). The evolution of two stacks in bounded space and random walks in a triangle. In International symposium on mathematical foundations of computer science (pp. 325–340). Springer.

  16. Foley, R. D., & McDonald, D. R. (2005). Large deviations of a modified Jackson network: Stability and rough asymptotics, The Annals of Applied Probability, 15(1B), 519–541.

    Google Scholar 

  17. Frater, M. R., Lennon, T. M., & Anderson, B. D. O. (1991). Optimally efficient estimation of the statistics of rare events in queueing networks. IEEE Transactions on Automatic Control, 36(12), 1395–1405.

    Google Scholar 

  18. Freidlin, M. I., & Wentzell, A. D. (1998). Random perturbation of dynamical systems (2nd ed.). New York: Springer-Verlag Telos.

    Google Scholar 

  19. Glasserman, P., & Kou, S.-G. (1995). Analysis of an importance sampling estimator for tandem queues. ACM Transactions on Modeling and Computer Simulation (TOMACS), 5(1), 22–42.

    Google Scholar 

  20. Ignatiouk-Robert, I. (2000). Large deviations of Jackson networks. Annals of Applied Probability, 10, 962–1001.

    Google Scholar 

  21. Ignatiouk-Robert, I., & Loree, C. (2010). Martin boundary of a killed random walk on a quadrant. The Annals of Probability, 38(3), 1106–1142.

    Google Scholar 

  22. Ignatyuk, I. A., Malyshev, V. A., & Scherbakov, V. V. (1994). Boundary effects in large deviation problems. Russian Mathematical Surveys, 49(2), 41–99.

    Google Scholar 

  23. Juneja, S., & Nicola, V. (2005). Efficient simulation of buffer overflow probabilities in Jackson networks with feedback. ACM Transcations on Modeling and Computer Simulation (TOMACS), 15(4), 281–315.

    Google Scholar 

  24. Kroese, D. P., & Nicola, V. (2002). Efficient simulation of a tandem Jackson network. ACM Transactions on Modeling and Computer Simulation (TOMACS), 12(2), 119–141.

    Google Scholar 

  25. Kurkova, I. A., & Malyshev, V. A. (1998). Martin boundary and elliptic curves. Markov Process and Related Fields, 4(2), 203–272.

    Google Scholar 

  26. Lax, P. D. (1996). Linear algebra. Pure and applied mathematics. Hoboken: Wiley-Interscience.

    Google Scholar 

  27. Louchard, G., & Schott, R. (1991). Probabilistic analysis of some distributed algorithms. Random Structures & Algorithms, 2(2), 151–186.

    Google Scholar 

  28. Maier, R. S. (1991). Colliding stacks: A large deviations analysis. Random Structures & Algorithms, 2(4), 379–420.

    Google Scholar 

  29. Maier, R. S. (1993). Large fluctuations in stochastically perturbed nonlinear systems: Applications in computing. In Proceedings 1992 Complex Systems Summer School (pp. 501–517). Addison-Wesley.

  30. McDonald, D. R. (1999). Asymptotics of first passage times for random walk in an orthant. Annals of Applied Probability, 9, 110–145.

    Google Scholar 

  31. Nicola, V., & Zaburnenko, T. (2007). Efficient importance sampling heuristics for the simulation of population overflow in Jackson networks. ACM Transactions on Modeling and Computer Simulation (TOMACS), 17(2), 10.

    Google Scholar 

  32. Parekh, S., & Walrand, J. (1989). A quick simulation method for excessive backlogs in networks of queues. IEEE Transactions on Automatic Control, 34(1), 54–66.

    Google Scholar 

  33. Prabhu, N. U., & Zhu, Y. (1989). Markov-modulated queueing systems. Queueing Systems, 5(1–3), 215–245.

    Google Scholar 

  34. Ridder, A. (2009). Importance sampling algorithms for first passage time probabilities in the infinite server queue. European Journal of Operational Research, 199(1), 176–186.

    Google Scholar 

  35. Serre, D. (2010). Matrices: Theory and applications. Graduate texts in mathematics (2nd ed., Vol. 216). New York: Springer.

    Google Scholar 

  36. Sezer, A. D. (2005). Dynamic importance sampling for queueing networks. Ph.D. thesis, Brown University Division of Applied Mathematics.

  37. Sezer, A. D. (2007). Asymptotically optimal importance sampling for Jackson networks with a tree topology. Preprint

  38. Sezer, A. D. (2009). Importance sampling for a Markov modulated queuing network. Stochastic Processes and their Applications, 119(2), 491–517.

    Google Scholar 

  39. Sezer, A. D. (2010). Asymptotically optimal importance sampling for Jackson networks with a tree topology. Queueing Systems, 6(2), 103–117.

    Google Scholar 

  40. Sezer, A. D. (2015). Exit probabilities and balayage of constrained random walks.

  41. Sezer, A. D. (2018). Approximation of excessive backlog probabilities of two tandem queues. Journal of Applied Probability, 55(3), 968–997.

    Google Scholar 

  42. Ünlü, K. D., & Sezer, A. D. (2019). Excessive backlog probabilities of two parallel queues. Annals of Operations Research.

  43. Yao, A. C. (1981). An analysis of a memory allocation scheme for implementing stacks. SIAM Journal on Computing, 10(2), 398–403.

    Google Scholar 

Download references

Author information



Corresponding author

Correspondence to Ali Devin Sezer.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Two lemmas

Two lemmas

For a square matrix \({\varvec{G}}\), let \({\varvec{G}}^{i,j}\) denote the matrix obtained by removing the ith row and jth column of \({\varvec{G}}\).

Lemma A.1

For \(n_0 \in \{2,3,\ldots \}\), suppose \({\varvec{G}}\) is an \(n_0 \times n_0\) irreducible and aperiodic matrix with nonnegative entries. Then \(\det \left( (\Lambda _1({\varvec{G}}) {\varvec{I}} - {\varvec{G}})^{i,i}\right) > 0\) for all \(i \in \{1,2,\ldots ,n_0\}\), where \({\varvec{I}}\) is the \(n_0 \times n_0\) identity matrix.


The argument is the same for all \( i \in \{1,2,\ldots ,n_0\}\); so it suffices to argue for \(i=1\). Suppose the claim is not true and

$$\begin{aligned} \det \left( (\Lambda _1({\varvec{G}}){\varvec{I}} - {\varvec{G}})^{1,1}\right) \le 0. \end{aligned}$$

Consider the function \(u \mapsto g(u) = \det \left( ( u{\varvec{I}} -{\varvec{G}})^{1,1} \right) \), \(u \ge 0\). The multilinearity and continuity of \(\det \) implies \(\lim _{u\nearrow \infty } g(u) = \infty \). This implies that if (101) is true there must be \(u_0 \ge \Lambda _1({\varvec{G}})\) such that

$$\begin{aligned} \det \left( (u_0 {\varvec{I}} - {\varvec{G}})^{1,1} \right) = 0. \end{aligned}$$

The matrix \({\varvec{G}}^{1,1}\) is nonnegative, therefore, it has a largest eigenvalue \(\Lambda _1({\varvec{G}}^{1,1})\) with an eigenvector \({\varvec{v}}_1 \ge 0\). The equality (102) implies

$$\begin{aligned} \Lambda _1({\varvec{G}}^{1,1}) \ge u_0 \ge \Lambda _1({\varvec{G}}). \end{aligned}$$

That \({\varvec{G}}\) is irreducible and aperiodic implies that \({\varvec{G}}^{n_0}\) is strictly positive; its largest eigenvalue is

$$\begin{aligned} \Lambda _1({\varvec{G}}^{n_0}) = \Lambda _1({\varvec{G}})^{n_0}. \end{aligned}$$

The matrix \(({\varvec{G}}^{n_0})^{1,1}\) has strictly positive entries and therefore its largest eigenvalue \(\Lambda _1( ({\varvec{G}}^{n_0})^{1,1})\) has an eigenvalue \({\varvec{v}}_2\) with strictly positive entries. For two vectors \(x,y \in {{\mathbb {R}}}^d\), let \(x \ge y\) and \(x > y\) denote componentwise comparison. The inequality

$$\begin{aligned} ({\varvec{G}}^{n_0})^{1,1} \ge ({\varvec{G}}^{1,1})^{n_0} \end{aligned}$$


$$\begin{aligned} ({\varvec{G}}^{n_0})^{1,1} {\varvec{v}}_1 \ge \Lambda _1({\varvec{G}}^{1,1})^{n_0} {\varvec{v}}_1. \end{aligned}$$

On the other hand

$$\begin{aligned} \Lambda _1(({\varvec{G}}^{n_0})^{1,1}) = \sup \{ c: \exists x \in {\mathbb R}^{n_0-1}_+, ({\varvec{G}}^{n_0})^{1,1}x \ge cx \}, \end{aligned}$$

(see Lax 1996, Proof of Theorem 1, Chapter 16). This and (104) imply

$$\begin{aligned} \Lambda _1(({\varvec{G}}^{n_0})^{1,1}) \ge \Lambda _1({\varvec{G}}^{1,1})^{n_0}. \end{aligned}$$

Define \({\varvec{v}}_3 = [ 1; {\varvec{v}}_2] \in {{\mathbb {R}}}^{n_0}\); it follows from \(({\varvec{G}}^{n_0})^{1,1} {\varvec{v}}_2 = \Lambda _1(({\varvec{G}}^{n_0}){1,1}) {\varvec{v}}_2\), the strict positivity of the components of \({\varvec{G}}^{n_0}\) and \({\varvec{v}}_2\) that one can choose \(\delta > 0\) small enough so that

$$\begin{aligned} {\varvec{G}}^{n_0} {\varvec{v}}_3 > \left( \Lambda _1(({\varvec{G}}^{n_0})^{1,1}+ \delta \right) {\varvec{v}}_3; \end{aligned}$$

This and

$$\begin{aligned} \Lambda _1({\varvec{G}}^{n_0})= \sup \{ c: \exists x \in {\mathbb R}^{n_0}_+, {\varvec{G}}^{n_0}x \ge cx \} \end{aligned}$$


$$\begin{aligned} \Lambda _1({\varvec{G}}^{n_0}) > \Lambda _1\left( ({\varvec{G}}^{n_0})^{1,1}\right) . \end{aligned}$$

The last inequality, (106) and (103) imply

$$\begin{aligned} \Lambda _1({\varvec{G}})^{n_0}= \Lambda _1({\varvec{G}}^{n_0}) > \Lambda _1(({\varvec{G}}^{n_0})^{1,1}) \ge \Lambda _1({\varvec{G}}^{1,1})^{n_0} \ge \Lambda _1({\varvec{G}})^{n_0}, \end{aligned}$$

which is a contradiction. \(\square \)

In our analysis we need the following fact from Sezer (2009); its proof is elementary and follows from the multilinearity of the determinant function and the previous lemma.

Lemma A.2

Let \({\varvec{G}}\) be an aperiodic and irreducible transition matrix. Then the row vector whose ith component equals \(\det \left( ({\varvec{I}} - {\varvec{G}})^{i,i}\right) \) is the unique (upto scaling by a positive number) left eigenvector associated with the eigenvalue 1 of \({\varvec{G}}\).

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Başoğlu Kabran, F., Sezer, A.D. Approximation of the exit probability of a stable Markov modulated constrained random walk. Ann Oper Res (2020).

Download citation


  • Markov modulation
  • Regime switch
  • Multidimensional constrained random walks
  • Exit probabilities
  • Rare events
  • Queueing systems
  • Characteristic surface
  • Superharmonic functions
  • Affine transformation

Mathematics Subject Classification

  • Primary 60G50
  • Secondary 60G40
  • 60F10
  • 60J45