# Approximation of the exit probability of a stable Markov modulated constrained random walk

## Abstract

Let X be the constrained random walk on $${\mathbb {Z}}_+^2$$ having increments (1, 0), $$(-\,1,1)$$, $$(0,-\,1)$$ with jump probabilities $$\lambda (M_k)$$, $$\mu _1(M_k)$$, and $$\mu _2(M_k)$$ where M is an irreducible aperiodic finite state Markov chain. The process X represents the lengths of two tandem queues with arrival rate $$\lambda (M_k)$$, and service rates $$\mu _1(M_k)$$, and $$\mu _2(M_k)$$; the process M represents the random environment within which the system operates. We assume that the average arrival rate with respect to the stationary measure of M is less than the average service rates, i.e., X is assumed stable. Let $$\tau _n$$ be the first time when the sum of the components of X equals n for the first time. Let Y be the random walk on $${{\mathbb {Z}}} \times {{\mathbb {Z}}}_+$$ having increments $$(-\,1,0)$$, (1, 1), $$(0,-\,1)$$ with probabilities $$\lambda (M_k)$$, $$\mu _1(M_k)$$, and $$\mu _2(M_k)$$. Supposing that the queues share a joint buffer of size n, $$p_n =P_{(x_n,m)}(\tau _n < \tau _0)$$ is the probability that this buffer overflows during a busy cycle of the system. To the best of our knowledge, the only methods currently available for the approximation of $$p_n$$ are classical large deviations analysis giving the exponential decay rate of $$p_n$$ and rare event simulation. Let $$\tau$$ be the first time the components of Y are equal. For $$x \in {{\mathbb {R}}}_+^2$$, $$x(1) + x(2) < 1$$, $$x(1) > 0$$, and $$x_n = \lfloor nx \rfloor$$, we show that $$P_{(n-x_n(1),x_n(2),m)}( \tau < \infty )$$ approximates $$P_{(x_n,m)}(\tau _n < \tau _0)$$ with exponentially vanishing relative error as $$n\rightarrow \infty$$. For the analysis we define a characteristic matrix in terms of the jump probabilities of (XM). The 0-level set of the characteristic polynomial of this matrix defines the characteristic surface; conjugate points on this surface and the associated eigenvectors of the characteristic matrix are used to define (sub/super) harmonic functions which play a fundamental role both in our analysis and the computation/approximation of $$P_{(y,m)}(\tau < \infty )$$.

This is a preview of subscription content, log in to check access.

1. 1.

Note that $$\log ([(\beta ,\alpha ), y ]) = \langle (\log (\beta ),\log (\alpha )), ((y(1)-y(2)),y(2)) \rangle$$ where $$\langle \cdot ,\cdot \rangle$$ denotes the standard inner product in $${{\mathbb {R}}}^2$$; the notation $$[(\beta ,\alpha ), y ]$$ was chosen with this connection to the inner product in mind. It also leads to much easier to read formulas because multiindex variables will be substituted for $$\beta$$ and $$\alpha$$, see, e.g., display (92).

## References

1. Berman, A., & Plemmons, R. J. (1994). Nonnegative matrices in the mathematical sciences (Vol. 9). Philadelphia: SIAM.

2. Blanchet, J. (2013). Optimal sampling of overflow paths in Jackson networks. Mathematics of Operations Research, 38(4), 698–719.

3. Blanchet, J. H., Leder, K., & Glynn, P. W. (2008). Efficient simulation of light-tailed sums: An old folk song sung to a faster new tune. In Monte Carlo and Quasi-Monte Carlo Methods (Vol. 2009, pp. 227–248).

4. Borovkov, A. A., & Mogul’skii, A. A. (2001). Large deviations for Markov chains in the positive quadrant. Russian Mathematical Surveys, 56(5), 803–916.

5. Comets, F., Delarue, F., & Schott, R. (2009). Large deviations analysis for distributed algorithms in an ergodic Markovian environment. Applied Mathematics and Optimization, 60(3), 341–396.

6. De Boer, P.-T. (2006). Analysis of state-independent importance-sampling measures for the two-node tandem queue. ACM Transactions on Modeling and Computer Simulation (TOMACS), 16(3), 225–250.

7. De Boer, P.-T., Kroese, D. P., & Rubenstein, R. Y. (2004). A fast cross-entropy method for estimating buffer overflows in queueing networks. Management Science, 50(7), 883–895.

8. Dean, T., & Dupuis, P. (2009). Splitting for rare event simulation: A large deviation approach to design and analysis. Stochastic Processes and Their applications, 119(2), 562–587.

9. Dean, T., & Dupuis, P. (2011). The design and analysis of a generalized RESTART/DPR algorithm for rare event simulation. Annals of Operations Research, 189(1), 63–102.

10. Dupuis, P., Leder, K., & Wang, H. (2007). Importance sampling for sums of random variables with regularly varying tails. ACM Transactions on Modeling and Computer Simulation (TOMACS), 17(3), 14.

11. Dupuis, P., Sezer, A. D., & Wang, H. (2007). Dynamic importance sampling for queueing networks. Annals of Applied Probability, 17(4), 1306–1346.

12. Dupuis, P., & Wang, H. (2004). Importance sampling, large deviations and differential games. Stochastics : An International Journal of Probability and Stochastic Processes, 76(6), 481–508.

13. Dupuis, P., & Wang, H. (2009). Importance sampling for Jackson networks. Queueing Systems, 62(1–2), 113–157.

14. Durrett, R. (2010). Probability: Theory and examples (4th ed.). Cambridge: Cambridge University Press.

15. Flajolet, P. (1986). The evolution of two stacks in bounded space and random walks in a triangle. In International symposium on mathematical foundations of computer science (pp. 325–340). Springer.

16. Foley, R. D., & McDonald, D. R. (2005). Large deviations of a modified Jackson network: Stability and rough asymptotics, The Annals of Applied Probability, 15(1B), 519–541.

17. Frater, M. R., Lennon, T. M., & Anderson, B. D. O. (1991). Optimally efficient estimation of the statistics of rare events in queueing networks. IEEE Transactions on Automatic Control, 36(12), 1395–1405.

18. Freidlin, M. I., & Wentzell, A. D. (1998). Random perturbation of dynamical systems (2nd ed.). New York: Springer-Verlag Telos.

19. Glasserman, P., & Kou, S.-G. (1995). Analysis of an importance sampling estimator for tandem queues. ACM Transactions on Modeling and Computer Simulation (TOMACS), 5(1), 22–42.

20. Ignatiouk-Robert, I. (2000). Large deviations of Jackson networks. Annals of Applied Probability, 10, 962–1001.

21. Ignatiouk-Robert, I., & Loree, C. (2010). Martin boundary of a killed random walk on a quadrant. The Annals of Probability, 38(3), 1106–1142.

22. Ignatyuk, I. A., Malyshev, V. A., & Scherbakov, V. V. (1994). Boundary effects in large deviation problems. Russian Mathematical Surveys, 49(2), 41–99.

23. Juneja, S., & Nicola, V. (2005). Efficient simulation of buffer overflow probabilities in Jackson networks with feedback. ACM Transcations on Modeling and Computer Simulation (TOMACS), 15(4), 281–315.

24. Kroese, D. P., & Nicola, V. (2002). Efficient simulation of a tandem Jackson network. ACM Transactions on Modeling and Computer Simulation (TOMACS), 12(2), 119–141.

25. Kurkova, I. A., & Malyshev, V. A. (1998). Martin boundary and elliptic curves. Markov Process and Related Fields, 4(2), 203–272.

26. Lax, P. D. (1996). Linear algebra. Pure and applied mathematics. Hoboken: Wiley-Interscience.

27. Louchard, G., & Schott, R. (1991). Probabilistic analysis of some distributed algorithms. Random Structures & Algorithms, 2(2), 151–186.

28. Maier, R. S. (1991). Colliding stacks: A large deviations analysis. Random Structures & Algorithms, 2(4), 379–420.

29. Maier, R. S. (1993). Large fluctuations in stochastically perturbed nonlinear systems: Applications in computing. In Proceedings 1992 Complex Systems Summer School (pp. 501–517). Addison-Wesley.

30. McDonald, D. R. (1999). Asymptotics of first passage times for random walk in an orthant. Annals of Applied Probability, 9, 110–145.

31. Nicola, V., & Zaburnenko, T. (2007). Efficient importance sampling heuristics for the simulation of population overflow in Jackson networks. ACM Transactions on Modeling and Computer Simulation (TOMACS), 17(2), 10.

32. Parekh, S., & Walrand, J. (1989). A quick simulation method for excessive backlogs in networks of queues. IEEE Transactions on Automatic Control, 34(1), 54–66.

33. Prabhu, N. U., & Zhu, Y. (1989). Markov-modulated queueing systems. Queueing Systems, 5(1–3), 215–245.

34. Ridder, A. (2009). Importance sampling algorithms for first passage time probabilities in the infinite server queue. European Journal of Operational Research, 199(1), 176–186.

35. Serre, D. (2010). Matrices: Theory and applications. Graduate texts in mathematics (2nd ed., Vol. 216). New York: Springer.

36. Sezer, A. D. (2005). Dynamic importance sampling for queueing networks. Ph.D. thesis, Brown University Division of Applied Mathematics.

37. Sezer, A. D. (2007). Asymptotically optimal importance sampling for Jackson networks with a tree topology. Preprint http://arxiv.org/abs/0708.3260.

38. Sezer, A. D. (2009). Importance sampling for a Markov modulated queuing network. Stochastic Processes and their Applications, 119(2), 491–517.

39. Sezer, A. D. (2010). Asymptotically optimal importance sampling for Jackson networks with a tree topology. Queueing Systems, 6(2), 103–117.

40. Sezer, A. D. (2015). Exit probabilities and balayage of constrained random walks. https://arxiv.org/abs/1506.08674.

41. Sezer, A. D. (2018). Approximation of excessive backlog probabilities of two tandem queues. Journal of Applied Probability, 55(3), 968–997.

42. Ünlü, K. D., & Sezer, A. D. (2019). Excessive backlog probabilities of two parallel queues. Annals of Operations Research. https://doi.org/10.1007/s10479-019-03324-w.

43. Yao, A. C. (1981). An analysis of a memory allocation scheme for implementing stacks. SIAM Journal on Computing, 10(2), 398–403.

## Author information

Authors

### Corresponding author

Correspondence to Ali Devin Sezer.

### Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Two lemmas

### Two lemmas

For a square matrix $${\varvec{G}}$$, let $${\varvec{G}}^{i,j}$$ denote the matrix obtained by removing the ith row and jth column of $${\varvec{G}}$$.

### Lemma A.1

For $$n_0 \in \{2,3,\ldots \}$$, suppose $${\varvec{G}}$$ is an $$n_0 \times n_0$$ irreducible and aperiodic matrix with nonnegative entries. Then $$\det \left( (\Lambda _1({\varvec{G}}) {\varvec{I}} - {\varvec{G}})^{i,i}\right) > 0$$ for all $$i \in \{1,2,\ldots ,n_0\}$$, where $${\varvec{I}}$$ is the $$n_0 \times n_0$$ identity matrix.

### Proof

The argument is the same for all $$i \in \{1,2,\ldots ,n_0\}$$; so it suffices to argue for $$i=1$$. Suppose the claim is not true and

\begin{aligned} \det \left( (\Lambda _1({\varvec{G}}){\varvec{I}} - {\varvec{G}})^{1,1}\right) \le 0. \end{aligned}
(101)

Consider the function $$u \mapsto g(u) = \det \left( ( u{\varvec{I}} -{\varvec{G}})^{1,1} \right)$$, $$u \ge 0$$. The multilinearity and continuity of $$\det$$ implies $$\lim _{u\nearrow \infty } g(u) = \infty$$. This implies that if (101) is true there must be $$u_0 \ge \Lambda _1({\varvec{G}})$$ such that

\begin{aligned} \det \left( (u_0 {\varvec{I}} - {\varvec{G}})^{1,1} \right) = 0. \end{aligned}
(102)

The matrix $${\varvec{G}}^{1,1}$$ is nonnegative, therefore, it has a largest eigenvalue $$\Lambda _1({\varvec{G}}^{1,1})$$ with an eigenvector $${\varvec{v}}_1 \ge 0$$. The equality (102) implies

\begin{aligned} \Lambda _1({\varvec{G}}^{1,1}) \ge u_0 \ge \Lambda _1({\varvec{G}}). \end{aligned}
(103)

That $${\varvec{G}}$$ is irreducible and aperiodic implies that $${\varvec{G}}^{n_0}$$ is strictly positive; its largest eigenvalue is

\begin{aligned} \Lambda _1({\varvec{G}}^{n_0}) = \Lambda _1({\varvec{G}})^{n_0}. \end{aligned}

The matrix $$({\varvec{G}}^{n_0})^{1,1}$$ has strictly positive entries and therefore its largest eigenvalue $$\Lambda _1( ({\varvec{G}}^{n_0})^{1,1})$$ has an eigenvalue $${\varvec{v}}_2$$ with strictly positive entries. For two vectors $$x,y \in {{\mathbb {R}}}^d$$, let $$x \ge y$$ and $$x > y$$ denote componentwise comparison. The inequality

\begin{aligned} ({\varvec{G}}^{n_0})^{1,1} \ge ({\varvec{G}}^{1,1})^{n_0} \end{aligned}

implies

\begin{aligned} ({\varvec{G}}^{n_0})^{1,1} {\varvec{v}}_1 \ge \Lambda _1({\varvec{G}}^{1,1})^{n_0} {\varvec{v}}_1. \end{aligned}
(104)

On the other hand

\begin{aligned} \Lambda _1(({\varvec{G}}^{n_0})^{1,1}) = \sup \{ c: \exists x \in {\mathbb R}^{n_0-1}_+, ({\varvec{G}}^{n_0})^{1,1}x \ge cx \}, \end{aligned}
(105)

(see Lax 1996, Proof of Theorem 1, Chapter 16). This and (104) imply

\begin{aligned} \Lambda _1(({\varvec{G}}^{n_0})^{1,1}) \ge \Lambda _1({\varvec{G}}^{1,1})^{n_0}. \end{aligned}
(106)

Define $${\varvec{v}}_3 = [ 1; {\varvec{v}}_2] \in {{\mathbb {R}}}^{n_0}$$; it follows from $$({\varvec{G}}^{n_0})^{1,1} {\varvec{v}}_2 = \Lambda _1(({\varvec{G}}^{n_0}){1,1}) {\varvec{v}}_2$$, the strict positivity of the components of $${\varvec{G}}^{n_0}$$ and $${\varvec{v}}_2$$ that one can choose $$\delta > 0$$ small enough so that

\begin{aligned} {\varvec{G}}^{n_0} {\varvec{v}}_3 > \left( \Lambda _1(({\varvec{G}}^{n_0})^{1,1}+ \delta \right) {\varvec{v}}_3; \end{aligned}

This and

\begin{aligned} \Lambda _1({\varvec{G}}^{n_0})= \sup \{ c: \exists x \in {\mathbb R}^{n_0}_+, {\varvec{G}}^{n_0}x \ge cx \} \end{aligned}

imply

\begin{aligned} \Lambda _1({\varvec{G}}^{n_0}) > \Lambda _1\left( ({\varvec{G}}^{n_0})^{1,1}\right) . \end{aligned}

The last inequality, (106) and (103) imply

\begin{aligned} \Lambda _1({\varvec{G}})^{n_0}= \Lambda _1({\varvec{G}}^{n_0}) > \Lambda _1(({\varvec{G}}^{n_0})^{1,1}) \ge \Lambda _1({\varvec{G}}^{1,1})^{n_0} \ge \Lambda _1({\varvec{G}})^{n_0}, \end{aligned}

which is a contradiction. $$\square$$

In our analysis we need the following fact from Sezer (2009); its proof is elementary and follows from the multilinearity of the determinant function and the previous lemma.

### Lemma A.2

Let $${\varvec{G}}$$ be an aperiodic and irreducible transition matrix. Then the row vector whose ith component equals $$\det \left( ({\varvec{I}} - {\varvec{G}})^{i,i}\right)$$ is the unique (upto scaling by a positive number) left eigenvector associated with the eigenvalue 1 of $${\varvec{G}}$$.

## Rights and permissions

Reprints and Permissions