Queueing Systems

, Volume 50, Issue 2–3, pp 255–297 | Cite as

Dynamic Safety-Stocks for Asymptotic Optimality in Stochastic Networks

  • Sean Meyn


This paper concerns control of stochastic networks using state-dependent safety-stocks.

Three examples are considered: a pair of tandem queues; a simple routing model; and the Dai-Wang re-entrant line. In each case, a single policy is proposed that is independent of network load ρ•. The following conclusions are obtained for the controlled network in each case, where the finite constant K0 is independent of load: The policy is fluid-scale asymptotically optimal, and satisfies the bound
$$ eta_ast \le \eta \le \eta_\ast + K_0 \log(\eta_\ast),\quad 0 \le \rho_\bullet \le 1, $$
where η* is the optimal steady-state cost.

These results are based on the construction of an approximate solution to the average-cost dynamic programming equations using a perturbation of the value function for an associated fluid model.

Moreover, a new technique is introduced to obtain fluid-scale asymptotic optimality for general networks modeled in discrete time. The proposed policy is myopic with respect to a perturbation of the value function for the fluid model.


queueing networks routing scheduling optimal control 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Nicole Bäuerle, Optimal control of queueing networks: An approach via fluid models, Adv. in Appl. Probab. 34(2) (2002) 313–328.CrossRefGoogle Scholar
  2. [2]
    S.L. Bell and R.J. Williams, Dynamic scheduling of a system with two parallel servers in heavy traffic with complete resource pooling: Asymptotic optimality of a continuous review threshold policy, Ann. Appl. Probab. 11 (2001) 608–649.CrossRefGoogle Scholar
  3. [3]
    M. Bramson, State space collapse with application to heavy traffic limits for multiclass queueing networks, Queueing Systems: Theory and Applications 30 (1998) 89–148.CrossRefGoogle Scholar
  4. [4]
    H. Chen and D.D. Yao, Fundamentals of Queueing Networks: Performance, Asymptotics, and Optimization (Springer-Verlag, New York, 2001) Stochastic Modelling and Applied Probability.Google Scholar
  5. [5]
    M. Chen, C. Pandit and S.P. Meyn, in search of sensitivity in network optimization, Queueing Systems: Theory and Applications 44(4) (2003) 313–363.CrossRefGoogle Scholar
  6. [6]
    D.P. Connors, G. Feigin and D. Yao, Scheduling semiconductor lines using a fluid network model, IEEE Trans. on Robotics and Automation 10 (1994) 88–98.CrossRefGoogle Scholar
  7. [7]
    J.G. Dai and W. Lin, Maximum pressure policies in stochastic processing networks, Operations Research, 53(2) to appear.Google Scholar
  8. [8]
    J.G. Dai and Y. Wang, Nonexistence of Brownian models of certain multiclass queueing networks, Queueing Systems: Theory and Applications 13 (1993) 41–46.CrossRefMathSciNetGoogle Scholar
  9. [9]
    A. Ephremides and L. Tassiulas, Stability properties of constrained queueing systems and scheduling policies for maximum throughput in multihop radio networks, IEEE Trans. Automat. Control 37 (1992) 1936–1949.CrossRefMathSciNetGoogle Scholar
  10. [10]
    A. Gajrat, A. Hordijk and A. Ridder, Large deviations analysis of the fluid approximation for a controllable tandem queue, Ann. Appl. Probab. 13 (2003) 1423–1448.CrossRefGoogle Scholar
  11. [11]
    L. Georgiadis, W. Szpankowski and L. Tassiulas, A scheduling policy with maximal stability region for ring networks with spatial reuse, Queueing Systems: Theory and Applications 19(1/2) (1995) 131–148.CrossRefMathSciNetGoogle Scholar
  12. [12]
    P.W. Glynn and S.P. Meyn, A Liapounov bound for solutions of the Poisson equation, Ann. Probab. 24(2) (1996) 916–931.CrossRefGoogle Scholar
  13. [13]
    L. Green, A queueing system with general-use and limited-use servers, Oper. Res. 33(1) (1985) 168–182.Google Scholar
  14. [14]
    B. Hajek, Optimal control of two interacting service stations, IEEE Trans. Automat. Control, AC-29 (1984) 491–499.CrossRefGoogle Scholar
  15. [15]
    J. M. Harrison, Brownian Motion and Stochastic Flow Systems (Robert E. Krieger Publishing Co. Inc., Malabar, FL, 1990) Reprint of the 1985 original.Google Scholar
  16. [16]
    J.M. Harrison and M.J. López, Heavy traffic resource pooling in parallel-server systems, Queueing Systems: Theory and Applications 33 (1999) 339–368.CrossRefMathSciNetGoogle Scholar
  17. [17]
    S.G. Henderson, S.P. Meyn and V. B. Tadić, Performance evaluation and policy selection in multiclass networks, Discrete Event Dynamic Systems: Theory and Applications, 13(1/2) (2003) 149–189. Special issue on learning, optimization and decision making (invited).Google Scholar
  18. [18]
    F.P. Kelly and C.N. Laws, Dynamic routing in open queueing networks: Brownian models, cut constraints and resource pooling, QUESTA, 13 (1993) 47–86.Google Scholar
  19. [19]
    L. Kleinrock, Queueing Systems Volume 1: Theory (Wiley, 1975).Google Scholar
  20. [20]
    G. Koole, On the pathwise optimal Bernoulli routing policy for homogeneous parallel servers, Math. Operations Res. 21 (1996) 469–476.Google Scholar
  21. [21]
    V.G. Kulkarni, Modeling and Analysis of Stochastic Systems (Texts in Statistical Science Series. Chapman and Hall Ltd., London, 1995).Google Scholar
  22. [22]
    H.J. Kushner, Heavy Traffic Analysis of Controlled Queueing and Communication Networks (Springer-Verlag, New York, 2001). Stochastic Modelling and Applied Probability.Google Scholar
  23. [23]
    C.N. Laws, Dynamic routing in queueing networks, PhD thesis, Cambridge University, Cambridge, UK, 1990.Google Scholar
  24. [24]
    C. Maglaras, Dynamic scheduling in multiclass queueing networks: Stability under discrete-review policies, Queueing Systems: Theory and Applications 31 (1999) 171–206.CrossRefGoogle Scholar
  25. [25]
    A. Mandelbaum and A.L. Stolyar, Scheduling flexible servers with convex delay costs: heavy-traffic optimality of the generalized cμ-rule, Operations Res. 52(6) (2004) 836–855.CrossRefGoogle Scholar
  26. [26]
    S.P. Meyn, The policy iteration algorithm for average reward Markov decision processes with general state space, IEEE Trans. Automat. Control 42(12) (1997) 1663–1680.CrossRefGoogle Scholar
  27. [27]
    S.P. Meyn, Stability and optimization of queueing networks and their fluid models, in: Mathematics of Stochastic Manufacturing Systems (Williamsburg, VA, 1996) pp. 175–199. Amer. Math. Soc., Providence, RI, 1997.Google Scholar
  28. [28]
    S.P. Meyn, Sequencing and routing in multiclass queueing networks. Part I: Feedback regulation, SIAM J. Control Optim. 40(3) (2001) 741–776.CrossRefGoogle Scholar
  29. [29]
    S.P. Meyn, Sequencing and routing in multiclass queueing networks. Part II: Workload relaxations, SIAM J. Control Optim. 42(1) (2003) 178–217.CrossRefGoogle Scholar
  30. [30]
    S.P. Meyn, Workload models for Stochastic networks: Value functions and performance evaluation, in IEEE Transactions on Automatic Control, July 2005.Google Scholar
  31. [31]
    S.P. Meyn and R.L. Tweedie, Markov Chains and Stochastic Stability (Springer-Verlag, London, 1993). online:
  32. [32]
    T. Osogami, M. Harchol-Balter, A. Scheller-Wolf and L. Zhang, Exploring threshold-base policies for load sharing, in: Proceedings of the Forty-second Annual Allerton Conference on Communication, Control, and Computing, 2004.Google Scholar
  33. [33]
    S. Stidham, Jr. and R. Weber, A survey of Markov decision models for control of networks of queues, Queueing Systems: Theory and Applications 13(1/3) (1993) 291–314.CrossRefGoogle Scholar
  34. [34]
    A. L. Stolyar, Maxweight scheduling in a generalized switch: State space collapse and workload minimization in heavy traffic, Adv. Appl. Probab. 14(1) (2004) 1–53.CrossRefGoogle Scholar
  35. [35]
    V. Tadic and S.P. Meyn, Asymptotic properties of two time-scale stochastic approximation algorithms with constant step sizes, in: American Control Conference, 2003.Google Scholar
  36. [36]
    L. Tassiulas and A. Ephremides, Stability properties of constrained queueing systems and scheduling policies for maximum throughput in multihop radio networks, IEEE Trans. Automat. Control 37(12) (1992) 1936–1948.CrossRefMathSciNetGoogle Scholar
  37. [37]
    Y.-C. Teh and A.R. Ward, Critical thresholds for dynamic routing in queueing networks, Queueing Systems: Theory and Applications 42(3) (2002) 297–316.CrossRefGoogle Scholar
  38. [38]
    M.H. Veatch, Fluid analysis of arrival routing, IEEE Trans. Automat. Control AC-46 (2001) 1254–1257.CrossRefMathSciNetGoogle Scholar
  39. [39]
    G. Weiss, Optimal draining of a fluid re-entrant line, in: Volume 71 of IMA volumes in Mathematics and its Applications, eds. Frank Kelly and Ruth Williams (New York, Springer-Verlag, 1995) pp. 91–103.Google Scholar
  40. [40]
    R.J. Williams, Diffusion approximations for open multiclass queueing networks: sufficient conditions involving state space collapse, Queueing Systems: Theory and Applications, 30(1/2) (1998) 27–88.CrossRefGoogle Scholar

Copyright information

© Springer Science + Business Media, Inc. 2005

Authors and Affiliations

  1. 1.Department of Electrical and Computer Engineering and the Coordinated Sciences LaboratoryUniversity of Illinois at Urbana-ChampaignUrbanaUSA

Personalised recommendations