Value Function Approximation in Complex Queueing Systems

  • Sandjai BhulaiEmail author
Part of the International Series in Operations Research & Management Science book series (ISOR, volume 248)


The application of Markov decision theory to the control of queueing systems often leads to models with enormous state spaces. Hence, direct computation of optimal policies with standard techniques and algorithms is almost impossible for most practical models. A convenient technique to overcome this issue is to use one-step policy improvement. For this technique to work, one needs to have a good understanding of the queueing system under study, and its (approximate) value function under policies that decompose the system into less complicated systems. This warrants the research on the relative value functions of simple queueing models, that can be used in the control of more complex queueing systems. In this chapter we provide a survey of value functions of basic queueing models and show how they can be applied to the control of more complex queueing systems.

Key words

One-step policy improvement Relative value function Complex queueing systems Approximate dynamic programming 


  1. 1.
    E. Altman, Applications of Markov decision processes in communication networks: a survey, in Handbook of Markov Decision Processes, edited by E.A. Feinberg, A. Shwartz (Kluwer, Dordrecht, 2002)Google Scholar
  2. 2.
    S. Asmussen, O. Nerman, M. Olsson, Fitting phase type distributions via the EM algorithm. Scand. J. Stat. 23, 419–441 (1996)Google Scholar
  3. 3.
    J.S. Baras, D.-J. Ma, A.M. Makowski, k competing queues with geometric service requirements and linear costs: the μ c rule is always optimal. Syst. Control Lett. 6, 173–180 (1985)Google Scholar
  4. 4.
    R. Bellman, Adaptive Control Processes: A Guided Tour (Princeton University Press, Princeton, NJ, 1961)CrossRefGoogle Scholar
  5. 5.
    S. Bhulai, On the value function of the M/Cox(r)/1 queue. J. Appl. Probab. 43 (2), 363–376 (2006)Google Scholar
  6. 6.
    S. Bhulai, G.M. Koole, On the structure of value functions for threshold policies in queueing models. J. Appl. Probab. 40, 613–622 (2003)CrossRefGoogle Scholar
  7. 7.
    S. Bhulai, F.M. Spieksma, On the uniqueness of solutions to the Poisson equations for average cost Markov chains with unbounded cost functions. Math. Meth. Oper. Res. 58 (2), 221–236 (2003)CrossRefGoogle Scholar
  8. 8.
    S. Bhulai, A.C. Brooms, F.M. Spieksma, On structural properties of the value function for an unbounded jump Markov process with an application to a processor-sharing retrial queue. Queueing Syst. 76 (4), 425–446 (2014)CrossRefGoogle Scholar
  9. 9.
    P. Chevalier, N. Tabordon, R. Shumsky, Routing and staffing in large call centers with specialized and fully flexible servers. Working paper (2004)Google Scholar
  10. 10.
    E. Hyytiä, J. Virtamo, Dynamic routing and wavelength assignment using first policy iteration. Technical Report COST 257, Helsinki University of Technology (2000)Google Scholar
  11. 11.
    G.M. Koole, Stochastic Scheduling and Dynamic Programming. CWI Tract, vol. 113 (CWI, Amsterdam, 1995)Google Scholar
  12. 12.
    S.A. Lippman, Applying a new device in the optimization of exponential queueing systems. Oper. Res. 23, 687–710 (1975)CrossRefGoogle Scholar
  13. 13.
    R.A. Marie, Calculating equilibrium probabilities for λ(m)∕C k∕1∕N queues, in Proceedings of the International Symposium on Computer Performance Modeling (1980)Google Scholar
  14. 14.
    R.E. Mickens, Difference Equations: Theory and Applications (Chapman & Hall, London, 1990)Google Scholar
  15. 15.
    J.M. Norman, Heuristic Procedures in Dynamic Programming (Manchester University Press, Manchester, 1972)Google Scholar
  16. 16.
    E.L. Örmeci, Dynamic admission control in a call center with one shared and two dedicated service facilities. IEEE Trans. Autom. Control 49, 1157–1161 (2004)CrossRefGoogle Scholar
  17. 17.
    T.J. Ott, K.R. Krishnan, Separable routing: a scheme for state-dependent routing of circuit switched telephone traffic. Ann. Oper. Res. 35, 43–68 (1992)CrossRefGoogle Scholar
  18. 18.
    M.L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming (Wiley, New York, 1994)CrossRefGoogle Scholar
  19. 19.
    H. Rummukainen, J. Virtamo, Polynomial cost approximations in markov decision theory based call admission control. IEEE/ACM Trans. Netw. 9 (6), 769–779 (2001)CrossRefGoogle Scholar
  20. 20.
    S.A.E. Sassen, H.C. Tijms, R.D. Nobel, A heuristic rule for routing customers to parallel servers. Statistica Neerlandica 51, 107–121 (1997)CrossRefGoogle Scholar
  21. 21.
    R.F. Serfozo, An equivalence between continuous and discrete time Markov decision processes. Oper. Res. 27, 616–620 (1979)CrossRefGoogle Scholar
  22. 22.
    F.M. Spieksma, Geometrically ergodic Markov chains and the optimal control of queues. Ph.D. thesis, Leiden University (1990)Google Scholar
  23. 23.
    S. Stidham Jr., R.R. Weber, A survey of Markov decision models for control of networks of queues. Queueing Syst. 13, 291–314 (1993)CrossRefGoogle Scholar
  24. 24.
    H.C. Tijms, Stochastic Models: An Algorithmic Approach (Wiley, New York, 1994)Google Scholar
  25. 25.
    D. Towsley, R.H. Hwang, J.F. Kurose, MDP routing for multi-rate loss networks. Comput. Netw. ISDN 34, 241–261 (2000)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Faculty of SciencesVrije Universiteit AmsterdamAmsterdamThe Netherlands

Personalised recommendations