An Optimal Adaptive Scheme for Two Competing Queues with Constraints

  • Adam Shwartz
  • Armand M. Makowski
Part of the Lecture Notes in Control and Information Sciences book series (LNCIS, volume 83)


Two types of traffic, e. g., voice and data, share a single synchronous and noisy communication channel. This situation is modelled as a system of two discrete-time queues with geometric service requirements which compete for the attention of a single server. The problem is cast as one in Markov decision theory with long-run average cost and constraint. An optimal strategy is identified that possesses a simple structure, and its implementation is discussed in terms of an adaptive algorithm of the stochastic-approximations type. The proposed algorithm is extremely simple, recursive and easily implementable, with no a priori knowledge of the actual values of the statistical parameters. The derivation of the results combines martingale arguments, results on Markov chains, O.D.E. characterization of the limit of stochastic approximations and methods from weak convergence. The ideas developed here are of independent interest and should prove useful in studying broad classes of constrained Markov decision problems.


Stochastic Approximation Queue Size Optimal Bias Uniform Integrability Stochastic Approximation Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    G. Fayolle and R. Iasnogorodski, “Two coupled processors: The reduction to a Riemann-Hilbert problem,” Z. Wahr. verw. Gebiete vol. 47, pp. 325–351 (1979).CrossRefzbMATHMathSciNetGoogle Scholar
  2. [2]
    J. S. Baras, A. J. Dorsey, and A. M. Makowski, “Discrete time competing queues with geometric service requirements: stability, parameter estimation and adaptive control,” SIAM J. Control Opt., Under revision. Invited paper to the ORSA/TIMS National Meeting, San-Francisco, Cali-fornia, (May 1984).Google Scholar
  3. [3]
    J. S. Baras, A. J. Dorsey, and A. M. Makowski, “Two competing queues with geometric service requirements and linear costs: the mu-c rule is often optimal,” Adv. Appl. Prob. vol. 17, pp. 186–209 (March 1985).CrossRefzbMATHMathSciNetGoogle Scholar
  4. [4]
    P. Billingsley, Convergence of Probability Measures, John Wiley, New York (1968).zbMATHGoogle Scholar
  5. [5]
    D. P. Heyman and M. J. Sobel, Stochastic Models in Operations Research, Volume II: Stochastic Optimization, MacGraw-Hill, New York (1984).Google Scholar
  6. [6]
    S. Karlin and H. M. Taylor, A First Course in Stochastic Processes, Academic Press, New York (1975).zbMATHGoogle Scholar
  7. [7]
    P. R. Kumar, “A survey of some results in stochastic adaptive control,” SIAM J. Control Opt. vol. 23, no. 3, pp. 329–380 (May 1985).CrossRefzbMATHGoogle Scholar
  8. [8]
    H. J. Kushner and A. Shwartz, “An invariant measure approach to the convergence of Stochas-tic’ Approximations with state-dependent noise,” SIAM J. Control Opt. vol. 22, no. 1, pp. 13–27 (January 1984).CrossRefzbMATHMathSciNetGoogle Scholar
  9. [9]
    H. J. Kushner and A. Shwartz, “Weak convergence and asymptotic properties of adaptive filters with constant gains,” IEEE Trans. Info. Theory vol. IT-30, no. 1, pp. 177–182 (March 1984).MathSciNetGoogle Scholar
  10. [10]
    H. J. Kushner and A. Shwartz, “Stochastic Approximations in Hilbert space: identification and optimization of linear continuous-parameter systems,” SIAM J. Control Opt. vol. 23, no. 5, pp. 774–793 (September 1985).CrossRefzbMATHMathSciNetGoogle Scholar
  11. [11]
    P. Mandl, “Estimation and control in Markov chains,” Adv. Appl. Prob. vol. 6, pp. 40–60 (1974).CrossRefzbMATHMathSciNetGoogle Scholar
  12. [12]
    P. Nain and K. W. Ross, Optimal Priority Assignment with Constraints, Rapport de Recherche No. 346, INRIA - Rocquencourt, France (November 1984).Google Scholar
  13. [13]
    H. Robbins and S. Monro, “A Stochastic Approximation method,” Ann. Math. Stat. vol. 22, pp. 400–407 (1931).CrossRefMathSciNetGoogle Scholar
  14. [14]
    K. W. Ross, Constrained Markov Decision Processes with Queueing Applications, Ph. D. thesis, Computer, Information and Control Engineering, University of Michigan (1985).Google Scholar
  15. [15]
    S. M. Ross, Applied Probability Models with Optimization Applications, Holden-Day, San Fran-cisco (1970).zbMATHGoogle Scholar
  16. [16]
    S. M. Ross, Introduction to Stochastic Dynamic Programming, Academic Press (1984).Google Scholar
  17. [17]
    A. Shwartz and A. M. Makowsii, Adaptive schemes for server allocation in systems of compet-ing queues with constraints, Systems Research Report, In preparation, Systems Research Center, University of Maryland at Colleg Park. (1986)Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 1986

Authors and Affiliations

  • Adam Shwartz
    • 1
  • Armand M. Makowski
    • 2
  1. 1.Electrical Engineering DepartmentTechnion-Israel Institute of TechnologyHaifaISRAEL
  2. 2.Electrical Engineering Department and Systems Research CenterUniversity of MarylandCollege ParkUSA

Personalised recommendations