Optimal Distributed Uplink Channel Allocation: A Constrained MDP Formulation

  • Vivek S. Borkar
  • Joy Kuri
Part of the Annals of the International Society of Dynamic Games book series (AISDG, volume 11)


Several users share a common channel for transmission, which has an average rate constraint. The packets not yet transmitted are queued. The problem of optimal channel allocation to minimize the average sum queue occupancy subject to this constraint splits into individual Markov decision processes (MDPs) coupled through the Lagrange multiplier for the rate constraint. This multiplier can be computed by an on-line stochastic gradient ascent in a centralized or distributed manner. This gives a stochastic dynamic version of Kelly’s decomposition. A learning scheme is also presented.


Queue Length Congestion Control Markov Decision Process Stochastic Approximation Communication Constraint 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abounadi, J., Bertsekas, D.P., Borkar, V.S.: Learning Algorithms for Markov Decision Processes with Average Cost. SIAM Journal on Control and Optimization 40(3), 681–698 (2001)MATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Agarwal, M., Borkar, V.S., Karandikar, A.: Structural Properties of Optimal Tansmission Policies over a Randomly Varying Channel. IEEE Transactions on Automatic Control 53(6), 1476–1491 (2008)CrossRefMathSciNetGoogle Scholar
  3. 3.
    Altman, E.: Constrained Markov Decision Processes. Chapman and Hall/CRC, Boca Raton, Florida, USA (1999)MATHGoogle Scholar
  4. 4.
    Andrews, M., Kumaran, K., Ramanan, K., Stolyar, A., Whiting, P., Vijayakumar, R.: Providing Quality of Service over a Shared Wireless Link. IEEE Communications Magazine 39(2), 150–153 (2001)CrossRefGoogle Scholar
  5. 5.
    Borkar, V.S.: Control of Markov Chains with Long-run Average Cost Criterion: the Dynamic Programming Equations. SIAM Journal on Control and Optimization 27(3), 642–657 (1989)MATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Borkar, V.S.: Ergodic Control of Markov Chains with Constraints – the General Case. SIAM Journal on Control and Optimization 32(1), 176–186 (1994)MATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    Borkar, V.S.: An Actor-Critic Algorithm for Constrained Markov Decision Processes. Systems & Control Letters 54(3), 207–213 (2005)MATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Borkar, V.S.: Stochastic Approximation: A Dynamic Viewpoint. Hindustan Book Agency, New Delhi, India and Cambridge University Press, Cambridge, UK (2008)Google Scholar
  9. 9.
    Borkar, V.S.: Reinforcement Learning: A Bridge Between Numerical Methods and Markov Chain Monte Carlo. In: N.S.N. Sastry, T.S.S.R.K. Rao, M. Delampady and B. Rajeev (ed.) Perspectives in Mathematical Sciences I: Probability and Statistics, Statistical Sciences and Interdisciplinary Research, vol. 7, chap. 4, pp. 71–91. World Scientific, Singapore (2009)Google Scholar
  10. 10.
    Buche, R., Kushner, H.J.: Control of Mobile Communication Systems with Time-varying Channels via Stability Methods. IEEE Transactions on Automatic Control 49(11), 1954–1962 (2004)CrossRefMathSciNetGoogle Scholar
  11. 11.
    Bui, L., Eryilmaz, A., Srikant, R., Wu, X.: Joint Asyncronous Congestion Control and Distributed Scheduling for Multihop Wireless Networks. In: Proceedings of INFOCOM (2006)Google Scholar
  12. 12.
    Chen, L., Low, S.H., Chiang, M., Doyle, J.C.: Cross-layer Congestion Control, Routing and Scheduling Design in Ad Hoc Wireless Networks. In: Proceedings of INFOCOM (2006)Google Scholar
  13. 13.
    Eryilmaz, A., Srikant, R.: Joint Congestion Control, Routing and MAC for Fairness and Stability in Wireless Networks. IEEE Journal on Selected Areas in Communications 24(8), 1514–1524 (2006)Google Scholar
  14. 14.
    Eryilmaz, A., Srikant, R.: Fair Resource Allocation in Wireless Networks using Queue-Length-based Scheduling and Congestion Control. IEEE/ACM Transactions on Networking 15(6), 1333–1344 (2007)CrossRefGoogle Scholar
  15. 15.
    Gupta, A., Lin, X., Srikant, R.: Low Complexity Distributed Scheduling Algorithms for Wireless Networks. In: Proceedings of INFOCOM (2007)Google Scholar
  16. 16.
    Hernández-Lerma, O., Lasserre, J.B.: Discrete-Time Markov Control Processes. Springer, New York (1996)Google Scholar
  17. 17.
    Hernández-Lerma, O., Lasserre, J.B.: Further Topics in Discrete-Time Markov Control Processes. Springer, New York (1999)Google Scholar
  18. 18.
    Kelly, F.P.: Charging and Rate Control for Elastic Traffic. European Transactions on Telecommunications 8(1), 33–37 (1997)CrossRefGoogle Scholar
  19. 19.
    Kelly, F.P., Maulloo, A., Tan, D.: Rate Control for Communication Networks: Shadow Prices, Proportional Fairness and Stability. Journal of the Operational Research Society 49(3), 237–252 (1998)MATHGoogle Scholar
  20. 20.
    Kunniyur, S., Srikant, R.: End-to-end Congestion Control Schemes: Utility Functions, Random Losses and ECN marks. IEEE/ACM Transactions on Networking 11(5), 689–702 (2003)CrossRefGoogle Scholar
  21. 21.
    Lin, X., Shroff, N., Srikant, R.: A Tutorial on Cross-Layer Optimization in Wireless Networks. IEEE Journal on Selected Areas in Communications 24(8), 1452–1463 (2006)CrossRefGoogle Scholar
  22. 22.
    Low, S.H.: A Duality Model of TCP and Queue Management Algorithms. IEEE/ACM Transactions on Networking 11(4), 525–536 (2003)CrossRefMathSciNetGoogle Scholar
  23. 23.
    Low, S.H., Lapsley, D.: Optimization Flow Control-I: Basic Algorithm and Convergence. IEEE/ACM Transactions on Networking 7(6), 861–874 (1999)CrossRefGoogle Scholar
  24. 24.
    Manne, A.: Linear Programming and Sequential Decisions. Management Science 6(3), 259–267 (1960)MATHCrossRefMathSciNetGoogle Scholar
  25. 25.
    Milgrom, P., Segal, I.: Envelope Theorems for Arbitrary Choice Sets. Econometrica 70(2), 583–601 (2002)MATHCrossRefMathSciNetGoogle Scholar
  26. 26.
    Puterman, M.L.: Markov Decision Processes. Wiley-Interscience, Hoboken, New Jersey (1994)MATHCrossRefGoogle Scholar
  27. 27.
    Salodkar, N., Bhorkar, A., Karandikar, A., Borkar, V.S.: An Online Learning Algorithm for Energy Efficient Delay Constrained Scheduling over a Fading Channel. IEEE Journal on Selected Areas in Communications 26(4), 732–742 (2008)CrossRefGoogle Scholar
  28. 28.
    Salodkar, N., Karandikar, A., Borkar, V.S.: A Stable Online Algorithm for Energy-Efficient Multiuser Scheduling. IEEE Transactions on Mobile Computing 9(10), 1391–1406 (2010)CrossRefGoogle Scholar
  29. 29.
    Shakkottai, S., Stolyar, A.: Scheduling of a Shared Time-Varying Channel: the Exponential Rule Stability. In: Proceedings of INFORMS Applied Probability Conference (2001)Google Scholar
  30. 30.
    Shakkottai, S., Stolyar, A.: Scheduling for Multiple Flows sharing a Time-varying Channel: the Exponential Rule. Translations of the American Mathematical Society 207 (a volume in memory of F. Karpelevich) (2002)Google Scholar
  31. 31.
    Stolyar, A.L.: Maximizing Queueing Network Utility subject to Stability: Greedy Primal-Dual Algorithm. Queueing Systems: Theory and Applications (QUESTA) 50(4), 401–457 (2005)Google Scholar
  32. 32.
    Tan, L., Zhang, X., Andrew, L.L.H., Zukerman, M.: Price-based Max-Min Fair Rate Allocation in Wireless Multi-hop Networks. IEEE Communications Letters 10(1), 31–33 (2006)CrossRefGoogle Scholar
  33. 33.
    Tassiulas, L., Ephremides, A.: Stability Properties of Constrained Queueing Systems and Scheduling Policies for Maximum Throughput in Multihop Radio Networks. IEEE Transactions on Automatic Control 37(12), 1936–1948 (1992)MATHCrossRefMathSciNetGoogle Scholar
  34. 34.
    Watkins, C.: Learning from Delayed Rewards. Ph.D. thesis, Cambridge University, UK (1989)Google Scholar
  35. 35.
    Wu, X., Srikant, R., Perkins, J.R.: Queue-Length Stability of Maximal Greedy Schedules in Wireless Networks. IEEE Transactions on Mobile Computing 6(6), 595–605 (2007)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.School of Technology and Computer ScienceTata Institute of Fundamental ResearchMumbaiIndia
  2. 2.Centre for Electronics Design and TechnologyIndian Institute of ScienceBangaloreIndia

Personalised recommendations