Wireless Networks

, Volume 24, Issue 4, pp 1251–1264 | Cite as

A multi-state Q-learning based CSMA MAC protocol for wireless networks

  • Hossein Bayat-Yeganeh
  • Vahid Shah-Mansouri
  • Hamed Kebriaei


Due to the shared nature of wireless channels, competition among the nodes to access media is inevitable. P-persistent carrier sense multiple access (CSMA) is a medium access control scheme widely used for resource allocation in wireless environments. The probability of transmission highly affects the throughput. We model the wireless nodes as agents in a network game. The strategy of an agent is defined as the probability of transmission. Agents don’t have a priori information about the network (e.g., number of nodes, other agents strategies) and learn their optimal strategy using the historical sensory information including the number of collisions or successful transmissions. In this paper, a multi-state reinforcement learning (RL) method is designed for this purpose. The main challenge in designing an RL agent is to define the states of the environment from agent’s perspective. For this purpose, in this paper, various state representations are proposed in a multi-state Q-learning model. This leads to different agents personalities ranging from cautious agents with risk aversion to aggressive risky agents. We also propose agents with combined personalities of cautiousness and aggressiveness. The performance of the proposed Q-learning agents with different state definitions in comparison with each other and also other benchmarking agents is examined via comprehensive simulation results.


Carrier sense multiple access (CSMA) Multi-state reinforcement learning Persistent probability 


  1. 1.
    Felegyhazi, M., & Hubaux, J-P. (2006). Game theory in wireless networks: A tutorial, Technical report LCA-REPORT-2006-002, EPFL.Google Scholar
  2. 2.
    Alfano, G., Garetto, M., & Leonardi, E. (2013). New directions into the stochastic geometry analysis of dense CSMA networks. IEEE Transactions on Mobile Computing, 13(2), 324–336.CrossRefGoogle Scholar
  3. 3.
    Busson, A., & Chelius, G. (2014). Capacity and interference modeling of CSMA/CA networks using SSI point processes. Telecommunication Systems, 57(1), 25–39.CrossRefGoogle Scholar
  4. 4.
    Cohen, K., Nedic, A., & Srikant, R. (2015). Distributed learning algorithms for spectrum sharing in spatial random access wireless networks. Available online: Scholar
  5. 5.
    Chu, Yi, Kosunalp, S., Mitchell, P. D., Grace, D., & Clarke, Tim. (2015). Application of reinforcement learning to medium access control for wireless sensor networks. Engineering Applications of Artificial Intelligence, 46, 23–32.CrossRefGoogle Scholar
  6. 6.
    Du, Q., & Zhang, X. (2009). Game-theoretic approach for QoS-aware resource competition in wireless networks. In Proceedings of IEEE MILCOM.Google Scholar
  7. 7.
    Wu, D., & Negi, R. (2003). Effective capacity: A wireless link model for support of quality of service. IEEE Transactions on Wireless Communications, 2(4), 630–643.Google Scholar
  8. 8.
    Chen, L., Low, S. H., & Doyle, J. C. (2010). Random access game and medium access control design. IEEE/ACM Transactions Networking, 18(4), 1303–1316.CrossRefGoogle Scholar
  9. 9.
    Cui, T., Chen, L., & Low, S. H. (2008). A Game-theoretic framework for medium access control. IEEE Transactions on Selected Areas in Communications, 26(7), 1116.CrossRefGoogle Scholar
  10. 10.
    Cho, Y., Hwang, C-S., & Tobagi, F A. (2008). Design of robust random access protocols for wireless networks using game theoretic models. In Proceedings of IEEE INFOCOM, Phoenix, AZ.Google Scholar
  11. 11.
    Tanenbaum, A. S. (2010). Computer networks (pp. 141–148). Eagle Cliffs, US: Prentice Hall.Google Scholar
  12. 12.
    Ghazvini, M., Movahedinia, N., Jamshidi, K., & Moghim, N. (2013). Game theory applications in CSMA methods. IEEE Communications Surveys and Tutorials, 15(3), 1062–1087.CrossRefGoogle Scholar
  13. 13.
    Akkarajitsakul, K., Hossain, E., Niyato, D., & Kim, D. (2012). Game theoretic approaches for multiple access in wireless networks: A survey. IEEE Communication Surveys and Tutorials, 13(3), 372–395.CrossRefGoogle Scholar
  14. 14.
    Zhang, Y., & Lazos, L. (2013) Countering selfish misbehavior in multi-channel MAC protocols. In Proceedings of IEEE Infocom.Google Scholar
  15. 15.
    Huang, J. W., & Krishnamurthy, V. (2010). Transmission control in cognitive radio as a markovian dynamic game: Structural result on randomized threshold policies. IEEE Transactions on Communications, 58(1), 300–310.CrossRefGoogle Scholar
  16. 16.
    Huang, J. W., & Krishnamurthy, V. (2009). Game theoretic issues in cognitive radio systems. Journal of Communications, 4(10), 790–802.Google Scholar
  17. 17.
    Wu, Y., Wang, B., & Liu, K.J.R. (2008). Repeated spectrum sharing game with self-enforcing truth-telling mechanism. In Proceedings of IEEE ICC (pp. 3583–3587).Google Scholar
  18. 18.
    Inaltekin, H., & Wicker, S. (2006). The analysis of a game theoretic MAC protocol for wireless networks. In Proceedings of IEEE SECON (pp. 296–305).Google Scholar
  19. 19.
    Cho, Y., & Tobagi, F. A. (2008). Cooperative and non-cooperative ALOHA games with channel capture. In Proceedings of IEEE GLOBCOM (pp. 1–6).Google Scholar
  20. 20.
    Cho, Y., Hwang, C. S., and Tobagi, F. A. (2008). Design of robust random access protocols for wireless networks using game theoretic models. In Proceedings IEEE INFOCOM (pp. 1750–1758).Google Scholar
  21. 21.
    Tian, J. (2014). Game-theory model based on carrier sense multiple access protocol in wireless network. Journal of Networks, 9(6), 1603–1609.CrossRefGoogle Scholar
  22. 22.
    Nuggehalli, P., Sarkar, M., Kulkarni, K., Rao, R. R.(2008). A game-theoretic analysis of QoS in wireless MAC. In Proceedings IEEE INFOCOM (pp. 1903–1911).Google Scholar
  23. 23.
    Wang, D., Comaniciu, C., Minn, H., & Al-Dhahir, N. (2008). A game-theoretic approach for exploiting multiuser diversity in cooperative slotted aloha. IEEE Transactions on Wireless Communications, 7(11), 4215–4225.CrossRefGoogle Scholar
  24. 24.
    Tembine, H., Altaian, E., & El-Azouzi, R.(2007). Delayed evolutionary game dynamics applied to medium access control. In Proceedings of IEEE international conference on mobile adhoc and sensor systems.Google Scholar
  25. 25.
    Ma, R. T. B., Misra, V., & Rubenstein, D. (2009). An analysis of generalized slotted-ALOHA protocols. IEEE/ACM Transactions on Networking, 17(3), 936–949.CrossRefGoogle Scholar
  26. 26.
    Zhang, G., Zhang, H., & Zhao, L. (2007). A novel MAC scheme for wireless LANs from the perspective of game theory. In Proceedings of IET conference on wireless, mobile and sensor networks.Google Scholar
  27. 27.
    Chen, L., Low, S. H., & Doyle, J. C. (2007). Contention control: A game-theoretic approach. In Proceedings of 46th IEEE conference on decision and control.Google Scholar
  28. 28.
    Felegyhazi, M., Cagalj, M., & Hubaux, J.-P. (2009). Efficient MAC in cognitive radio systems: A game-theoretic approach. IEEE Transactions on Wireless Communications, 8(4), 1984–1995.CrossRefGoogle Scholar
  29. 29.
    Felegyhazi, M. (2007). Noncooperative behavior in wireless networks, Ph.D. dissertation, EPFL, Switzerland.Google Scholar
  30. 30.
    Konorski, J. (2006). A game-theoretic study of CSMA/CA under a backoff attack. IEEE/ACM Transactions on Networking, 14(6), 1167–1178.CrossRefGoogle Scholar
  31. 31.
    Li, H., Grace, D., & Mitchell, P.D. (2010). Collision reduction in cognitive radio using multichannel 1-persistent CSMA combined with reinforcement learning. In Proceedings of fifth international conference on cognitive radio oriented wireless network and communications, France.Google Scholar
  32. 32.
    Bao, S., & Fujii, T. (2011). Q-learning based p-persistent CSMA MAC protocol for secondary user of cognitive radio networks. In IEEE Third international conference on intelligent networking and collaborative systems (INCoS).Google Scholar
  33. 33.
    Shah, S M., Krishna C, A., & Sharma, V. (2016). Resource allocation in a MAC with and without security via game theoretic learning. InIEEE information theory and applicaitons (ITA) Workshop, CA: San Diego.Google Scholar
  34. 34.
    MacKenzie, A. B., & Wicker, S. B. (2001). Game theory and the design of self-configuring, adaptive wireless networks. IEEE communications magazine (pp. 126-131).Google Scholar
  35. 35.
    Watkins, C., & Dayan, P. (1992). Q-learning. Technical Note Machine learning, 8(3–4), 279–292.zbMATHGoogle Scholar
  36. 36.
    Sutton, Richard S., & Barto, Andrew G. (1998). Reinforcement learning: An introduction. Cambridge: MIT press.Google Scholar
  37. 37.
    Darmona, E., & Waldeckb, R. (2005). Convergence of reinforcement learning to Nash equilibrium: A search-market experiment. Physica, 335, 119–130.MathSciNetCrossRefGoogle Scholar
  38. 38.
    Szepesvári, C. (2010). Algorithms for reinforcement learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 4(1), 1–103.CrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.School of Engineering Science, College of EngineeringUniversity of TehranTehranIran
  2. 2.School of Electrical and Computer EngineeringUniversity of TehranTehranIran
  3. 3.School of Computer Science, Institute for Research in Fundamental Sciences (IPM)TehranIran

Personalised recommendations