Skip to main content
SpringerLink
Log in

A multi-state Q-learning based CSMA MAC protocol for wireless networks

  • Published:
Wireless Networks Aims and scope Submit manuscript

Abstract

Due to the shared nature of wireless channels, competition among the nodes to access media is inevitable. P-persistent carrier sense multiple access (CSMA) is a medium access control scheme widely used for resource allocation in wireless environments. The probability of transmission highly affects the throughput. We model the wireless nodes as agents in a network game. The strategy of an agent is defined as the probability of transmission. Agents don’t have a priori information about the network (e.g., number of nodes, other agents strategies) and learn their optimal strategy using the historical sensory information including the number of collisions or successful transmissions. In this paper, a multi-state reinforcement learning (RL) method is designed for this purpose. The main challenge in designing an RL agent is to define the states of the environment from agent’s perspective. For this purpose, in this paper, various state representations are proposed in a multi-state Q-learning model. This leads to different agents personalities ranging from cautious agents with risk aversion to aggressive risky agents. We also propose agents with combined personalities of cautiousness and aggressiveness. The performance of the proposed Q-learning agents with different state definitions in comparison with each other and also other benchmarking agents is examined via comprehensive simulation results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Felegyhazi, M., & Hubaux, J-P. (2006). Game theory in wireless networks: A tutorial, Technical report LCA-REPORT-2006-002, EPFL.

  2. Alfano, G., Garetto, M., & Leonardi, E. (2013). New directions into the stochastic geometry analysis of dense CSMA networks. IEEE Transactions on Mobile Computing, 13(2), 324–336.

    Article  Google Scholar 

  3. Busson, A., & Chelius, G. (2014). Capacity and interference modeling of CSMA/CA networks using SSI point processes. Telecommunication Systems, 57(1), 25–39.

    Article  Google Scholar 

  4. Cohen, K., Nedic, A., & Srikant, R. (2015). Distributed learning algorithms for spectrum sharing in spatial random access wireless networks. Available online: http://arxiv.org/abs/1507.05664.

  5. Chu, Yi, Kosunalp, S., Mitchell, P. D., Grace, D., & Clarke, Tim. (2015). Application of reinforcement learning to medium access control for wireless sensor networks. Engineering Applications of Artificial Intelligence, 46, 23–32.

    Article  Google Scholar 

  6. Du, Q., & Zhang, X. (2009). Game-theoretic approach for QoS-aware resource competition in wireless networks. In Proceedings of IEEE MILCOM.

  7. Wu, D., & Negi, R. (2003). Effective capacity: A wireless link model for support of quality of service. IEEE Transactions on Wireless Communications, 2(4), 630–643.

    Google Scholar 

  8. Chen, L., Low, S. H., & Doyle, J. C. (2010). Random access game and medium access control design. IEEE/ACM Transactions Networking, 18(4), 1303–1316.

    Article  Google Scholar 

  9. Cui, T., Chen, L., & Low, S. H. (2008). A Game-theoretic framework for medium access control. IEEE Transactions on Selected Areas in Communications, 26(7), 1116.

    Article  Google Scholar 

  10. Cho, Y., Hwang, C-S., & Tobagi, F A. (2008). Design of robust random access protocols for wireless networks using game theoretic models. In Proceedings of IEEE INFOCOM, Phoenix, AZ.

  11. Tanenbaum, A. S. (2010). Computer networks (pp. 141–148). Eagle Cliffs, US: Prentice Hall.

    Google Scholar 

  12. Ghazvini, M., Movahedinia, N., Jamshidi, K., & Moghim, N. (2013). Game theory applications in CSMA methods. IEEE Communications Surveys and Tutorials, 15(3), 1062–1087.

    Article  Google Scholar 

  13. Akkarajitsakul, K., Hossain, E., Niyato, D., & Kim, D. (2012). Game theoretic approaches for multiple access in wireless networks: A survey. IEEE Communication Surveys and Tutorials, 13(3), 372–395.

    Article  Google Scholar 

  14. Zhang, Y., & Lazos, L. (2013) Countering selfish misbehavior in multi-channel MAC protocols. In Proceedings of IEEE Infocom.

  15. Huang, J. W., & Krishnamurthy, V. (2010). Transmission control in cognitive radio as a markovian dynamic game: Structural result on randomized threshold policies. IEEE Transactions on Communications, 58(1), 300–310.

    Article  Google Scholar 

  16. Huang, J. W., & Krishnamurthy, V. (2009). Game theoretic issues in cognitive radio systems. Journal of Communications, 4(10), 790–802.

    Google Scholar 

  17. Wu, Y., Wang, B., & Liu, K.J.R. (2008). Repeated spectrum sharing game with self-enforcing truth-telling mechanism. In Proceedings of IEEE ICC (pp. 3583–3587).

  18. Inaltekin, H., & Wicker, S. (2006). The analysis of a game theoretic MAC protocol for wireless networks. In Proceedings of IEEE SECON (pp. 296–305).

  19. Cho, Y., & Tobagi, F. A. (2008). Cooperative and non-cooperative ALOHA games with channel capture. In Proceedings of IEEE GLOBCOM (pp. 1–6).

  20. Cho, Y., Hwang, C. S., and Tobagi, F. A. (2008). Design of robust random access protocols for wireless networks using game theoretic models. In Proceedings IEEE INFOCOM (pp. 1750–1758).

  21. Tian, J. (2014). Game-theory model based on carrier sense multiple access protocol in wireless network. Journal of Networks, 9(6), 1603–1609.

    Article  Google Scholar 

  22. Nuggehalli, P., Sarkar, M., Kulkarni, K., Rao, R. R.(2008). A game-theoretic analysis of QoS in wireless MAC. In Proceedings IEEE INFOCOM (pp. 1903–1911).

  23. Wang, D., Comaniciu, C., Minn, H., & Al-Dhahir, N. (2008). A game-theoretic approach for exploiting multiuser diversity in cooperative slotted aloha. IEEE Transactions on Wireless Communications, 7(11), 4215–4225.

    Article  Google Scholar 

  24. Tembine, H., Altaian, E., & El-Azouzi, R.(2007). Delayed evolutionary game dynamics applied to medium access control. In Proceedings of IEEE international conference on mobile adhoc and sensor systems.

  25. Ma, R. T. B., Misra, V., & Rubenstein, D. (2009). An analysis of generalized slotted-ALOHA protocols. IEEE/ACM Transactions on Networking, 17(3), 936–949.

    Article  Google Scholar 

  26. Zhang, G., Zhang, H., & Zhao, L. (2007). A novel MAC scheme for wireless LANs from the perspective of game theory. In Proceedings of IET conference on wireless, mobile and sensor networks.

  27. Chen, L., Low, S. H., & Doyle, J. C. (2007). Contention control: A game-theoretic approach. In Proceedings of 46th IEEE conference on decision and control.

  28. Felegyhazi, M., Cagalj, M., & Hubaux, J.-P. (2009). Efficient MAC in cognitive radio systems: A game-theoretic approach. IEEE Transactions on Wireless Communications, 8(4), 1984–1995.

    Article  Google Scholar 

  29. Felegyhazi, M. (2007). Noncooperative behavior in wireless networks, Ph.D. dissertation, EPFL, Switzerland.

  30. Konorski, J. (2006). A game-theoretic study of CSMA/CA under a backoff attack. IEEE/ACM Transactions on Networking, 14(6), 1167–1178.

    Article  Google Scholar 

  31. Li, H., Grace, D., & Mitchell, P.D. (2010). Collision reduction in cognitive radio using multichannel 1-persistent CSMA combined with reinforcement learning. In Proceedings of fifth international conference on cognitive radio oriented wireless network and communications, France.

  32. Bao, S., & Fujii, T. (2011). Q-learning based p-persistent CSMA MAC protocol for secondary user of cognitive radio networks. In IEEE Third international conference on intelligent networking and collaborative systems (INCoS).

  33. Shah, S M., Krishna C, A., & Sharma, V. (2016). Resource allocation in a MAC with and without security via game theoretic learning. InIEEE information theory and applicaitons (ITA) Workshop, CA: San Diego.

  34. MacKenzie, A. B., & Wicker, S. B. (2001). Game theory and the design of self-configuring, adaptive wireless networks. IEEE communications magazine (pp. 126-131).

  35. Watkins, C., & Dayan, P. (1992). Q-learning. Technical Note Machine learning, 8(3–4), 279–292.

    MATH  Google Scholar 

  36. Sutton, Richard S., & Barto, Andrew G. (1998). Reinforcement learning: An introduction. Cambridge: MIT press.

    Google Scholar 

  37. Darmona, E., & Waldeckb, R. (2005). Convergence of reinforcement learning to Nash equilibrium: A search-market experiment. Physica, 335, 119–130.

    Article  MathSciNet  Google Scholar 

  38. Szepesvári, C. (2010). Algorithms for reinforcement learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 4(1), 1–103.

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Engineering Science, College of Engineering, University of Tehran, Tehran, Iran

    Hossein Bayat-Yeganeh

  2. School of Electrical and Computer Engineering, University of Tehran, P.O. Box 14395-515, Tehran, Iran

    Vahid Shah-Mansouri & Hamed Kebriaei

  3. School of Computer Science, Institute for Research in Fundamental Sciences (IPM), P.O. Box 19395-5746, Tehran, Iran

    Hamed Kebriaei

Authors
  1. Hossein Bayat-Yeganeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vahid Shah-Mansouri
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hamed Kebriaei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hamed Kebriaei.

Additional information

This research was in part supported by a grant from the Institute for Research in Fundamental Sciences (IPM) (No. CS1395-4-02)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bayat-Yeganeh, H., Shah-Mansouri, V. & Kebriaei, H. A multi-state Q-learning based CSMA MAC protocol for wireless networks. Wireless Netw 24, 1251–1264 (2018). https://doi.org/10.1007/s11276-016-1402-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11276-016-1402-0

Keywords

Search

Navigation