Wireless Personal Communications

, Volume 101, Issue 1, pp 465–490 | Cite as

A Reinforcement Learning Based Routing in Cognitive Radio Networks for Primary Users with Multi-stage Periodicity

  • Mahsa Soheil Shamaee
  • Mohammad Ebrahim ShiriEmail author
  • Masoud Sabaei


Designing an efficient routing protocol for cognitive radio networks is critical due to the dynamic behavior of the primary users. Based on empirical studies, the primary users activity on the licensed channels has periodicity comprised of several stages, and that the model of primary users activity may change during different stages. This paper has identified two main challenges facing designers: how to transmit packets via a stable route, and how to ensure imposing of minimal interference on the primary users. To address these, we propose a routing protocol which is based on a generalized version of Q-learning and which exploits the said model of primary users behavior. We divide the infinite time horizon into cycles (corresponding to periods), then break each cycle into several sub-cycles (corresponding to stages), making an assumption that the statistical model parameters of the primary users activity will not change during a sub-cycle. The extensive simulations confirm that our proposed routing approach outperforms existing schemes in terms of throughput and minimal interference. The paper also verifies that imposing of significant interference on the primary users and degradation of QoS of secondary users stem from lack of attention to the multi-stage periodic behavior of primary users.


Cognitive radio Routing Reinforcement learning Multi-stage periodicity 


  1. 1.
    FCC, E. (2003). Docket no 03-222 notice of proposed rule making and order.Google Scholar
  2. 2.
    Mitola, J. (2000). Cognitive radio—An integrated agent architecture for software defined radio.Google Scholar
  3. 3.
    Haykin, S. (2005). Cognitive radio: Brain-empowered wireless communications. IEEE Journal on Selected Areas in Communications, 23(2), 201–220.CrossRefGoogle Scholar
  4. 4.
    Akyildiz, I. F., Lee, W. Y., & Chowdhury, K. R. (2009). Crahns: Cognitive radio ad hoc networks. AD Hoc Networks, 7(5), 810–836.CrossRefGoogle Scholar
  5. 5.
    Al-Rawi, H., & Yau, A. (2013). Routing in distributed cognitive radio networks: A survey. Wireless Personal Communications, 69(4), 1983–2020.CrossRefGoogle Scholar
  6. 6.
    Caleffi, M., & Cacciapuoti, A. S. (2016). Optimal database access for tv white space. IEEE Transactions on Communications, 64(1), 83–93.CrossRefGoogle Scholar
  7. 7.
    Cacciapuoti, A. S., Caleffi, M., & Paura, L. (2016). Optimal strategy design for enabling the coexistence of heterogeneous networks in tv white space. IEEE Transactions on Vehicular Technology, 65(9), 7361–7373.CrossRefGoogle Scholar
  8. 8.
    Wellens, M., RiihijäRvi, J., & MäHöNen, P. (2009). Empirical time and frequency domain models of spectrum use. Physical Communication, 2(1), 10–32.CrossRefGoogle Scholar
  9. 9.
    Macone, D., Oddi, G., & Pietrabissa, A. (2013). Mq-routing: Mobility-, gps-and energy-aware routing protocol in manets for disaster relief scenarios. Ad Hoc Networks, 11(3), 861–878.CrossRefGoogle Scholar
  10. 10.
    Shiang, H. P., & Van der Schaar, M. (2009). Distributed resource management in multihop cognitive radio networks for delay-sensitive transmission. IEEE Transactions on Vehicular Technology, 58(2), 941–953.CrossRefGoogle Scholar
  11. 11.
    Xia, B., Wahab, M. H., Yang, Y., Fan, Z., & Sooriyabandara, M. (2009). Reinforcement learning based spectrum-aware routing in multi-hop cognitive radio networks. In 2009. CROWNCOM’09. 4th international conference on cognitive radio oriented wireless networks and communications (pp. 1–5). IEEE.Google Scholar
  12. 12.
    Zheng, K., Li, H., Qiu, R. C., & Gong, S. (2012). Multi-objective reinforcement learning based routing in cognitive radio networks: Walking in a random maze. In 2012 international conference on computing, networking and communications (ICNC) (pp. 359–363). IEEE.Google Scholar
  13. 13.
    Al-Rawi, H., Yau, A., Mohamad, H., Ramli, N., & Hashim W., et al. (2014). A reinforcement learning-based routing scheme for cognitive radio ad hoc networks. In 2014 7th IFIP wireless and mobile networking conference (WMNC) (pp. 1–8). IEEE.Google Scholar
  14. 14.
    Wang, W., Kwasinski, A., & Han, Z. (2014). Routing based on layered stochastic games for multi-hop cognitive radio networks. In 2014 IEEE wireless communications and networking conference (WCNC) (pp. 63–68). IEEE.Google Scholar
  15. 15.
    Saleem, Y., Yau, A., Mohamad, H., Ramli, N., & Rehmani, M. H. (2015). Smart: A spectrum-aware cluster-based routing scheme for distributed cognitive radio networks. Computer Networks, 91, 196–224.CrossRefGoogle Scholar
  16. 16.
    Geirhofer, S., Tong, L., & Sadler, B. M. (2008). Cognitive medium access: Constraining interference based on experimental models. IEEE Journal on Selected Areas in Communications, 26(1), 95–105.CrossRefGoogle Scholar
  17. 17.
    Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction (Vol. 1). Cambridge: MIT Press.Google Scholar
  18. 18.
    Xing, X., Jing, T., Huo, Y., Li, H., & Cheng, X. (2013). Channel quality prediction based on Bayesian inference in cognitive radio networks. In INFOCOM, 2013 proceedings IEEE (pp. 1465–1473). IEEE.Google Scholar
  19. 19.
    Varga, A. (2010). Omnet++ user manual.Google Scholar
  20. 20.
    Bayhan, S., & Alagöz, F. (2012). Distributed channel selection in crahns: A non-selfish scheme for mitigating spectrum fragmentation. Ad Hoc Networks, 10(5), 774–788.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Mathematics and Computer ScienceAmirkabir University of TechnologyTehranIran
  2. 2.Department of Computer Engineering and Information TechnologyAmirkabir University of TechnologyTehranIran

Personalised recommendations