Advertisement

A Survey and Performance Evaluation of Reinforcement Learning Based Spectrum Aware Routing in Cognitive Radio Ad Hoc Networks

  • Rashmi Naveen RajEmail author
  • Ashalatha Nayak
  • M. Sathish Kumar
Article
  • 16 Downloads

Abstract

Cognitive radio technology is an assuring solution for under-utilization of licensed spectrum bands and overcrowding of unlicensed spectrum bands, in which secondary user is permitted to access the primary users’ spectrum in an opportunistic manner. Opportunistic access of the spectrum requires complex changes across all the layers of a network protocol stack. Cognitive radio has to be an autonomous agent in order to configure itself to dynamic spectrum environment. And, the characteristics of reinforcement learning, a subfield of artificial intelligence in which the agent learns the surrounding operating environment through continuous interaction and takes an optimum decision on the fly, is in compliance with features of self-organized cognitive radio ad hoc network. Therefore, reinforcement learning is an appropriate option for incorporating intelligence and self-adaptivity into cognitive radio. This paper provides a comprehensive survey on the application of reinforcement learning for efficient spectrum aware routing in cognitive radio ad hoc network. The preliminaries of cognitive radio ad hoc networks and reinforcement learning are first introduced, and a review is investigated in the proposed research area along with a discussion on open research challenges with an aim to promote research. From the survey, reinforcement learning incorporated cognitive radio can learn the unknown primary user network model and the learned model can be then used for finding a suitable route to meet the Quality of Service requirements. With this in mind, the paper also proposes a multi-objective reinforcement learning based spectrum aware routing protocol with an aim to maximize the probability of successful transmission using a minimum hop path. The simulated results prove the performance of the algorithm.

Keywords

Cognitive radio Reinforcement learning Cognitive radio ad hoc network Opportunistic access Artificial intelligence Spectrum aware routing protocol Multi-objective 

Notes

References

  1. 1.
    J. Mitola. Cognitive radio for flexible mobile multimedia communications. In Proceedings of International Workshop on Mobile Multimedia Communications, San Diego, CA, 1999.Google Scholar
  2. 2.
    Spectrum policy task force report. Federal Communication Commission, 2002.Google Scholar
  3. 3.
    J. Marinho and E. Monteiro, Cognitive radio: survey on communication protocols, spectrum decision issues, and future research directions, Wireless Networks, Vol. 18, No. 2, pp. 147–164, 2012.CrossRefGoogle Scholar
  4. 4.
    I. F. Akyildiz, B. F. Lo and R. Balakrishnan, Cooperative spectrum sensing in cognitive radio networks: a survey, Physical Communication, Vol. 4, No. 1, pp. 40–62, 2011.CrossRefGoogle Scholar
  5. 5.
    I. F. Akyildiz, W. Lee and K. R. Chowdhury, CRAHNs: cognitive radio ad hoc networks, Ad Hoc Networks, Vol. 7, No. 5, pp. 810–836, 2009.CrossRefGoogle Scholar
  6. 6.
    K. Patil, R. Prasad, and K. Skouby. A survey of worldwide spectrum occupancy measurement campaign for cognitive radio. In Proceedings of International Conference on Devices and Communications (ICDeCom), India, pp. 1–5, Feb. 2011.Google Scholar
  7. 7.
    J. Xue, Z. Feng, and P. Zhang (2013) Spectrum occupancy measurements and analysis in beijing. IERI Procedia, 4:295–302. In 2013 International Conference on Electronic Engineering and Computer Science (EECS 2013).Google Scholar
  8. 8.
    S. D. Barnes, P. A. Jansen van Vuuren and B. T. Maharaj, Spectrum occupancy investigation: measurements in south africa, Measurement, Vol. 46, No. 9, pp. 3098–3112, 2013.CrossRefGoogle Scholar
  9. 9.
    S. D. Barnes, P. R. Botha and B. T. Maharaj, Spectral occupation of TV broadcast bands: measurement and analysis, Measurement, Vol. 93, No. Supplement C, pp. 272–277, 2016.CrossRefGoogle Scholar
  10. 10.
    D. Das and S. Das, A survey on spectrum occupancy measurement for cognitive radio, Wireless Personal Communications, Vol. 85, No. 4, pp. 2581–2598, 2015.CrossRefGoogle Scholar
  11. 11.
    A. Ali and W. Hamouda, Advances on spectrum sesning for cognitive radio networks: theory and applications, IEEE Communication Surveys and Tutorials, Vol. 19, No. 2, pp. 1277–1304, 2017.CrossRefGoogle Scholar
  12. 12.
    G. I. Tsiropoulos, O. A. Dobre, M. H. Ahmed and K. F. Baddour, Radio resource allocation techniques for efficient spectrum access in cognitive radio network, IEEE Communication Surveys and Tutorials, Vol. 18, No. 1, pp. 824–847, 2016.CrossRefGoogle Scholar
  13. 13.
    K. Kumar, A. Prakash and R. Trpathi, Spectrum handoff in cognitive radio networks: a classification and comprehensive survey, Journal of Network and Computer Applications, Vol. 61, pp. 161–188, 2016.CrossRefGoogle Scholar
  14. 14.
    M. Youssef, M. Ibrahim, M. Abdelatif, L. Chen and A. V. Vasilakos, Routing metrics of cognitive radio networks: a survey, IEEE Communications Surveys and Tutorials, Vol. 16, No. 1, pp. 92–109, 2014.CrossRefGoogle Scholar
  15. 15.
    K. Singh and S. Moh, Routing protocols in cognitive radio ad hoc networks: a comprehensive review, Journal of Network and Computer Applications, Vol. 72, pp. 28–37, 2016.CrossRefGoogle Scholar
  16. 16.
    Y. Saleem and M. Rehmani, Primary radio user activity models for cognitive radio networks: a survey, Journal of Network and Computer Applications, Vol. 43, pp. 1–16, 2014.CrossRefGoogle Scholar
  17. 17.
    R. K. Sharma and D. B. Rawat, Advances on security threats and countermeasures for cognitive radio networks: a survey, IEEE Communication Surveys and Tutorials, Vol. 17, No. 2, pp. 1023–1043, 2015.CrossRefGoogle Scholar
  18. 18.
    Y. Liang, K. Chen, G. Y. Li and P. Mahonen, Cognitive radio networking and communications: an overview, IEEE Transactions on Vehicular Technology, Vol. 60, No. 7, pp. 3386–3407, 2011.CrossRefGoogle Scholar
  19. 19.
    J. Wang, M. Ghosh and K. Challapali, Emerging cognitive radio applications: a survey, IEEE Communications Magazine, Vol. 49, No. 3, pp. 74–81, 2011.CrossRefGoogle Scholar
  20. 20.
    P. C. Jain. Rural wireless broad band internet access in wireless regional area network using cognitive radio. In Proceedings of International conference on signal processing and communication, Noida, India, 2013.Google Scholar
  21. 21.
    K. Katzis and H. Ahmadi, Challenges Implementing Internet of Things (IoT) Using Cognitive Radio Capabilities in 5G Mobile Networks, Springer International Publishing, Cham, 2016. pp. 55–76.Google Scholar
  22. 22.
    A. Athar, M. H. Rehmani and A. Rachedi, Cognitive-radio-based Internet of Things: applications, architectures, spectrum related functionalities, and future research directions, IEEE Wireless Communications, Vol. 24, No. 3, pp. 17–25, 2017.CrossRefGoogle Scholar
  23. 23.
    S. Sengupta and K. P. Subbalakshmi, Open research issues in multi-hop cognitive radio networks, IEEE Communications Magazine, Vol. 51, No. 4, pp. 168–176, 2013.CrossRefGoogle Scholar
  24. 24.
    T. Yucek and H. Arslan, A survey of spectrum sensing algorithms for cognitive radio applications, IEEE Communications Surveys and Tutorials, Vol. 11, No. 1, pp. 116–130, 2009.CrossRefGoogle Scholar
  25. 25.
    S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach3rd ed., Prentice Hall Press, Upper Saddle River, 2009.zbMATHGoogle Scholar
  26. 26.
    M. Bkassiny, Y. Li and S. K. Jayaweera, A survey on machine learning techniques in cognitive radios, IEEE Communications Surveys and Tutorials, Vol. 15, No. 3, pp. 1136–1159, 2013.CrossRefGoogle Scholar
  27. 27.
    J. Qadir, Artificial intelligence based cognitive routing for cognitive radio networks, Artificial Intelligence Review, Vol. 45, No. 1, pp. 25–96, 2016.CrossRefGoogle Scholar
  28. 28.
    I. Macaluso, D. Finn, B. Ozgul and L. A. DaSilva, Complexity of spectrum activity, IEEE Journal on Selected Areas in Communications, Vol. 31, No. 11, pp. 2237–2248, 2013.CrossRefGoogle Scholar
  29. 29.
    Richard S. Sutton and Andrew G. Barto, Introduction to Reinforcement Learning, vol. 1st, MIT Press, Cambridge, 1998.zbMATHGoogle Scholar
  30. 30.
    S. J. Gershman and N. D. Daw, Reinforcement learning and episodic memory in humans and animals: an integrative framework, Annual Review on Psychology, Vol. 68, pp. 101–128, 2017.CrossRefGoogle Scholar
  31. 31.
    W. Wang, A. Kwasinski, D. Niyato and Z. Han, A survey on model-free strategy learning in cognitive wireless network, IEEE Communications Surveys and Tutorials, Vol. 18, No. 3, pp. 1717–1757, 2016.CrossRefGoogle Scholar
  32. 32.
    H. A. A. Al-Rawi, M. A. Ng and K. A. Yau, Application of reinforcement learning to routing in distributed wireless networks: a review, Artificial Intelligence Review, Vol. 48, No. 3, pp. 381–416, 2015.CrossRefGoogle Scholar
  33. 33.
    O. Sigaud and O. Buffer, Markov Decision Processes in Artificial Intelligence, Wiley, Hoboken, 2012.Google Scholar
  34. 34.
    D. M. Roijers, P. Vamplew, S. Whilson and R. Dazeley, A survey of multi-objective sequential decision making, Journal of Artificial Intelligence Research, Vol. 48, pp. 67–113, 2013.CrossRefMathSciNetzbMATHGoogle Scholar
  35. 35.
    W. Wang. Multi-objective sequential decision making. Ph. D Thesis, 2014.Google Scholar
  36. 36.
    H. Li, Adaptive Learning in Cognitive Radio, Springer, Singapore, 2019. pp. 1083–1121.Google Scholar
  37. 37.
    B. Xia, M. H. Wahab, Y. Yang, Z. Fan, and M. Sooriyabandara. Reinforcement learning based spectrum aware routing in multi hop cognitive radio networks. In Proceedings of Fourth International Conference on CROWNCOM, Hanover, Germany, 2009.Google Scholar
  38. 38.
    M. D. Felice, K. R. Chowdhury, A. Kassler, W. Kim and L. Bononi, End-to-end protocols for cognitive radio ad hoc networks: an evaluation study, Performance Evaluation, Vol. 68, No. 9, pp. 859–875, 2011.CrossRefGoogle Scholar
  39. 39.
    C. Wu, K. Chowdhury, and M. Di Felice. Spectrum management of cognitive radio using multi-agent reinforcement learning. In Proceedings of 9th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2010), Toronto, Canada, pp. 1705–1712, May 2010.Google Scholar
  40. 40.
    S. Shamaee, M. E. Shiri, M. Ebrahim and M. Sabaei, A reinforcement learning based routing in cognitive radio networks for primary users with multi-stage periodicity, Wireless Personal Communications, Vol. 101, No. 1, pp. 465–490, 2018.CrossRefGoogle Scholar
  41. 41.
    Z. Yin, Y. Wang and C. Wu, Reinforcement learning spectrum management paradigm in cognitive radio using novel state and action sets, Procedia Computer Science, Vol. 129, pp. 433–437, 2018.CrossRefGoogle Scholar
  42. 42.
    C . R. Stevenson, G. Chouinard, Z. Lei, W. Hu, S. J. Shellhammer and W. Caldwell, IEEE 802.22: the first cognitive radio wireless regional area network standard, Communications Magazine, Vol. 47, No. 1, pp. 130–138, 2009.CrossRefGoogle Scholar
  43. 43.
    H. A. Al-Rawi and L. L. A. Yau. Route selection for minimizing interference to primary users in cognitive radio networks: a reinforcement learning approach. In Proceedings of Symposium on Computational Intelligence for Communication Systems and Networks, 2013.Google Scholar
  44. 44.
    H. A. A. Al-Rawi, K. A. Yau, H. Mphammad, N. Ramli and W. Hashim, Reinforcement learning for routing in cognitive radio ad hoc networks, The Scientific World Journal, Vol. 2014, p. 22, 2014.CrossRefGoogle Scholar
  45. 45.
    B. Pourpeighambar, M. Deghan and M. Sabaei, Multi-agent learning based routing for delay minimization in cognitive radio networks, Journal of Network and Computer Appilcations, Vol. 84, pp. 82–92, 2017.CrossRefGoogle Scholar
  46. 46.
    B. Pourpeighambar, M. Dehghan and M. Sabaei, Non-cooperative reinforcement learning based routing in cognitive radio networks, Computer Communications, Vol. 106, pp. 11–23, 2017.CrossRefGoogle Scholar
  47. 47.
    S. S. Barve and P. Kulkarni, Multi agent reinforcement learning based opportunistic routing in cognitive radio sensor network, Mobile Network Application, Vol. 19, No. 6, pp. 720–730, 2014.CrossRefGoogle Scholar
  48. 48.
    A. C. Talay and D. T. Altilar, Self adaptive routing for dynamic spectrum access in cognitive radio networks, Journal of Network and Computer Applications, Vol. 36, No. 4, pp. 1140–1151, 2013.CrossRefGoogle Scholar
  49. 49.
    Y. Saleem, K. A. Yau, H. Mohamad, N. Ramli, M. H. Rahmani and Q. Ni, Clustering and reinforcement-learning-based routing for cognitive radio networks, IEEE Wireless Communications, Vol. 24, No. 4, pp. 146–151, 2017.CrossRefGoogle Scholar
  50. 50.
    M. Musavi, K. Yau, A. R. Syed, H. Mohamad and N. B. Ramli, Route selection over clustered cognitive radio networks: an experimental evaluation, Computer Communications, Vol. 129, pp. 138–151, 2018.CrossRefGoogle Scholar
  51. 51.
    K. Zheng, H. Li, R. C. Qiu, and S. Gong. Multi objective reinforcement learning based routing in cognitive radio networks: Walking in a random maze. In Proceedings of International Conference on Computing, Networking and Communications, Maui, USA, 2012.Google Scholar
  52. 52.
    C. Messikh and N. Zarour. Towards a multi-objective reinforcement learning based routing protocol for cognitive radio networks. In 2018 International Conference on Smart Communications in Network Technologies (SaCoNeT), pp. 84–89, Oct 2018.Google Scholar
  53. 53.
    B. F. Lo, A survey of common control channel design in cognitive radio networks, Physical Communication, Vol. 4, pp. 26–39, 2011.CrossRefGoogle Scholar
  54. 54.
    H. A. A. Al-Rawi, K. A. Yau, H. Mohamad, N. Ramli, and W. Hashim. A reinforcement learning based routing scheme for cognitive radio ad hoc networks. In Proceedings of Seventh International Conference on Wireless and Mobile Networking Conference, Vilamoura, 2014.Google Scholar
  55. 55.
    C. Liu, X. Xu and D. Hu, Multiobjective reinforcement learning: a comprehensive overview, IEEE Transactions on Systems Man and Cybernetics Systems, Vol. 45, No. 3, pp. 385–398, 2015.CrossRefGoogle Scholar
  56. 56.
    Y. Wang, G. Zheng, H. Ma, Y. Li and J. Li, A joint channel selection and routing protocol for cognitive radio network, Wireless Communications and Mobile Computing, Vol. 2018, p. 7, 2018.Google Scholar
  57. 57.
    A. R. Syed, K. A. Yau, J. Qadir, H. Mohamad, N. B. Ramli and S. L. Keoh, Route selection for multi hop cognitive radio network using reinforcement learning: an experimental study, IEEE Access, Vol. 4, pp. 6304–6324, 2016.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Information and Communication TechnologyManipal Institute of Technology, Manipal Academy of Higher Education (MAHE)ManipalIndia
  2. 2.Department of Computer and Science EngineeringManipal Institute of Technology, Manipal Academy of Higher Education (MAHE)ManipalIndia
  3. 3.Department of Electronics and CommunicationManipal Institute of Technology, Manipal Academy of Higher Education (MAHE)ManipalIndia

Personalised recommendations