Meeting Dynamic User Demand with Transmission Cost Awareness: CT-MAB RL Based Network Selection

  • Zhiyong DuEmail author
  • Bin Jiang
  • Qihui Wu
  • Yuhua Xu
  • Kun Xu


The access and transmission in wireless networks not only satisfy user demand for service delivery, but also incur cost in terms of fee and energy consumption, which poses the concern of cost-performance ratio. This chapter studies the cost-performance ratio optimization with dynamic traffic types in dynamic and uncertain HWN. To balance the QoE reward and transmission cost for different traffic types on the fly, the problem is formulated as a continuous-time multi-armed bandit (CT-MAB) model. A traffic-aware online network selection algorithm (ONES) is designed to match typical traffic types (user demand) with respective optimal networks in terms of QoE. In addition, we exploit the correlation feature among multiple traffic types to improve the learning capability, which inspires us to propose another two efficient algorithms: decoupled online network selection algorithm (D-ONES) and virtual multiplexing ONES (VM-ONES). Simulation results demonstrate that our online network selection algorithms achieve better QoE reward rate over non- learning-based algorithms and learning-based algorithms without QoE considerations.


  1. 1.
    Trestian R, Ormond O, Muntean G (2012) Game theory-based network selection: solutions and challenges. IEEE Commun Surv Tut 2(99):1–20Google Scholar
  2. 2.
    Hou J, Brien DCO (2006) Vertical handover-decision-making algorithm using fuzzy logic for the integrated Radio-and-OW system. IEEE Trans Wirel Commun 5(1):176–185CrossRefGoogle Scholar
  3. 3.
    Martinez-Morales JD, Pineda-Ricoand U, Stevens-Navarro E (2010) Performance comparison between MADM algorithms for vertical handoff in 4G networks. In: Proceedings of the 7th international conference on electrical engineering computing science and automatic control (CCE)Google Scholar
  4. 4.
    Piamrat K, Ksentini A, Viho C et al (2008) QoE-based network selection for multimedia users in IEEE 802.11 wireless networks. In: IEEE local computer networks (LCN)Google Scholar
  5. 5.
    Stevens-Navarro E, Lin Y, Wong VWS (2008) An MDP-based vertical handoff decision algorithm for heterogeneous wireless networks. IEEE Trans Veh Technol 57(2):1243–1254CrossRefGoogle Scholar
  6. 6.
    György A, Kocsis L et al (2007) Continuous time associative bandit problems. In: Proceedings of the 20th international joint conference on artificial intelligence (IJCAI)Google Scholar
  7. 7.
    Wu Q, Du Z, Yang P, Yao Y, Wang J (2016) Traffic-aware online network selection in heterogeneous wireless networks. IEEE Trans Veh Technol 65(1):381–397CrossRefGoogle Scholar
  8. 8.
    Haddad M, Elayoubi SE, Altman E et al (2011) A hybrid approach for radio resource management in heterogeneous cognitive networks. IEEE J Sel Areas Commun 29(4):831–842CrossRefGoogle Scholar
  9. 9.
    Reis AB, Chakareski J, Kassler A et al (2010) Distortion optimized multi-service scheduling for next-generation wireless mesh networks. In: IEEE INFOCOMGoogle Scholar
  10. 10.
    Manzoor A, Umar T (2011) User utility function as quality of experience (QoE). In: The 10th international conference on networks (ICN)Google Scholar
  11. 11.
    Ferguson TS (2008) Optimal stopping and applications.
  12. 12.
    Auer P, Cesa-Bianchi N, Fischer P (2002) Finite-time analysis of the multiarmed bandit problem. Mach Learn 47:235–256CrossRefGoogle Scholar
  13. 13.
    Anandkumar A, Michael N et al (2011) Distributed algorithms for learning and cognitive medium access with logarithmic regret. IEEE J Sel Areas Commun 29(4):731–745CrossRefGoogle Scholar
  14. 14.
    ITU-T Recommendation G.107 (1998) The E-model: a computational model for use in transmission planning.
  15. 15.
    Sengupta S, Chatterjee M, Ganguly S (2008) Improving quality of VoIP streams over WiMax. IEEE Trans Mobile Comput 57(2):145–156MathSciNetCrossRefGoogle Scholar
  16. 16.
    Kelly FP (1997) Charging and rate control for elastic traffic. Eur Trans Telecommun 8:33–37CrossRefGoogle Scholar
  17. 17.
    Raychaudhuri D, Mandayam NB (2012) Frontiers of wireless and mobile communications. Proc IEEE 100(4):824–840CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  • Zhiyong Du
    • 1
    Email author
  • Bin Jiang
    • 1
  • Qihui Wu
    • 2
  • Yuhua Xu
    • 3
  • Kun Xu
    • 1
  1. 1.National University of Defense TechnologyChangshaChina
  2. 2.Nanjing University of Aeronautics and AstronauticsNanjingChina
  3. 3.Army Engineering University of PLANanjingChina

Personalised recommendations