Online Learning and Optimization for Computation Offloading in D2D Edge Computing and Networks

  • Guanhua Qiao
  • Supeng Leng
  • Yan Zhang


This paper introduces a framework of device-to-device edge computing and networks (D2D-ECN), a new paradigm for computation offloading and data processing with a group of resource-rich devices towards collaborative optimization between communication and computation. However, the computation process of task intensive applications would be interrupted when capacity-limited battery energy run out. In order to tackle this issue, the D2D-ECN with energy harvesting technology is applied to provide a green computation network and guarantee service continuity. Specifically, we design a reinforcement learning framework in a point-to-point offloading system to overcome challenges of the dynamic nature and uncertainty of renewable energy, channel state and task generation rates. Furthermore, to cope with high-dimensionality and continuous-valued action of the offloading system with multiple cooperating devices, we propose an online approach based on Lyapunov optimization for computation offloading and resource management without priori energy and network information. Numerical results demonstrate that our proposed scheme can reduce system operation cost with low task execution time in D2D-ECN.


D2D-ECN Energy harvesting Computation offloading Resource management Reinforcement learning Lyapunpv optimization 



This work is supported by the joint fund of the Ministry of Education of China and China Mobile (MCM 20160304), the Fundamental Research Funds for the Central Universities, China (ZYGX2016Z011), and EU H2020 Project COSAFE (MSCA-RISE-2018-824019).


  1. 1.
    Gubbi J, Buyya R, Marusic S, Palaniswami M (2013) Internet of things (IoT): a vision, architectural elements, and future directions. Future Generat Comput Syst 29(7):1645–1660CrossRefGoogle Scholar
  2. 2.
    ETSI ISG (2015) Mobile edge computing a key technology towards 5G. White Paper 11:1–16Google Scholar
  3. 3.
    Pu L, Chen C, Xu J, Fu X (2016) D2D fogging: an energy-efficient and incentive-aware task offloading framework via network-assisted D2D collaboration. IEEE J Sel Areas Commun 34(12):3887–3901CrossRefGoogle Scholar
  4. 4.
    Liu F et al (2013) Gearing resource-poor mobile devices with powerful clouds: architectures, challenges, and applications. IEEE Wireless Commun 20(3):14–22CrossRefGoogle Scholar
  5. 5.
    Zhang K et al (2016) Energy-ffficient offloading for mobile edge computing in 5G heterogeneous networks. IEEE Access 4:5896–5907CrossRefGoogle Scholar
  6. 6.
    Shih Y-Y, Chung W-H, Pang A-C, Chiu T-C, Wei H-Y (2017) Enabling low-latency applications in fog-radio access networks. IEEE Netw 31(1):52–58CrossRefGoogle Scholar
  7. 7.
    Zhang K, Mao Y, Leng S, He Y, Zhang Y (2017) Mobile-edge computing for vehicular networks: a promising network paradigm with predictive off-loading. IEEE Vehi Tech Maga 12(2):36–44CrossRefGoogle Scholar
  8. 8.
    Wang X, Leng S, Yang K (2017) Social-aware edge caching in fog radio access networks. IEEE Access 5:8492–8501CrossRefGoogle Scholar
  9. 9.
    Chen X, Pu L, Gao L, Wu W, Wu D (2017) Exploiting massive D2D collaboration for energy-efficient mobile edge computing. IEEE Commun Mag 24(4):64–71Google Scholar
  10. 10.
    Ti N, Le L (2017) Computation offloading leveraging computing resources from edge cloud and mobile peers. In: Proceedings of the IEEE Int. Commun. Confe (ICC)Google Scholar
  11. 11.
    Meng X, Wang W, Zhang Z (2017) Delay-constrained hybrid computation offloading with cloud and fog computing. IEEE Access 5:21355–21367CrossRefGoogle Scholar
  12. 12.
    Garlatova M, Wallwater A, Zussman G (2011) Networking low-power energy harvesting devices: measurements and algorithms. In: Proceedings of the IEEE Conf. Comput. Commun. (INFOCOM), pp 1602–1610Google Scholar
  13. 13.
    Dhillon H, Li Y, Nuggehalli P, Pi Z, Andrews J (2014) Fundamentals of heterogeneous cellular networks with energy harvesting. IEEE Trans Wireless Commun 13(5):2782–2797CrossRefGoogle Scholar
  14. 14.
    Mao Y, Zhang J, Letaief KB (2016) Dynamic computation offloading for Mobile-Edge computing with energy harvesting devices. IEEE J Sel Areas Commun 34(12):3590–3605CrossRefGoogle Scholar
  15. 15.
    Fan B, Leng S, Yang K (2016) A dynamic bandwidth allocation algorithm in mobile networks with big data of users and networks. IEEE Netw Maga 30(1):6–10CrossRefGoogle Scholar
  16. 16.
    Mastronarde N, Schaar M (2011) Fast reinforcement learning for energy-efficient wireless communication. IEEE Trans Sig Process 50(12):6262–6266MathSciNetCrossRefGoogle Scholar
  17. 17.
    Wei Y, Yu FR, Song M, Han Z (2017) User scheduling and resource allocation in HetNets with hybrid energy supply: an actor-critic reinforcement learning approach. IEEE Trans Wire Commun 17(1):680–692CrossRefGoogle Scholar
  18. 18.
    Xu J, Chen L, Ren S (2017) Online learning for offloading and autoscaling in energy harvesting mobile edge computing. IEEE Trans Cogn Commun Netw 3(3):361–373CrossRefGoogle Scholar
  19. 19.
    Mao Y, You C, Zhang J, Huang K, Letaief KB (2017) A servey on mobile edge computing: the communication perspective. IEEE Commun Survey Tuts 19(4):2322–2358CrossRefGoogle Scholar
  20. 20.
    Miettinen AP, Nurminen JK (2010) Energy efficiency of mobile clients in cloud computing. In: Proceedings of the USENIX Conf. Hot Topics Cloud Comput. (HotCloud), Boston, MA, USA, pp 1–7Google Scholar
  21. 21.
    Wang Q, Leng S, Fu H, Zhang Y (2012) An IEEE 802.11p-based multichannle MAC scheme with channel coordination for vehicular Ad Hoc networks. IEEE Trans Intell Trans Sys 13(2):449–458CrossRefGoogle Scholar
  22. 22.
    Rajan D, Sabharwal A, Aazhang B (2004) Delay-bounded packet scheduling of bursty traffic over wireless channels. IEEE Trans Inform Theory 50(1):125–144MathSciNetCrossRefGoogle Scholar
  23. 23.
    Burd TD, Brodersen RW (1996) Processor design for portable systems. Kluwer J VLSI Signal Process Syst 13(2):203–221CrossRefGoogle Scholar
  24. 24.
    Altman E (1999) Constrained Markov Decision Process Chapman and Hall/CRCGoogle Scholar
  25. 25.
    Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, CambridgeGoogle Scholar
  26. 26.
    Salodkar N, Bhorkar A, Karandikar A, Borkar VS (2008) An on-line learning algorithm for energy efficient delay constrained scheduling over a fading channel. IEEE J Sel Areas Commun 26(4):732–742CrossRefGoogle Scholar
  27. 27.
    Neely MJ (2010) Stochastic network optimization with application to communication and queueing systems, San rafael, CA USA: Morgan claypoolGoogle Scholar
  28. 28.
    Gibilisco P, Hiai F, Petz D (2009) Quantum covariance, quantum fisher information, and the uncertainty relations. IEEE Trans Inform Theory 55(1):439–443MathSciNetCrossRefGoogle Scholar
  29. 29.
    Gorlatova M, Wallwater A, Zussma G (2012) Networking low-power energy harvesting devices: Measurements and algorithms. IEEE Trans Mobile Comput 12(9):1853–1865CrossRefGoogle Scholar
  30. 30.
    Mitchell TM (1997) Machine learning McGraw-HillGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of Information & Communication EngineeringUniversity of Electronic Science and Technology of ChinaChengduChina
  2. 2.Department of InformaticsUniversity of OsloOsloNorway

Personalised recommendations