Skip to main content

Advertisement

SpringerLink
Log in

A multi-objective trade-off framework for cloud resource scheduling based on the Deep Q-network algorithm

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Cloud computing, now a mature computing service, provides an efficient and economical solution for processing big data. As such, it attracts a lot of attention in academia and plays an important role in industrial applications. With the recent increase in the scale of cloud computing data centers, and improvements in user service quality requirements, the structure of the whole cloud system has become more complex, which has also made the resource scheduling management of these systems more challenging. Thus, the goal of this research was to resolve the conflict between cloud service providers (CSPs) who aim to minimize energy costs and those who seek to optimize service quality. Based on the excellent environmental awareness and online adaptive decision-making ability of deep reinforcement learning (DRL), we proposed an online resource scheduling framework based on the Deep Q-network (DQN) algorithm. The framework could make a trade-off of the two optimization objectives of energy consumption and task makespan by adjusting the proportion of the reward of different optimization objectives. Experimental results showed that this framework could effectively be used to make a trade-off of the two optimization objectives of energy consumption and task makespan, and exhibited obvious optimization effects compared with the baseline algorithm. Therefore, our proposed framework can dynamically adjust the optimization objective of the system according to the different requirements of the cloud system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Abbreviations

CSP:

Cloud service provider

DRL:

Deep reinforcement learning

RL:

Reinforcement learning

DQN:

Deep Q-network

QoS:

Quality of service

MoPSO:

Multi-objective particle swarm optimization

GA:

Genetic algorithm

ACO:

Ant colony optimization

MOO:

Multi-objective optimization

VM:

Virtual machine

AWT:

Average waiting time

SLA:

Server level agreement

ETE:

End-to-end

DAG:

Directed acyclic graphs

RR:

Round-robin

MEC:

Mobile edge computing

References

  1. Lin, W.W., Qi, D.Y., et al.: Review of cloud computing resource scheduling. Comput. Sci. 39(10), 1–6 (2012)

    Google Scholar 

  2. Ramezani, F., Lu, J., Hussain, F.: Task scheduling optimization in cloud computing applying multi-objective particle swarm optimization. In: Service-Oriented Computing. ICSOC 2013. Lecture Notes in Computer Science, vol. 8274, pp. 237–251. Springer (2013)

  3. Liu, J., Luo, X.G., Zhang, X.M., et al.: Job scheduling algorithm for cloud computing based on particle swarm optimization. Adv. Mater. Res. 662, 957–960 (2013)

    Article  Google Scholar 

  4. Alkayal, E.S., Jennings, N.R., Abulkhair, M.F.: Efficient task scheduling multi-objective particle swarm optimization in cloud computing. In: 2016 IEEE 41st Conference on Local Computer Networks Workshops (LCN Workshops), pp. 17–24 (2016)

  5. Verma, A., Kaushal, S.: A hybrid multi-objective particle swarm optimization for scientific workflow scheduling. Parallel Comput. 62, 1–19 (2017)

    Article  MathSciNet  Google Scholar 

  6. Agarwal, Mohit, Srivastava, G.M.S.: Genetic algorithm enabled particle swarm optimization (PSOGA) based task scheduling in cloud computing environment. Int. J. Inf. Technol. Decis. Mak. 17(3), 1237–1267 (2018)

    Article  Google Scholar 

  7. Barrett, E., Howley, E., Duggan, J.: Applying reinforcement learning towards automating resource allocation and application scalability in the cloud. Concurr. Comput. 25(12), 1656–1674 (2013)

    Article  Google Scholar 

  8. Farahnakian, F., Liljeberg, P., Plosila, J.: Energy-efficient virtual machines consolidation in cloud data centers using reinforcement learning. In: 22nd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, pp. 500–507 (2014)

  9. Cui, D., Ke, W., Peng, Z., Zuo, J.: Multiple DAGs workflow scheduling algorithm based on reinforcement learning in cloud computing. In: Computational Intelligence and Intelligent Systems. ISICA 2015. Communications in Computer and Information Science, vol. 575, pp. 305–311 (2016)

  10. Sharma, A.R., Kaushik, P., et al.: Deep and reinforcement learning in natural language processing. In: Proceedings of the 2017 International Conference on Computing, Communication and Automation (ICCCA), pp. 350–354 (2017)

  11. Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Playing atari with deep reinforcement learning. In: Proceedings of Workshops at the 26th Neural Information Processing Systems. Computer Science, pp. 201–220 (2013)

  12. Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)

    Article  Google Scholar 

  13. Phaniteja, S., Dewangan P.: A deep reinforcement learning approach for dynamically stable inverse kinematics of humanoid robots. In: Proceedings of the 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 1818–1823 (2017)

  14. Bitsakos, C., Konstantinou, I., et al.: DERP: A deep reinforcement learning cloud system for elastic resource provisioning. In: 2018 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), vol. 1, pp. 21–29 (2018)

  15. Huang, L., Feng, X., Zhang, C., et al.: Deep reinforcement learning-based joint task offloading and bandwidth allocation for multi-user mobile edge computing. Digit. Commun. Netw. 5(1), 10–17 (2019)

    Article  Google Scholar 

  16. Huang, L., Feng, X., Feng, A., et al.: Distributed deep learning-based offloading for mobile edge computing networks. Mobile Networks and Applications, pp. 1–8 (2018)

  17. Huang, L., Feng, X., et al.: Multi-server multi-user multi-task computation offloading for mobile edge computing networks. Sensors 19(6), 1446 (2019)

    Article  Google Scholar 

  18. Tsai, C.W., Huang, W.C., Chiang, M.H., et al.: A hyper-heuristic scheduling algorithm for cloud. IEEE Trans. Cloud Comput. 2(2), 236–250 (2014)

    Article  Google Scholar 

  19. Salza, P., Ferrucci, F., Sarro, F.: Develop, deploy and execute parallel genetic algorithms in the cloud. In: Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion, ACM, pp. 121–122 (2016)

  20. Li, H.H., Chen, Z.G., Zhan, Z.H., et al.: Renumber coevolutionary multiswarm particle swarm optimization for multi-objective workflow scheduling on cloud computing environment. In: Companion Publication of the Conference on Genetic & Evolutionary Computation, ACM, pp. 1419–1420 (2015)

  21. Yu, Z., Fang, L.I., Tao, Z., et al.: Task scheduling algorithm based on genetic ant colony algorithm in cloud computing environment. Comput. Eng. Appl. 50(6), 51–55 (2014)

    Google Scholar 

  22. Zhang, H., Li, P., Zhou, Z., Yu, X.: A PSO-based hierarchical resource scheduling strategy on cloud computing. In: International Conference on Trustworthy Computing and Services, pp. 325–332. Springer (2013)

  23. Xue, S., Li, M., Xu, X., Chen, J., Xue, S.: An ACO-LB algorithm for task scheduling in the cloud environment. J. Softw. 9(2), 466–473 (2014)

    Google Scholar 

  24. Gao, Y., Guan, H., Qi, Z., et al.: A multi-objective ant colony system algorithm for virtual machine placement in cloud computing. J. Comput. Syst. Sci. 79(8), 1230–1242 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  25. Zuo, L., Shu, L., Dong, S., et al.: A multi-objective optimization scheduling method based on the ant colony algorithm in cloud computing. IEEE Access 3, 2687–2699 (2015)

    Article  Google Scholar 

  26. Chen, S., Wu, J., Lu, Z.: A cloud computing resource scheduling policy based on genetic algorithm with multiple fitness. In: IEEE International Conference on Computer & Information Technology, pp. 177–184 (2012)

  27. Duan, H., Chen, C., Min, G., Wu, Y.: Energy-aware scheduling of virtual machines in heterogeneous cloud computing systems. Future Gener. Comput. Syst. 74, 142–150 (2017)

    Article  Google Scholar 

  28. Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: a survey. Int. J. Robot. Res. 32(11), 1238–1274 (2013)

    Article  Google Scholar 

  29. Peng, Z., Cui, D., Zuo, J., et al.: Random task scheduling scheme based on reinforcement learning in cloud computing. Clust. Comput. 18(4), 1595–1607 (2015)

    Article  Google Scholar 

  30. Peng, Z., Cui, D., Zuo, J., et al.: Research on cloud computing resources provisioning based on reinforcement learning. Math. Probl. Eng. 9, 1–12 (2015)

    Google Scholar 

  31. Liu, J.Z., Zhang, Y.X., Zhou, Y.Z., et al.: Aggressive resource provisioning for ensuring QoS in virtualized environments. IEEE Trans. Cloud Comput. 3(2), 119–131 (2015)

    Article  Google Scholar 

  32. Peng, Z., Cui, D., Ma, Y., et al.: A reinforcement learning-based mixed job scheduler scheme for cloud computing under SLA constraint. In: 2016 IEEE 3rd International Conference on Cyber Security and Cloud Computing (CSCloud), pp. 142–147 (2016)

  33. Liu, Q., Zhan, J.W., Zhang, Z.Z., et al.: A survey on deep reinforcement learning. Chin. J. Comput. 41(1), 1–27 (2018)

    Google Scholar 

  34. Mao, H., Alizadeh, M., Menache, I., et al.: Resource management with deep reinforcement learning. In: ACM Workshop on Hot Topics in Networks. ACM, pp. 50–56 (2016)

  35. Lin, J.P., Peng, Z.Z., Cui, D.D.: Deep reinforcement learning for multi-resource cloud job scheduling. In: Neural Information Processing. ICONIP 2018. Lecture Notes in Computer Science, Springer, vol. 11303, pp. 289–302 (2018)

  36. Liu, N., Li, Z., Xu, Z., et al.: A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning. In: 37th IEEE International Conference on Distributed Computing (ICDCS 2017), Computer Science, pp. 34–56 (2017)

  37. Cheng, M., Li, J., Nazarian, S.: Deep reinforcement learning-based resource provisioning and task scheduling for cloud service providers. In: 2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 129–134 (2018)

  38. Quan, L., Wang, Z., Ren, F.: A novel two-layered reinforcement learning for task offloading with tradeoff between physical machine utilization rate and delay. Future Internet 10, 60 (2018)

    Article  Google Scholar 

  39. Gao, Y., Wang, Y., Gupta, S.K., Pedram, M.: An energy and deadline aware resource provisioning, scheduling and optimization framework for cloud systems. In: Proceedings of the Ninth IEEE/ACM/IFIP International Conference on Hardware/Software Codesign and System Synthesis, pp. 1–10. IEEE Press (2013)

  40. Pedram, M.: Energy-efficient datacenters. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 31, 1465–1484 (2012)

    Article  Google Scholar 

  41. Google cluster data. [Online]. https://github.com/google/cluster-data

Download references

Acknowledgements

The work presented in this paper was supported by National Natural Science Foundation of China (61772145, 61672174). Jianpeng Lin and Delong Cui are corresponding authors.

Author information

Authors and Affiliations

  1. College of Computer and Electronic Information, Guangdong University of Petrochemical Technology, Maoming, China

    Zhiping Peng, Jianpeng Lin, Delong Cui, Qirui Li & Jieguang He

Authors
  1. Zhiping Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianpeng Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Delong Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qirui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jieguang He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jianpeng Lin or Delong Cui.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Peng, Z., Lin, J., Cui, D. et al. A multi-objective trade-off framework for cloud resource scheduling based on the Deep Q-network algorithm. Cluster Comput 23, 2753–2767 (2020). https://doi.org/10.1007/s10586-019-03042-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-019-03042-9

Keywords

Search

Navigation