Advertisement

PDRM: A Probability Distribution Based Resource Management for Batch Workloads in Heterogeneous Cluster

  • Jun Zhou
  • Dan FengEmail author
  • Fang Wang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11783)

Abstract

Resource consumption prediction and dynamic resource provision based on historical consumption are common methods to improve cluster resource utilization, however they have to face the challenge of fluctuation in resource consumption for accurate prediction. We propose PDRM, an efficient resource management scheme based on resource consumption probability distribution for batch workloads to deal with this dilemma. Based on the common sense that the same type of tasks have similar resource consumption on the same node, we get the resource consumption probability distribution of each type of task to describe the fluctuations in its resource consumption. Based on the resource consumption distribution function, we can allocate resources precisely for tasks. Experimental results demonstrate that PDRM achieves good performance for various application in the heterogeneous cluster. PDRM can effectively improve resource utilization and reduce job completion time.

Keywords

Resource management Big data Gaussian distribution Heterogeneous 

References

  1. 1.
    Alipourfard, O., Liu, H.H., Chen, J., Venkataraman, S., Yu, M., Zhang, M.: CherryPick: adaptively unearthing the best cloud configurations for big data analytics. In: 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2017), pp. 469–482. USENIX Association, Boston (2017)Google Scholar
  2. 2.
    Cai, L., Qi, Y., Wei, W., Wu, J., Li, J.: mrMoulder: a recommendation-based adaptive parameter tuning approach for big data processing platform. Future Gener. Comput. Syst. 93(1), 570–582 (2019)CrossRefGoogle Scholar
  3. 3.
    Delimitrou, C., Kozyrakis, C.: Quasar: resource-efficient and QoS-aware cluster management. In: Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 127–144. ACM, New York (2014)Google Scholar
  4. 4.
    Mohan, A., Kaseb, A.S., Lu, Y., Hacker, T.: Adaptive resource management for analyzing video streams from globally distributed network cameras. IEEE Trans. Cloud Comput. 1 (2018) Google Scholar
  5. 5.
    Reiss, C., Tumanov, A., Ganger, G.R., Katz, R.H., Kozuch, M.A.: Heterogeneity and dynamicity of clouds at scale: Google trace analysis. In: Proceedings of the Third ACM Symposium on Cloud Computing, pp. 7:1–7:13. ACM, New York (2012)Google Scholar
  6. 6.
    Zhang, Y., Prekas, G., Fumarola, G.M., Fontoura, M., Goiri, I.n., Bianchini, R.: History-based harvesting of spare cycles and storage in large-scale datacenters. In: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, pp. 755–770. USENIX Association, Berkeley (2016)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2019

Authors and Affiliations

  1. 1.School of Computer Science and TechnologyHuazhong University of Science and TechnologyWuhanChina
  2. 2.Wuhan National Laboratory for OptoelectronicsWuhanChina
  3. 3.Key Laboratory of Information Storage System, Engineering Research Center of data storage systems and TechnologyMinistry of Education of ChinaWuhanChina

Personalised recommendations