Skip to main content
SpringerLink
Log in
Book cover

International Conference on Green, Pervasive, and Cloud Computing

GPC 2017: Green, Pervasive, and Cloud Computing pp 460–474Cite as

A Profit-Maximum Resource Allocation Approach for Mapreduce in Data Centers

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 10232))

Abstract

Resource allocation for Mapreduce data processing poses difficult challenges to system administrators in data centers. The extreme scale of Mapreduce applications require an efficiently profitable resource allocation algorithm that minimizes the energy consumption cost while maintaining the highest level of performance. In this paper, we propose a profit-maximum model that minimizes the cost of energy consumption and makespan. By adopting a minimum-weight b-matching rounding algorithm (MBRA) to find an integer solution, then assign map/reduce tasks to individual slots to build a complete resource allocation. Finally, we perform experiments on real workload to evaluate the profit-maximum model and analyze the performance of our proposed algorithm. The results show that MBRA is able to find a near-optimal integer solution that maximizes the profit per unit time in a lower runtime, and it is up to 30%~70% in profit that is better than the current heuristic scheduling algorithm and the rounding algorithm.

This is a preview of subscription content, log in via an institution.

References

  1. Moseley, B., Dasgupta, A., Kumar, R., Sarlos, T.: On scheduling in map-reduce and flow-shops. In: Proceedings of the 23rd Annual ACM Symposium on Parallelism in Algorithms and Architectures, pp. 289–298 (2011)

    Google Scholar 

  2. Tang, S., Lee, B.S., He, B.: Dynamic job ordering and slot configurations for mapreduce workloads. IEEE Trans. Serv. Comput. 9(1), 4–17 (2016)

    Article  Google Scholar 

  3. Zhu, Y., Jiang, Y., Wu, W., Ding, L.: Minimizing makespan and total completion time in mapreduce-like systems. In: Proceedings of the 33rd Annual IEEE International Conference on Computer Communications (INFOCOM 2014), pp. 2166–2174 (2014)

    Google Scholar 

  4. Kolb, L., Thor, A., Rahm, E.: Load balancing for mapreduce-based entity resolution. In: Proceedings of the 28th IEEE International Conference on Data Engineering, pp. 618–629 (2012)

    Google Scholar 

  5. Wang, W., Zhu, K., Ying, L., Tan, J., Zhang, L.: MapTask scheduling in mapreduce with data locality: throughput and heavy-traffic optimality. IEEE/ACM Trans. Netw. 24(1), 190–203 (2016)

    Article  Google Scholar 

  6. Wang, X., Wang, Y., Zhu, H.: Energy-efficient task scheduling model based on MapReduce for cloud computing using genetic algorithm. J. Comput. 7(12), 2962–2970 (2012)

    Article  Google Scholar 

  7. Mashayekhy, L., Nejad, M.M., Grosu, D., Zhang, Q., Shi, W.: Energy-aware scheduling of mapreduce jobs for big data applications. IEEE Trans. Parallel Distrib. Syst. 26(10), 2720–2733 (2015)

    Article  Google Scholar 

  8. Tarplee, K.M., Maciejewski, A.A., Siegel, H.J.: Energy-aware profit maximizing scheduling algorithm for heterogeneous computing systems. In: Proceedings of the 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2014), pp. 595–603 (2014)

    Google Scholar 

  9. Ren, Z.J., Wan, J., Shi, W.S., Xu, X.H., Zhou, M.: Workload analysis, implications, and optimization on a production hadoop cluster: a case study on taobao. IEEE Trans. Serv. Comput. 7(2), 307–321 (2014)

    Article  Google Scholar 

  10. Verma, A., Cherkasova, L., Campbell, R.H.: Two sides of a coin: optimizing the schedule of mapreduce jobs to minimize their makespan and improve cluster performance. In: Proceedings of the 20th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, pp. 11–18 (2012)

    Google Scholar 

  11. Song, J., Wang, Z., Li, T.T., Yu, G.: Energy consumption optimization data placement algorithm for mapreduce system. J. softw. 26(8), 2091–2110 (2015)

    MathSciNet  Google Scholar 

  12. Lin, B., Li, S.S., Liao, X.K., Meng, L.B., Liu, X.D., Huang, H.: Seadown: SLA-aware size-scaling power management in heterogeneous mapreduce cluster. J. Comput. 36(5), 977–987 (2013)

    Google Scholar 

  13. Maheshwari, N., Nanduri, R., Varma, V.: Dynamic energy efficient data placement and cluster reconfiguration algorithm for mapreduce framework. Future Gener. Comput. Syst. 28(1), 119–127 (2012)

    Article  Google Scholar 

  14. Lin, J.C., Leu, F.Y., Chen, Y.: Impact of mapreduce policies on job completion reliability and job energy consumption. IEEE Trans. Parallel Distrib. Syst. 26(5), 1364–1378 (2015)

    Article  Google Scholar 

  15. Tian, F., Chen, K.: Towards optimal resource provisioning for running mapreduce programs in public clouds. In: Proceedings of 2011 IEEE International Conference on Cloud Computing (CLOUD 2011), pp. 155–162 (2011)

    Google Scholar 

  16. Palanisamy, B., Singh, A., Liu, L.: Cost-effective resource provisioning for mapreduce in a cloud. IEEE Trans. Parallel Distrb. Syst. 26(5), 1265–1279 (2015)

    Article  Google Scholar 

  17. Young, B.D., Apodaca, J., Briceño, L.D., Smith, J., Pasricha, S., Maciejewski, A.A., Siegel, H.J., Khemka, B., Bahirat, S., Ramirez, A., Zou, Y.: Deadline and energy constrained dynamic resource allocation in a heterogeneous computing environment. J. Supercomput. 63(2), 326–347 (2013)

    Article  Google Scholar 

  18. Li, W.D., Liu, X., Zhang, X.J., Cai, X.B.: A Task-type-based algorithm for the energy-aware profit maximizing scheduling problem in heterogeneous computing systems. In: Proceedings of the 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2015), pp. 1107–1110 (2015)

    Google Scholar 

  19. Jansen, K., Porkolab, L.: Improved approximation schemes for scheduling unrelated parallel machines. Math. Oper. Res. 26(2), 324–338 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  20. Shmoys, D.B., Tardos, É.: An approximation algorithm for the generalized assignment problem. Math. Prog. 62(1–3), 461–474 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  21. Huang, B.C., Jebara, T.: Fast b-matching via sufficient selection belief propagation. In: Proceedings of AISTATS, pp. 361–369 (2011)

    Google Scholar 

Download references

Acknowledgments

This paper was supported by the National Natural Science Foundation of China (Nos. 61170222, 61662088, 11301466), the Natural Science Foundation of Yunnan Province of China (No. 2014FB114), and the Scientific Research Foundation of the Educational Department of Yunnan Province (No. 2015J0007).

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Yunnan University, Kunming, China

    Xiaolu Zhang, Weidong Li, Xi Liu & Xuejie Zhang

Authors
  1. Xiaolu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weidong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xuejie Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xuejie Zhang .

Editor information

Editors and Affiliations

  1. Hong Kong Polytechnic University, Hong Kong, Hong Kong

    Man Ho Allen Au

  2. University of Salerno, Fisciano, Italy

    Arcangelo Castiglione

  3. University of Texas at San Antonio, San Antonio, Texas, USA

    Kim-Kwang Raymond Choo

  4. University of Salerno, Fisciano, Italy

    Francesco Palmieri

  5. Providence University, Taichung City, Taiwan

    Kuan-Ching Li

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Zhang, X., Li, W., Liu, X., Zhang, X. (2017). A Profit-Maximum Resource Allocation Approach for Mapreduce in Data Centers. In: Au, M., Castiglione, A., Choo, KK., Palmieri, F., Li, KC. (eds) Green, Pervasive, and Cloud Computing. GPC 2017. Lecture Notes in Computer Science(), vol 10232. Springer, Cham. https://doi.org/10.1007/978-3-319-57186-7_34

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-57186-7_34

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-57185-0

  • Online ISBN: 978-3-319-57186-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation