Abstract
A Hadoop MapReduce cluster is an environment where multi-users, multi-jobs and multi-tasks share the same physical resources. Because of the competitive relationship among the jobs, we need to select the most suitable job to be sent to the cluster. In this paper we consider this problem as a two-level scheduling problem based on a detailed cost model. Then we abstract these scheduling problems into two games. And we solve these games in using some methods of game theory to achieve the solution. Our strategy improves the utilization efficiency of each type of the resources. And it can also avoid the unnecessary transmission of data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: Proceedings of the Sixth Symposium on Operating System Design and Implementation, Usenix Association, San Francisco, 6–8 Dec 2004
Apache Hadoop. http://hadoop.apache.org
Jiang, D., Ooi, B.C., et al.: The performance of MapReduce: an in-depth study. Proc. VLDB Endow. 3(1), 494–505 (2010)
Capacity Scheduler. http://hadoop.apache.org/common/docs/r0.20.2/capacity_scheduler.html
Fair Scheduler. http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html
Zaharia, M., Borthakur, D., et al.: Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In: Proceedings of the EuroSys’ 10 (2010)
Zaharia, M., Borthakur, D. et al.: Job scheduling for multi-user MapReduce clusters. Technical Report, EECS Department, University of California, Berkeley (2009)
He, C., Lu, Y., et al.: Matchmaking: a new MapReduce scheduling technique. In: Proceedings of the CloudCom ’11 (2011)
Verma, A., Cherkasova, L., et al.: ARIA: automatic resource inference and allocation for MapReduce Environments. In: Proceedings of the ICAC’ 11 (2011)
Polo, J., Carera, D., et al.: Performance-driven task co-scheduling for MapReduce environments. In: Proceedings of the NOMS’ (2010)
Fischer, M.J., Su, X., et al.: Assigning tasks for efficiency in Hadoop. In: Proceedings of the SPAA ’10 (2010)
Lin, X., Meng, Z., et al.: A practical performance model for Hadoop MapReduce, Cluster Computing Workshops (CLUSTER WORKSHOPS), IEEE International Conference (2012)
Khan, S.U., Ahmad, I.: Non-cooperative, semi-cooperative, and cooperative games based grid resource allocation. In: Parallel and Distributed Processing Symposium, pp. 101 (2006)
Kuhn, H.W.: The Hungarian method for the assignment problem. Bryn Mawr College, Pennsylvania
Acknowledgments
This work was supported by the National High Technology Research and Development Program of China (No. 2011AA010502) and the National Science and Technology Pillar Program (2012BAH07B01)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer Science+Business Media New York
About this paper
Cite this paper
Song, G., Yu, L., Meng, Z., Lin, X. (2013). A Game Theory Based MapReduce Scheduling Algorithm. In: Wong, W.E., Ma, T. (eds) Emerging Technologies for Information Systems, Computing, and Management. Lecture Notes in Electrical Engineering, vol 236. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-7010-6_33
Download citation
DOI: https://doi.org/10.1007/978-1-4614-7010-6_33
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-7009-0
Online ISBN: 978-1-4614-7010-6
eBook Packages: EngineeringEngineering (R0)