Abstract
In this paper, the task scheduling in MapReduce is considered for geo-distributed data centers on heterogeneous networks. Job deadlines and an adaptive heartbeat are concerned for data locality. With the data locality and deadline constraints, the task scheduling in the Map phase is formulated as an Assignment Problem (AP) in each heartbeat. The mapped jobs are allocated to the most suitable data centers by the earliest completion times (including both the data transfer and processing times) in the Reduce phase. A task scheduling framework TSH is proposed, in which the scheduling sequence of jobs is determined by the job deadlines, adaptive heartbeats by the processing times of tasks, and the schedule by the Hungarian algorithm. Three heuristics (TSHC, TSHA, and TSHB) are constructed based on TSH with various heartbeat intervals. Experimental results show that TSHB outperforms the other two in effectiveness with the least computation time.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Magnusson, J., Kvernvik, T.: Subscriber classification within telecom networks utilizing big data technologies and machine learning. In: Proceedings of the 1st International Workshop on Big Data, Streams and Heterogeneous Source Mining: Algorithms, Systems, Programming Models and Applications (2012)
Eagle, N., Macy, M., Claxton, R.: Network diversity and economic development. Science 328(5981), 1029–1031 (2010)
Hadoop. http://hadoop.apache.org/
Amazon Web Services. http://aws.amazon.com/
Tauer, G., Nagi, R.: A map-reduce lagrangian heuristic for multidimensional assignment problems with decomposable costs. Parallel Computing 39(11), 653–668 (2013)
Guo, Z., Fox, G., Zhou, M.: Investigation of data locality in mapreduce. In: Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) (2012)
Zaharia, M., Borthakur, D., Sen, S.J., Elmeleegy, K., Shenker, S., Stoica, I.: Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In: Proceedings of the 5th European Conference on Computer Systems (2010)
Fischer, M., Su, X., Yin, Y.: Assigning tasks for efficiency in Hadoop. In: Proceedings of the 22nd ACM Symposium on Parallelism in Algorithms and Architectures (2010)
Ibrahim, S., Jin, H., Lu, L., He, B., Antoniu, G., Wu, S.: Maestro: replica-aware map scheduling for mapreduce. In: 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) (2012)
Polo, J., Becerra, Y., Carrera, D., Steinder, M., Whalley, I., Torres, J., Ayguad, E.: Deadline-Based MapReduce Workload Management. IEEE Transactions on Network and Service Management 10(2), 231–244 (2013)
Dong, X., Wang, Y., Liao, H.: Scheduling mixed real-time and non-real-time applications in MapReduce Environment. In: 2011 IEEE 17th International Conference on Parallel and Distributed Systems (ICPADS) (2011)
Tang, Z., Zhou, J., Li, K., Li, R.: A MapReduce task scheduling algorithm for deadline constraints. Cluster Computing 16(4), 651–652 (2013)
Li, H., Wei, X., Fu, Q., Luo, Y.: MapReduce delay scheduling with deadline constraint. Practice and Experience, Concurrency and Computation (2013)
Yang, J., Li, X., Wang, D., Wang, J.: A Group Mining Method for Big Data on Distributed Vehicle Trajectories in WAN (2014, accepted)
Dou, A.J., Kalogeraki, V., Gunopulos, D., Mielikainen, T., Tuulos, V.: Misco: a MapReduce framework for mobile systems. In: Proceedings of the 3rd International Conference on Pervasive Technologies Related to Assistive Environments (2010)
Dou, A.J., Kalogeraki, V., Gunopulos, D., Mielikainen, T., Tuulos, V.: Data clustering on a network of mobile smartphones. In: 2011 IEEE/IPSJ 11th International Symposium on Applications and the Internet (SAINT) (2011)
Dou, A.J., Kalogeraki, V., Gunopulos, D., Mielikainen, T., Tuulos, V.: Scheduling for real-time mobile MapReduce systems. In: Proceedings of the 5th ACM International Conference on Distributed Event-Based System (2011)
Laurila, J.K., Gatica, P.D., Aad, I., Bornet, O., Do, T.M.T., Dousse O., Eberle J., Miettinen M.: The mobile data challenge: Big data for mobile computing research. Pervasive Computing. EPFL-CONF-192489 (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Wang, J., Li, X. (2015). Task Scheduling for MapReduce Based on Heterogeneous Networks. In: Zu, Q., Hu, B., Gu, N., Seng, S. (eds) Human Centered Computing. HCC 2014. Lecture Notes in Computer Science(), vol 8944. Springer, Cham. https://doi.org/10.1007/978-3-319-15554-8_23
Download citation
DOI: https://doi.org/10.1007/978-3-319-15554-8_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-15553-1
Online ISBN: 978-3-319-15554-8
eBook Packages: Computer ScienceComputer Science (R0)