Skip to main content
SpringerLink
Log in

Task Scheduling for MapReduce Based on Heterogeneous Networks

  • Conference paper
  • First Online:
Book cover Human Centered Computing (HCC 2014)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 8944))

Included in the following conference series:

Abstract

In this paper, the task scheduling in MapReduce is considered for geo-distributed data centers on heterogeneous networks. Job deadlines and an adaptive heartbeat are concerned for data locality. With the data locality and deadline constraints, the task scheduling in the Map phase is formulated as an Assignment Problem (AP) in each heartbeat. The mapped jobs are allocated to the most suitable data centers by the earliest completion times (including both the data transfer and processing times) in the Reduce phase. A task scheduling framework TSH is proposed, in which the scheduling sequence of jobs is determined by the job deadlines, adaptive heartbeats by the processing times of tasks, and the schedule by the Hungarian algorithm. Three heuristics (TSHC, TSHA, and TSHB) are constructed based on TSH with various heartbeat intervals. Experimental results show that TSHB outperforms the other two in effectiveness with the least computation time.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Magnusson, J., Kvernvik, T.: Subscriber classification within telecom networks utilizing big data technologies and machine learning. In: Proceedings of the 1st International Workshop on Big Data, Streams and Heterogeneous Source Mining: Algorithms, Systems, Programming Models and Applications (2012)

    Google Scholar 

  2. Eagle, N., Macy, M., Claxton, R.: Network diversity and economic development. Science 328(5981), 1029–1031 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  3. Hadoop. http://hadoop.apache.org/

  4. Amazon Web Services. http://aws.amazon.com/

  5. Tauer, G., Nagi, R.: A map-reduce lagrangian heuristic for multidimensional assignment problems with decomposable costs. Parallel Computing 39(11), 653–668 (2013)

    Article  Google Scholar 

  6. Guo, Z., Fox, G., Zhou, M.: Investigation of data locality in mapreduce. In: Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) (2012)

    Google Scholar 

  7. Zaharia, M., Borthakur, D., Sen, S.J., Elmeleegy, K., Shenker, S., Stoica, I.: Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In: Proceedings of the 5th European Conference on Computer Systems (2010)

    Google Scholar 

  8. Fischer, M., Su, X., Yin, Y.: Assigning tasks for efficiency in Hadoop. In: Proceedings of the 22nd ACM Symposium on Parallelism in Algorithms and Architectures (2010)

    Google Scholar 

  9. Ibrahim, S., Jin, H., Lu, L., He, B., Antoniu, G., Wu, S.: Maestro: replica-aware map scheduling for mapreduce. In: 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) (2012)

    Google Scholar 

  10. Polo, J., Becerra, Y., Carrera, D., Steinder, M., Whalley, I., Torres, J., Ayguad, E.: Deadline-Based MapReduce Workload Management. IEEE Transactions on Network and Service Management 10(2), 231–244 (2013)

    Article  Google Scholar 

  11. Dong, X., Wang, Y., Liao, H.: Scheduling mixed real-time and non-real-time applications in MapReduce Environment. In: 2011 IEEE 17th International Conference on Parallel and Distributed Systems (ICPADS) (2011)

    Google Scholar 

  12. Tang, Z., Zhou, J., Li, K., Li, R.: A MapReduce task scheduling algorithm for deadline constraints. Cluster Computing 16(4), 651–652 (2013)

    Article  Google Scholar 

  13. Li, H., Wei, X., Fu, Q., Luo, Y.: MapReduce delay scheduling with deadline constraint. Practice and Experience, Concurrency and Computation (2013)

    Google Scholar 

  14. Yang, J., Li, X., Wang, D., Wang, J.: A Group Mining Method for Big Data on Distributed Vehicle Trajectories in WAN (2014, accepted)

    Google Scholar 

  15. Dou, A.J., Kalogeraki, V., Gunopulos, D., Mielikainen, T., Tuulos, V.: Misco: a MapReduce framework for mobile systems. In: Proceedings of the 3rd International Conference on Pervasive Technologies Related to Assistive Environments (2010)

    Google Scholar 

  16. Dou, A.J., Kalogeraki, V., Gunopulos, D., Mielikainen, T., Tuulos, V.: Data clustering on a network of mobile smartphones. In: 2011 IEEE/IPSJ 11th International Symposium on Applications and the Internet (SAINT) (2011)

    Google Scholar 

  17. Dou, A.J., Kalogeraki, V., Gunopulos, D., Mielikainen, T., Tuulos, V.: Scheduling for real-time mobile MapReduce systems. In: Proceedings of the 5th ACM International Conference on Distributed Event-Based System (2011)

    Google Scholar 

  18. Laurila, J.K., Gatica, P.D., Aad, I., Bornet, O., Do, T.M.T., Dousse O., Eberle J., Miettinen M.: The mobile data challenge: Big data for mobile computing research. Pervasive Computing. EPFL-CONF-192489 (2012)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Southeast University, Nanjing, China

    Jia Wang

  2. Southeast University, Road No. 2, Nanjing, 210096, Jiangsu, China

    Xiaoping Li

Authors
  1. Jia Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoping Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoping Li .

Editor information

Editors and Affiliations

  1. Wuhan University of Technology, Wuhan, China

    Qiaohong Zu

  2. Fujitsu Laboratories of Europe, Middlesex, United Kingdom

    Bo Hu

  3. Fudan University, Shanghai, China

    Ning Gu

  4. Telecommunications and Information Communication Technology, National Institute of Posts, Phnom Penh, Cambodia

    Sopheap Seng

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Wang, J., Li, X. (2015). Task Scheduling for MapReduce Based on Heterogeneous Networks. In: Zu, Q., Hu, B., Gu, N., Seng, S. (eds) Human Centered Computing. HCC 2014. Lecture Notes in Computer Science(), vol 8944. Springer, Cham. https://doi.org/10.1007/978-3-319-15554-8_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-15554-8_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-15553-1

  • Online ISBN: 978-3-319-15554-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation