Skip to main content
SpringerLink
Log in

An Analysis of Resource-Aware Adaptive Scheduling for HPC Clusters with Hadoop

  • Conference paper
  • First Online:
Big Data Analytics

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 654))

  • 3936 Accesses

Abstract

High-Performance Computing (HPC) is one of the upcoming technologies that represent data-intensive and compute-intensive applications. HPC–on–Cloud is an added advantage to enhance the efficiency of massively parallel applications. Hadoop-MapReduce is a programming paradigm designed to process parallel data on cloud. The key to improve performance of Hadoop-MapReduce lies with Efficient Resource allocation and Scheduling. In this paper, we analyze the behaviour of resource-aware adaptive scheduling which aims to improve resource utilization in MapReduce clusters.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ekanayake, J., Qiu, X., Gunarathne, T., Beason, S., Fox, G.: High Performance Parallel Computing with Cloud and Cloud Technologies, vol. 34, pp. 20–38 (2010)

    Google Scholar 

  2. Ekanayake, J., Fox, G.: High Performance parallel computing with clouds and cloud technologies. 1st international conference on cloud computing, 19–21 Oct 2009

    Google Scholar 

  3. Wikipedia. High-performance computing. https://en.wikipedia.org/wiki/Supercomputer

  4. NIST Definition of Cloud Computing v15. csrc.nist.gov/groups/SNS/cloud-computing/cloud-def-v15.doc

  5. http://www.computer.org/web/computingnow/archive/september2012

  6. Ye, X., Lv, A., Zhao, L.: Research of High Performance Computing With Clouds, pp. 289–293 (2010)

    Google Scholar 

  7. Dean, J., Ghemawat: MapReduce: Simplified Data Processing On Large Clusters. Google Inc. (2004)

    Google Scholar 

  8. Wang, G.: Evaluating MapReduce System Performance:A Simulation Approach (2012)

    Google Scholar 

  9. Apache. Hadoop. http://hadoop.apache.org

  10. Hadoop: The Definitive Guide, Second Edition, by Tom White, O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472

    Google Scholar 

  11. Hadoop Distributed File System. http://hadoop.apache.org/hdfs

  12. Hadoop’s Fair Scheduler. https://hadoop.apache.org/docs/r1.2.1/fair_scheduler

  13. Hadoop’s Capacity Scheduler. http://hadoop.apache.org/core/docs/current/capacity_scheduler.html

  14. Polo, J., Castillo, C., Carrera, D., Becerra, Y., Whalley, I., Steinder, M., Torres, J., Ayguadé, E.: Resource-aware adaptive scheduling for MapReduce clusters. In Proceedings of the 12th ACM/IFIP/USENIX international conference on Middleware, in (2011)

    Google Scholar 

  15. http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.0/bk_getting-started-guide/content/ch_hdp1_getting_started_chp2_1.html

  16. Zhang, Q., Zhani, M.F., Yang, Y., Boutaba, R., Wong, B.: PRISM: Fine-grained resource-aware scheduling for mapreduce. (2015)

    Google Scholar 

  17. Wottrich, K., Bressoud, T.: The Performance Characteristics of MapReduceApplications on Scalable Clusters. MCURCSM (2011)

    Google Scholar 

  18. Dong, F., Akl, S.G.: Scheduling algorithms for grid computing: state of art and open problems. Queens University Technicalreport (2006)

    Google Scholar 

  19. Shivakumar, N., Rashmi, S., Basu, A.: Hadoop map reduce multi-job workloads using resource aware scheduler. IJARCS (2014)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, East Point College of Engineering and Technology, Bangalore, India

    S. Rashmi & Anirban Basu

Authors
  1. S. Rashmi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anirban Basu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S. Rashmi .

Editor information

Editors and Affiliations

  1. Jagan Institute of Management Studies, New Delhi, Delhi, India

    V. B. Aggarwal

  2. Department of Computer Science, University of Delhi, New Delhi, Delhi, India

    Vasudha Bhatnagar

  3. Microsoft Innovation Centre, Sri Aurobindo Institute of Technology, Indore, Madhya Pradesh, India

    Durgesh Kumar Mishra

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this paper

Cite this paper

Rashmi, S., Basu, A. (2018). An Analysis of Resource-Aware Adaptive Scheduling for HPC Clusters with Hadoop. In: Aggarwal, V., Bhatnagar, V., Mishra, D. (eds) Big Data Analytics. Advances in Intelligent Systems and Computing, vol 654. Springer, Singapore. https://doi.org/10.1007/978-981-10-6620-7_22

Download citation

  • DOI: https://doi.org/10.1007/978-981-10-6620-7_22

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-10-6619-1

  • Online ISBN: 978-981-10-6620-7

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation