Performance Analysis of Job Scheduling Algorithms on Hadoop Multi-cluster Environment

Abstract

In recent years, big data applications with scheduling algorithms have evolved lot due to the advancement of new technologies and techniques. We are living in digital data world where the data size is in terms of Exabyte or Pico Byte. This large volume of data is referred as big data. In today’s business environment, the performance of applications largely depends on the efficient retrieval of relevant data on time; the data analysis and retrieval of relevant data need to be done at faster rate. The traditional scheduling algorithms will not be efficient to handle such huge volume of data, considering the above facts managing big data applications and scheduling of big data on distributed architecture has become a challenging research area in the last three–four years. To process such huge volume of data, efficient scheduling algorithms need to be adopted to achieve better performance. The existing MapReduce implementation on Hadoop framework on single node cluster limits themselves to implement all the jobs on single node cluster. In this paper, we will discuss different scheduling techniques and their performance effects on a multimode clusters. The parameters considered for performance evaluation are CPU time, physical memory, and virtual memory. The main aim is to provide survey of different scheduling algorithms that can be used across distributed architecture to achieve better performance in analysis of big data considering YouTube dataset. The results interpret that capacity-based scheduling algorithm is more efficient as compared to FIFO and FAIR in terms of CPU cycles, physical and virtual memory utilization.

Appendix

1.1 Results screen shots: Implementation Results of FIFO, FAIR, and CAPACITY Scheduling Algorithms

The above screen shot interprets that for FIFO scheduling algorithm CPU execution time is 31,090 ms, physical memory (in Bytes) is 1,981,964,288, and virtual memory (in Bytes) is 15,903,952,896.

The above screen shot interprets that for FAIR scheduling algorithm CPU execution time (in ms) is 30,200, physical memory (in Bytes) is 1,844,121,600, and virtual memory (in Bytes) is 15,858,708,480.

The above screen shot interprets that for CAPACITY scheduling algorithm CPU execution time (in ms) is 28,900, physical memory (in Bytes) is 1,832,574,976, and virtual memory (in Bytes) is 15,913,111,552.