Advertisement

A Review of Scheduling Algorithms in Hadoop

  • Anil Sharma
  • Gurwinder SinghEmail author
Conference paper
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 597)

Abstract

In this epoch of data surge, big data is one of the significant areas of research being widely pondered over by computer science research community, and Hadoop is the broadly used tool to store and process it. Hadoop is fabricated to work effectively for the clusters having homogeneous environment but when the cluster environment is heterogeneous then its performance decreases which result in various challenges surfacing in the areas like query execution time, data movement cost, selection of best Cluster and Racks for data placement, preserving privacy, load distribution: imbalance in input splits, computations, partition sizes and heterogeneous hardware, and scheduling. The epicenter of Hadoop is scheduling and all incoming jobs are multiplexed on existing resources by the schedulers. Enhancing the performance of schedulers in Hadoop is very vigorous. Keeping this idea in mind as inspiration, this paper introduces the concept of big data, market share of popular vendors for big data, various tools in Hadoop ecosystem and emphasizing to study various scheduling algorithms for MapReduce model in Hadoop and make a comparison based on varied parameters.

Keywords

Big data Hadoop TaskTracker JobTracker Scheduling MapReduce 

References

  1. 1.
    Cox, M., Ellsworth, D.: Managing big data for scientific visualization. ACM Siggraph. 97, 5.1–5.17 (1997)Google Scholar
  2. 2.
    Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., Byers, A.H.: Big Data : The Next Frontier for Innovation, Competition, and Productivity (2011)Google Scholar
  3. 3.
    Zikopoulos, P.C., DeRoos, D., Parasuraman, K., Deutsch, T., Corrigan, D., Giles, J.: Harness the Power of Big Data. The McGraw-Hill Companies (2013)Google Scholar
  4. 4.
    Berman, J.J.: Principles of Big Data : Preparing, Sharing, and Analyzing Complex Information. Morgan Kaufmann Elsevier (2013)Google Scholar
  5. 5.
    Gantz, J., Reinsel, D.: Extracting Value from Chaos (2011)Google Scholar
  6. 6.
    Chen, M., Mao, S., Liu, Y.: Big Data: A Survey. Mob Netw Appl 19, 171–209 (2014)CrossRefGoogle Scholar
  7. 7.
    Reinsel, D., Gantz, J., Rydning, J.: The Digitization of the World- From Edge to Core (2018)Google Scholar
  8. 8.
    Kelly, J., Vellante, D., Floyer, D.: Big Data Market Size and Vendor Revenues (2012)Google Scholar
  9. 9.
    White, T.: Hadoop: The Definitive Guide. O’Reilly Media (2015)Google Scholar
  10. 10.
    Saraladevi, B., Pazhaniraja, N., Paul, P.V., Basha, M.S.S., Dhavachelvan, P.: Big Data and Hadoop-A Study in Security Perspective. Procedia Comput. Sci. 50, 596–601 (2015)CrossRefGoogle Scholar
  11. 11.
    Ji, C., Li, Y., Qiu, W., Awada, U., Li, K.: Big data processing in cloud computing environments. In: 2012 International Symposium on Pervasive Systems, Algorithms and Networks. pp. 17–23. IEEE (2012)Google Scholar
  12. 12.
    Song, Y.: Storing Big Data—The Rise of the Storage Cloud (2012)Google Scholar
  13. 13.
    Ghazi, M.R., Gangodkar, D.: Hadoop, MapReduce and HDFS: a developers perspective. Procedia Comput. Sci. 48, 45–50 (2015)CrossRefGoogle Scholar
  14. 14.
    Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The Hadoop distributed file system. In: 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies, MSST2010, pp. 1–10 (2010)Google Scholar
  15. 15.
    Martha, V.: Big Data processing algorithms. In: Mohanty, H., Bhuyan, P., Chenthati, D. (eds.) Studies in Big Data, pp. 61–92. Springer (2015)Google Scholar
  16. 16.
    Raj, E.D., Dhinesh Babu, L.D.: A two pass scheduling policy based resource allocation for mapreduce. In: Procedia Computer Science, International Conference on Information and Communication Technologies (ICICT 2014), pp. 627–634. Elsevier B.V. (2015)Google Scholar
  17. 17.
    He, B., Fang, W., Luo, Q., Govindaraju, N.K., Wang, T.: Mars. In: Proceedings of the 17th International Conference on Parallel Architectures and Compilation Techniques—PACT ’08, p. 260 (2008)Google Scholar
  18. 18.
    Marx, V.: Technology feature: the big challenges of Big Data. Nature 498, 255–260 (2013)CrossRefGoogle Scholar
  19. 19.
    Bhosale, H.S., Gadekar, D.P.: A review paper on Big Data and Hadoop. Int. J. Sci. Res. Publ. 4, 1–7 (2014)Google Scholar
  20. 20.
    Al-janabi, S.T.F., Rasheed, M.A.: Public-key cryptography enabled kerberos authentication. In: 2011 Developments in E-systems Engineering Public-Key, pp. 209–214. IEEE (2011)Google Scholar
  21. 21.
    Fadika, Z., Dede, E., Hartog, J., Govindaraju, M.: MARLA : MapReduce for heterogeneous clusters. In: 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, pp. 49–56. ACM (2012)Google Scholar
  22. 22.
    Mao, Y., Ling, J.: Research on load balance strategy based on grey prediction theory in cloud storage. In: 2nd International Conference on Electronic & Mechanical Engineering and Information Technology (EMEIT-2012), pp. 199–203. Atlantis Press, Paris, France (2012)Google Scholar
  23. 23.
    Ye, X., Huang, M., Zhu, D., Xu, P.: A novel blocks placement strategy for hadoop. In: Proceedings—2012 IEEE/ACIS 11th International Conference on Computer and Information Science, pp. 3–7. IEEE (2012)Google Scholar
  24. 24.
    Ling, J., Jiang, X.: Distributed storage method based on information dispersal algorithm. In: Proceedings—2013 2nd International Symposium on Instrumentation and Measurement, Sensor Network and Automation, IMSNA 2013, pp. 624–626. IEEE (2013)Google Scholar
  25. 25.
    Kumar, S.D.M., Shabeera, T.P.: Bandwidth-aware data placement scheme for Hadoop. In: 2013 IEEE Recent Advances in Intelligent Computational Systems (RAICS), pp. 64–67. IEEE (2013)Google Scholar
  26. 26.
    Fan, K., Zhang, D., Li, H., Yang, Y.: An adaptive feedback load balancing algorithm in HDFS. In: 2013 5th International Conference on Intelligent Networking and Collaborative Systems, pp. 23–29. IEEE (2013)Google Scholar
  27. 27.
    Lee, C.W., Hsieh, K.Y., Hsieh, S.Y., Hsiao, H.C.: A dynamic data placement strategy for Hadoop in heterogeneous environments. Big Data Res. 1, 14–22 (2014)CrossRefGoogle Scholar
  28. 28.
    Gao, Z., Liu, D., Yang, Y., Zheng, J., Hao, Y.: A load balance algorithm based on nodes performance in Hadoop cluster. In: APNOMS 2014—16th Asia-Pacific Network Operations and Management Symposium, pp. 1–4. IEEE (2014)Google Scholar
  29. 29.
    Lin, C.Y., Lin, Y.C.: A load-balancing algorithm for Hadoop distributed file system. In: Proceedings—2015 18th International Conference on Network-Based Information Systems, pp. 173–179. IEEE (2015)Google Scholar
  30. 30.
    Kim, D., Choi, E., Hong, J.: System information-based hadoop load balancing for heterogeneous clusters. In: RACS ’15 International Conference on Research in Adaptive and Convergent Systems, pp. 465–467. ACM (2015)Google Scholar
  31. 31.
    Islam, N.S., Lu, X., Shankar, D., Panda, D.K.D.K.: Triple-H : A hybrid approach to accelerate HDFS on HPC clusters with heterogeneous storage architecture. In: 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing Triple-H, pp 101–110. ACM (2015)Google Scholar
  32. 32.
    Wang, S., Zhou, H.: The research of MapReduce load balancing based on multiple partition algorithm. In: IEEE/ACM 9th International Conference on Utility and Cloud Computing, pp. 339–342. IEEE/ACM (2016)Google Scholar
  33. 33.
    Hou, X., Pal, D., Kumar T.K.A., Thomas, J.P., Liu, H.: Privacy preserving rack-based dynamic workload balancing for Hadoop MapReduce. In: IEEE 2nd International Conference on Big Data Security on Cloud, IEEE International Conference on High Performance and Smart Computing, IEEE International Conference on Intelligent Data and Security, pp. 30–35. IEEE (2016)Google Scholar
  34. 34.
    Nayahi, J.J.V., Kavitha, V.: Privacy and utility preserving data clustering for data anonymization and distribution on Hadoop. Futur. Gener. Comput. Syst. 74, 393–408 (2016)CrossRefGoogle Scholar
  35. 35.
    Song, Y., Shin, Y., Jang, M., Chang, J.: Design and implementation of HDFS data encryption scheme using ARIA algorithm on Hadoop. In: 4th International Conference on Big Data and Smart Computing (BigComp 2017), pp. 84–90. IEEE (2017)Google Scholar
  36. 36.
    Tao, D., Lin, Z., Wang, B.: Load feedback-based resource scheduling and dynamic migration-based data locality for virtual Hadoop clusters in OpenStack-based clouds. Tsinghua Sci. Technol. 22, 149–159 (2017)CrossRefGoogle Scholar
  37. 37.
    Guo, Z., Fox, G., Zhou, M., Ruan, Y.: Improving resource utilization in MapReduce. In: IEEE International Conference on Cluster Computing, pp. 402–410. IEEE (2012)Google Scholar
  38. 38.
    Zaharia, M., Konwinski, A., Joseph, A.D., Katz, R., Stoica, I.: Improving MapReduce performance in heterogeneous environments. In: 8th USENIX Symposium on Operating Systems Design and Implementation, pp. 29–42. USENIX Association (2008)Google Scholar
  39. 39.
    Kc, K., Anyanwu, K.: Scheduling Hadoop jobs to meet deadlines. In: 2nd IEEE International Conference on Cloud Computing Technology and Science Scheduling, pp. 388–392. IEEE (2010)Google Scholar
  40. 40.
    Dai, X., Bensaou, B.: Scheduling for response time in Hadoop MapReduce. In: IEEE ICC 2016 SAC Cloud Communications and Networking, pp. 3627–3632. IEEE (2016)Google Scholar
  41. 41.
    Cheng, D., Rao, J., Jiang, C., Zhou, X.: Resource and deadline-aware job scheduling in dynamic Hadoop Clusters. In: Proceedings—2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015, pp. 956–965 (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.School of Computer ApplicationsLovely Professional UniversityPunjabIndia

Personalised recommendations