WASC: Adapting Scheduler Configurations for Heterogeneous MapReduce Workloads

  • Siyi Wang
  • Fan ZhangEmail author
  • Rui Han
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 911)


MapReduce has emerged as a popular programming paradigm for data intensive computing in both scientific and commercial applications. On a MapReduce cluster, modern resource negotiation frameworks like Hadoop YARN and Mesos support scheduling of jobs submitted by multiple tenants. However, existing job schedulers lacks the automatic adaption to workload variations in their scheduling configuration, which is crucial for the jobs’ latencies because it determines how to share resources among the latest jobs in the system. The major challenge here is, to a MapReduce cluster scheduler, The performance of different configurations depends not only on the number of jobs in different queues, but also on their workload characteristics, which refer to the type and size of jobs. We introduce a workload-adaptive scheduling configuration (WASC) framework for heterogeneous MapReduce jobs. WASC identifies the optimal configuration for them by reasoning about their performances under different configurations.


MapReduce Workload heterogeneous Cluster schedulers Configurations 


  1. 1.
  2. 2.
  3. 3.
    Statistical workload injector for mapreduce (swim).
  4. 4.
  5. 5.
  6. 6.
    Alapati, S.R.: Expert Hadoop Administration: Managing, Tuning, and Securing Spark, YARN, and HDFS. Addison-Wesley Professional, Boston (2016)Google Scholar
  7. 7.
    Bhattacharya, A.A., Culler, D., Friedman, E., Ghodsi, A., Shenker, S., Stoica, I.: Hierarchical scheduling for diverse datacenter workloads. In: SoCC 2013, p. 4. ACM (2013)Google Scholar
  8. 8.
    Cai, C.X., Saeed, S., Gupta, I., et al.: Phurti: application and network-aware flow scheduling for multi-tenant MapReduce clusters. In: IC2E 2016, pp. 161–170. IEEE (2016)Google Scholar
  9. 9.
    Chen, Y., Alspaugh, S., Katz, R.: Interactive analytical processing in big data systems: a cross-industry study of MapReduce workloads. In: VLDB 2011, vol. 5, pp. 1802–1813 (2012)CrossRefGoogle Scholar
  10. 10.
    Cho, B., Rahman, M., Chajed, T., et al.: Natjam: design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters. In: SoCC 2013, p. 6. ACM (2013)Google Scholar
  11. 11.
    Davis, R., Hamscher, W.: Model-based reasoning: troubleshooting. Explor. Artif. Intell. 8, 297–346 (1988)CrossRefGoogle Scholar
  12. 12.
    Ghodsi, A., Zaharia, M., Hindman, B., Konwinski, A., Shenker, S., Stoica, I.: Dominant resource fairness: fair allocation of multiple resource types. In: NSDI 2011, vol. 11, pp. 24–24 (2011)Google Scholar
  13. 13.
    Herodotou, H., et al.: Starfish: a self-tuning system for big data analytics. In: CIDR 2011, pp. 261–272 (2011)Google Scholar
  14. 14.
    Kavulya, S., Tan, J., Gandhi, R., Narasimhan, P.: An analysis of traces from a production MapReduce cluster. In: CCGrid 2010, pp. 94–103. IEEE (2010)Google Scholar
  15. 15.
    Ren, Z., Xu, X., Wan, J., Shi, W., Zhou, M.: Workload characterization on a production Hadoop cluster: a case study on Taobao. In: IISWC 2012, pp. 3–13. IEEE (2012)Google Scholar
  16. 16.
    Srirama, S.N., Jakovits, P., Vainikko, E.: Adapting scientific computing problems to clouds using MapReduce. Future Gener. Comput. Syst. 28(1), 184–192 (2012)CrossRefGoogle Scholar
  17. 17.
    Verma, A., Cherkasova, L., Campbell, R.H.: ARIA: automatic resource inference and allocation for MapReduce environments. In: ICAC 2011, pp. 235–244. ACM (2011)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.Zhengzhou UniversityZhengzhouChina
  2. 2.Institute of Computing TechnologyChinese Academy of SciencesBeijingChina

Personalised recommendations