Abstract
This work describes the goals and impacts of a large reconfiguration of the job scheduling system, used in the Czech National Grid and Cloud infrastructure MetaCentrum, which was implemented in early 2014. MetaCentrum, as a “long-tail” oriented provider, serves a varied user-base consisting of both individual users and research groups. This imposes strict requirements on the robustness of job scheduling algorithms being employed, as the system must be capable of assigning a highly heterogeneous set of workloads to a similarly heterogeneous set of computational resources. Primary goals for MetaCentrum were always to provide efficient and fair resource utilization with respect to different users in the system. During the last few years, MetaCentrum has gone through a period of rapid growth (1,500 CPU cores in 2009 vs. 10,600 CPU cores in 2014) forcing us to re-evaluate our scheduling approaches, as the “old” configuration no longer satisfied our utilization and fairness demands. This re-evaluation was supported by a significant body of research, which included the proposal of new scheduling approaches as well as detailed simulations based on real-life complex workload traces. First of all, a new multi-resource aware fair-sharing algorithm (based on our recent research) was deployed, with the goal of improving fairness with respect to the growing heterogeneity of resources and users’ workloads. Second, the queue configuration of the entire system was completely reworked in order to decrease resource fragmentation and improve the utilization and the impact of fairness policies. This paper summarizes the effects of these changes using real-life data from the production system. Moreover, we publish complex workload traces from MetaCentrum that were used in this paper, since they represent a valuable source of data concerning a highly heterogeneous production system. Last but not least, we also present our advanced job scheduling simulator which is routinely used for testing of new scheduling strategies prior their deployment in the real system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Only major queues in the main system pool are considered. Auxiliary and specialized queues are omitted as well as all results coming from the second scheduler.
- 2.
To be more precise, not users but their jobs in a queue are then ordered according to corresponding \(F_{u}\) values.
- 3.
As was explained in Sect. 2.1, q_2w, q_1w, q_4d, q_2d, etc. queues now have larger pools of available resources compared to the original long queue.
- 4.
Jobs requesting less than 1 GB of RAM are not shown in Fig. 7 as they would end up “bellow” the baseline of the log. scale-shaped graph.
- 5.
Those exceptions are jobs lying under the main “diagonal”, i.e., in the lower central/right part of the plot. Such exceptions were expected as the new fair-sharing scheme may also (rarely) assign smaller penalties compared to the original single-resource aware mechanism.
- 6.
In case of the earlier period (October–December 2013)—which did not use the new fair-sharing mechanism—these affected jobs were detected using the Alea job scheduling simulator which is capable of emulating the new fair-sharing method.
- 7.
A detailed description of qsub semantics is available at: https://wiki.metacentrum.cz/wiki/Running_jobs_in_scheduler.
References
Adaptive Computing Enterprises, Inc., Maui Scheduler Administrator’s Guide, version 3.2, January 2014. http://docs.adaptivecomputing.com
Alea job scheduling simulator, February 2015. https://github.com/aleasimulator/
Chapin, S., et al.: Benchmarks and standards for the evaluation of parallel job schedulers. In: Feitelson, D.G., Rudolph, L. (eds.) JSSPP 1999. LNCS, vol. 1659, pp. 67–90. Springer, Heidelberg (1999). doi:10.1007/3-540-47954-6_4
Ghodsi, A., Zaharia, M., Hindman, B., Konwinski, A., Shenker, S., Stoica, I.: Dominant resource fairness: fair allocation of multiple resource types. In: 8th USENIX Symposium on Networked Systems Design and Implementation (2011)
Jackson, D., Snell, Q., Clement, M.: Core algorithms of the maui scheduler. In: Feitelson, D.G., Rudolph, L. (eds.) JSSPP 2001. LNCS, vol. 2221, pp. 87–102. Springer, Heidelberg (2001). doi:10.1007/3-540-45540-X_6
Joe-Wong, C., Sen, S., Lan, T., Chiang, M.: Multi-resource allocation: fairness-efficiency tradeoffs in a unifying framework. In: 31st Annual International Conference on Computer Communications (IEEE INFOCOM), pp. 1206–1214 (2012)
Klusáček, D., Chlumský, V., Rudová, H.: Planning and optimization in TORQUE resource manager. In: High Performance and Distributed Computing (HPDC). ACM (2015)
Klusáček, D., Rudová, H.: Alea 2 - job scheduling simulator. In: 3rd International ICST Conference on Simulation Tools and Technique (ICST) (2010)
Klusáček, D., Rudová, H.: Multi-resource aware fairsharing for heterogeneous systems. In: Cirne, W., Desai, N. (eds.) JSSPP 2014. LNCS, vol. 8828, pp. 53–69. Springer, Cham (2015). doi:10.1007/978-3-319-15789-4_4
Klusáček, D., Tóth, Š.: On interactions among scheduling policies: finding efficient queue setup using high-resolution simulations. In: Silva, F., Dutra, I., Santos Costa, V. (eds.) Euro-Par 2014. LNCS, vol. 8632, pp. 138–149. Springer, Cham (2014). doi:10.1007/978-3-319-09873-9_12
Lawson, B.G., Smirni, E.: Multiple-queue backfilling scheduling with priorities and reservations for parallel systems. In: Feitelson, D.G., Rudolph, L., Schwiegelshohn, U. (eds.) JSSPP 2002. LNCS, vol. 2537, pp. 72–87. Springer, Heidelberg (2002). doi:10.1007/3-540-36180-4_5
Lublin, U., Feitelson, D.G.: The workload on parallel supercomputers: modeling the characteristics of rigid jobs. J. Parallel Distrib. Comput. 63(11), 1105–1122 (2003)
MetaCentrum workload logs, February 2015. http://www.fi.muni.cz/ xklusac/workload/
Parallel workload models, February 2015. http://www.cs.huji.ac.il/labs/parallel/workload/models.html
Mu’alem, A.W., Feitelson, D.G.: Utilization, predictability, workloads, and user runtime estimates in scheduling the IBM SP2 with backfilling. IEEE Trans. Parallel Distrib. Syst. 12(6), 529–543 (2001)
Ohio Supercomputer Center. Batch Processing at OSC, February 2014. https://www.osc.edu/supercomputing/batch-processing-at-osc
PBS Works. PBS Professional 12.1, Administrator’s Guide, January 2014. http://www.pbsworks.com
Podolníková, G.: Configuration and presentation system of job scheduling simulator, Bachelor’s thesis (2014). http://is.muni.cz/th/396214/fi_b/Gabriela_Podolnikova_BP.pdf
Sempolinski, P., Thain, D.: A comparison and critique of Eucalyptus, OpenNebula and Nimbus. In: Proceedings of the 2010 IEEE Second International Conference on Cloud Computing Technology and Science (CLOUDCOM 2010), pp. 417–426. IEEE Computer Society (2010)
Skovira, J., Chan, W., Zhou, H., Lifka, D.: The EASY — loadleveler API project. In: Feitelson, D.G., Rudolph, L. (eds.) JSSPP 1996. LNCS, vol. 1162, pp. 41–47. Springer, Heidelberg (1996). doi:10.1007/BFb0022286
Sulistio, A., Cibej, U., Venugopal, S., Robic, B., Buyya, R.: A toolkit for modelling and simulating data grids: an extension to gridsim. Concurr. Comput. Pract. Exp. 20(13), 1591–1609 (2008)
Zakay, N., Feitelson, D.G.: Preserving user behavior characteristics in trace-based simulation of parallel job scheduling. In: 22nd Modeling, Analysis & Simulation of Computer & Telecommunication System (MASCOTS), pp. 51–60 (2014)
Acknowledgments
We highly appreciate the support of the Grant Agency of the Czech Republic under the grant No. P202/12/0306. The support provided by the programme “Projects of Large Infrastructure for Research, Development, and Innovations” LM2010005 funded by the Ministry of Education, Youth, and Sports of the Czech Republic is highly appreciated. The access to the MetaCentrum computing facilities and workloads is kindly acknowledged.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Klusáček, D., Tóth, Š., Podolníková, G. (2017). Real-Life Experience with Major Reconfiguration of Job Scheduling System. In: Desai, N., Cirne, W. (eds) Job Scheduling Strategies for Parallel Processing. JSSPP JSSPP 2015 2016. Lecture Notes in Computer Science(), vol 10353. Springer, Cham. https://doi.org/10.1007/978-3-319-61756-5_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-61756-5_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-61755-8
Online ISBN: 978-3-319-61756-5
eBook Packages: Computer ScienceComputer Science (R0)