Abstract
Current production resource management and scheduling systems often use some mechanism to guarantee fair sharing of computational resources among different users of the system. For example, the user who so far consumed small amount of CPU time gets higher priority and vice versa. The problem with such a solution is that it does not reflect other consumed resources like RAM, HDD storage capacity or GPU cores. Clearly, different users may have highly heterogeneous demands concerning aforementioned resources, yet they are all prioritized only with respect to consumed CPU time. In this paper we show that such a single resource-based approach is unfair and is no longer suitable for nowadays systems. We provide a survey of existing works that somehow try to deal with this situation and we closely analyze and evaluate their characteristics. Next, we propose new enhanced approaches that would allow the development of usable multi resource-aware user prioritization mechanisms. We demonstrate that different consumed resources can be weighted and combined together within a single formula which can be used to establish users’ priorities. Moreover, we show that when it comes to multiple resources, it is not always possible to find a suitable solution that would fulfill all fairness-related requirements.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
In Maui’s terminology, fairshare usage represents the metric of utilization measurement [10]. Typically, fairshare usage expresses the amount of consumed CPU time of a given user.
- 3.
In MetaCentrum, resources allocated (i.e., reserved) to a given job cannot be used by other jobs even if those resources are not fully used. Therefore, in the whole paper we measure CPU, RAM, etc., requirements as the amount of a given resource that has been allocated for a job, even if actual job’s requirements are smaller.
- 4.
Max-min penalty rule defines when \(P_j\) reaches its minimum and maximum. Apparently, no “real” job should ever receive minimum penalty since it always consumes some resources.
- 5.
References
Adaptive Computing Enterprises, Inc. Maui Scheduler Administrator’s Guide, version 3.2, February 2013. http://docs.adaptivecomputing.com
Adaptive Computing Enterprises, Inc. Moab workload manager administrator’s guide, version 7.2.1, February 2013. http://docs.adaptivecomputing.com
Adaptive Computing Enterprises, Inc. TORQUE Admininstrator Guide, version 4.2.0, February 2013. http://docs.adaptivecomputing.com
Apache.org. Hadoop Capacity Scheduler, February 2013. http://hadoop.apache.org/docs/r1.1.1/capacity_scheduler.html
Apache.org. Hadoop Fair Scheduler, February 2013. http://hadoop.apache.org/docs/r1.1.1/fair_scheduler.html
Blazewicz, J., Drozdowski, M., Markiewicz, M.: Divisible task scheduling - concept and verification. Parallel Comput. 25(1), 87–98 (1999)
Dolev, D., Feitelson, D.G., Halpern, J.Y., Kupferman, R., Linial, N.: No justified complaints: on fair sharing of multiple resources. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS ’12, pp. 68–75. ACM, New York (2012)
Ghodsi, A., Zaharia, M., Hindman, B., Konwinski, A., Shenker, S., Stoica, I.: Dominant resource fairness: fair allocation of multiple resource types. In: 8th USENIX Symposium on Networked Systems Design and Implementation (2011)
Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., Goldberg, A.: Quincy: fair scheduling for distributed computing clusters. In: SOSP’09 (2009)
Jackson, D.B., Snell, Q.O., Clement, M.J.: Core algorithms of the Maui scheduler. In: Feitelson, D.G., Rudolph, L. (eds.) JSSPP 2001. LNCS, vol. 2221, pp. 87–102. Springer, Heidelberg (2001)
Jain, R., Chiu, D.-M., Hawe, W.: A quantitative measure of fairness and discrimination for resource allocation in shared computer systems. Technical report TR-301, Digital Equipment Corporation (1984)
Joe-Wong, C., Sen, S., Lan, T., Chiang, M.: Multi-resource allocation: fairness-efficiency tradeoffs in a unifying framework. In: INFOCOM (2012)
Jones, J.P.: PBS Professional 7, administrator guide. Altair, April 2005
Kleban, S.D., Clearwater, S.H.: Fair share on high performance computing systems: what does fair really mean? In: Third IEEE International Symposium on Cluster Computing and the Grid (CCGrid’03), pp. 146–153. IEEE Computer Society (2003)
Klusáček, D., Ruda, M., Rudová, H.: New fairness and performance metrics for current grids. In: Cracow Grid Workshop, pp. 73–74. ACC Cyfronet AGH (2012)
MetaCentrum, February 2013. http://www.metacentrum.cz/
Mu’alem, A.W., Feitelson, D.G.: Utilization, predictability, workloads, and user runtime estimates in scheduling the IBM SP2 with backfilling. IEEE Trans. Parallel Distrib. Syst. 12(6), 529–543 (2001)
Ohio Supercomputer Center. Batch Processing at OSC, February 2013. https://www.osc.edu/supercomputing/batch-processing-at-osc
Shmueli, E., Feitelson, D.G.: Backfilling with lookahead to optimize the performance of parallel job scheduling. In: Feitelson, D.G., Rudolph, L., Schwiegelshohn, U. (eds.) JSSPP 2003. LNCS, vol. 2862, pp. 228–251. Springer, Heidelberg (2003)
Tsafrir, D., Etsion, Y., Feitelson, D.G.: Backfilling using system-generated predictions rather than user runtime estimates. IEEE Trans. Parallel Distrib. Syst. 18(6), 789–803 (2007)
Acknowledgments
We highly appreciate the support of the Grant Agency of the Czech Republic under the grant No. P202/12/0306. The access to the MetaCentrum computing facilities provided under the programme LM2010005 funded by the Ministry of Education, Youth, and Sports of the Czech Republic is highly appreciated. The Zewura workload log was kindly provided by the Czech NGI MetaCentrum. The access to the CERIT-SC computing and storage facilities provided under the programme Center CERIT Scientific Cloud, part of the Operational Program Research and Development for Innovations, reg. no. CZ. 1.05/3.2.00/08.0144 is appreciated.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Klusáček, D., Rudová, H., Jaroš, M. (2014). Multi Resource Fairness: Problems and Challenges. In: Desai, N., Cirne, W. (eds) Job Scheduling Strategies for Parallel Processing. JSSPP 2013. Lecture Notes in Computer Science(), vol 8429. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-43779-7_5
Download citation
DOI: https://doi.org/10.1007/978-3-662-43779-7_5
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-662-43778-0
Online ISBN: 978-3-662-43779-7
eBook Packages: Computer ScienceComputer Science (R0)