Abstract
Computational and data scientists at universities are often limited by the quantity and diversity of the shared resources available at their institution. Access cost for these resources are often uniform, that is, it is not differentiated based on job priority or resource requirements. This flat access policy on shared resources often lead to sub-optimal values for the institutions, and researchers with special requirements (i.e. GPU, large-memory, etc.) often have to wait significantly longer to get their job scheduled. A market-based resource trading in a multi-campus Compute Co-operative can lead to higher aggregated value for the co-operative as well as provide significant benefits for the individual institutions by scheduling jobs opportunistically when resources of one campus are over-subscribed and by placing jobs efficiently based on resource requirements. In this paper, we evaluate a resource allocation scheme in a multi-campus environment, (i.e. CCC [10]) based on job priority and resource cost, with the provision for resource trading between campuses. We collected real data traces from three (3) universities over a month and conducted a simulation to evaluate the effectiveness of our resource trading approach over the existing single institution flat rate allocation policy. Our simulation shows that, with CCC and market-based resource trading, the aggregated institutional value for the co-operative increases by 15% and the average wait time for the jobs reduce by 49%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Whether users pay or not is up to their institution. The users’ institution is responsible for paying in kind or paying with cash.
References
The genesis ii project, virginia center for grid research. http://genesis2.virginia.edu/wiki/. Accessed May 2018
Open grid forum. https://www.ogf.org/. Accessed May 2018
Abramson, D., Sosic, R., Giddy, J., Hall, B.: Nimrod: a tool for performing parametrised simulations using distributed workstations. In: Proceedings of the Fourth IEEE International Symposium on HPDC, pp. 112–121. IEEE (1995)
Altmann, J., et al.: GridEcon: a market place for computing resources. In: Altmann, J., Neumann, D., Fahringer, T. (eds.) GECON 2008. LNCS, vol. 5206, pp. 185–196. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-85485-2_15
Bachmann, F., Foster, I., Grimshaw, A., Lifka, D., Riedel, M., Tuecke, S.: XSEDE architecture level 3 decomposition. Version 0.972, June 2013
Berman, F., Wolski, R., Figueira, S., Schopf, J., Shao, G.: Application-level scheduling on distributed heterogeneous networks. In: Proceedings of the 1996 ACM/IEEE Conference on Supercomputing, pp. 39–39. IEEE (1996)
Buyya, R., Abramson, D., Venugopal, S.: The grid economy. Proc. IEEE 93(3), 698–714 (2005)
Buyya, R., Murshed, M.: GridSim: a toolkit for the modeling and simulation of distributed resource management and scheduling for grid computing. Concurr. Comput. Pract. Exp. 14(13–15), 1175–1220 (2002)
Grimshaw, A., et al.: An open grid services architecture primer. Computer 2, 27–34 (2009)
Grimshaw, A., Prodhan, M.A., Thomas, A., Stewart, C., Knepper, R.: Campus compute co-operative (CCC): a service oriented cloud federation. In: 2016 IEEE 12th International Conference on e-Science (e-Science), pp. 1–10. IEEE (2016)
Grimshaw, A.S., Wulf, W.A., et al.: The legion vision of a worldwide virtual computer. Commun. ACM 40(1), 39–45 (1997)
Klusáček, D., Rudová, H.: Alea 2 - job scheduling simulator. In: Proceedings of the 3rd International ICST Conference on Simulation Tools and Techniques (SIMUTools 2010). ICST (2010)
Koeritz, C.: Genesis ii omnibus reference manual. Technical report, Genesis II Group, University of Virginia (2012)
Mell, P., Grance, T.: The nist definition of cloud computing. Commun. ACM 53(6), 50 (2010)
Morgan, M.M., Grimshaw, A.S.: Genesis ii-standards based grid computing. In: Seventh IEEE International Symposium on Cluster Computing and the Grid, CCGRID 2007, pp. 611–618. IEEE (2007)
Padala, P., Harrison, C., Pelfort, N., Jansen, E., Frank, M.P., Chokkareddy, C.: Ocean: the open computation exchange and arbitration network, a market approach to meta computing. In: International Symposium on Parallel and Distributed Computing, p. 185. IEEE (2003)
Prodhan, M.A., Grimshaw, A.: Market-based on demand scheduling (MBoDS) in co-operative grid environment. In: Proceedings of the 2015 XSEDE Conference, p. 26. ACM (2015)
Regev, O., Nisan, N.: The popcorn market. Online markets for computational resources. Decis. Support. Syst. 28(1), 177–189 (2000)
Waldspurger, C.A., Hogg, T., Huberman, B.A., Kephart, J.O., Storn, W.S.: Spawn: a distributed computational economy. IEEE Trans. Softw. Eng. 18(2), 103–117 (1992)
Weitzel, D., Bockelman, B., Fraser, D., Pordes, R., Swanson, D.: Enabling campus grids with open science grid technology. In: Journal of Physics: Conference Series, vol. 331, p. 062025. IOP Publishing (2011)
Wolski, R., Plank, J.S., Bryan, T., Brevik, J.: G-commerce: market formulations controlling resource allocation on the computational grid. In: Proceedings of 15th International Symposium on Parallel and Distributed Processing, 8-pp. IEEE (2001)
Acknowledgments
We would like to thank several people, from Indiana University Craig Stewart and Rich Knepper for leading the Campus Bridging effort, and Matt Allen for gathering the Big Red job files. We would also like to Mark Gardner from Virginia Tech for his continuous support on the project and gathering the VT traces for us. This document was developed with support from National Science Foundation (NSF) grant OCI-1053575. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Prodhan, M.A.T., Grimshaw, A. (2019). An Evaluation of a Market Based Resource Trading in a Multi-campus Compute Co-operative (CCC). In: Coppola, M., Carlini, E., D’Agostino, D., Altmann, J., Bañares, J. (eds) Economics of Grids, Clouds, Systems, and Services. GECON 2018. Lecture Notes in Computer Science(), vol 11113. Springer, Cham. https://doi.org/10.1007/978-3-030-13342-9_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-13342-9_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-13341-2
Online ISBN: 978-3-030-13342-9
eBook Packages: Computer ScienceComputer Science (R0)