Abstract
The paper presents an overview of existing workload management systems and diagnoses difficulties in their use. Such systems require a complex configuration, and the process of ordering tasks to be carried out is burdened with many restrictions. A new solution is presented which supports the use of massive distributed computations. The work presents the process of designing, implementing and testing the workload management Shapp library, based on the HTCondor system. It implements a convenient application interface in the form of a dynamically linked library, which extends the capabilities of existing applications with a convenient mechanism allowing the use of massive distributed processing. Recursive computations in tree-like structure are possible using Shapp library.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Daszczuk, W.B., MieĆcicki, J., Grabski, W.: Distributed algorithm for empty vehicles management in personal rapid transit (PRT) network. J. Adv. Transp. 50(4), 608â629 (2016). https://doi.org/10.1002/atr.1365
Daszczuk, W.B.: Discrete event simulation of personal rapid transit (PRT) systems. Autobusy-TEST 17(3), 1302â1310 (2016). arXiv:1705.05237
Sfiligoi, I.: glideinWMSâa generic pilot-based workload management system. J. Phys. Conf. Ser. 119(6), 062044 (2008). https://doi.org/10.1088/1742-6596/119/6/062044
Anderson, D.P.: BOINC: a system for public-resource computing and storage. In: Buyya, R. (ed.) Fifth IEEE/ACM International Workshop on Grid Computing, Pittsburgh, PA, 8 November 2004, pp. 4â10. IEEE (2004). https://doi.org/10.1109/grid.2004.14
Aad, G.: The ATLAS experiment at the CERN large hadron collider. J. Instrum. 3, 407 (2008). https://iopscience.iop.org/article/10.1088/1748-0221/3/08/S08003/meta
MĂ©ndez, B.J.H.: SpaceScience@Home: authentic research projects that use citizen scientists. In: Garmany, C., Gibbs, M.G., Moody, J.W. (eds.) EPO and a Changing World: Creating Linkages and Expanding Partnerships, Chicago, IL 5â7 September 2007, pp. 219â226. ASP Press, San Francisco (2008). http://adsabs.harvard.edu/full/2008ASPC..389..219M
Patoli, M.Z., Gkion, M., Al-Barakati, A., Zhang, W., Newbury, P., White, M.: An open source Grid based render farm for Blender 3D. In: 2009 IEEE/PES Power Systems Conference and Exposition, Seattle, WA, 15â18 March 2009, pp. 1â6. IEEE (2009). https://doi.org/10.1109/psce.2009.4839978
Czarnul, P., Kuchta, J., Matuszek, M.: Parallel computations in the volunteerâbased comcute system. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., WaĆniewski, J. (eds.) International Conference on Parallel Processing and Applied Mathematics, PPAM 2013, Warsaw, Poland, 8â11 September 2013. LNCS, vol. 8384, pp. 261â271. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-55224-3_25
HTCondor. https://research.cs.wisc.edu/htcondor/
Pool, M.: distcc, a fast free distributed compiler. In: The Linux Conference, Las Vegas, NV, June 2004, pp. 1879â1885 (2004). https://fossies.org/linux/distcc/doc/web/distcc-lca-2004.pdf
Zhang, W.: Linux virtual server for scalable network services. In: Ottawa Linux Symposium, Ottawa, Canada, 22 July 2000, pp. 1â10 (2000). www.linuxvirtualserver.org/ols/lvs.pdf
Owsiany, M.: High availability in Linux System (in Polish: Wysoka dostÄpnoĆÄ w systemie Linux) (2003). http://marcin.owsiany.pl/studia/inf-4_rok/swn/referat.pdf
Cassen, A.: Keepalived. http://www.keepalived.org/
Nurmi, D., Wolski, R., Grzegorczyk, C., Obertelli, G., Soman, S., Youseff, L., Zagorodnov, D.: The eucalyptus open-source cloud-computing system. In: Cappello, F., Wang, C.-L., Buyya, R. (eds.) 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, Shanghai, China, 18â21 May 2009, pp. 124â131. IEEE (2009). https://doi.org/10.1109/ccgrid.2009.93
Raman, R., Livny, M., Solomon, M.: Matchmaking: distributed resource management for high throughput computing. In: Seventh International Symposium on High Performance Distributed Computing, Chicago, IL, 31 July 1998, pp. 140â146. IEEE (1998). https://doi.org/10.1109/hpdc.1998.709966
Santos, A., Almeida, F., Blanco, V.: Lightweight web services for high performance computing. In: Oquendo, F. (ed.) European Conference on Software Architecture, Aranjuez, Spain, 24â26 September 2007. LNCS, vol. 4758, pp. 225â236. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-75132-8_18
Raicu, I., Foster, I.T., Zhao, Y.: Many-task computing for grids and supercomputers. In: Workshop on Many-Task Computing on Grids and Supercomputers, Austin, TX, 17 November 2008, pp. 1â11. IEEE (2008). https://doi.org/10.1109/mtags.2008.4777912
Satyanarayana, K.C., Gani, R., Abildskov, J.: Polymer property modeling using grid technology for design of structured products. Fluid Phase Equilib. 261(1â2), 58â63 (2007). https://doi.org/10.1016/j.fluid.2007.07.058
Zakrzewska, K., Bouvier, B., Michon, A., Blanchet, C., Lavery, R.: ProteinâDNA binding specificity: a grid-enabled computational approach applied to single and multiple protein assemblies. Phys. Chem. Chem. Phys. 11(45), 10712 (2009). https://doi.org/10.1039/b910888m
Bird, I.: Computing for the Large Hadron Collider. Annu. Rev. Nucl. Part. Sci. 61(1), 99â118 (2011). https://doi.org/10.1146/annurev-nucl-102010-130059
Raicu, I., Foster, I., Zhao, Y., Szalay, A., Little, P., Moretti, C.M., Chaudhary, A., Thain, D.: Towards data intensive many-task computing. In: Kosar, T. (ed.) Data Intensive Distributed Computing: Challenges and Solutions for Large-Scale Information Management, pp. 28â73. IGI Global (2012). https://doi.org/10.4018/978-1-61520-971-2.ch002
Nishimura, H., Timossi, C.: Mono for cross-platform control system environment. In: 6th International Workshop on Personal Computers and Particle Accelerator Controls, Newport News, VA, 24â27 September 2006 (2006). https://escholarship.org/uc/item/3hn297s0
Kolici, V., Herrero, A., Xhafa, F.: On the performance of oracle grid engine queuing system for computing intensive applications. J. Inf. Process. Syst. 10(4), 491â502 (2014). https://doi.org/10.3745/JIPS.01.0004
Foster, I., Kesselman, C.: Globus: a metacomputing infrastructure toolkit. Int. J. Supercomput. Appl. High Perform. Comput. 11(2), 115â128 (1997). https://doi.org/10.1177/109434209701100205
Krieger, M.T., Torreno, O., Trelles, O., KranzlmĂŒller, D.: Building an open source cloud environment with auto-scaling resources for executing bioinformatics and biomedical workflows. Futur. Gener. Comput. Syst. 67, 329â340 (2017). https://doi.org/10.1016/j.future.2016.02.008
Yoo, A.B., Jette, M.A., Grondona, M.: SLURM: simple linux utility for resource management. In: Feitelson, D., Rudolph, L., Schwiegelshohn, U. (eds.) Job Scheduling Strategies for Parallel Processing, pp. 44â60. Springer, Heidelberg (2003). https://doi.org/10.1007/10968987_3
Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the Condor experience. Concurr. Comput. Pract. Exp. 17(2â4), 323â356 (2005). https://doi.org/10.1002/cpe.938
Asagba, P., Ogheneovo, E.: Qualities of grid computing that can last for ages. J. Appl. Sci. Environ. Manag. 12(4) (2010). https://doi.org/10.4314/jasem.v12i4.55218
Georgatos, F., Gkamas, V., Ilias, A., Kouretis, G., Varvarigos, E.: A grid-enabled CPU scavenging architecture and a case study of its use in the greek school network. J. Grid Comput. 8(1), 61â75 (2010). https://doi.org/10.1007/s10723-009-9143-2
Galecki, T.: The environment of support of a massive distributed computing (in Polish: Srodowisko wsparcia masowego przetwarzania rozproszonego), BSc thesis, Warsaw University of Technology, Institute of Computer Science, 50p. (2019). http://repo.bg.pw.edu.pl/index.php/pl/r#/info/bachelor/WUTcac04f4e732f434590a18a4b4d6fcf68/?r=diploma&tab=&lang=pl
Ossher, H., Kaplan, M., Harrison, W., Katz, A., Kruskal, V.: Subject-oriented composition rules. ACM SIGPLAN Not. 30(10), 235â250 (1995). https://doi.org/10.1145/217839.217864
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
GaĆecki, T., Daszczuk, W.B. (2019). Shapp: Workload Management System for Massive Distributed Calculations. In: Silhavy, R. (eds) Software Engineering Methods in Intelligent Algorithms. CSOC 2019. Advances in Intelligent Systems and Computing, vol 984. Springer, Cham. https://doi.org/10.1007/978-3-030-19807-7_31
Download citation
DOI: https://doi.org/10.1007/978-3-030-19807-7_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-19806-0
Online ISBN: 978-3-030-19807-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)