Advertisement

Adaptation and Deployment of PanDA Task Management System for a Private Cloud Infrastructure

  • Oleg IakushkinEmail author
  • Daniil Malevanniy
  • Alexander Bogdanov
  • Olga Sedova
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10408)

Abstract

Management of computational infrastructure is a complicated task which, often employs user workloads delivery across multiple clusters. Criteria for such tasks distribution may vary: priority, transport costs, utilization of data, node capabilities, etc.

Such process happens to tasks devoted to the simulation and analysis of the results of high-energy physics experiments at CERN. For task distribution on massive data streams obtained during ATLAS experiment, “Production ANd Distributed Analysis system” (PanDA) was developed. It performs management of workloads delivery and execution in a geographically distributed cluster environment. This paper is devoted to the deployment of PanDA server in a private cluster setting.

This paper presents architecture and its implementation that allows, to run and embed PanDA system into existing computational solutions. It consists of a container, that isolates PanDA server its dependencies and environment from other system processes and an embedded Web interface which simplifies task management for end-users. In other words, our approach is focused on PanDA system deployment speed up by means of security layer simplification, containerization and stateless client web service implementation. System was tested on a heterogeneous geographically distributed Azure cloud nodes.

Keywords

Grid computing User interface API Virtualization Deploying 

Notes

Acknowledgments.

This research was partially supported by Russian Foundation for Basic Research grant (project no. 16-07-01113). Microsoft Azure for Research Award (http://research.microsoft.com/en-us/projects/azure/) as well as the resource center “Computer Center of SPbU” (http://cc.spbu.ru/en) provided computing resources. The authors would like to acknowledge the Reviewers for the valuable recommendations that helped in the improvement of this paper.

References

  1. 1.
    Bogdanov, A.V., Degtyarev, A., Stankova, E.N.: Example of a potential grid technology application in shipbuilding. In: 2007 International Conference on Computational Science and Its Applications (ICCSA 2007), pp. 3–8 (2007)Google Scholar
  2. 2.
    Borodin, M., De, K., Garcia, J., Golubkov, D., Klimentov, A., Maeno, T., Vaniachine, A., et al.: Scaling up ATLAS production system for the LHC run 2 and beyond: project ProdSys2. J. Phys. Conf. Ser. 664, 062005 (2015). IOP PublishingCrossRefGoogle Scholar
  3. 3.
    De, K., Klimentov, A., Maeno, T., Nilsson, P., Oleynik, D., Panitkin, S., Petrosyan, A., Schovancova, J., Vaniachine, A., Wenaus, T.: The future of panda in atlas distributed computing. J. Phys. Conf. Ser. 664, 062035 (2015). IOP PublishingCrossRefGoogle Scholar
  4. 4.
    Dworak, A., Ehm, F., Charrue, P., Sliwinski, W.: The new cern controls middleware. J. Phys. Conf. Ser. 396, 012017 (2012). IOP PublishingCrossRefGoogle Scholar
  5. 5.
    Gankevich, I., Gaiduchok, V., Gushchanskiy, D., Tipikin, Y., Korkhov, V., Degtyarev, A., Bogdanov, A., Zolotarev, V.: Virtual private supercomputer: design and evaluation. In: Ninth International Conference on Computer Science and Information Technologies Revised Selected Papers, pp. 1–6 (2013)Google Scholar
  6. 6.
    Gankevich, I., Korkhov, V., Balyan, S., Gaiduchok, V., Gushchanskiy, D., Tipikin, Y., Degtyarev, A., Bogdanov, A.: Constructing virtual private supercomputer using virtualization and cloud technologies. In: Murgante, B., et al. (eds.) ICCSA 2014. LNCS, vol. 8584, pp. 341–354. Springer, Cham (2014). doi: 10.1007/978-3-319-09153-2_26 Google Scholar
  7. 7.
    Grishkin, V., Iakushkin, O.: Middleware transport architecture monitoring: topology service. In: 2014 20th International Workshop on Beam Dynamics and Optimization (BDO), pp. 1–2 (2014)Google Scholar
  8. 8.
    Iakushkin, O.: Cloud middleware combining the functionalities of message passing and scaling control. In: EPJ Web of Conferences, vol. 108 (2016)Google Scholar
  9. 9.
    Iakushkin, O., Grishkin, V.: Messaging middleware for cloud applications: extending brokerless approach. In: 2014 2nd International Conference on Emission Electronics (ICEE), pp. 1–4 (2014)Google Scholar
  10. 10.
    Iakushkin, O., Sedova, O., Valery, G.: Application control and horizontal scaling in modern cloud middleware. In: Gavrilova, M.L., Tan, C.J.K. (eds.) Transactions on Computational Science XXVII. LNCS, vol. 9570, pp. 81–96. Springer, Heidelberg (2016). doi: 10.1007/978-3-662-50412-3_6 CrossRefGoogle Scholar
  11. 11.
    Iakushkin, O., Grishkin, V.: Unification of control in p2p communication middleware: towards complex messaging patterns. In: AIP Conference Proceedings, vol. 1648, no. 1, p. 040004 (2015)Google Scholar
  12. 12.
    Iakushkin, O., Shichkina, Y., Sedova, O.: Petri nets for modelling of message passing middleware in cloud computing environments. In: Gervasi, O., et al. (eds.) ICCSA 2016. LNCS, vol. 9787, pp. 390–402. Springer, Cham (2016). doi: 10.1007/978-3-319-42108-7_30 CrossRefGoogle Scholar
  13. 13.
    Johnston, W.E., Dart, E., Ernst, M., Tierney, B.: Enabling high throughput in widely distributed data management and analysis systems: lessons from the LHC. In: TERENA Networking Conference (TNC) (2013)Google Scholar
  14. 14.
    Klimentov, A., Buncic, P., De, K., Jha, S., Maeno, T., Mount, R., Nilsson, P., Oleynik, D., Panitkin, S., Petrosyan, A., et al.: Next generation workload management system for big data on heterogeneous distributed computing. J. Phys. Conf. Ser. 608, 012040 (2015). IOP PublishingGoogle Scholar
  15. 15.
    Korenkov, V., Pelevanyuk, I., Zrelov, P., Tsaregorodtsev, A.: Accessing distributed computing resources by scientific communities using dirac services (2016)Google Scholar
  16. 16.
    Maeno, T., De, K., Klimentov, A., Nilsson, P., Oleynik, D., Panitkin, S., Petrosyan, A., Schovancova, J., Vaniachine, A., Wenaus, T., et al.: Evolution of the atlas panda workload management system for exascale computational science. J. Phys. Conf. Ser. 513, 032062 (2014). IOP PublishingGoogle Scholar
  17. 17.
    Maeno, T., De, K., Panitkin, S.: Pd2p: Panda dynamic data placement for atlas. J. Phys. Conf. Ser. 396, 032070 (2012). IOP PublishingCrossRefGoogle Scholar
  18. 18.
    Maeno, T.: Panda: distributed production and distributed analysis system for atlas. J. Phys. Conf. Ser. 119, 062036 (2008). IOP PublishingCrossRefGoogle Scholar
  19. 19.
    Maeno, T., De, K., Wenaus, T., Nilsson, P., Stewart, G., Walker, R., Stradling, A., Caballero, J., Potekhin, M., Smith, D., et al.: Overview of atlas panda workload management. J. Phys. Conf. Ser. 331, 072024 (2011). IOP PublishingCrossRefGoogle Scholar
  20. 20.
    Shichkina, Y., Degtyarev, A., Gushchanskiy, D., Iakushkin, O.: Application of optimization of parallel algorithms to queries in relational databases. In: Gervasi, O., et al. (eds.) ICCSA 2016. LNCS, vol. 9787, pp. 366–378. Springer, Cham (2016). doi: 10.1007/978-3-319-42108-7_28 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Oleg Iakushkin
    • 1
    Email author
  • Daniil Malevanniy
    • 1
  • Alexander Bogdanov
    • 1
  • Olga Sedova
    • 1
  1. 1.Saint-Petersburg State UniversitySt. PetersburgRussia

Personalised recommendations