Journal of Grid Computing

, Volume 14, Issue 4, pp 687–703 | Cite as

A Meta-Brokering Framework for Science Gateways

  • Krisztian Karoczkai
  • Attila Kertesz
  • Peter Kacsuk


Recently scientific communities produce a growing number of computation-intensive applications, which calls for the interoperation of distributed infrastructures including Clouds, Grids and private clusters. The European SHIWA and ER-flow projects have enabled the combination of heterogeneous scientific workflows, and their execution in a large-scale system consisting of multiple Distributed Computing Infrastructures. One of the resource management challenges of these projects is called parameter study job scheduling. A parameter study job of a workflow generally has a large number of input files to be consumed by independent job instances. In this paper we propose a meta-brokering framework for science gateways to support the execution of such workflows. In order to cope with the high uncertainty and unpredictable load of the utilized distributed infrastructures, we introduce the so called resource priority services. These tools are capable of determining and dynamically updating priorities of the available infrastructures to be selected for job instances. Our evaluations show that this approach implies an efficient distribution of job instances among the available computing resources resulting in shorter makespan for parameter study workflows.


Meta-brokering Interoperability Distributed infrastructures Workflows 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Wiggins, A.: Success-Abandonment-Classification workflow at myExperiment. Online: (2012)
  2. 2.
    SHaring Interoperable Workflows for large-scale scientific simulations on Available DCIs (SHIWA) EU FP7 project.Online: (2012)
  3. 3.
    Building a European Research Community through Interoperable Workflows and Data (ER-flow) Eu FP7 project. Online: (2013)
  4. 4.
    Rubio-Montero, A.J., Huedo, E., Castejon, F., Mayo-Garcia, R.: GWpilot: Enabling multi-level scheduling in distributed infrastructures with GridWay and pilot jobs. Fut. Gener. Comput. Syst. 45, 25–52 (2015)CrossRefGoogle Scholar
  5. 5.
    Hirales-Carbajal, A., Tchernykh, A., Yahyapour, R., Gonzalez-Garcia, J.L., Roblitz, T., Ramirez-Alcaraz, J.M.: Multiple workflow scheduling strategies with user run time estimates on a grid. Journal of Grid Computing (2012)Google Scholar
  6. 6.
    Oprescu, A., Kielmann, T.: Bag-of-tasks scheduling under budget constraints. CloudCom, 351–359 (2010)Google Scholar
  7. 7.
    Silberstein, M., Sharov, A., Geiger, D., Schuster, A.: GridBot, execution of bags of tasks in multiple grids. In: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis (SC ’09) (2009)Google Scholar
  8. 8.
    Kacsuk, P., Farkas, Z., Kozlovszky, M., Hermann, G., Balasko, Á., Karóczkai, K., Márton, I.: WS-PGRADE/gUSE generic DCI gateway framework for a large variety of user communities. J. Grid Comput. 10(4), 601–630 (2012)CrossRefGoogle Scholar
  9. 9.
    Lee, C.B., Schwartzman, Y., Hardy, J., Snavely, A.: Are user runtime estimates inherently inaccurate? Springer LNCS 3277, 253–263 (2005)Google Scholar
  10. 10.
    Fan, Y., Pamidighantam, S., Smith, W.: Incorporating job predictions into the SEAGrid science gateway. ACM, NY, USA (2014)Google Scholar
  11. 11.
    Resource Priority Service for gUSE. Online: (2015)
  12. 12.
    Ullman, J.D.: NP-complete scheduling problems. J. Comput. Syst. Sci. 10(3), 384–393 (1975)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Schwiegelshohn, U., Tchernykh, A., Yahyapour, R.: Online scheduling in grids. 22nd IEEE International Symposium on Parallel and Distributed Processing (IPDPS 2008), pp. 1–10 (2008)Google Scholar
  14. 14.
    Casanova, H., et al.: Heuristics for scheduling parameter sweep applications in grid environments. Heterogeneous Computing Workshop, 2000. (HCW 2000) Proceedings. 9th. IEEE (2000)Google Scholar
  15. 15.
    Muthucumaru, M., Ali, S., Siegal, H.J., Hensgen, D., Freund, R.F.: Dynamic matching and scheduling of a class of independent tasks onto heterogeneous computing systems. In: Heterogeneous Computing Workshop, 1999. (HCW’99) Proceedings, pp 30–44. IEEE (1999)Google Scholar
  16. 16.
    Lucas-Simarro, J.L., Moreno-Vozmediano, R., Montero, R.S., Llorente, I.M.: Scheduling strategies for optimal service deployment across multiple clouds. Future Generation Computer Systems,  10.1016/j.future.2012.01.007 (2012)
  17. 17.
    Kertesz, A., Kacsuk, P.: GMBS: A new middleware service for making grids interoperable. Fut. Gener. Comput. Syst. 16, 542–553 (2010)CrossRefGoogle Scholar
  18. 18.
    Assuncao, M.D., Buyya, R., Venugopal, S.: InterGrid: A case for internetworking islands of grids. Concurrency and Computation: Practice and Experience (CCPE) (2007)Google Scholar
  19. 19.
    Buyya, R., Ranjan, R., Calheiros, R.N.: InterCloud: Utility-oriented federation of cloud computing environments for scaling of application services. Lect. Notes Comput. Sci. Algorithm. Architectures Parallel Process. 6081 (2010)Google Scholar
  20. 20.
    Buyya, R., Ranjan, R.: Special section: Federated resource management in grid and cloud computing systems. Fut. Gener. Comput. Syst. 26, 1189–1191 (2010)CrossRefGoogle Scholar
  21. 21.
    Kertesz, A., Maros, G., Dombi, J.D.: Multi-job meta-brokering in distributed computing infrastructures using pliant logic. In: Proceedings of the 22th Euromicro International Conference on Parallel, Distributed and Network-Based Computing (PDP’14), pp 138–145. IEEE CS, Turin, Italy (2014)Google Scholar
  22. 22.
    Karoczkai, K., Kertesz, A., Kacsuk, P.: Brokering solution for science gateways using multiple distributed computing infrastructures. In: 7th International Workshop on Science Gateways (IWSG). doi: 10.1109/IWSG.2015.12, pp 28–33, 3–5 (2015)
  23. 23.
    Bacso, G., Kis, T., Visegradi, A., Kertesz, A., Nemeth, Z.S.: A set of successive job allocation models in distributed computing infrastructures (2015). doi:doi: 10.1007/s10723-015-9347-6  10.1007/s10723-015-9347-6
  24. 24.
  25. 25.
    Korkhov, V., Krefting, D., Kukla, T., et al.: Exploring workflow interoperability for neuroimage analysis on the SHIWA platform. J Grid Comput. 11, 505 (2013). doi: 10.1007/s10723-013-9262-7 CrossRefGoogle Scholar
  26. 26.
    Kiss, T.: Science gateways for the broader take-up of distributed computing infrastructures. J Grid Comput. 10, 599 (2012). doi: 10.1007/s10723-012-9245-0 CrossRefGoogle Scholar
  27. 27.
    Liu, J., Pacitti, E., Valduriez, P., et al.: A survey of data-intensive scientific workflow management. J Grid Comput. 13, 457 (2015). doi: 10.1007/s10723-015-9329-8 CrossRefGoogle Scholar
  28. 28.
    Kacsuk, P. (ed.): Science Gateways for Distributed Computing Infrastructures: Development Framework and Exploitation by Scientific User Communities, Springer. pp. 301 (2014)Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2016

Authors and Affiliations

  • Krisztian Karoczkai
    • 1
  • Attila Kertesz
    • 1
  • Peter Kacsuk
    • 1
  1. 1.Laboratory of Parallel and Distributed SystemsMTA SZTAKIBudapestHungary

Personalised recommendations