Abstract
The pipeline software architecture pattern is commonly used in many application domains to structure a software system. A pipeline comprises a sequence of processing steps that progressively transform data to some desired outputs. As pipeline-based systems are required to handle increasingly large volumes of data and provide high throughput services, simple scripting-based technologies that have traditionally been used for constructing pipelines do not scale. In this paper we describe the MeDICI Integration Framework (MIF), which is specifically designed for building flexible, efficient and scalable pipelines that exploit distributed services as elements of the pipeline. We explain the core runtime and development infrastructures that MIF provides, and demonstrate how MIF has been used in two complex applications to improve performance and modifiability.
Chapter PDF
References
Buschmann, F., Meunier, R., Rohnert, H., Sommerlad, P., Stal, M.: Pattern-Oriented Software Architecture: A System of Patterns, vol. 1. Wiley, Chichester (2009)
Yang, X., Bruin, R.P., Dove, M.T.: Developing an End-to-End Scientific Workflow. Computing in Science and Engineering, 52–61 (May/June 2010)
Shah, A.R., Singhal, M., Gibson, T.D., Sivaramakrishnan, C., Waters, K.M., Gorton, I.: An extensible, scalable architecture for managing bioinformatics data and analysis. In: IEEE 4th International Conference on e-Science, Indianapolis, Indiana, December 7-12, pp. 190–197. IEEE Computer Society, Los Alamitos (2008)
Gorton, I., Huang, Z., Chen, Y., Kalahar, B., Jin, S., Chavarria-Miranda, D., Baxter, D., Feo, J.: A High-Performance Hybrid Computing Approach to Massive Contingency Analysis in the Power Grid. In: Fifth IEEE International Conference on e-Science and Grid Computing, pp. 277–283. IEEE, Los Alamitos (2009)
Chase, J.M., Gorton, I., Sivaramakrishnan, C., Almquist, J.P., Wynne, A.S., Chin, G., Critchlow, T.J.: Kepler + MeDICi - Service-Oriented Scientific Workflow Applications. In: 2009 IEEE Congress on Services - Part I (Services-1 2009), pp. 275–282. IEEE, Los Alamitos (2009)
Barker, A., van Hemert, J.: Scientific Workflow: A Survey and Research Directions. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Wasniewski, J. (eds.) PPAM 2007. LNCS, vol. 4967, pp. 746–753. Springer, Heidelberg (2008)
Kiebel, G.R., Auberry, K.J., Jaitly, N., Clark, D., Monroe, M.E., Peterson, E.S., Tolic, N., Anderson, G.A., Smith, R.D.: PRISM: A Data Management System for High-Throughput Proteomics. Proteomics 6(6), 1783–1790 (2006)
Best, D.M., Bohn, S., Love, D., Wynne, A., Pike, W.A.: Real-time visualization of network behaviors for situational awareness. In: Proceedings of the Seventh international Symposium on Visualization For Cyber Security, VizSec 2010, Ottawa, Ontario, Canada, September 14, pp. 79–90. ACM, New York (2010)
Wilde, M., Foster, I., Iskra, K., Beckman, P., Zhang, Z., Espinosa, A., Hategan, M., Clifford, B., Raicu, I.: Parallel Scripting for Applications at the Petascale and Beyond. Computer 42(11) (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Gorton, I., Wynne, A., Liu, Y. (2011). Engineering High Performance Service-Oriented Pipeline Applications with MeDICi. In: Maximilien, E.M., Rossi, G., Yuan, ST., Ludwig, H., Fantinato, M. (eds) Service-Oriented Computing. ICSOC 2010. Lecture Notes in Computer Science, vol 6568. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-19394-1_10
Download citation
DOI: https://doi.org/10.1007/978-3-642-19394-1_10
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-19393-4
Online ISBN: 978-3-642-19394-1
eBook Packages: Computer ScienceComputer Science (R0)