Abstract
Current scientific workflows consist of generally several components either integrated in situ or as completely independent, asynchronous components using centralized storage as an interface. Neither of these approaches are likely to scale well into Exascale. Instead, separate applications and services will be launched using online communication to link these components of the scientific discovery process. Our experiences with coupling multiple, independent MPI applications, each with separate processing phases, exposes limitations preventing use of some of the optimized mechanisms within the MPI standard. In this regard, we have identified two shortcomings with current MPI implementations. First, MPI intercommunicators offer a mechanism to communicate across application boundaries, but do not address the impact this operating mode has on possible programming models for each separate application. Second, MPI_Probe offers a way to interleave both local messaging and remote messages, but has limitations as MPI_Bcast and other collective calls are not supported by MPI_Probe thus limiting use of optimize collective calls in this operating mode.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Docan, C., Parashar, M., Klasky, S.: DataSpaces: An interaction and coordination framework for coupled simulation workflows. In: HPDC 2010: Proceedings of the 18th International Symposium on High Performance Distributed Computing (2010)
Droegemeier, K., Chandrasekar, V., Clark, R., Gannon, D., Graves, S., Joseph, E., Ramamurthy, M., Wilhelmson, R., Brewster, K., Domenico, B., Leyton, T., Morris, V., Murray, D., Plale, B., Ramachandran, R., Reed, D., Rushing, J., Weber, D., Wilson, A., Xue, M., Yalda, S.: Linked environments for atmospheric discovery (lead): A cyberinfrastructure for mesoscale meteorology research and education. In: 20th Conf. on Interactive Information Processing Systems for Meteorology, Oceanography, and Hydrology, Seattle, WA (January 2004), http://www.cs.indiana.edu/dde/papers/droegemeierIIPS2004.pdf
Hertel Jr., E.S., Bell, R.L., Elrick, M.G., Farnsworth, A.V., Kerley, G.I., McGlaun, J.M., Petney, S.V., Silling, S.A., Taylor, P.A., Yarrington, L.: CTH: A software family for multi-dimensional shock physics analysis. In: Brun, R., Dumitrescu, L.D. (eds.) Proceedings of the 19th International Symposium on Shock Physics, Marseille, France, vol. 1, pp. 377–382 (July 1993), http://sherpa.sandia.gov/9231home/pdfpapers/issw.pdf
Ludäscher, B., Altintas, I., Berkley, C., Higgins, D., Jaeger, E., Jones, M., Lee, E.A., Tao, J., Zhao, Y.: Scientific workflow management and the kepler system: Research articles. Concurr. Comput.: Pract. Exper. 18(10), 1039–1065 (2006)
Malewicz, G., Foster, I., Rosenberg, A., Wilde, M.: A tool for prioritizing DAGMan jobs and its evaluation. In: 2006 15th IEEE International Symposium on High Performance Distributed Computing, pp. 156–168 (2006)
Moreland, K., Lepage, D., Koller, D., Humphreys, G.: Remote rendering for ultrascale data. Journal of Physics: Conference Series 125(1), 012096 (2008), http://stacks.iop.org/1742-6596/125/i=1/a=012096
Mullender, S.J., Leslie, I.M., McAuley, D.: Operating-system support for distributed multimedia. In: Proceedings of the 1994 Summer USENIX Technical Conference, pp. 209–219 (1994)
Oldfield, R.A., Widener, P., Maccabe, A.B., Ward, L., Kordenbrock, T.: Efficient data-movement for lightweight I/O. In: Proceedings of the 2006 International Workshop on High Performance I/O Techniques and Deployment of Very Large Scale I/O Systems, Barcelona, Spain (September 2006), http://doi.ieeecomputersociety.org/10.1109/CLUSTR.2006.311897
Riedel, M., Eickermann, T., Habbinga, S., Frings, W., Gibbon, P., Mallmann, D., Wolf, F., Streit, A., Lippert, T., Schiffmann, W., Ernst, A., Spurzem, R., Nagel, W.: Computational steering and online visualization of scientific applications on large-scale hpc systems within e-science infrastructures. In: IEEE International Conference on e-Science and Grid Computing, pp. 483–490 (December 2007)
UCAR: Local data manager, http://www.unidata.ucar.edu/software/ldm
Wozniak, J.M., Latham, R., Lang, S., Son, S.W., Ross, R.: C-mpi: A dht implementation for grid and hpc environments. In: EuroMPI (2009)
Zheng, F., Abbasi, H., Docan, C., Lofstead, J., Klasky, S., Liu, Q., Parashar, M., Podhorszki, N., Schwan, K., Wolf, M.: PreDatA- preparatory data analytics on Peta-Scale machines. In: Proceedings of 24th IEEE International Parallel and Distributed Processing Symposium, Atlanta, Georgia (April 2010)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Lofstead, J., Dayal, J. (2012). Extending MPI to Better Support Multi-application Interaction. In: Träff, J.L., Benkner, S., Dongarra, J.J. (eds) Recent Advances in the Message Passing Interface. EuroMPI 2012. Lecture Notes in Computer Science, vol 7490. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33518-1_32
Download citation
DOI: https://doi.org/10.1007/978-3-642-33518-1_32
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-33517-4
Online ISBN: 978-3-642-33518-1
eBook Packages: Computer ScienceComputer Science (R0)