Abstract
With simulation codes becoming more powerful, using more and more resources, and producing larger and larger data, monitoring or post-processing simulation data in-situ has obvious advantages over the conventional approach of saving to – and reloading data from – the file system. The time it takes to write and then read the data from disk is a significant bottleneck for both the simulation and subsequent post-processing. In order to be able to post-process data as efficiently as possible with minimal disruption to the simulation itself, we have developed a parallel virtual file driver for the HDF5 library which acts as an MPI-IO virtual file layer, allowing the simulation to write in parallel to remotely located distributed shared memory instead of writing to disk.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Hierarchical Data Format (HDF5), http://hdf.ncsa.uiuc.edu/
Abbasi, H., Wolf, M., Eisenhauer, G., Klasky, S., Schwan, K., Zheng, F.: Datastager: scalable data staging services for petascale applications. In: HPDC 2009: Proceedings of the 18th ACM international symposium on High performance distributed computing, pp. 39–48. ACM, New York (2009)
Allen, G., Benger, W., Dramlitsch, T., Goodale, T., Christian Hege, H., Lanfermann, G., Merzky, A., Radke, T., Seidel, E.: Cactus Grid Computing: Review of Current Development (2001)
Borrill, J., Oliker, L., Shalf, J., Shan, H.: Investigation of leading HPC I/O performance using a scientific-application derived benchmark. In: Lumpe, M., Vanderperren, W. (eds.) SC 2007, pp. 1–12. ACM, New York (2007), doi:10.1145/1362622.1362636
Chilan, C.M., Yang, M., Cheng, A., Arber, L.: Parallel I/O Performance Study with HDF5, a Scientific Data Package. Tech. rep., National Center for Supercomputing Applications, University of Illinois, Urbana-Champaign (2006)
Clarke, J.A.: Emulating Shared Memory to Simplify Distributed-Memory Programming. IEEE Comput. Sci. Eng. 4(1), 55–62 (1997)
Clarke, J.A., Mark, E.R.: Enhancements to the eXtensible Data Model and Format (XDMF). In: HPCMP-UGC 2007: Proceedings of the 2007 DoD High Performance Computing Modernization Program Users Group Conference, pp. 322–327. IEEE Computer Society Press, Washington (2007)
Clarke, J.A., Namburu, R.R.: A Generalized Method for One-Way Coupling of CTH and Lagrangian Finite-Element Codes With Complex Structures Using the Interdisciplinary Computing Environment. Tech. rep., US Army Research Laboratory, Aberdeen Proving Ground, Md. (November 2004), ARL-TN-230
Docan, C., Parashar, M., Klasky, S.: DART: a substrate for high speed asynchronous data IO. In: HPDC 2008: Proceedings of the 17th international symposium on High performance distributed computing., pp. 219–220. ACM, New York (2008), doi:10.1145/1383422.1383454
Henderson, A.: ParaView Guide, A Parallel Visualization Application. Kitware Inc. (2005), http://www.paraview.org
Lofstead, J.F., Klasky, S., Schwan, K., Podhorszki, N., Jin, C.: Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS). In: CLADE 2008: Proceedings of the 6th international workshop on Challenges of large applications in distributed environments, pp. 15–24. ACM, New York (2008)
Yang, M., Koziol, Q.: Using collective IO inside a high performance IO software package - HDF5. Tech. rep., National Center for Supercomputing Applications, University of Illinois, Urbana-Champaign (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Soumagne, J., Biddiscombe, J., Clarke, J. (2010). An HDF5 MPI Virtual File Driver for Parallel In-situ Post-processing. In: Keller, R., Gabriel, E., Resch, M., Dongarra, J. (eds) Recent Advances in the Message Passing Interface. EuroMPI 2010. Lecture Notes in Computer Science, vol 6305. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15646-5_7
Download citation
DOI: https://doi.org/10.1007/978-3-642-15646-5_7
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-15645-8
Online ISBN: 978-3-642-15646-5
eBook Packages: Computer ScienceComputer Science (R0)