Abstract
As Earth systems simulations grow in sophistication and complexity, developers need to be concerned not only about computational constraints but also about storage resources. When dealing with the large data sets produced by high-resolution simulations, the storage subsystem must have both the capacity to store the data and the capability to access that data efficiently. The computing facilities of today and tomorrow will provide increasing computational power, but storage capabilities are not increasing at a corresponding rate. One challenge for all applications will be how to best manage and mitigate the growing input/output (I/O) bottleneck.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Actually NFS isn’t a file system at all, but rather a file system protocol; but for our purposes in this section it is easiest to just think of it as a file system.
References
Braam PJ (2003) The lustre storage architecture. Technical Report, Cluster File Systems, Inc., http://lustre.org/docs/lustre.pdf
Ching A, Choudhary A, Coloma K, Liao W, Ross R, Gropp W (2003a) Noncontiguous i/o accesses through MPI-IO. In: Proceedings of the third IEEE/ACM international symposium on cluster computing and the grid (CCGrid2003)
Ching A, Choudhary A, Liao W, Ross R, Gropp W (2002) Noncontiguous i/o through pvfs. In: Proceedings of the 2002 IEEE international conference on cluster computing
Ching A, Choudhary A, Liao W, Ross R, Gropp W (2003a) Efficient structured data access in parallel file systems. In: Proceedings of cluster 2003, Hong Kong
Gropp W, Lusk E, Thakur R (1999) Using MPI-2: Advanced features of the message-passing interface. MIT Press, Cambridge http://mitpress.mit.edu/book-home.tcl?isbn=0262571331
Li J, Keng Liao W, Choudhary A, Ross R, Thakur R, Gropp W, Latham R, Siegel A, Gallagher B, Zingale M (2003) Parallel netCDF: A high-performance scientific I/O interface. In: Proceedings of SC2003: high performance networking and computing, IEEE Computer Society Press, Phoenix, AZ http://www.sc-conference.org/sc2003/paperpdfs/pap258.pdf
Ma X, Winslett M, Lee J, Yu S (2003) Improving MPI-IO output performance with active buffering plus threads. In: Proceedings of the 2003 international parallel and distributed processing symposium, IEEE, pp 10
Nagle D, Serenyi D, Matthews A (2004) The panasas activescale storage cluster: Delivering scalable high bandwidth storage. In: SC ’04: Proceedings of the 2004 ACM/IEEE conference on supercomputing, IEEE Computer Society, Washington, DC, USA, p 53, http://dx.doi.org/10.1109/SC.2004.57
Prost JP, Treumann R, Hedges R, Jia B, Koniges A (2001) MPI-IO/GPFS, an optimized implementation of MPI-IO on top of GPFS. In: Proceedings of SC2001
PVFS development team (2008) The PVFS parallel file system. http://www.pvfs.org/
Rew RK, Hartnett EJ, Caron J (2006) NetCDF-4: Software implementing an enhanced data model for the geosciences. In: 22nd international conference on interactive information processing systems for meteorology, oceanography and hydrology, AMS
Schmuck F, Haskin R (2002) GPFS: A shared-disk file system for large computing clusters. In: First USENIX conference on File and Storage Technologies (FAST’02), Monterey, CA
Shepard L, Eppe E (2006) SGI infinite storage shared filesystem CXFS: A high-performance, Multi-OS filesystem from SGI
Thakur R, Gropp W, Lusk E (1999) Data sieving and collective I/O in ROMIO. In: Proceedings of the seventh symposium on the frontiers of massively parallel computation, IEEE Computer Society Press, pp 182–189, http://www.mcs.anl.gov/ thakur/papers/romio-coll.ps
The HDF Group (2008) HDF5. http://www.hdfgroup.org
The MPI Forum (1997) MPI-2: extensions to the message-passing interface. The MPI Forum, http://www.mpi-forum.org/docs/docs.html
Yu H, Sahoo RK, Howson C, Almasi G, Castanos JG, Gupta M, Moreira JE, Parker JJ, Engelsiepen TE, Ross R, Thakur R, Latham R, Gropp WD (2006) High performance file I/O for the bluegene/l supercomputer. In: Proceedings of the 12th international symposium on high-performance computer architecture (HPCA-12). http://www.mcs.anl.gov/ thakur/papers/bgl-io.pdf
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2013 The Author(s)
About this chapter
Cite this chapter
Latham, R., Ross, R. (2013). Parallel I/O Basics. In: Earth System Modelling - Volume 4. SpringerBriefs in Earth System Sciences. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-36464-8_2
Download citation
DOI: https://doi.org/10.1007/978-3-642-36464-8_2
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-36463-1
Online ISBN: 978-3-642-36464-8
eBook Packages: Earth and Environmental ScienceEarth and Environmental Science (R0)