Abstract
The processing of MPI-IO operations can be controlled via the MPI API using file hints, which are passed to the MPI library as MPI info objects. A file hint can affect how the MPI library accesses the file on the file system level, it can set buffer sizes, turn special optimizations on and off or whatever parameters the MPI implementation provides. However, experience shows that file hints are rarely used for reasons that will be discussed in the paper. We present a new approach which dynamically determines the optimal setting for file hints related to collective MPI-IO operations. The chosen settings adapt to the actual file access pattern, the topology of the MPI processes and the available memory resources and consider the characteristics of the underlying file system. We evaluate our approach which has been implemented in MPI/SX, NEC’s MPI implementation for the SX series of vector supercomputers.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Ahmad Faraj, X.Y., Lowenthal, D.: STARMPI: Self Tuned Adaptive Routines for MPI Collective Operations. In: Proceedings of the 20th ACM International Conference on Supercomputing, Queensland, Australia (June 2006)
Dongarra, J., Eijkhout, V.: Self-Adapting Numerical Software and Automatic Tuning of Heuristics. In: Sloot, P.M.A., Abramson, D., Bogdanov, A.V., Gorbachev, Y.E., Dongarra, J., Zomaya, A.Y. (eds.) ICCS 2003. LNCS, vol. 2660. Springer, Heidelberg (2003)
Message-Passing Interface Forum. MPI-2.0: Extensions to the Message-Passing Interface, ch. 9. MPI Forum (June 1997)
Nitzberg, B., Lo, V.: Collective buffering: Improving parallel I/O performance. In: Proceedings of the Sixth IEEE International Symposium on High Performance Distributed Computing, pp. 148–157. IEEE Computer Society Press, Los Alamitos (1997)
Norcott, W.D., Capps, D.: Iozone Filesystem Benchmark (March 2006), http://www.iozone.org
Ohtani, A., Aono, H., Tomaru, H.: A File Sharing Method for Storage Area Network and its Performance Verification. NEC Research & Development 44(1), 85–90 (2003)
Prost, J.-P., Treumann, R., Hedges, R., Jia, B., Koniges, A.: MPI-IO/GPFS, an Optimized Implementation of MPI-IO on top of GPFS. In: Proceedings of the 2001 ACM/IEEE conference on Supercomputing, Denver, Association for Computing Machinery (November 2001)
Rabenseifner, R., Koniges, A.E., Prost, J.-P., Hedges, R.: The Parallel Effective I/O Bandwidth Benchmark: b_eff_io, ch. 4. Kogan Page Ltd. (2004)
Thakur, R., Gropp, W., Lusk, E.: Data sieving and collective I/O in ROMIO. In: Proceedings of the Seventh Symposium on the Frontiers of Massively Parallel Computation, pp. 182–189. IEEE Computer Society Press, Los Alamitos (1999)
Thakur, R., Gropp, W., Lusk, E.: On Implementing MPI-IO Portably and with High Performance. In: Proceedings of the Sixth Workshop on Input/Output in Parallel and Distributed Systems, May 1999, pp. 23–32 (1999)
Vadhiyar, S.S., Fagg, G.E., Dongarra, J.: Automatically Tuned Collective Communications. In: Proceedings of the 2000 ACM/IEEE conference on Supercomputing, Dallas, USA (November 2000)
Worringen, J.: Experiment Management and Analysis with perfbase. In: Proceedings of the IEEE International Conference on Cluster Computing, Boston, September 2005. IEEE Computer Society Press, Los Alamitos (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Worringen, J. (2006). Self-adaptive Hints for Collective I/O. In: Mohr, B., Träff, J.L., Worringen, J., Dongarra, J. (eds) Recent Advances in Parallel Virtual Machine and Message Passing Interface. EuroPVM/MPI 2006. Lecture Notes in Computer Science, vol 4192. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11846802_32
Download citation
DOI: https://doi.org/10.1007/11846802_32
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-39110-4
Online ISBN: 978-3-540-39112-8
eBook Packages: Computer ScienceComputer Science (R0)