Hint Controlled Distribution with Parallel File Systems

  • Hipolito Vasquez Lucas
  • Thomas Ludwig
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3666)


The performance of scientific parallel programs with high file-I/O-activity running on top of cluster computers strongly depends on the qualitative and quantitative characteristics of the requested I/O-accesses. It also depends on the corresponding mechanisms and policies being used at the parallel file system level. This paper presents the motivation and design of a set of MPI-IO-hints. These hints are used to select the distribution function with which a parallel file system manipulates an opened file. The implementation of a new physical distribution function called varstrip_dist is also presented in this article. This function is proposed based upon spatial characteristics presented by I/O-access patterns observed at the application level.


Access Pattern Metadata Server Request Size Strip Size Parallel File System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Patterson, D.A., Chen, P.M.: Storage Performance - Metrics and Benchmarks (1998),
  2. 2.
    Patterson, D.A., Chen, P.M.: Maximizing Performance in a Striped Disk Array. In: Proc. 17th Annual Symposium on Computer Architecture (17th ISCA 1990), Computer Architecture News, pp. 322–331 (1990)Google Scholar
  3. 3.
    Hsu, W.W., Smith, A.J.: Characteristics of I/O traffic in personal computer and server workloads. IBM Syst. J. 42, 347–372 (2003)CrossRefGoogle Scholar
  4. 4.
    Hsu, W.W., Smith, A.J.: The performance impact of I/O optimizations and disk improvements. IBM Journal of Research and Development 48, 255–289 (2004)CrossRefGoogle Scholar
  5. 5.
    Sterling, T.: An Overview of Cluster Computing. Beowulf Cluster Computing with Linux., pp. 15–29 (2002)Google Scholar
  6. 6.
  7. 7.
  8. 8.
    Ligon, W.B., Ross, R.B.: Implementation and Performance of a Parallel File System for High Performance Distributed Applications. In: Proceedings of the Fifth IEEE International Symposium on High Performance Distributed Computing, pp. 471–480 (1996)Google Scholar
  9. 9.
    Ross, R.B., Carns, P.H., Ligon III, W.B., Latham, R.: Using the Parallel Virtual File System (2002),
  10. 10.
    Madhyastha, T.M.: Automatic Classification of Input/Output Access Patterns. PhD Thesis (1997)Google Scholar
  11. 11.
    Madhyastha, T.M., Reed, D.A.: Exploiting Global Input/Output Access Pattern Classification. In: Proceedings of SC 1997: High Performance Networking and Computing (1997)Google Scholar
  12. 12.
    Thakur, R., Gropp, W., Lusk, E.: On implementing MPI-IO portably and with high performance. In: Proceedings of the 6th Workshop on I/O in Parallel and Distributed Systems (IOPADS 1999), pp. 23–32 (1999)Google Scholar
  13. 13.
    Thakur, R.S., Gropp, W., Lusk, E.: A Case for ung MPI’s derived datatypes to improve I/O Performance. In: Proceedings of Supercomputing 1998, CD-ROM (1998)Google Scholar
  14. 14.
    Rabenseifner, R., Koniges, A.E., Prost, J.-P., Hedges, R.: The Parallel Effective I/O Bandwidth Benchmark: b_eff_io. Parallel I/O for Cluster Computing, 107–132 (2004)Google Scholar
  15. 15.
    Miller, E.L., Katz, R.H.: Input/output behavior of supercomputing applications. SC, 567–576 (1991)Google Scholar
  16. 16.
  17. 17.
    Gropp, W., Lusk, E., Rajeev, T.: Using MPI-2: Advanced Features of the Message-Passing Interface, pp. 15–16 (1999)Google Scholar
  18. 18.
    Patterson, D., Gibson, G., Randy, K.: A case for redundant arrays of inexpensive disks (RAID). In: Proceedings of the ACM SIGMOD International Conference on Management of Data, pp. 109–116 (1988)Google Scholar
  19. 19.
    PVFS Development Team: PVFS 2 Concepts: the new guy’s guide to PVFS. PVFS 2 Documentation (2004)Google Scholar
  20. 20.
    PVFS Development Team: PVFS 2 Distribution Design Notes. PVFS 2 Documentation (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Hipolito Vasquez Lucas
    • 1
  • Thomas Ludwig
    • 1
  1. 1.Parallele und Verteilte Systeme, Institut für InformatikRuprecht-Karls-Universität HeidelbergHeidelbergGermany

Personalised recommendations