Advertisement

Parallel File Systems

  • Robert Ross
  • Philip Carns
  • David Metheny
Chapter
Part of the International Series in Operations Research & Management Science book series (ISOR, volume 132)

Abstract

The success of a CDI Grid is dependent upon the design of its storage infrastructure. As seen in Chapter 7, processing in this environment revolves around the simultaneous movement and transformation of data on many compute elements. Effective storage solutions combine hardware and software to meet these needs. The storage hardware selected must provide enough raw throughput for the expected workloads. Typical storage hardware architectures also often provide some redundancy to help in creating a fault tolerant system. Storage software, specifically file systems, must organize this storage hardware into a single logical space, provide efficient mechanisms for accessing that space, and hide common hardware failures from compute elements.

Keywords

File System Message Passing Interface Access Pattern File Region Internet Engineer Task Force 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. Barroso L, Dean J, Holzle U (2003) Web search for a planet: The Google cluster architecture. In: IEEE Micro, vol. 23, issue 2, pp 22-28.CrossRefGoogle Scholar
  2. Culler D, Singh J, Gupta A (1999) Parallel Computer Architecture: A Hardware/Software Approach. Morgan Kaufmann, San Francisco, CA.Google Scholar
  3. Dean J, Ghemawat S (2004) MapReduce: Simplified data processing on large clusters. In: OSDI’04, 6th Symposium on Operating Systems Design and Implementation, Sponsored by USENIX in cooperation with ACM SIGOPS, pp 137–150.Google Scholar
  4. EMC Corporation (2006) Deploying Celerra MPFSi in high-performance computing environments. EMC White Paper.Google Scholar
  5. Fridella S, Jiang X, Black D (2003) Elements of a scalable network file system protocol. In: NFS Extensions for Parallel Storage Workshop.Google Scholar
  6. Ghemawat S, Gobioff H, Leung S (2003) The Google File System. In: ACM SIGOPS Operating Systems Review Volume 37, Issue 5, pp 29-43.CrossRefGoogle Scholar
  7. Gropp W, Lusk E, Skjellum A (1999) Using MPI: Portable Parallel Programming with the Message-Passing Interface. MIT Press, Cambridge, MA.Google Scholar
  8. The Hadoop Distributed File System, http://lucene.apache.org/hadoop/hdfs_design.html.
  9. Hildebrand D, Honeyman P (2005) Exporting storage systems in a scalable manner with pNFS. In: Proceedings of the 22nd IEEE - 13th NASA Goddard (MSST2005) Conference on Mass Storage Systems and Technologies, Monterey, California.Google Scholar
  10. IEEE/ANSI Std. 1003.1 (1996) Portable operating system interface (POSIX) part 1: System application program interface (API) [C Language].Google Scholar
  11. Message Passing Interface Forum (1997) MPI-2: Extensions to the Message-Passing Interface.Google Scholar
  12. Patterson D, Gibson G, Katz R (1988) A case for redundant arrays of inexpensive disks (RAID). In: Proceedings of the ACM SIGMOD International Conference on Management of Data, pp 108-116.Google Scholar
  13. Purakayastha A, Ellis C, Kotz D, Nieuwejaar N, Best M (1995) Characterizing parallel file-access patterns on a large-scale multiprocessor. In: Proceedings of the Ninth International Parallel Processing Symposium. pp 165-172.Google Scholar
  14. Schroeder B, Gibson G (2007) Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you? In: 5th USENIX Conference on File and Storage Technologies.Google Scholar
  15. Shepler S, Callaghan B, Robinson D, Thurlow R, Sun Microsystems, Inc., Beame C, Hummingbird Ltd. Eisler M, Noveck D, Network Appliance, Inc. (2003) Network File System (NFS) version 4 protocol. Network Working Group RFC 3530, Internet Engineering Task Force.Google Scholar
  16. Thakur R, Gropp W, Lusk E (1999) On implementing MPI-IO portably and with high performance. In: Proceedings of the Sixth Workshop on I/O in Parallel and Distributed Systems, Atlanta, Georgia, pp 23-32.Google Scholar
  17. Welch B, Halevy B, Goodson G, Black D, Adamson A (2005) pNFS operations. Internet Engineering Task Force Internet-Draft.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Robert Ross
    • 1
  • Philip Carns
    • 1
  • David Metheny
    • 2
  1. 1.Mathematics and Computer Science DivisionArgonne National LaboratoryArgonneUSA
  2. 2.Acxiom Corporation ConwayConwayUSA

Personalised recommendations