Abstract
The performance of metadata processing in large distributed file systems currently presents larger challenges than scaling of data throughput. The paper presents a novel, distributed benchmark called DMetabench for measuring the performance of metadata operations (e.g. file creation). DMetabench runs in environments with potentially thousands of nodes and allows an assessment of the scalability of metadata operations. Additionally, precise run-time performance data is preserved which allows for a better understanding of performance artifacts. Validation results from production file systems at the Leibniz Supercomputing Centre (LRZ) are provided and discussed. Possible applications of knowledge about metadata performance scaling include the choice of an optimal parallelization strategy for metadata-itensive workload in a specific runtime environment.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Callaghan, B., Pawlowski, B., Staubach, P.: NFS Version 3 Protocol Specification (1995), http://www.ietf.org/rfc/rfc1813.txt
Campbell, R.: Managing AFS: The Andrew File System. Prentice-Hall, Englewood Cliffs (1998)
Shepard, L., Eppe, E.: SGI InfiniteStorage Shares Filesystem CXFS: A High-Performance, Multi-OS Filesystem from SGI. Technical report, Silicon Graphics (2006)
Cluster File Systems, Inc.: Lustre 1.6 Operations Manual (2007)
The Open Group: The Single UNIX Specification, Version 3. Technical report (2004)
Eisler, M., Corbett, P., Kazar, M., Nydick, D.S., Wagner, J.C.: Data ONTAP GX: A Scalable Storage Cluster. In: Proceedings of FAST 2007 (2007)
Hitz, D., Lau, J., Malcolm, M.: File System Design for an NFS File Server Appliance. Technical report, Network Appliance (TR 3002)
Sweeney, A., Doucette, D., Hu, W., Anderson, C., Nishimoto, M., Peck, G.: Scalability in the XFS File System. In: Proceedings of the USENIX 1996 Technical Conference, San Diego, CA, USA, pp. 1–14 (22–26 1996)
Norcott, W.D., Capps, D.: Iozone Filesystem Benchmark (2006), http://www.iozone.org/
Intel Corporation: Iometer (1998), http://www.iometer.org/
Rabenseifner, R., Koniges, A.E., Prost, J.P., Hedges, R.: The Parallel Effective I/O Bandwidth Benchmark: b_eff_io. Technical report, High-Performance Computing Center, HLRS (2001)
Howard, J., Kazar, M., Menees, S., Nichols, D., Satyanarayanan, M., Sidebotham, R., West, M.: Scale and Performance in a Distributed File System. ACM Transactions on Computer Systems 6, 51–81 (1988)
Katcher, J.: PostMark: A new file system benchmark. Technical report 3022, Network Appliance (1997)
SPEC: SPECsfs 2008 User’s Guide. Technical Report Version 1.0, Standard Performance Evaluation Corporation (SPEC) (2008)
McDougall, R., Mauro, J.: Filebench tutorial (2006), http://www.solarisinternals.com/si/tools/filebench
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Biardzki, C., Ludwig, T. (2009). Analyzing Metadata Performance in Distributed File Systems. In: Malyshkin, V. (eds) Parallel Computing Technologies. PaCT 2009. Lecture Notes in Computer Science, vol 5698. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-03275-2_2
Download citation
DOI: https://doi.org/10.1007/978-3-642-03275-2_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-03274-5
Online ISBN: 978-3-642-03275-2
eBook Packages: Computer ScienceComputer Science (R0)