Abstract
Efficient metadata management is a critical aspect of overall system performance in large distributed storage systems. Directory subtree partitioning and traditional hashing are two common techniques used for managing metadata in such systems, but both have some shortcomings, for example, in directory subtree partitioning method, the root node easily becomes a bottleneck, and for traditional hashing, a large directory hierarchy must be maintained and traversed. In this paper, we present a new method called directory-level-based metadata management that has high efficiency while avoiding the shortcomings of traditional methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ousterhout JK, Costa HD, Harrison D, Kunze JA, Kupfer M and Thompson JG (1985) A trace-driven analysis of the Unix 4.2 BSD file system. In: Proceedings of the 10th ACM symposium on operating systems principles (SOSP’85), December, pp 15–24
Morris JH, Satyanarayanan M, Conner MH, Howard JH, Rosenthal DSH, Smith FD (1986) Andrew: a distributed personal computing environment. Commun ACM 29(3):184–201
Levy E, Silberschatz A (1990) Distributed file systems: concepts and examples. ACM Comput Surv 22(4)
Corbett PF, Feitelso DG (1996) The Vesta parallel file system. ACM Trans Comput Syst 14(3):225–264
Braam PJ Lustre whitepaper. http://wenku.baidu.com/view/72b0fb1cfad6195f312ba642.html
Pawlowski B, Juszczak C, Staubach P, Smith C, Lebel D and Hitz D (1994) Nfs version3: design and implementation. In: Proceedings of the summer 1994 USENIX technical conference, pp 137–151
Satyanarayanan M, Kistler JJ, Kumar P, Okasaki ME, Siegel EH, Steere DC (1990) Coda: a highly available file system for a distributed workstation environment. IEEE Trans Comput 39(4):447–459
Nelson MN, Welch BB, Ousterhout JK (1988) Caching in the sprite network file system. ACM Trans Comput Syst 6(1):134–154
Adya A, Bolosky W, Castro M, Cermak G, Chaiken R, Douceur J, Howell J, Lorch J, Theimer M, Wattenhofer RP (2002) Farsite: federated, available and reliable storage for an incompletely trusted environment. ACM SIGOPS Oper Syst Rev 36:1–14
Chervenak A, Palavalli N, Bharathi S, Kesselman C, Schwartzkopf R (2004) Performance and scalability of a replica location service. In: Proceedings of HPDC
Braam PJ, Nelson PA (1999) Removing bottlenecks in distributed file systems: Coda & intermezzo as examples. In: Proceedings of Linux expo
Brandt SA, Miller EL, Long DDE, Xue L (2003) Efficient metadata management in large distributed storage systems. In: Proceedings of MSST
Weil S, Pollack K, Brandt SA, Miller EL (2004) Dynamic metadata management for petabyte-scale file systems. In: Proceedings of the 2004 ACM/IEEE conference on supercomputing (SC’04)
Weil S, Brandt SA, Miller EL, Long DDE, Maltzahn C (2006) Ceph: a scalable, high-performance distributed file system. In: Proceedings of the 7th conference on operating systems design and implementation (OSDI’06)
Weil S, Brandt SA, Miller EL, Maltzahn C (2006) Crush: controlled, scalable, decentralized placement of replicated data. In: Proceedings of the 2006 ACM/IEEE conference on supercomputing (SC’06)
Honicky RJ, Miller EL (2004) Replication under scalable hashing: a family of algorithms for scalable decentralized data distribution. In: Proceedings of IEEE IPDPS, April 2004
Fan L, Cao P, Almeida J and Broder AZ (2000) Summary cache: a scalable wide area web cache sharing protocol. IEEE/ACM Trans Netw 8(3)
Zhu Y, Jiang H and Wang J (2004) Hierarchical Bloom filter arrays (HBA): a novel, scalable metadata management system for large cluster-based storage. In: Proceedings of IEEE cluster computing
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer Science+Business Media B.V.
About this paper
Cite this paper
Ran, L., Jin, H. (2011). An Efficient Metadata Management Method in Large Distributed Storage Systems. In: Park, J., Jin, H., Liao, X., Zheng, R. (eds) Proceedings of the International Conference on Human-centric Computing 2011 and Embedded and Multimedia Computing 2011. Lecture Notes in Electrical Engineering, vol 102. Springer, Dordrecht. https://doi.org/10.1007/978-94-007-2105-0_34
Download citation
DOI: https://doi.org/10.1007/978-94-007-2105-0_34
Published:
Publisher Name: Springer, Dordrecht
Print ISBN: 978-94-007-2104-3
Online ISBN: 978-94-007-2105-0
eBook Packages: EngineeringEngineering (R0)