Abstract
With the increase of large number of data, multi-clusters structure has been adopted for data storage. Therefore, the demand for file metadata sharing across global scale clusters is quickly rising. However, today’s software can not provide features to reflect these desires well. In this paper, We propose and develop a metadata sharing management method called Shedder. First Shedder can allow customized multi-clusters metadata sharing structure. Next, Shedder can provide highly efficient global synchronization for all clusters. Finally, Shedder allows customized user view generated from global namespace. Our evaluation for Shedder shows that Shedder provides low latency for global synchronization. Dynamic transformation from global namespace to customized user view also has low time cost for different size of workloads.
This paper is supported by the Hi-tech Research and Development Program of China (863 Program) under Grant No. 2011AA01A205, the National Natural Science Foundation of China under Grant No. 61370059, the National Natural Science Foundation of China under Grant No. 61003015, the Doctoral Fund of Ministry of Education of China under Grant No. 20101102110018, Beijing Natural Science Foundation under Grant No. 4122042 and the fund of the State Key Laboratory of Software Development Environment under Grant No. SKLSDE-2012ZX-23.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
SETI@HOME project, http://setiathome.berkeley.edu/
Foster, I., Zhao, Y., Raicu, I., Lu, S.: Cloud Computing and Grid Computing 360-Degree Compared. In: Grid Computing Environments Workshop, GCE 2008 (2008)
Page, L., Brin, S., Motwani, R., Winograd, T.: The PageRank Citation Ranking: Bringing Order to the Web. Technical Report. Stanford InfoLab
Corbett, J.C., Dean, J., Epstein, M., et al.: Spanner: Google’s Globally-Distributed Database. In: 10th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2012 (2012)
Schmuck, F., Haskin, R.: GPFS: A Shared-Disk File System for Large Computing Clusters. In: Proceedings of the Conference on File and Storage Technologies (FAST 2002), Monterey, CA, January 28-30, pp. 231–244 (2002)
Weil, S.A., Brandt, S.A., Miller, E.L., Long, D.D.E.: Ceph: A Scalable, High-Performance Distributed File System. In: OSDI 2006 Proceedings of the 7th Symposium on Operating Systems Design and Implementation, pp. 307–320 (2006)
Fadden, S.: An Introduction to GPFS Version 3.5. IBM Systems and Technology Group
Lloyd, W., Freedman, M.J., Kaminskyy, M., Andersen, D.G.: Don’t Settle for Eventual: Scalable Causal Consistency for Wide-Area Storage with COPS. In: Proceedings of the 23rd ACM Symposium on Operating Systems Principles, SOSP 2011 (2011)
Kraska, T., Pang, G., Franklin, M.J., Madden, S., Fekete, A.: MDCC: Multi-data center consistency. In: Proceedings of the 8th ACM European Conference on Computer Systems, EuroSys 2013, pp. 113–126 (2013)
Lamport, L.: The part-time parliament. ACM TOCS, 16(2) (1998)
Baker, J., Bond, C., Corbett, J.C., Furman, J., Khorlin, A., Larson, J., Leon, J.-M., Li, Y., Lloyd, A., Yushprakh, V.: Megastore: Providing scalable, highly available storage for interactive services. In: CIDR (January 2011)
Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: A Distributed Storage System for Structured Data. In: OSDI 2006: Seventh Symposium on Operating System Design and Implementation, Seattle, WA (November 2006)
Lakshman, A., Malik, P.: Cassandra - A Decentralized Structured Storage System. ACM SIGOPS Operating Systems Review Archive 44(2), 35–40 (2010)
Ghemawat, S., Gobioff, H., Leung, S.-T.: The Google File System. In: SOSP 2003, Bolton Landing, New York, USA, October 19-22 (2003)
Gray, J.N.: Notes on Database Operating Systems. In: Flynn, M.J., Jones, A.K., Opderbeck, H., Randell, B., Wiehle, H.R., Gray, J.N., Lagally, K., Popek, G.J., Saltzer, J.H. (eds.) Operating Systems. LNCS, vol. 60, pp. 393–481. Springer, Heidelberg (1978)
Palmieri, F., Pardi, S.: Towards a federated Metropolitan Area Grid environment: The SCoPE network-aware infrastructure. Future Generation Computer Systems 26(8), 1241–1256 (2010)
Esposito, C., Ficco, M., Palmieri, F., Castiglione, A.: Interconnecting Federated Clouds by Using Publish-Subscribe Service. Cluster Computing (2013)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer International Publishing Switzerland
About this paper
Cite this paper
Hao, Q., Zhong, Q., Ruan, L., Zhang, Z., Xiao, L. (2013). Shedder: A Metadata Sharing Management Method across Multi-clusters. In: Kołodziej, J., Di Martino, B., Talia, D., Xiong, K. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2013. Lecture Notes in Computer Science, vol 8285. Springer, Cham. https://doi.org/10.1007/978-3-319-03859-9_6
Download citation
DOI: https://doi.org/10.1007/978-3-319-03859-9_6
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-03858-2
Online ISBN: 978-3-319-03859-9
eBook Packages: Computer ScienceComputer Science (R0)