Abstract
Developing and deploying a global and scalable data access service is a challenging task. We assume that the globalization is achieved by creating and maintaining appropriate metadata while the scalability is achieved by limiting the number of entities taking part in keeping the metadata consistency. In this paper, we present different consistency and synchronization models for various metadata types chosen for implementation of global and scalable data access service.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Baud, J.P.B., Casey, J., Lemaitre, S., Nicholson, C., Smith, D., Stewart, G.: LCG data management: from EDG to EGEE. In: UK e-Science All Hands Meeting, Nottingham, UK (2005). http://www.allhands.org.uk/2005/proceedings/papers/475.pdf
Drago, I., Mellia, M., Munafo, M., Sperotto, A., Sadre, R., Pras, A.: Inside dropbox: understanding personal cloud storage services. In: Proceedings of the 2012 ACM Conference on Internet Measurement, IMC 2012, pp. 481–494. ACM, New York (2012)
Dutka, Ł., Wrzeszcz, M., Lichoń, T., Słota, R., Zemek, K., Trzepla, K., Opiola, Ł., Słota, R., Kitowski, J.: Onedata - a step forward towards globalization of data access for computing infrastructures. In: Proceedings of the International Conference on Computational Science, ICCS 2015, Computational Science at the Gates of Nature, Reykjavík, Iceland, pp. 2843–2847, 1–3 June 2015
Hünich, D., Müller-Pfefferkorn, R.: Managing large datasets with iRODS - a performance analysis. In: Proceedings of the 2010 International Multiconference on Computer Science and Information Technology (IMCSIT), pp. 647–654. IEEE (2010)
Pacheco, L., Halalai, R., Schiavoni, V., Pedone, F., Riviere, E., Felber, P.: GlobalFS: a strongly consistent multi-site file system. In: Proceedings of the 35th IEEE Symposium on Reliable Distributed Systems, SRDS 2016, Budapest, Hungary, 26–29 September 2016, pp. 147–156. IEEE Computer Society (2016). http://dblp.uni-trier.de/db/conf/srds/srds2016.html#PachecoHSPRF16
Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The hadoop distributed file system. In: Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), MSST 2010, pp. 1–10. IEEE Computer Society, Washington (2010). https://doi.org/10.1109/MSST.2010.5496972
Slota, R., Nikolow, D., Skalkowski, K., Kitowski, J.: Management of data access with quality of service in PL-Grid environment. Comput. Inf. 31(2), 463–479 (2012). http://www.cai.sk/ojs/index.php/cai/article/view/950
Słota, R., Nikolow, D., Skitał, Ł., Kitowski, J.: Implementation of replication methods in the grid environment. In: Sloot, P.M.A., Hoekstra, A.G., Priol, T., Reinefeld, A., Bubak, M. (eds.) EGC 2005. LNCS, vol. 3470, pp. 474–484. Springer, Heidelberg (2005). https://doi.org/10.1007/11508380_49
Słota, R., Skitał, Ł., Nikolow, D., Kitowski, J.: Algorithms for automatic data replication in grid environment. In: Wyrzykowski, R., Dongarra, J., Meyer, N., Waśniewski, J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 707–714. Springer, Heidelberg (2006). https://doi.org/10.1007/11752578_85
Thain, D., Livny, M.: Parrot: an application environment for data-intensive computing. Scalable Comput. Pract. Exp. 6(3), 9–18 (2005)
Viotti, P., Vukolić, M.: Consistency in non-transactional distributed storage systems. ACM Comput. Surv. 49(1), 19:1–19:34 (2016). https://doi.org/10.1145/2926965
Wang, F., Oral, S., Shipman, G., Drokin, O., Wang, T., Huang, I.: Understanding Lustre Filesystem Internals. Technical report ORNL/TM-2009/117, Oak Ridge National Lab., National Center for Computational Sciences (2009)
Weil, S.A., Brandt, S.A., Miller, E.L., Long, D.D.E., Maltzahn, C.: Ceph: a scalable, high-performance distributed file system. In: Proceedings of the 7th Symposium on Operating Systems Design and Implementation (OSDI), pp. 307–320 (2006)
Weil, S.A., Brandt, S.A., Miller, E.L., Maltzahn, C.: CRUSH: controlled, scalable, decentralized placement of replicated data. In: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC 2006, Tampa. ACM, New York (2006). https://doi.org/10.1145/1188455.1188582
Wrzeszcz, M., Trzepla, K., Słota, R., Zemek, K., Lichoń, T., Opioła, Ł., Nikolow, D., Dutka, Ł., Słota, R., Kitowski, J.: Metadata organization and management for globalization of data access with Onedata. In: Wyrzykowski, R., Deelman, E., Dongarra, J., Karczewski, K., Kitowski, J., Wiatr, K. (eds.) PPAM 2015. LNCS, vol. 9573, pp. 312–321. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-32149-3_30
Zhang, J., Wu, Y., Chung, Y.C.: PROAR: a weak consistency model for Ceph. In: 2016 IEEE 22nd International Conference on Parallel and Distributed Systems (ICPADS), pp. 347–353 (2016)
Acknowledgements
This research is supported partly by the European Regional Development Fund program no. POIG.02.03.00-12-137/13 as part of the PLGrid Core. R. G. Słota, D. Nikolow and J. Kitowski acknowledge AGH-UST statutory Grant no. 11.11.230.337. Support by IndigoDC project no. RIA 653549 is also acknowledged by Ł. Dutka, M. Wrzeszcz. T. Lichoń acknowledges Polish Ministry of Science and Higher Education under AGH University of Science and Technology, Faculty of Computer Science, Electronics and Telecommunications statutory project no. 15.11.230.317, and owes special thanks to ACK Cyfronet AGH computing center for providing computing and storage infrastructure for this research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Wrzeszcz, M. et al. (2018). Consistency Models for Global Scalable Data Access Services. In: Wyrzykowski, R., Dongarra, J., Deelman, E., Karczewski, K. (eds) Parallel Processing and Applied Mathematics. PPAM 2017. Lecture Notes in Computer Science(), vol 10777. Springer, Cham. https://doi.org/10.1007/978-3-319-78024-5_41
Download citation
DOI: https://doi.org/10.1007/978-3-319-78024-5_41
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-78023-8
Online ISBN: 978-3-319-78024-5
eBook Packages: Computer ScienceComputer Science (R0)