Abstract
UNICORE is a state of the art and well tested Grid middleware, designed for seamless and secure access to distributed resources, applications and data, in an easy to use fashion. A wide variety of UNICORE applications for example in bio-informatics generate and compute huge amounts of data. These large amounts of data are not easy to manage reliably and efficiently with the default UNICORE storage system which is using a standard file system. Hence, the current UNICORE does not support a scalable distributed storage system so far. We have integrated Apache Hadoop and its supported distributed storage/file systems into the UNICORE storage management service. Thus allows to build a UNICORE storage system providing data replication, disaster recovery, durability and elasticity. In this paper we will present the architecture and operation of a prototype called UniHadoop, which provides the integration of UNICORE and the distributed storage systems (DSS) supported by Hadoop and highlight its potential in usage scenarios.
Chapter PDF
Similar content being viewed by others
Keywords
- Storage System
- Storage Management
- Hadoop Distribute File System
- Open Grid Service Architecture
- Distribute Storage System
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Newman, H.B., Ellisman, M.H., Orcutt, J.A.: Dataintensive e-science frontier research. Communications of the ACM 46, 68–77 (2003)
Breuer, D., Erwin, D., Mallmann, D., Menday, R., Romberg, M., Sander, V., Schuller, B., Wieder, P.: Scientific Computing with UNICORE. In: Wolf, D., Münster, G., Kremer, M. (eds.) NIC Symposium 2004, Forschungszentrum Jülich; proceedings (2004)
Streit, A., Erwin, D., Mallmann, D., Menday, R., Rambadt, M., Riedel, M., Romberg, M., Schuller, B., Wieder, P.: Unicore - from project results to production grids. In: Grid Computing and New Frontiers of High Performance Processing (2005)
Foster, I., Kesselman, C., Nick, J.M., Tuecke, S.: The physiology of the grid. In: Berman, F., Hey, G.C.F.A.J.G. (eds.) Grid Computing, vol. pages 217-249, John Wiley & Sons, Chichester (2003)
Riedel, M., Schuller, B., Mallmann, D., Menday, R., Streit, A., Tweddell, B., Shahbaz Memon, M., Shiraz Memon, A., Demuth, B., Lippert, T., Snelling, D., van den Berghe, S., Li, V., Drescher, M., Geiger, A., Ohme, G., Vanni, A., Cacciari, C., Lanzarini, S., Malfetti, P., Benedyczak, K., Bala, P., Ratering, R., Lukichev, A.: Web services interfaces and open standards integration into the european unicore 6 grid middleware. In: Proc. Eleventh International IEEE EDOC Conference Workshop EDOC ’07, October 15–16, pp. 57–60 (2007)
Wsrf-technical committee, http://www.oasisopen.org/committees/wsrf/
Unicore clients, http://www.unicore.eu/unicore/architecture/client-layer.php. (accessed on 23.05.09)
Soap specification, http://www.w3.org/TR/soap/
Web services interoperability (ws-i), http://www.ws-i.org
Welch, V., Foster, I., Kesselman, C., Mulmo, O., Pearlman, L., Gawor, J., Meder, S., Siebenlist, F.: X.509 proxy certificates for dynamic delegation (2004)
Howard, S.G., Gobioff, H., tak Leung, S.: The google file system (2003)
Apache software foundation. hadoop distributed file system, http://hadoop.apache.org/core/
Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters (2008)
Hadoop pemissions, http://hadoop.apache.org/common/docs/ (accessed on 26-05-09)
Cloudstore, http://kosmosfs.sourceforge.net/
Ripeanu, M., Iamnitchi, A.: S4: A simple storage service for sciences
Amazon s3 with hadoop, http://wiki.apache.org/hadoop/AmazonS3 (accessed on 26-05-09)
Hadoop with amazon ec2, http://wiki.apache.org/hadoop/AmazonEC2 (accessed on 26-05-09)
Apache software foundation. pig software, http://hadoop.apache.org/pig/ (accessed on 26-05-09)
Olston, C., Reed, B., Srivastava, U., Kumar, R., Tomkins, A.: Pig latin: a not-so-foreign language for data processing (2008)
Apache software foundation. hive, http://wiki.apache.org/hadoop/Hive (accessed on 26-05-09)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bari, W., Memon, A.S., Schuller, B. (2010). Enhancing UNICORE Storage Management Using Hadoop Distributed File System. In: Lin, HX., et al. Euro-Par 2009 – Parallel Processing Workshops. Euro-Par 2009. Lecture Notes in Computer Science, vol 6043. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-14122-5_39
Download citation
DOI: https://doi.org/10.1007/978-3-642-14122-5_39
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-14121-8
Online ISBN: 978-3-642-14122-5
eBook Packages: Computer ScienceComputer Science (R0)