Abstract
MapReduce is a programming model that allows users the parallel processing of large data sets into a cluster. One of its major implementation is the Apache Hadoop framework that couples both big data storage and processing features. In this paper, we aim to make Hadoop Cloud-like and more resilient adding a further level of parallelization by means of cooperation of federated Clouds. Such an approach allows Cloud providers to elastically scale up/down the system used for parallel job processing. More specifically, we present a system prototype integrating the Hadoop framework and CLEVER, a Message Oriented Middleware supporting federated Cloud environments. In addition, in order to minimize overhead of data transmission among federated Clouds, we considered a shared memory system based on the Amazon S3 Cloud Storage Provider.Experimental results highlight the major factors involved for job deployment in a federated Cloud environment.
Chapter PDF
Similar content being viewed by others
References
Panarello, A., Celesti, A., Fazio, M., Villari, M., Puliafito, A.: A Requirements Analysis for IaaS Cloud Federation. In: 4th International Conference on Cloud Computing and Services Science, Barcelona, Spain (2014)
Petruch, K., Stantchev, V., Tamm, G.: A survey on it-governance aspects of cloud computing. IJWGS 7(3), 268–303 (2011)
Yuan, Y., Wang, H., Wang, D., Liu, J.: On interference-aware provisioning for cloud-based big data processing. In: 2013 IEEE/ACM 21st International Symposium on Quality of Service (IWQoS), pp. 1–6 (June 2013)
Rochwerger, B., Breitgand, D., Epstein, A., Hadas, D., Loy, I., Nagin, K., Tordsson, J., Ragusa, C., Villari, M., Clayman, S., Levy, E., Maraschini, A., Massonet, P., Muñoz, H., Tofetti, G.: Reservoir - when one cloud is not enough. Computer 44, 44–51 (2011)
Kertesz, A., Kecskemeti, G., Marosi, A., Oriol, M., Franch, X., Marco, J.: Integrated monitoring approach for seamless service provisioning in federated clouds. In: 2012 20th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP), pp. 567–574 (February 2012)
Toosi, A.N., Calheiros, R.N., Buyya, R.: Interconnected cloud computing environments: Challenges, taxonomy, and survey. ACM Comput. Surv. 47, 7:1–7:47 (2014)
The Apache Hadoop project: the open-source software for reliable, scalable, distributed computing, http://hadoop.apache.org/
Gahlawat, M., Sharma, P.: Survey of virtual machine placement in federated clouds. In: 2014 IEEE International Advance Computing Conference (IACC), pp. 735–738 (February 2014)
Zaharia, M., Konwinski, A., Joseph, A.D., Katz, R., Stoica, I.: Improving mapreduce performance in heterogeneous environments. In: 8th USENIX Conference on Operating Systems Design and Implementation, OSDI 2008, pp. 29–42. USENIX Association, Berkeley (2008)
Ahmad, F., Chakradhar, S.T., Raghunathan, A., Vijaykumar, T.N.: Tarazu: Optimizing mapreduce on heterogeneous clusters. SIGARCH Comput. Archit. News 40, 61–74 (2012)
Heintz, B., Wang, C., Chandra, A., Weissman, J.: Cross-phase optimization in mapreduce. In: Proceedings of the 2013 IEEE International Conference on Cloud Engineering, IC2E 2013, pp. 338–347. IEEE Computer Society, Washington, DC (2013)
Gandhi, R., Xie, D., Hu, Y.C.: Pikachu: How to rebalance load in optimizing mapreduce on heterogeneous clusters. In: USENIX Conference on Annual Technical Conference, USENIX ATC 2013, pp. 61–66. USENIX Association, Berkeley (2013)
Rao, S., Ramakrishnan, R., Silberstein, A., Ovsiannikov, M., Reeves, D.: Sailfish: A framework for large scale data processing. In: Proceedings of the Third ACM Symposium on Cloud Computing, SoCC 2012, pp. 4:1–4:14. ACM, New York (2012)
Fazio, M., Celesti, A., Puliafito, A., Villari, M.: A message oriented middleware for cloud computing to improve efficiency in risk management systems. Scalable Computing: Practice and Experience (SCPE) 14, 201–213 (2013)
Celesti, A., Fazio, M., Villari, M.: Se clever: A secure message oriented middleware for cloud federation. In: IEEE Symposium on Computers and Communications (ISCC), pp. 35–40 (July 2013)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Panarello, A., Fazio, M., Celesti, A., Puliafito, A., Villari, M. (2014). Cloud Federation to Elastically Increase MapReduce Processing Resources. In: Lopes, L., et al. Euro-Par 2014: Parallel Processing Workshops. Euro-Par 2014. Lecture Notes in Computer Science, vol 8806. Springer, Cham. https://doi.org/10.1007/978-3-319-14313-2_9
Download citation
DOI: https://doi.org/10.1007/978-3-319-14313-2_9
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-14312-5
Online ISBN: 978-3-319-14313-2
eBook Packages: Computer ScienceComputer Science (R0)