Abstract
The exponential growth of data first presented challenges to cutting-edge businesses such as Goggle, Yahoo, Amazon, Microsoft, Facebook, and Twitter. Data volumes to be processed by cloud applications are growing much faster than computing power. This growth demands new strategies for processing and analyzing information. Hadoop MapReduce has become a powerful computation model that addresses those problems. MapReduce is a programming model that enables easy development of scalable parallel applications to process vast amounts of data on large clusters. Through a simple interface with two functions, map and reduce, this model facilitates parallel implementation of many real world tasks such as data processing for search engines and machine learning. Earlier versions of Hadoop MapReduce had several performance problems like connection between map to reduce task, data overload and slow processing. In this paper, we propose a modified MapReduce architecture – MapReduce Agent (MRA) – that resolves those performance problems. MRA can reduce completion time, improve system utilization, and give better performance. MRA employs multi-connection which resolves error recovery with a Q-chained load balancing system. In this paper, we also discuss various applications and implementations of the MapReduce programming model in cloud environments.
This research (Grants NO. 2013-140-10047118) was supported by the 2013 Industrial Technology Innovation Project Funded by Ministry Of Science, ICT and Future Planning.
The source code for HOP can be downloaded from http://code.google.com/p/hop.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dean, J., Ghemawat, S.: MapReduce: Simplified dataprocessing on large clusters. In: OSDI (2004)
SAM-3 Information Technology – SCSI Architecture Model 3, Working Draft, T10 Project 1561-D, Revision7 (2003)
Allayear, S.M., Park, S.S.: iSCSI multi-connection and error recovery method for remote storage system in mobile appliance. In: Gavrilova, M.L., Gervasi, O., Kumar, V., Tan, C., Taniar, D., Laganá, A., Mun, Y., Choo, H. (eds.) ICCSA 2006. LNCS, vol. 3981, pp. 641–650. Springer, Heidelberg (2006)
Condie, T., Conway, N., Alvaro, P., Hellerstein, J.M.: UC Berkeley: MapReduce Online. Khaled Elmeleegy, Russell Sears (Yahoo! Research)
Allayear, S.M., Park, S.S.: iSCSI protocol adaptation with NAS system via wireless environment. In: International Conference on Consumer Electronics (ICCE), Las Vegus, USA (2008)
RFC 3270. http://www.ietf.org/rfc/rfc3720.txt
Daneshyar, S., Razmjoo, M.: Large-Scale Data Processing Using Mapreduce in Cloud Computing Environment
Changqing Ji∗†, Yu Li‡, Wenming Qiu‡, Uchechukwu Awada‡, Keqiu Li‡ : Big Data Processing in Cloud Computing Environments
Rabi Prasad Padhy: Big Data Processing with Hadoop-MapReduce in Cloud Systems
Chan, J.O.: An Architecture for Big Data Analytics
Hellerstein, J.M., Haas, P.J., Wang, H.J.: Online aggregation. In: SIGMOD (1997)
Caceres, R., Iftode, L.: Improving the performance of reliable transport protocols inMobile computing environments. IEEE JSAC
Laurila, J.K., Blom, J., Dousse, O., Gatica-Perez, D.: The Mobile Data Challenge: Big Data for Mobile Computing Research
Satyanarayanan, M.: Mobile computing: the next decade. In: Proceedings of the 1st ACM Workshop on Mobile Cloud Computing & Services: Social Networks and Beyond (MCS) June 2010
Verma, A., Zea, N., Cho, B., Gupta, I., Campbell, R.H.: Breaking the MapReduce Stage Barrier*
Stokely, M.: Histogram tools for distributions of large data sets
Lu, L., Shi, X., Jin, H., Wang, Q., Yuan, D., Wu, S.: Morpho: A decoupled MapReduce framework for elastic cloud computing
Hao, C., Ying, Q.: Research of Cloud Computing based on the Hadoop platform. Chengdu, China, pp. 181–184, 21-23 October 2011
Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R.H., Konwinski, A., Lee, G., Patterson, D.A., Rabkin, A., Stoica, I., Zaharia, M.: Above the Clouds: a Berkeley View of Cloud Computing, Tech. Rep., University of California at Berkeley (2009)
Palanisamy, B., Singh, A., Liu, L., Jain, B.,: Purlieus: locality-aware resource allocation for MapReduce in a cloud. In: Proceedings of the ACM/IEEE Conference on High Performance Computing Networking, Storage and Analysis, SC 2011, Seattle, WA, USA (2011)
Lu, L., Jin, H., Shi, X., Fedak, G.: Assessing MapReduce for internet computing: a comparison of Hadoop and BitDew-MapReduce. In: Proceedings of the 13th ACM/IEEE International
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Allayear, S.M., Salahuddin, M., Hossain, D., Park, S.S. (2015). The Emergence of Modified Hadoop Online-Based MapReduce Technology in Cloud Environments. In: Rabl, T., Sachs, K., Poess, M., Baru, C., Jacobson, HA. (eds) Big Data Benchmarking. WBDB 2014. Lecture Notes in Computer Science(), vol 8991. Springer, Cham. https://doi.org/10.1007/978-3-319-20233-4_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-20233-4_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-20232-7
Online ISBN: 978-3-319-20233-4
eBook Packages: Computer ScienceComputer Science (R0)