Abstract
MapReduce framework is widely used in massive data processing, such as financial prediction, online marketing, and so on. Multicore processor is a great platform to implement MapReduce because of its inherent parallelism and flexibility. This book chapter extracts features of MapReduce applications, and proposes a software–hardware co-design framework based on a multi-core processor to improve the performance of MapReduce applications. Experimental results show that the MapReduce framework with hardware accelerators speeds up by 40 times at maximum compared to the pure software solution, and the proposed Topo-MapReduce speeds up further by 29% at maximum compared to the original MapReduce.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
M. Saecker, V. Markl, Big data analytics on modern hardware architectures: a technology survey, in Business Intelligence (Springer, Berlin/Heidelberg, 2013), pp. 125–149
J. Dean, S. Ghemawat, MapReduce: simplified data processing on large clusters, in Proceedings of Operating Systems Design and Implementation, San Francisco, CA (2004), pp. 137–150
L. Zhou, Z. Yu, J. Lin, S. Zhu, W. Shi, H. Zhou, K. Song, X. Zeng, Acceleration of Naive-Bayes algorithm on multicore processor for massive text classification, in International Symposium on Integrated Circuits (ISIC) (2014), pp. 244–247
Spark the fastest open source engine for sorting a petabyte, https://databricks.com/blog/2014/10/10/spark-petabyte-sort.html
C. Ranger, R. Raghuraman, A. Penmetsa, et al., Evaluating MapReduce for multi-core and multiprocessor systems, in IEEE 13th International Symposium on High Performance Computer Architecture, 2007. HPCA 2007 (IEEE, Washington, 2007), pp. 13–24
Z. Yu, K. You, R. Xiao, H. Quan, P. Ou, Y. Ying, H. Yang, M. Jing, X. Zeng, An 800MHz 320mW 16-core processor with message-passing and shared-memory inter-core communication mechanisms, in IEEE International Solid-State Circuits Conference (ISSCC) (2012), pp. 64–65
P. Ou, J. Zhang, H. Quan, Y. Li, M. He, Z. Yu, X. Yu, S. Cui, J. Feng, S. Zhu, J. Lin, M. Jing, X. Zeng, Z. Yu, A 65nm 39GOPS/W 24-core processor with 11Tb/s/W packet controlled circuit-switched double-layer network-on-chip and heterogeneous execution array, in IEEE International Solid-State Circuits Conference (ISSCC) (2013), pp. 56–57
V. Felipe, et al., GPU-NB: a fast CUDA-based implementation of Naive Bayes, in Proceedings - Symposium on Computer Architecture and High Performance Computing (2013), pp. 168–175
Acknowledgements
This work was supported by grants from Huawei Corporation, and SYSU-CMU Shunde International Joint Research Institute.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this chapter
Cite this chapter
Zhou, L., Yu, Z. (2017). Acceleration of MapReduce Framework on a Multicore Processor. In: Chattopadhyay, A., Chang, C., Yu, H. (eds) Emerging Technology and Architecture for Big-data Analytics. Springer, Cham. https://doi.org/10.1007/978-3-319-54840-1_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-54840-1_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-54839-5
Online ISBN: 978-3-319-54840-1
eBook Packages: EngineeringEngineering (R0)