Abstract
Running MapReduce framework for massive data processing on a cluster of commodity hardware requires enormous resource, especially high CPU and memory occupation. To enhance the commodity hardware performance without physical update and topology change, the highly parallel and dynamically configurable FPGA can be dedicated to provide feasible supplements in computation running as coprocessor to CPU. This paper presents a MapReduce Framework on FPGA accelerated commodity hardware. In our framework, a cluster of worker nodes is designed for MapReduce framework, and each worker node consists of commodity hardware and special hardware. CPU base worker runs the major communications with other worker node and tasks, while FPGA base worker operates extended mapreduce tasks process to speed up the computation process. Due to internal pipeline in computing operations, FPGA base worker offers the high performance which enhances 10x faster task processing. Furthermore, CPU base worker can reconfigure FPGA chip immediately when it fails. In this period, data will be migrated and continuously processed in commodity hardware. Meanwhile a local memory in commodity hardware is implemented to recover the lost data. Moreover, most frequent computation modules are provided in FPGA module library which are convenient for user to configure operations in special hardware. Experimental results proves that our framework offers high performance and flexibility in applications.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. In: OSDI 2004: Sixth Symposium on Operating System Design and Implementation, pp. 10–23 (2004)
Apache Group, Apache Hadoop (2008), http://hadoop.apache.org
Chu, C.-T., Kim, S.K., Lin, Y.-A., Yu, Y., Bradski, G., Ng, A.Y., Olukotun, K.: Map-Reduce for Machine Learning on Multicore, MapReduce: Simplified Data Processing on Large Clusters. In: Advances in Neural Information Processing Systems, pp. 281–288. MIT Press (2007)
Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., Kozyrakis, C.: Evaluating MapReduce for multi-core and multiprocessor systems. In: HPCA 2007: IEEE 13th International Symposium on High-Performance Computer Architecture, pp. 13–24 (2007)
Yoo, R.M., Romano, A., Kozyrakis, C.: Phoenix rebirth: Scalable MapReduce on a large-scale shared-memory system. In: IISWC 2009: 2009 IEEE International Symposium on Workload Characterization, pp. 198–207 (2009)
He, B., Fang, W., Luo, Q., Govindaraju, N.K., Wang, T.: Mars: A MapReduce framework on graphics processors. In: PACT 2008: 17th International Conference on Parallel Architectures and Compilation Techniques, pp. 260–269 (2008)
Yeung, J.H.C., Tsang, C.C., Tsoi, K.H., Kwan, B.S.H., Cheung, C.C.C., Chan, A.P.C., Leong, P.H.W.: Map-reduce as a Programming Model for Custom Computing Machines. In: FCCM 2008: 16th IEEE Symposium on Field-Programmable Custom Computing Machines, pp. 149–159 (2008)
Yi, S., Wang, B., Yan, J., Wang, Y., Xu, N., Yang, H.: FPMR: MapReduce framework on FPGA. In: FPGA 2010: Proceedings of the 18th Annual ACM/SIGDA International Symposium on Field Programmable Gate Arrays, pp. 93–102 (2010)
NetFPGA Group (2010), http://netfpga.org
Condie, T., Conway, N., Alvaro, P., Hellerstein, J.M., Elmeleegy, K., Sears, R.: MapReduce Online, EECS Department, University of California, Berkeley, Tech-Rep. UCB/EECS-2009-136 (2009), http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-136.html
Mao, Y., Morris, R., Frans Kaashoek, M.: Optimizing MapReduce for Multicore Architectures, Computer Science and Artificial Intelligence Lab (CSAIL), Massachusetts Institute of Technology, Tech-Rep. MIT-CSAIL-TR-2010-020 (2010)
Herbordt, M.C., Van Court, T., Gu, Y., Sukhwani, B., Conti, A., Model, J., Di Sabello, D.: Achieving High Performance with FPGA-Based Computing. Computer 40(3), 50–57 (2007)
Unnikrishnan, D., Vadlamani, R., Liao, Y., Dwaraki, A., Crenne, J., Gao, L., Tessier, R.: Scalable network virtualization using FPGAs. In: FPGA 2010: Proceedings of the ACM/SIGDA International Symposium on Field Programmable Gate Arrays, pp. 219–228 (2010)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Yin, D., Li, G., Huang, Kd. (2012). Scalable MapReduce Framework on FPGA Accelerated Commodity Hardware. In: Andreev, S., Balandin, S., Koucheryavy, Y. (eds) Internet of Things, Smart Spaces, and Next Generation Networking. ruSMART NEW2AN 2012 2012. Lecture Notes in Computer Science, vol 7469. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-32686-8_26
Download citation
DOI: https://doi.org/10.1007/978-3-642-32686-8_26
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-32685-1
Online ISBN: 978-3-642-32686-8
eBook Packages: Computer ScienceComputer Science (R0)