Abstract
In this paper, we propose an accelerator for Hadoop appliances. Data servers receive significant data traffic because of several services based on networks. Processing such big data requires sufficient communication bandwidth for big data management. In order to increase communication bandwidth, compression algorithms are adopted in Hadoop appliances, although additional computation overhead is required. Our accelerator compresses data from a PCIe (Peripheral Component Interconnect Express) interface, thus reducing the size of data that should be transmitted through a network. As a result, computation overhead of the main processor in a server is decreased and communication bandwidth is increased.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Hadoop: http://hadoop.apache.org/
Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The Hadoop Distributed File System. IEEE Mass Storage Systems and Technologies (MSST), 1–10 (2010)
Thusoo, A., Sen Sarma, J., Jain, N., Shao, Z., Chakka, P., Zhang, N., Antony, S., Liu, H., Murthy, R.: Hive – A Petabyte Scale Data Warehouse Using Hadoop. In: IEEE Data Engineering (ICDE), pp. 996–1005 (2010)
Ghemawat, S., Gobioff, H., Leung, S.-T.: The Google File System. In: The 19th Symposium on Operating Systems Principles, pp. 29–43 (2003)
Seo, S., Jang, I., Woo, K., Kim, I., Kim, J.-S., Maeng, S.: HPMR: Prefetching and Pre-shuffling in Shared MapReduce Computation Environment. In: IEEE Cluster Computing and Workshops, CLUSTER 2009, pp. 1–8 (2009)
Guo, G., Qui, S., Ye, Z., Wang, B., Fang, L., Lu, M., See, S., Mao, R.: GPU-Accelerated Adaptive Compression Framework for Genomics Data. In: IEEE International Conference on Big Data, pp. 181–186 (2013)
Kim, H., Choi, S., Yoo, H.-J.: A Low Power 16-bit RISC with Lossless Compression Accelerator for Body Sensor Network System. In: ASSCC Solid-State Circuits Conference, pp. 207–210 (2006)
Yan, J., Luo, R., Gao, R., Xu, N.-Y.: An Efficient Lossless Compression Method for Internet Search Data in Hardware Accelerators. In: World Congress on Computer Science and Information Engineering, pp. 453–457 (2009)
Chasapis, K., Dolz, M.F., Kuhn, M., Ludwig, T.: Evaluating Power-Performance Benefits of Data Compression in HPC Storage Servers. In: International Conference on Smart Grids, Green Communications and IO Energy-aware Technologies, pp. 29–34 (2014)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Kim, S.D. et al. (2014). Compression Accelerator for Hadoop Appliance. In: Hsu, R.CH., Wang, S. (eds) Internet of Vehicles – Technologies and Services. IOV 2014. Lecture Notes in Computer Science, vol 8662. Springer, Cham. https://doi.org/10.1007/978-3-319-11167-4_41
Download citation
DOI: https://doi.org/10.1007/978-3-319-11167-4_41
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-11166-7
Online ISBN: 978-3-319-11167-4
eBook Packages: Computer ScienceComputer Science (R0)