An Efficient Data Extracting Method Based on Hadoop
- 1.4k Downloads
As an open-source big data solutions, Hadoop ecosystem have been widely accepted and applied. However, how to import large amounts of data in a short time from the traditional relational database to hadoop become a major challenge for ETL (Extract-Transform-Load)stage of big data processing. This paper presents an efficient parallel data extraction method based on hadoop, using MapReduce computation engine to call JDBC(The Java Database Connectivity) interface for data extraction. Among them, for the problem of multi-Map segmentation during the data input, this paper presents a dynamic segmentation algorithm for Map input based on range partition, can effectively avoid data tilt, making the input data is distributed more uniform in each Map. Experimental results show that the proposed method with respect to the ETL tool Sqoop which also using the same calculation engine of MapReduce is more uniform in dividing the input data and take less time when extract same datas.
KeywordsETL Hadoop MapReduce Big data Range Partition
This work is supported by the Core Electronic Devices, High-end Generic Chips and Basic Software of National Science and Technology Major Projects of China,No.2013ZX01039002.
- 2.White, T.: Hadoop: The Definitive Guide. O’Reilly Media Inc., Sebastopol (2012)Google Scholar
- 3.Vavilapalli, V.K., Murthy, A.C., Douglas, C., Agarwal, S., Konar, M., Evans, R., Graves, T., Lowe, J., Shah, H., Seth, S., et al.: Apache hadoop yarn: yet another resource negotiator. In: Proceedings of the 4th Annual Symposium on Cloud Computing, p. 5. ACM (2013)Google Scholar
- 5.Casters, M., Bouman, R., Van Dongen, J.: Pentaho Kettle Solutions: Building Open Source ETL Solutions with Pentaho Data Integration. Wiley, Indianapolis (2010)Google Scholar
- 7.Azarmi, B.: Talend for Big Data. Packt Publishing Ltd, Birmingham (2014)Google Scholar
- 9.Jain, N., Liao, G., Willke, T.L.: Graphbuilder: scalable graph etl framework. In: First International Workshop on Graph Data Management Experiences and Systems, ACM (2013)Google Scholar
- 11.Chen, J., Le, J.: Programming model based on mapreduce for importing big table into hdfs. J. Comput. Appl. 33(9), 2486–2489, 2561 (2013)Google Scholar
- 13.Ting, K., Cecho, J.J.: Apache Sqoop Cookbook. O’Reilly Media, Inc., CA (2013)Google Scholar