Abstract
Data intensive applications are widely existed, such as massive data mining, search engine and high-throughput computing in bioinformatics, etc. Data processing becomes a bottleneck as the scale keeps bombing. However, the cost of processing the large scale dataset increases dramatically in traditional relational database, because traditional technology inclines to adopt high performance computer. The boost of cloud computing brings a new solution for data processing due to the characteristics of easy scalability, robustness, large scale storage and high performance. It provides a cost effective platform to implement distributed parallel data processing algorithms. In this paper, we proposed CPLDP (Cloud based Parallel Large Data Processing System), which is an innovative MapReduce based parallel data processing system developed to satisfy the urgent requirements of large data processing. In CPLDP system, we proposed a new method called operation dependency analysis to model data processing workflow and furthermore, reorder and combine some operations when it is possible. Such optimization reduces intermediate file read and write. The performance test proves that the optimization of processing workflow can reduce the time and intermediate results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Wikipedia entry: computer data processing, http://en.wikipedia.org/wiki/Data_processing
Phillip, B.G.: Data-Rich Computing: Where It’s At. In: Data-Intensive Computing Symposium (March 26, 2008), http://research.yahoo.com/news/2104
Christian, T., Thomas, R.: Data Intensive Computing How SGI Altix ICE and Intel Xeon Processor 5500 Series[Code-name Nehalem] help Sustain HPC Efficiency Amid Explosive Data Growth. Silicon Graphics Inc. http://www.sgi.com/pdfs/4154.pdf
Xu, M., Gao, D., Deng, C., Luo, Z.G., Sun, S.L.: Cloud Computing Boosts Business Intelligence of Telecommunication Industry. In: Jaatun, M.G., Zhao, G., Rong, C. (eds.) CloudCom 2009. LNCS, vol. 5931, Springer, Heidelberg (2009)
Wikipedia entry, http://en.wikipedia.org/wiki/Extract,_transform,_load
Pete, C., Julian, C., Randy, K., Thomas, K., Thomas, R., Colin, S., Rüdiger, W.: CRISP-DM 1.0 - Step-by-step data mining guide.
Jeffrey, D., Sanjay, G.: MapReduce: Simplified Data Processing on Large Cluster. In: Proc. 6th Symposium on Operating Systems Design and Implementation, San Francisco, pp. 13–149 (2004)
Sanjay, G., Howard, G., Shunk-Tak, L.: The Google File System. In: Proc.19th Symposium on Operating System Principles, pp. 29–43. Lake George, New York (2003)
Hadoop, A.: http://www.hadoop.com
Miner, A.: http://www.eti.hku.hk/alphaminer/
Pike, R., Dorward, S., Griesemer, R., Quinlan, S.: Interpreting the data: Parallel analysis with Sawzall. Scientific Programming Journal 13(4) (2005)
Christopher, O., Benjamin, R., Utkarsh, S., Ravi, K., Andrew, T.: Yahoo! Research Pig Latin: A Not-So-Foreign Language for Data Processing. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of data (2008)
DryadLINQ, http://research.microsoft.com/research/sv/DryadLINQ/
Isard, M.: Dryad: Distributed data-parallel programs from sequential building blocks. In: European Conference on Computer Systems (EuroSys), Lisbon, Portugal, pp. 59–72 (2007)
Yuan, Y., Michael, I., Dennis, F., Mihai, B., Úlfar, E., Pradeep, K.G., Jon, C.: DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language. In: Symposium on Operating System Design and Implementation (OSDI), San Diego, CA (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhong, Z., Li, M., Chang, J., Zhou, L., Huang, J.Z., Feng, S. (2010). CPLDP: An Efficient Large Dataset Processing System Built on Cloud Platform. In: Cao, L., Zhong, J., Feng, Y. (eds) Advanced Data Mining and Applications. ADMA 2010. Lecture Notes in Computer Science(), vol 6441. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-17313-4_2
Download citation
DOI: https://doi.org/10.1007/978-3-642-17313-4_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-17312-7
Online ISBN: 978-3-642-17313-4
eBook Packages: Computer ScienceComputer Science (R0)