Research and Application of DBSCAN Algorithm Based on Hadoop Platform
Along with the rapid development of information age, more and more data can be obtained from the Internet, it is very difficult to get useful information and knowledge from these huge amounts of data. On the foundation of the existing algorithm based on DBSCAN, a new improved incremental DBSCAN clustering algorithm is proposed. Combining with cloud computing open source framework Hadoop, the improved algorithm use the programming model of MapReduce which can easy write distributed applications and simplify distributed programme to divide a huge amounts of data elements into chunks and distribute the chunks across the cluster and run the algorithm as a MapReduce job, in this way, this improved algorithm of data mining is integrated with framework Hadoop by the DBSCAN clustering algorithm. When data manipulation (add or delete) has occurred in the database, what we need to do is to mine the mutative data and merge the similar clusters, and ultimately form the final knowledge mining.Compared with single node server serial arithmetic and the overall mining, the time delay of data processing will be reduced. In the last part,the paper verified the effectiveness by experiments and data analysis.
KeywordsHadoop MapReduce DBSCAN Algorithm Incremental Mining
Unable to display preview. Download preview PDF.
- 1.Armbrust, M., Fox, A., Griffith, R., et al.: Above the clouds:A berkely view of cloud computing. University of California, Berkely (2009)Google Scholar
- 2.Xu, G., Xu, F., Ma, H.: Deploying and researching Hadoop in virtual machines. In: IEEE International Conference on Digital Object Identifier Automation and Logistics (ICAL), pp. 395–399 (2012)Google Scholar
- 3.Kurazumi, S., Tsumura, T., Saito, S., Matsuo, H.: Dynamic Processing Slots Scheduling for I/O Intensive Jobs of Hadoop MapReduce. In: Third International Conference on Networking and Computing (ICNC), pp. 288–292 (2012)Google Scholar
- 4.He, Y., Tan, H., Luo, W., Mao, H., Ma, D., Feng, S., Fan, J.: MR-DBSCAN: An Efficient Parallel Density-Based Clustering Algorithm Using MapReduce. In: 2011 IEEE 17th International Conference on Parallel and Distributed Systems (ICPADS), pp. 473–480 (2011)Google Scholar
- 5.Yue, C., Jinsheng, Y.: Text Clustering Based on Improved DBSCAN Algorithm. Computer Engineering 37(12), 50–52 (2011)Google Scholar
- 8.Hanxi, L.: Research Community Mining Based on DBSCAN Algorithm. Computer Applications and Software 26(9), 110–113 (2009)Google Scholar
- 9.Wenfeng, L., Xiaoxia, Q.: Study of Chameleon Clustering Algorithm and implementation in Weka. Computer Systems & Applications 19(12), 246–250 (2010)Google Scholar
- 10.Shenyi, J., Guansong, P., Lisha, Z.: Chameleon Algorithm is Improved. Journal of Chinese Computer Systems 31(8), 1643–1646 (2010)Google Scholar