Advertisement

Research and Application of DBSCAN Algorithm Based on Hadoop Platform

  • Xiufen Fu
  • Yaguang Wang
  • Yanna Ge
  • Peiwen Chen
  • Shaohua Teng
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8351)

Abstract

Along with the rapid development of information age, more and more data can be obtained from the Internet, it is very difficult to get useful information and knowledge from these huge amounts of data. On the foundation of the existing algorithm based on DBSCAN, a new improved incremental DBSCAN clustering algorithm is proposed. Combining with cloud computing open source framework Hadoop, the improved algorithm use the programming model of MapReduce which can easy write distributed applications and simplify distributed programme to divide a huge amounts of data elements into chunks and distribute the chunks across the cluster and run the algorithm as a MapReduce job, in this way, this improved algorithm of data mining is integrated with framework Hadoop by the DBSCAN clustering algorithm. When data manipulation (add or delete) has occurred in the database, what we need to do is to mine the mutative data and merge the similar clusters, and ultimately form the final knowledge mining.Compared with single node server serial arithmetic and the overall mining, the time delay of data processing will be reduced. In the last part,the paper verified the effectiveness by experiments and data analysis.

Keywords

Hadoop MapReduce DBSCAN Algorithm Incremental Mining 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Armbrust, M., Fox, A., Griffith, R., et al.: Above the clouds:A berkely view of cloud computing. University of California, Berkely (2009)Google Scholar
  2. 2.
    Xu, G., Xu, F., Ma, H.: Deploying and researching Hadoop in virtual machines. In: IEEE International Conference on Digital Object Identifier Automation and Logistics (ICAL), pp. 395–399 (2012)Google Scholar
  3. 3.
    Kurazumi, S., Tsumura, T., Saito, S., Matsuo, H.: Dynamic Processing Slots Scheduling for I/O Intensive Jobs of Hadoop MapReduce. In: Third International Conference on Networking and Computing (ICNC), pp. 288–292 (2012)Google Scholar
  4. 4.
    He, Y., Tan, H., Luo, W., Mao, H., Ma, D., Feng, S., Fan, J.: MR-DBSCAN: An Efficient Parallel Density-Based Clustering Algorithm Using MapReduce. In: 2011 IEEE 17th International Conference on Parallel and Distributed Systems (ICPADS), pp. 473–480 (2011)Google Scholar
  5. 5.
    Yue, C., Jinsheng, Y.: Text Clustering Based on Improved DBSCAN Algorithm. Computer Engineering 37(12), 50–52 (2011)Google Scholar
  6. 6.
    Dean, J., Ghemawat, S.: MapReduce.:Simplified Data Processing on Large Clusters. Communications of the ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
  7. 7.
    Aggarwal, C.C., Yu, P.S.: A Survey of Uncertain Data Algorithms and Applications. IEEE Transactions on Knowledge and Data Engineering 21, 609–623 (2009)CrossRefGoogle Scholar
  8. 8.
    Hanxi, L.: Research Community Mining Based on DBSCAN Algorithm. Computer Applications and Software 26(9), 110–113 (2009)Google Scholar
  9. 9.
    Wenfeng, L., Xiaoxia, Q.: Study of Chameleon Clustering Algorithm and implementation in Weka. Computer Systems & Applications 19(12), 246–250 (2010)Google Scholar
  10. 10.
    Shenyi, J., Guansong, P., Lisha, Z.: Chameleon Algorithm is Improved. Journal of Chinese Computer Systems 31(8), 1643–1646 (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Xiufen Fu
    • 1
  • Yaguang Wang
    • 1
  • Yanna Ge
    • 1
  • Peiwen Chen
    • 1
  • Shaohua Teng
    • 1
  1. 1.School of ComputerGuangdong University of TechnologyGuangzhouP.R.China

Personalised recommendations