KMDT: A Hybrid Cluster Approach for Anomaly Detection Using Big Data

  • Santosh Thakur
  • Ramesh Dharavath
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 701)


In the current digital era, huge data are being generated in a voluminous state from different sources. This lead towards a processing repository called Big Data. Managing and processing such data in parallel clusters is a big challenge. To capture this problem, in this paper, we propose a hybrid algorithm for cluster analysis using the Spark framework for analyzing the Big Data instances. The proposed algorithm is the combination of two machine learning techniques namely, K-Means (KM) and C5.0 Decision Tree (DT). As per the factor of cluster, euclidean distance is used to find the nearest cluster and the related DT is built for each cluster using C5.0 DT algorithm. The inferences of the DT are used to classify each anomaly and the normal instances of the large datasets. Experimental results show that the proposed hybrid algorithm outperforms with other existing algorithms and produces better classification accuracy for anomaly detection.


Hadoop Spark K-means Decision tree Big Data 



This work is partially supported by Indian Institute of Technology (ISM), Govt. of India. The authors wish to express their gratitude and thanks to the Department of Computer Science and Engineering, Indian Institute of Technology (ISM), Dhanbad, India for providing their support in arranging necessary computing facilities.


  1. 1.
    Hayes, M.A., Capretz, M.A.: Contextual anomaly detection in big sensor data. In: 2014 IEEE International Congress on Big Data (BigData Congress), June, pp. 64–71. IEEE (2014)Google Scholar
  2. 2.
    Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. (CSUR) 41(3), 15 (2009)CrossRefGoogle Scholar
  3. 3.
    Rettig, L., Khayati, M., Cudré-Mauroux, P., Piórkowski, M.: Online anomaly detection over Big Data streams. In: 2015 IEEE International Conference on Big Data (Big Data), October, pp. 1113–1122. IEEE (2015)Google Scholar
  4. 4.
    Hayes, M.A., Capretz, M.A.: Contextual anomaly detection framework for big sensor data. J. Big Data 2(1), 2 (2015)CrossRefGoogle Scholar
  5. 5.
    Bottesch, T., Bühler, T., Kächele, M.: Speeding up k-means by approximating Euclidean distances via block vectors. In: International Conference on Machine Learning, June, pp. 2578–2586 (2016)Google Scholar
  6. 6.
    Rani, M.S., Xavier, S.B.: A hybrid intrusion detection system based on C5. 0 decision tree and one-class SVM. Int. J. Current Eng. Technol. 5(3) (2015)Google Scholar
  7. 7.
    Patil, N., Lathi, R., Chitre, V.: Comparison of C5.0 & CART classification algorithms using pruning technique. Int. J. Eng. Res. Technol. 1(4) (2012)Google Scholar
  8. 8.
    Pandya, R., Pandya, J.: C5. 0 algorithm to improved decision tree with feature selection and reduced error pruning. Int. J. Comput. Appl. 117(16) (2015)Google Scholar
  9. 9.
    Wu, X., Kumar, V., Quinlan, J.R., Ghosh, J., Yang, Q., Motoda, H., McLachlan, G.J., Ng, A., Liu, B., Philip, S.Y., Zhou, Z. H.: Top 10 algorithms in data mining. Knowl. Inf. Syst. 14(1), 1–37 (2008)CrossRefGoogle Scholar
  10. 10.
    Maulik, U., Bandyopadhyay, S.: Performance evaluation of some clustering algorithms and validity indices. IEEE Trans. Pattern Anal. Mach. Intell. 24(12), 1650–1654 (2002)CrossRefGoogle Scholar
  11. 11.
    Shafeeq, A., Hareesha, K.S.: Dynamic clustering of data with modified k-means algorithm. In: Proceedings of the 2012 Conference on Information and Computer Networks, pp. 221–225 (2012)Google Scholar
  12. 12.
    Rao, K.H., Srinivas, G., Damodhar, A., Krishna, M.V.: Implementation of anomaly detection technique using machine learning algorithms. Int. J. Comput. Sci. Telecommun. 2(3), 25–31 (2011)Google Scholar
  13. 13.
    Muniyandi, A.P., Rajeswari, R., Rajaram, R.: Network anomaly detection by cascading k-Means clustering and C4. 5 decision tree algorithm. Proc. Eng. 30, 174–182 (2012)CrossRefGoogle Scholar
  14. 14.
    Ghanem, T.F., Elkilani, W.S., Abdul-Kader, H.M.: A hybrid approach for efficient anomaly detection using metaheuristic methods. J. Adv. Res. 6(4), 609–619 (2015)CrossRefGoogle Scholar
  15. 15.
    Shilton, A., Rajasegarar, S., Palaniswami, M.: Combined multiclass classification and anomaly detection for large-scale wireless sensor networks. In: 2013 IEEE Eighth International Conference on Intelligent Sensors, Sensor Networks and Information Processing, April, pp. 491–496. IEEE (2013)Google Scholar
  16. 16.
    ccFraud Dataset: Apr. 2017. Accessed 01 June 2017
  17. 17.
    Kamaruddin, S., Ravi, V.: Credit card fraud detection using Big Data analytics: use of PSOAANN based one-class classification. In: Proceedings of the International Conference on Informatics and Analytics, August, p. 33. ACM (2016)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringIndian Institute of Technology (ISM)DhanbadIndia

Personalised recommendations