KMDT: A Hybrid Cluster Approach for Anomaly Detection Using Big Data
In the current digital era, huge data are being generated in a voluminous state from different sources. This lead towards a processing repository called Big Data. Managing and processing such data in parallel clusters is a big challenge. To capture this problem, in this paper, we propose a hybrid algorithm for cluster analysis using the Spark framework for analyzing the Big Data instances. The proposed algorithm is the combination of two machine learning techniques namely, K-Means (KM) and C5.0 Decision Tree (DT). As per the factor of cluster, euclidean distance is used to find the nearest cluster and the related DT is built for each cluster using C5.0 DT algorithm. The inferences of the DT are used to classify each anomaly and the normal instances of the large datasets. Experimental results show that the proposed hybrid algorithm outperforms with other existing algorithms and produces better classification accuracy for anomaly detection.
KeywordsHadoop Spark K-means Decision tree Big Data
This work is partially supported by Indian Institute of Technology (ISM), Govt. of India. The authors wish to express their gratitude and thanks to the Department of Computer Science and Engineering, Indian Institute of Technology (ISM), Dhanbad, India for providing their support in arranging necessary computing facilities.
- 1.Hayes, M.A., Capretz, M.A.: Contextual anomaly detection in big sensor data. In: 2014 IEEE International Congress on Big Data (BigData Congress), June, pp. 64–71. IEEE (2014)Google Scholar
- 3.Rettig, L., Khayati, M., Cudré-Mauroux, P., Piórkowski, M.: Online anomaly detection over Big Data streams. In: 2015 IEEE International Conference on Big Data (Big Data), October, pp. 1113–1122. IEEE (2015)Google Scholar
- 5.Bottesch, T., Bühler, T., Kächele, M.: Speeding up k-means by approximating Euclidean distances via block vectors. In: International Conference on Machine Learning, June, pp. 2578–2586 (2016)Google Scholar
- 6.Rani, M.S., Xavier, S.B.: A hybrid intrusion detection system based on C5. 0 decision tree and one-class SVM. Int. J. Current Eng. Technol. 5(3) (2015)Google Scholar
- 7.Patil, N., Lathi, R., Chitre, V.: Comparison of C5.0 & CART classification algorithms using pruning technique. Int. J. Eng. Res. Technol. 1(4) (2012)Google Scholar
- 8.Pandya, R., Pandya, J.: C5. 0 algorithm to improved decision tree with feature selection and reduced error pruning. Int. J. Comput. Appl. 117(16) (2015)Google Scholar
- 11.Shafeeq, A., Hareesha, K.S.: Dynamic clustering of data with modified k-means algorithm. In: Proceedings of the 2012 Conference on Information and Computer Networks, pp. 221–225 (2012)Google Scholar
- 12.Rao, K.H., Srinivas, G., Damodhar, A., Krishna, M.V.: Implementation of anomaly detection technique using machine learning algorithms. Int. J. Comput. Sci. Telecommun. 2(3), 25–31 (2011)Google Scholar
- 15.Shilton, A., Rajasegarar, S., Palaniswami, M.: Combined multiclass classification and anomaly detection for large-scale wireless sensor networks. In: 2013 IEEE Eighth International Conference on Intelligent Sensors, Sensor Networks and Information Processing, April, pp. 491–496. IEEE (2013)Google Scholar
- 16.ccFraud Dataset: Apr. 2017. http://packages.revolutionanalytics.com/datasets/. Accessed 01 June 2017
- 17.Kamaruddin, S., Ravi, V.: Credit card fraud detection using Big Data analytics: use of PSOAANN based one-class classification. In: Proceedings of the International Conference on Informatics and Analytics, August, p. 33. ACM (2016)Google Scholar