Skip to main content
SpringerLink
Log in

KMDT: A Hybrid Cluster Approach for Anomaly Detection Using Big Data

  • Conference paper
  • First Online:
Information and Decision Sciences

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 701))

Abstract

In the current digital era, huge data are being generated in a voluminous state from different sources. This lead towards a processing repository called Big Data. Managing and processing such data in parallel clusters is a big challenge. To capture this problem, in this paper, we propose a hybrid algorithm for cluster analysis using the Spark framework for analyzing the Big Data instances. The proposed algorithm is the combination of two machine learning techniques namely, K-Means (KM) and C5.0 Decision Tree (DT). As per the factor of cluster, euclidean distance is used to find the nearest cluster and the related DT is built for each cluster using C5.0 DT algorithm. The inferences of the DT are used to classify each anomaly and the normal instances of the large datasets. Experimental results show that the proposed hybrid algorithm outperforms with other existing algorithms and produces better classification accuracy for anomaly detection.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Hayes, M.A., Capretz, M.A.: Contextual anomaly detection in big sensor data. In: 2014 IEEE International Congress on Big Data (BigData Congress), June, pp. 64–71. IEEE (2014)

    Google Scholar 

  2. Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. (CSUR) 41(3), 15 (2009)

    Article  Google Scholar 

  3. Rettig, L., Khayati, M., Cudré-Mauroux, P., Piórkowski, M.: Online anomaly detection over Big Data streams. In: 2015 IEEE International Conference on Big Data (Big Data), October, pp. 1113–1122. IEEE (2015)

    Google Scholar 

  4. Hayes, M.A., Capretz, M.A.: Contextual anomaly detection framework for big sensor data. J. Big Data 2(1), 2 (2015)

    Article  Google Scholar 

  5. Bottesch, T., Bühler, T., Kächele, M.: Speeding up k-means by approximating Euclidean distances via block vectors. In: International Conference on Machine Learning, June, pp. 2578–2586 (2016)

    Google Scholar 

  6. Rani, M.S., Xavier, S.B.: A hybrid intrusion detection system based on C5. 0 decision tree and one-class SVM. Int. J. Current Eng. Technol. 5(3) (2015)

    Google Scholar 

  7. Patil, N., Lathi, R., Chitre, V.: Comparison of C5.0 & CART classification algorithms using pruning technique. Int. J. Eng. Res. Technol. 1(4) (2012)

    Google Scholar 

  8. Pandya, R., Pandya, J.: C5. 0 algorithm to improved decision tree with feature selection and reduced error pruning. Int. J. Comput. Appl. 117(16) (2015)

    Google Scholar 

  9. Wu, X., Kumar, V., Quinlan, J.R., Ghosh, J., Yang, Q., Motoda, H., McLachlan, G.J., Ng, A., Liu, B., Philip, S.Y., Zhou, Z. H.: Top 10 algorithms in data mining. Knowl. Inf. Syst. 14(1), 1–37 (2008)

    Article  Google Scholar 

  10. Maulik, U., Bandyopadhyay, S.: Performance evaluation of some clustering algorithms and validity indices. IEEE Trans. Pattern Anal. Mach. Intell. 24(12), 1650–1654 (2002)

    Article  Google Scholar 

  11. Shafeeq, A., Hareesha, K.S.: Dynamic clustering of data with modified k-means algorithm. In: Proceedings of the 2012 Conference on Information and Computer Networks, pp. 221–225 (2012)

    Google Scholar 

  12. Rao, K.H., Srinivas, G., Damodhar, A., Krishna, M.V.: Implementation of anomaly detection technique using machine learning algorithms. Int. J. Comput. Sci. Telecommun. 2(3), 25–31 (2011)

    Google Scholar 

  13. Muniyandi, A.P., Rajeswari, R., Rajaram, R.: Network anomaly detection by cascading k-Means clustering and C4. 5 decision tree algorithm. Proc. Eng. 30, 174–182 (2012)

    Article  Google Scholar 

  14. Ghanem, T.F., Elkilani, W.S., Abdul-Kader, H.M.: A hybrid approach for efficient anomaly detection using metaheuristic methods. J. Adv. Res. 6(4), 609–619 (2015)

    Article  Google Scholar 

  15. Shilton, A., Rajasegarar, S., Palaniswami, M.: Combined multiclass classification and anomaly detection for large-scale wireless sensor networks. In: 2013 IEEE Eighth International Conference on Intelligent Sensors, Sensor Networks and Information Processing, April, pp. 491–496. IEEE (2013)

    Google Scholar 

  16. ccFraud Dataset: Apr. 2017. http://packages.revolutionanalytics.com/datasets/. Accessed 01 June 2017

  17. Kamaruddin, S., Ravi, V.: Credit card fraud detection using Big Data analytics: use of PSOAANN based one-class classification. In: Proceedings of the International Conference on Informatics and Analytics, August, p. 33. ACM (2016)

    Google Scholar 

Download references

Acknowledgements

This work is partially supported by Indian Institute of Technology (ISM), Govt. of India. The authors wish to express their gratitude and thanks to the Department of Computer Science and Engineering, Indian Institute of Technology (ISM), Dhanbad, India for providing their support in arranging necessary computing facilities.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Indian Institute of Technology (ISM), Dhanbad, 826004, India

    Santosh Thakur & Ramesh Dharavath

Authors
  1. Santosh Thakur
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ramesh Dharavath
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ramesh Dharavath .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Engineering, PVP Siddhartha Institute of Technology, Vijayawada, Andhra Pradesh, India

    Suresh Chandra Satapathy

  2. Departamento de Engenharia Mecânica, Universidade do Porto, Porto, Portugal

    Joao Manuel R.S. Tavares

  3. Department of Electronics and Communication Engineering, SRMGPC, Lucknow, Uttar Pradesh, India

    Vikrant Bhateja

  4. School of Computer Application, KIIT University, Bhubaneswar, Odisha, India

    J. R. Mohanty

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Thakur, S., Dharavath, R. (2018). KMDT: A Hybrid Cluster Approach for Anomaly Detection Using Big Data. In: Satapathy, S., Tavares, J., Bhateja, V., Mohanty, J. (eds) Information and Decision Sciences. Advances in Intelligent Systems and Computing, vol 701. Springer, Singapore. https://doi.org/10.1007/978-981-10-7563-6_18

Download citation

  • DOI: https://doi.org/10.1007/978-981-10-7563-6_18

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-10-7562-9

  • Online ISBN: 978-981-10-7563-6

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation