Abstract
With the rising of enormous size in any field be in astronomy, health or education. Storage, analysis, and processing of big data are not only the big issue but security of these huge data is now big concern in the field of academia and industry. Network intrusion detection systems detect and stop network behaviors that interrupt or threaten network security. Supervised learning methods for intrusion detection are not so effective in detection of attacks of big data. Unsupervised models are mainly used for the detection of events or attributes that occur together. A new parallel K-medoid clustering method and k-nearest neighbor classification techniques are proposed for intrusion detection for huge amount of data. We have used NSL-KDD and UNSW-NB-15 datasets for experimental work. The results are compared with the proposed method. Experiments are performed in Hadoop environment and the performance is evaluated using accuracy, precision and confusion matrix.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Scarfone, K., Mell, P.: Guide to Intrusion Detection and Prevention Systems (IDPS). National Institute of Standards and Technology, Gaithersburg (2007)
Bilal, M., et al.: Big data in the construction industry: a review of present status, opportunities, and future trends. Adv. Eng. Inform. 30(3), 500–521 (2016)
Ghemawat, S., Gobioff, H., Leung, S.: The Google file system. In: Proceedings of ACM Symposium on Operating Systems Principles, Lake George, pp. 29–43 (2003)
Lee, K.-H., Lee, Y.-J., Choi, H., Chung, Y.D., Moon, B.: Parallel data processing with mapreduce: a survey. ACM SIGMOD Rec. 40(4), 11–20 (2012)
Marchal, S., Jiangz, X., State, R., Engel, T.: A big data architecture for large scale security monitoring. In: IEEE International Congress on Big Data, pp. 56–63 (2014)
Jianliang, M., Haikun, S., Ling, B.: The application on intrusion detection based on K-means cluster algorithm. In: International Forum on Information Technology and Application, vol. 1, pp. 150–152 (2009)
Wang, L., Jones, R.: Big data analytics for network intrusion detection: a survey. Int. J. Netw. Commun. 7(1), 24–31 (2017)
Sekar, R., et al.: Specification-based anomaly detection: a new approach for detecting network intrusions. In Proceedings of 9th ACM Conference on Computer Communications Security (CCS), pp. 265–274 (2002)
Brown, Z.: Big data. 2nd edn. (2014). http://technologyadvice.com/category/big-data/
Siddiqui, M.K., Naahid, S.: Analysis of KDD CUP 99 dataset using clustering based data mining. Int. J. Database Theory Appl. 6, 23–34 (2013)
NSL-KDD dataset. http://iscx.ca/NSL-KDD/(2006)
https://web.archive.org/web/20150205070216/http://nsl.cs.unb.ca/NSL-KDD/
Moustafa, N., Slay, J.: UNSW-NB15: a comprehensive data set for network intrusion detection systems. Inf. Secur. J. A Glob. Perspect. 25(1), 1–6 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Dahiya, P., Srivastava, D.K. (2018). A Comparative Evolution of Unsupervised Techniques for Effective Network Intrusion Detection in Hadoop. In: Singh, M., Gupta, P., Tyagi, V., Flusser, J., Ören, T. (eds) Advances in Computing and Data Sciences. ICACDS 2018. Communications in Computer and Information Science, vol 906. Springer, Singapore. https://doi.org/10.1007/978-981-13-1813-9_28
Download citation
DOI: https://doi.org/10.1007/978-981-13-1813-9_28
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-1812-2
Online ISBN: 978-981-13-1813-9
eBook Packages: Computer ScienceComputer Science (R0)