Abstract
Anomaly based intrusion detection systems classify network traffic into normal and malicious categories. The intrusion detection system raises an alert when maliciousness is detected in the traffic. A security administrator inspects these alerts and takes corrective action to protect the network from intrusions and unauthorized access. Manual inspection of the alerts is also necessary because anomaly based intrusion detection systems have a high false positive rate. The alerts can be in very large number and their manual inspection is a challenging task. We propose an extension for anomaly based intrusion detection system which automatically groups malicious IP flows into different attack clusters. Our technique creates attack clusters from a training set of unlabeled IP flows using unsupervised learning. Every attack cluster consists of malicious IP flows which are similar to each other. We analyze IP flows in every cluster and assign an attack label to them. After the clusters are created, an incoming malicious IP flow is compared with all clusters and the label of the closest cluster is assigned to the IP flow. The intrusion detection system uses labeled flows to raise consolidated anomaly alert for a set of similar IP flows. This approach significantly reduces the overall number of alerts and also generates a high-level map of attack population. We use unsupervised learning techniques for automatic clustering of IP flows. Unsupervised learning is advantageous over supervised learning because the availability of a labeled training set for supervised learning is not always guaranteed. Three unsupervised learning techniques, k-means, self-organizing maps (SOM) and DBSCAN are considered for clustering of malicious IP flows. We evaluated our technique on a flow-based data-set containing different types of malicious flows. Experimental results show that our scheme gives good performance and places majority of the IP flows in correct attack clusters.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
References
Alaidaros, H., Mahmuddin, M., Al Mazari, A.: An overview of flow-based and packet-based intrusion detection performance in high speed networks. In: Proceedings of the International Arab Conference on Information Technology (2011)
Amoli, P.V., Hamalainen, T.: A real time unsupervised NIDS for detecting unknown and encrypted network attacks in high speed network. In: 2013 IEEE International Workshop on Measurements and Networking Proceedings (M&N), pp. 149–154. IEEE (2013)
Bateni, M., Baraani, A., Ghorbani, A.: Using artificial immune system and fuzzy logic for alert correlation. IJ Netw. Secur. 15(3), 190–204 (2013)
Benferhat, S., Boudjelida, A., Tabia, K., Drias, H.: An intrusion detection and alert correlation approach based on revising probabilistic classifiers using expert knowledge. Appl. Intell. 38(4), 520–540 (2013)
Birant, D., Kut, A.: St-DBSCAN: an algorithm for clustering spatial-temporal data. Data Knowl. Eng. 60(1), 208–221 (2007)
Bolzoni, D., Etalle, S., Hartel, P.H.: Panacea: automating attack classification for anomaly-based network intrusion detection systems. In: Kirda, E., Jha, S., Balzarotti, D. (eds.) RAID 2009. LNCS, vol. 5758, pp. 1–20. Springer, Heidelberg (2009). doi:10.1007/978-3-642-04342-0_1
Boukhtouta, A., Mokhov, S.A., Lakhdari, N.-E., Debbabi, M., Paquet, J.: Network malware classification comparison using DPI and flow packet headers. J. Comput. Virol. Hacking Tech. 12, 69–100 (2015)
Chang, V., Kuo, Y.-H., Ramachandran, M.: Cloud computing adoption framework: a security framework for business clouds. Future Gener. Comput. Syst. 57, 24–41 (2016)
Chang, V., Ramachandran, M.: Towards achieving data security with the cloud computing adoption framework. IEEE Trans. Serv. Comput. 9(1), 138–151 (2016)
Claise, B., Trammell, B., Aitken, P. (eds.): Specification of the IP flow information export (IPFIX) protocol for the exchange of flow information. Technical report, STD 77, RFC 7011, September 2013
Dharamkar, B., Singh, R.R.: Cyber-attack classification using improved ensemble technique based on support vector machine and neural network. Int. J. Comput. Appl. 103(11), 1–7 (2014)
Garcia-Teodoro, P., Diaz-Verdejo, J., Maciá-Fernández, G., Vázquez, E.: Anomaly-based network intrusion detection: techniques, systems and challenges. Comput. Secur. 28(1), 18–28 (2009)
Ghahramani, Z.: Unsupervised learning. In: Bousquet, O., von Luxburg, U., Rätsch, G. (eds.) Advanced Lectures on Machine Learning, vol. 3176, pp. 72–112. Springer, Heidelberg (2004). doi:10.1007/978-3-540-28650-9_5
Golling, M., Hofstede, R., Koch, R.: Towards multi-layered intrusion detection in high-speed networks. In: 2014 6th International Conference on Cyber Conflict (CyCon 2014), pp. 191–206. IEEE (2014)
Haddadi, F., Khanchi, S., Shetabi, M., Derhami, V.: Intrusion detection and attack classification using feed-forward neural network. In: 2010 Second International Conference on Computer and Network Technology (ICCNT), pp. 262–266. IEEE (2010)
Hastie, T., Tibshirani, R., Friedman, J.: Unsupervised learning. In: The Elements of Statistical Learning, pp. 1–101 (2009)
Hoque, N., Bhuyan, M.H., Baishya, R.C., Bhattacharyya, D., Kalita, J.K.: Network attacks: taxonomy, tools and systems. J. Netw. Comput. Appl. 40, 307–324 (2014)
Hurtik, P., Hodakova, P., Perfilieva, I., Liberts, M., Asmuss, J.: Network attack detection and classification by the F-transform. In: 2015 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pp. 1–6. IEEE (2015)
Husak, M., Velan, P., Vykopal, J.: Security monitoring of http traffic using extended flows. In: 2015 10th International Conference on Availability, Reliability and Security (ARES), pp. 258–265. IEEE (2015)
Jin, X., Han, J.: Partitional clustering. In: Sammut, C., Webb, G.I. (eds.) Encyclopedia of Machine Learning, p. 766. Springer, Boston (2010)
Koch, R.: Towards next-generation intrusion detection. In: 2011 3rd International Conference on Cyber Conflict (ICCC), pp. 1–18. IEEE (2011)
Laskov, P., Düssel, P., Schäfer, C., Rieck, K.: Learning intrusion detection: supervised or unsupervised? In: Roli, F., Vitulano, S. (eds.) ICIAP 2005. LNCS, vol. 3617, pp. 50–57. Springer, Heidelberg (2005). doi:10.1007/11553595_6
Li, B., Springer, J., Bebis, G., Gunes, M.H.: A survey of network flow applications. J. Netw. Comput. Appl. 36(2), 567–581 (2013)
Liao, H.-J., Lin, C.-H.R., Lin, Y.-C., Tung, K.-Y.: Intrusion detection system: a comprehensive review. J. Netw. Comput. Appl. 36(1), 16–24 (2013)
Nazeer, K.A., Sebastian, M.: Improving the accuracy and efficiency of the k-means clustering algorithm. In: Proceedings of the World Congress on Engineering, vol. 1, pp. 1–3 (2009)
Pakhira, M.K.: Finding number of clusters before finding clusters. Procedia Technol. 4, 27–37 (2012)
Rieck, K., Trinius, P., Willems, C., Holz, T.: Automatic analysis of malware behavior using machine learning. J. Comput. Secur. 19(4), 639–668 (2011)
Rokach, L., Maimon, O.: Clustering methods. In: Maimon, O., Rokach, L. (eds.) Data Mining and Knowledge Discovery Handbook, pp. 321–352. Springer, Heidelberg (2005). doi:10.1007/0-387-25465-X_15
Shrivastava, P., Gupta, H.: A review of density-based clustering in spatial data. Int. J. Adv. Comput. Res. (IJACR) 2, 200–202 (2012)
Song, J., Takakura, H., Okabe, Y., Nakao, K.: Toward a more practical unsupervised anomaly detection system. Inf. Sci. 231, 4–14 (2013)
Sperotto, A., Pras, A.: Flow-based intrusion detection. In: 2011 IFIP/IEEE International Symposium on Integrated Network Management (IM), pp. 958–963. IEEE (2011)
Sperotto, A., Sadre, R., Vliet, F., Pras, A.: A labeled data set for flow-based intrusion detection. In: Nunzi, G., Scoglio, C., Li, X. (eds.) IPOM 2009. LNCS, vol. 5843, pp. 39–50. Springer, Heidelberg (2009). doi:10.1007/978-3-642-04968-2_4
Umer, M.F., Khiyal, M.S.H.: Classification of textual documents using learning vector quantization. Inf. Technol. J. 6(1), 154–159 (2007)
Van Hulle, M.M.: Self-organizing maps. In: Rozenberg, G., Bäck, T., Kok, J.N. (eds.) Handbook of Natural Computing, pp. 585–622. Springer, Heidelberg (2012). doi:10.1007/978-3-540-92910-9_19
Wang, L., Leckie, C., Ramamohanarao, K., Bezdek, J.: Automatically determining the number of clusters in unlabeled data sets. IEEE Trans. Knowl. Data Eng. 21(3), 335–350 (2009)
Wu, S.X., Banzhaf, W.: The use of computational intelligence in intrusion detection systems: a review. Appl. Soft Comput. 10(1), 1–35 (2010)
Xu, D., Tian, Y.: A comprehensive survey of clustering algorithms. Ann. Data Sci. 2(2), 165–193 (2015)
Ram Naresh Yadav, B.V., Satyanarayana, B., Vasumathi, D.: A vector space model approach for web attack classification using machine learning technique. In: Satapathy, S.C., Raju, K.S., Mandal, J.K., Bhateja, V. (eds.) Proceedings of the Second International Conference on Computer and Communication Technologies. AISC, vol. 381, pp. 363–373. Springer, Heidelberg (2016). doi:10.1007/978-81-322-2526-3_38
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Umer, M.F., Sher, M. (2017). Automatic Clustering of Malicious IP Flow Records Using Unsupervised Learning. In: Chang, V., Ramachandran, M., Walters, R., Wills, G. (eds) Enterprise Security. ES 2015. Lecture Notes in Computer Science(), vol 10131. Springer, Cham. https://doi.org/10.1007/978-3-319-54380-2_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-54380-2_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-54379-6
Online ISBN: 978-3-319-54380-2
eBook Packages: Computer ScienceComputer Science (R0)