Abstract
Network anomaly detection relies on intrusion detection systems based on knowledge databases. However, building this knowledge may take time as it requires manual inspection of experts. Actual detection systems are unable to deal with 0-day attack or new user’s behavior and in consequence they may fail in correctly detecting intrusions. Unsupervised network anomaly detectors overcome this issue as no previous knowledge is required. In counterpart, these systems may be very slow as they need to learn traffic’s pattern in order to acquire the necessary knowledge to detect anomalous flows. To improve speed, these systems are often only exposed to sampled traffic, harmful traffic may then avoid the detector examination. In this paper, we propose to take advantage of new distributed computing framework in order to speed up an Unsupervised Network Anomaly Detector Algorithm, UNADA. The evaluation shows that the execution time can be improved by a factor of 13 allowing UNADA to process large traces of traffic in real time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Apache spark - lightning-fast cluster computing. https://spark.apache.org/ (accessed April 29, 2015)
Grid5000. https://www.grid5000.fr (accessed April 29, 2015)
Andrade, G., Ramos, G., Madeira, D., Sachetto, R., Ferreira, R., Rocha, L.: G-dbscan: A GPU accelerated algorithm for density-based clustering. Procedia Computer Science, 369–378 (2013)
Brauckhoff, D., Tellenbach, B., Wagner, A., May, M., Lakhina, A.: Impact of packet sampling on anomaly detection metrics. In: Proc. of the 6th ACM SIGCOMM Conference on Internet Measurement, pp. 159–164 (2006)
Casas, P., Mazel, J., Owezarski, P.: Unsupervised network intrusion detection systems: Detecting the unknown without knowledge. Computer Communications, 772–783 (2012)
Celenk, M., Conley, T., Willis, J., Graham, J.: Anomaly detection and visualization using fisher discriminant clustering of network entropy. In: Third International Conference on Digital Information Management, pp. 216–220, November 2008
Dewaele, G., Fukuda, K., Borgnat, P., Abry, P., Cho, K.: Extracting hidden anomalies using sketch and non gaussian multiresolution statistical detection procedures. In: Proc. of the 2007 Workshop on Large Scale Attack Defense, pp. 145–152. ACM (2007)
Ester, M., peter Kriegel, H., S, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise, pp. 226–231. AAAI Press (1996)
Ester, M., Kriegel, H.P., Sander, J., Wimmer, M., Xu, X.: Incremental clustering for mining in a data warehousing environment. In: Proc. of the 24rd International Conference on Very Large Data Bases, pp. 323–333 (1998)
Fahad, A., Alshatri, N., Tari, Z., Alamri, A., Khalil, I., Zomaya, A., Foufou, S., Bouras, A.: A survey of clustering algorithms for big data: Taxonomy and empirical analysis. IEEE Transactions on Emerging Topics in Computing, 267–279, September 2014
Fontugne, R., Mazel, J., Fukuda, K.: Hashdoop: a mapreduce framework for network anomaly detection. In: INFOCOM WKSHPS, pp. 494–499, April 2014
Fontugne, R., Fukuda, K.: A hough-transform-based anomaly detector with an adaptive time interval. SIGAPP Appl. Comput. Rev., 41–51 (2011)
Gu, Y., McCallum, A., Towsley, D.: Detecting anomalies in network traffic using maximum entropy estimation. In: Proc. of the 5th ACM SIGCOMM Conference on Internet Measurement, pp. 32–32 (2005)
Kanda, Y., Fukuda, K., Sugawara, T.: Evaluation of anomaly detection based on sketch and pca. In: GLOBECOM 2010, pp. 1–5. IEEE (2010)
Kriegel, H.P., Kroger, P., Zimek, A.: Clustering high-dimensional data: A survey on subspace clustering, pattern-based clustering, and correlation clustering. ACM Trans. Knowl. Discov. Data (2009)
Lakhina, A., Crovella, M., Diot, C.: Diagnosing network-wide traffic anomalies. In: Proc. of ACM SIGCOMM 2004, pp. 219–230, Auguest 2004
Patcha, A., Park, J.M.: An overview of anomaly detection techniques: Existing solutions and latest technological trends. Comput. Netw., 3448–3470 (2007)
Portnoy, L., Eskin, E., Stolfo, S.: Intrusion detection with unlabeled data using clustering. In: Proc. of ACM CSS Workshop on Data Mining Applied to Security, pp. 5–8 (2001)
Wei, X., Huang, H., Tian, S.: A grid-based clustering algorithm for network anomaly detection. In: The First International Symposium on Data, Privacy, and E-Commerce, ISDPE 2007, pp. 104–106, November 2007
Xin, R.S., Rosen, J., Zaharia, M., Franklin, M.J., Shenker, S., Stoica, I.: Shark: SQL and rich analytics at scale. In: Proc. of the 2013 ACM SIGMOD International Conference on Management of Data, pp. 13–24 (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Dromard, J., Roudière, G., Owezarski, P. (2015). Unsupervised Network Anomaly Detection in Real-Time on Big Data. In: Morzy, T., Valduriez, P., Bellatreche, L. (eds) New Trends in Databases and Information Systems. ADBIS 2015. Communications in Computer and Information Science, vol 539. Springer, Cham. https://doi.org/10.1007/978-3-319-23201-0_22
Download citation
DOI: https://doi.org/10.1007/978-3-319-23201-0_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-23200-3
Online ISBN: 978-3-319-23201-0
eBook Packages: Computer ScienceComputer Science (R0)