Abstract
In this Chapter we give an overview of statistical methods for anomaly detection (AD), thereby targeting an audience of practitioners with general knowledge of statistics. We focus on the applicability of the methods by stating and comparing the conditions in which they can be applied and by discussing the parameters that need to be set.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Darpa intrusion detection evaluation data set, http://www.ll.mit.edu/mission/communications/ist/corpora/ideval
Kdd cup (1999), data, http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html
Arthur, D., Vassilvitskii, S.: k-means++: the advantages of careful seeding. In: SODA 2007: Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 1027–1035. Society for Industrial and Applied Mathematics, Philadelphia (2007)
Barford, P., Kline, J., Plonka, D., Ron, A.: A signal analysis of network traffic anomalies. In: Proceedings of the 2nd ACM SIGCOMM Workshop on Internet Measurment, IMW 2002, pp. 71–82. ACM, New York (2002)
Borgnat, P., Dewaele, G., Fukuda, K., Abry, P., Cho, K.: Seven years and one day: Sketching the evolution of internet traffic. In: INFOCOM (April 2009)
Bouzida, Y., Cuppens, F., Cuppens-Boulahia, N.A., Gombault, S.N.: Efficient intrusion detection using principal component analysis. In: 3ème Conférence sur la Sécurité et Architectures Réseaux, La Londe, France, Juin, RSM - Dépt. Réseaux, Sécurité et Multimédia (Institut Télécom-Télécom Bretagne) (2004)
Breunig, M.M., Kriegel, H.-P., Ng, R.T., Sander, J.: Lof: Identifying density-based local outliers. ACM SIGMOD Record 29(2), 93–104 (2000)
Brodsky, B., Darkhovsky, B.: Nonparametric Methods in Change-point Problems. Kluwer (1993)
Brown, C., Cowperthwaite, A., Hijazi, A., Somayaji, A.: Analysis of the 1999 darpa/lincoln laboratory ids evaluation data with netadhict. In: CISDA 2009: Proceedings of the Second IEEE International Conference on Computational Intelligence for Security and Defense Applications, pp. 67–73. IEEE Press, Piscataway (2009)
Bucklew, J.: Large Deviation Techniques in Decision, Simulation, andEstimation. Wiley (1985)
Burgess, M., Haugerud, H., Straumsnes, S., Reitan, T.: Measuring system normality. ACM Trans. Comput. Syst. 20(2), 125–160 (2002)
Callegari, C., Gazzarrini, L., Giordano, S., Pagano, M., Pepe, T.: A novel multi time-scales pca-based anomaly detection system. In: 2010 International Symposium on Performance Evaluation of Computer and Telecommunication Systems, SPECTS (2010)
Callegari, C., Gazzarrini, L., Giordano, S., Pagano, M., Pepe, T.: When randomness improves the anomaly detection performance. In: Proceedings of 3rd International Symposium on Applied Sciences in Biomedical and Communication Technologies, ISABEL (2010)
Callegari, C., Giordano, S., Pagano, M.: Application of Wavelet Packet Transform to Network Anomaly Detection. In: Balandin, S., Moltchanov, D., Koucheryavy, Y. (eds.) NEW2AN 2008. LNCS, vol. 5174, pp. 246–257. Springer, Heidelberg (2008)
Callegari, C., Giordano, S., Pagano, M., Pepe, T.: On the use of sketches and wavelet analysis for network anomaly detection. In: IWCMC 2010: Proceedings of the 6th International Wireless Communications and Mobile Computing Conference, pp. 331–335. ACM, New York (2010)
Callegari, C., Giordano, S., Pagano, M., Pepe, T.: Combining sketches and wavelet analysis for multi time-scale network anomaly detection. Computers & Security 30(8), 692–704 (2011)
Callegari, C., Giordano, S., Pagano, M., Pepe, T.: Detecting heavy change in the heavy hitter distribution of network traffic. In: IWCMC, pp. 1298–1303. IEEE Press (2011)
Callegari, C., Giordano, S., Pagano, M., Pepe, T.: Detecting anomalies in backbone network traffic: a performance comparison among several change detection methods. IJSNet 11(4), 205–214 (2012)
Carl, G., Brooks, R.R., Rai, S.: Wavelet based denial-of-service detection. Computers & Security 25(8), 600–615 (2006)
Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: A survey. ACM Comput. Surv. 41(3), 15:1–15:58 (2009)
Charikar, M., Chen, K., Farach-Colton, M.: Finding frequent items in data streams. In: Proc. VLDB Endow, pp. 693–703 (2002)
Chatzigiannakis, V., Papavassiliou, S., Androulidakis, G.: Improving network anomaly detection effectiveness via an integrated multi-metric-multi-link (m3l) pca-based approach. Security and Communication Networks 2(3), 289–304 (2009)
Chen, J., Gupta, A.: Testing and locating variance change points with application to stock prices. J. Am. Statist. Assoc. 92, 739–747 (1997)
Cheung-Mon-Chan, P., Clerot, F.: Finding hierarchical heavy hitters with the count min sketch. In: Proceedings of 4th International Workshop on Internet Performance, Simulation, Monitoring and Measurement, IPS-MOME (2006)
Coifman, R.R., Wickerhauser, M.V.: Entropy-based algorithms for best basis selection. IEEE Transactions on Information Theory 38(2), 713–718 (1992)
Cormode, G., Muthukrishnan, S.: What’s hot and what’s not: Tracking most frequent items dynamically. In: Proceedings of ACM Principles of Database Systems, pp. 296–306 (2003)
Cormode, G., Muthukrishnan, S.: What’s new: Finding significant differences in network data streams. In: Proc. of IEEE Infocom, pp. 1534–1545 (2004)
Cormode, G., Muthukrishnan, S.: An improved data stream summary: the count-min sketch and its applications. Journal of Algorithms 55(1), 58–75 (2005)
Cormode, G., Muthukrishnan, S., Srivastava, D.: Finding hierarchical heavy hitters in data streams. In: Proc. of VLDB, pp. 464–475 (2003)
Dainotti, A., Pescape, A., Ventre, G.: Wavelet-based detection of dos attacks. In: Proceedings of Global Telecommunications Conference, GLOBECOM 2006, pp. 1–6. IEEE (2006)
D’Alconzo, A., Coluccia, A., Romirer-Maierhofer, P.: Distribution-based anomaly detection in 3g mobile networks: from theory to practice. Int. J. Netw. Manag. 20(5), 245–269 (2010)
Daubechies, I.: Orthonormal bases of compactly supported wavelets. Communications on Pure and Applied Mathematics 41, 909–996 (1988)
Daubechies, I.: Ten lectures on Wavelets. CBMS-NSF Series in Applied Mathematics, vol. 61. SIAM, Philadelphia (1992)
Dembo, A., Zeitouni, O.: Large Deviations Techniques and Applications. Springer (1998)
Denning, D.E.: An intrusion-detection model. IEEE Transactions on Software Engineering 13(2), 222–232 (1987)
Dewaele, G., Fukuda, K., Borgnat, P., Abry, P., Cho, K.: Extracting hidden anomalies using sketch and non gaussian multiresolution statistical detection procedures. In: LSAD 2007: Proceedings of the 2007 Workshop on Large Scale Attack Defense, pp. 145–152. ACM, New York (2007)
Ensafi, R., Dehghanzadeh, S., Akbarzadeh, T.M.R.: Optimizing fuzzy k-means for network anomaly detection using pso. In: AICCSA 2008: Proceedings of the 2008 IEEE/ACS International Conference on Computer Systems and Applications, pp. 686–693. IEEE Computer Society, Washington, DC (2008)
Ertöz, L., Eilertson, E., Lazarevic, A., Tan, P.N., Kumar, V., Srivastava, J.P., Dokas, P.: MINDS - Minnesota Intrusion Detection System. MIT Press (2004)
Eskin, E., Arnold, A., Prerau, M., Portnoy, L., Stolfo, S.: A geometric framework for unsupervised anomaly detection: Detecting intrusions in unlabeled data. In: Applications of Data Mining in Computer Security. Kluwer (2002)
Estan, C., Varghese, G.: New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice. ACM Transactions on Computer Systems 21, 270–313 (2003)
Ester, M., Kriegel, H.-P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise, pp. 226–231. AAAI Press (1996)
Fox, K.L., Henning, R.R., Reed, J.H., Simonian, R.P.: A neural network approach towards intrusion detection. In: Proc. 13th National Computer Security Conference. Information Systems Security. Standards - the Key to the Future, vol. I, pp. 124–134 (1990)
Maier, G., Feldmann, A., Paxson, V., Allman, M.: On dominant characteristics of residential broadband internet traffic. In: IEEE IMC (2009)
Gao, J., Hu, G., Yao, X.: Anomaly detection of network traffic based on wavelet packet (2006)
Gilbert, A.C.: Multiscale analysis and data networks. Applied and Computational Harmonic Analysis 10, 185–202 (2001)
Hodge, V., Austin, J.: A survey of outlier detection methodologies. Artif. Intell. Rev. 22(2), 85–126 (2004)
Hsu, D.: Tests for variance shift at an unknown time point. Appl. Statist. 26, 279–284 (1977)
Huang, P., Feldmann, A., Willinger, W.: A non-instrusive, wavelet-based approach to detecting network performance problems. In: IMW 2001: Proceedings of the 1st ACM SIGCOMM Workshop on Internet Measurement, pp. 213–227 (2001)
Inclán, C., Tiao, G.: Use of cumulative sums of squares for retrospective detection of changes of variance. J. Am. Statist. Assoc. 89, 913–923 (1994)
Zaki, M.J., Sequeira, K.: Admit: Anomaly-base data mining for intrusions. In: 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Jul. 2002)
Karp, R.M., Papadimitriou, C.H., Shenker, S.: A simple algorithm for finding frequent elements in streams and bags. ACM Transactions on Database Systems 28 (2003)
Kim, S.S., Narasimha Reddy, A.L., Vannucci, M.: Detecting traffic anomalies using discrete wavelet transform. In: Proceedings of International Conference on Information Networking (ICOIN), Busan, Korea, pp. 1375–1384 (2003)
Lakhina, A.: Diagnosing network-wide traffic anomalies. In: ACM SIGCOMM, pp. 219–230 (2004)
Lakhina, A., Crovella, M., Diot, C.: Characterization of network-wide anomalies in traffic flows. In: ACM Internet Measurement Conference, pp. 201–206 (2004)
Lakhina, A., Crovella, M., Diot, C.: Mining anomalies using traffic feature distributions. In: ACM SIGCOMM (2005)
Lakhina, A., Papagiannaki, K., Crovella, M., Christophe, D., Kolaczyk, E.D., Taft, N.: Structural analysis of network traffic flows. In: Proceedings of the Joint International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS 2004/Performance 2004, pp. 61–72. ACM, New York (2004)
Lazarevic, A., Ozgur, A., Ertoz, L., Srivastava, J., Kumar, V.: A comparative study of anomaly detection schemes in network intrusion detection. In: Proceedings of the Third SIAM International Conference on Data Mining (2003)
Leland, W.E., Taqqu, M.S., Willinger, W., Wilson, D.V.: On the self-similar nature of ethernet traffic (extended version). IEEE/ACM Trans. Netw. 2(1), 1–15 (1994)
Lin, S.-Y., Liu, J.-C., Zhao, W.: Adaptive cusum for anomaly detection and its application to detect shared congestion. Texas A&M University. Technical Report TAMU-CS-TR-2007-1-2 (2007)
Liu, Y., Zhang, L., Guan, Y.: Sketch-based streaming pca algorithm for network-wide traffic anomaly detection. In: Proceedings of International Conference on Distributed Computing Systems (2010)
Lorden, G.: Procedures for reacting to a change in distribution. Ann. Math. Statist. 42, 1897–1908 (1971)
Lu, W., Ghorbani, A.: Network anomaly detection based on wavelet analysis. EURASIP Journal on Advances in Signal Processing (1), 837601 (2009)
Mallat, S.G.: A theory for multiresolution signal decomposition: The wavelet representation. IEEE Transactions on Pattern Analysis and Machine Intelligence 11(7), 674–693 (1989)
Mandjes, M.: Large Deviations for Gaussian Queues. Wiley (2007)
Mandjes, M., Zuraniewski, P.: M/g/∞ transience, and its applications to overload detection. Performance Evaluation 68, 507–527 (2011)
Manku, G.S., Motwani, R.: Approximate frequency counts over data streams. In: VLDB, pp. 346–357 (2002)
Mata, F., Zuraniewski, P., Mandjes, M., Mellia, M.: Anomaly detection in voip traffic with trends. In: Proceedings of the 24th International Teletraffic Congress (2012)
Matteoli, S., Diani, M., Corsini, G.: A tutorial overview of anomaly detection in hyperspectral images. IEEE Aerospace and Electronic Systems Magazine 25(7), 5–28 (2010)
Münz, G., Carle, G.: Application of forecasting techniques and control charts for traffic anomaly detection. In: Proceedings of the 19th ITC Specialist Seminar on Network Usage and Traffic (2008)
Munz, G., Li, S., Carle, G.: Traffic anomaly detection using k-means clustering. In: GI/ITG-Workshop MMBnet (2007)
Muthukrishnan, S.: Data streams: algorithms and applications. In: Proceedings of the Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 413–413. Society for Industrial and Applied Mathematics, Philadelphia (2003)
Oldmeadow, J., Ravinutala, S., Leckie, C.: Adaptive Clustering for Network Intrusion Detection. In: Dai, H., Srikant, R., Zhang, C. (eds.) PAKDD 2004. LNCS (LNAI), vol. 3056, pp. 255–259. Springer, Heidelberg (2004)
Page, E.: Continuous inspection scheme. Biometrika 41, 100–115 (1954)
Pollak, M.: Optimal detection of a change in distribution. Ann. Statist. 13, 206–227 (1985)
Portnoy, L., Eskin, E., Stolfo, S.J.: Intrusion detection with unlabeled data using clustering. In: Proceedings of ACM CSS Workshop on Data Mining Applied to Security (November 2001)
Pukkawanna, S., Fukuda, K.: Combining sketch and wavelet models for anomaly detection. In: 2010 IEEE International Conference on Intelligent Computer Communication and Processing (ICCP), pp. 313–319 (August 2010)
Ramaswamy, S., Rastogi, R., Shim, K.: Efficient algorithms for mining outliers from large data sets. SIGMOD Rec. 29(2), 427–438 (2000)
Resnick, S.: Adventures in Stochastic Processes. Birkhäuser (2002)
Ricciato, F., Coluccia, A., D’Alconzo, A., Veitch, D., Borgnat, P., Abry, P.: On the role of flows and sessions in internet traffic modeling: an explorative toy-model. In: IEEE Globecom (2009)
Ricciato, F., Coluccia, A., D’Alconzo, A.: A review of dos attack models for 3g cellular networks from a system-design perspective. Computer Communications 33(5), 551–558 (2010)
Ringberg, H., Soule, A., Rexford, J., Diot, C.: Sensitivity of pca for traffic anomaly detection. SIGMETRICS Perform. Eval. Rev. 35(1), 109–120 (2007)
Roughan, M., Veitch, D., Abry, P.: Real-time estimation of the parameters of long-range dependence. IEEE/ACM Trans. Netw. 8(4), 467–478 (2000)
Schweller, R., Gupta, A., Parsons, E., Chen, Y.: Reversible sketches for efficient and accurate change detection over network data streams. In: Proceedings of the 4th ACM SIGCOMM Conference on Internet Measurement, IMC 2004, pp. 207–212. ACM, New York (2004)
Shiryaev, A.: On optimum methods in quickest detection problems. Theory Probab. Appl. 8, 22–46 (1963)
Shiryaev, A.: On Markov sufficient statistics in non-additive Bayes problems of sequential analysis. Theory Probab. Appl. 9, 604–618 (1964)
Shlens, J.: A tutorial on principal component analysis (December 2005), http://www.snl.salk.edu/~shlens/pub/notes/pca.pdf
Shyu, M., Chen, S., Sarinnapakorn, K., Chang, L.: A novel anomaly detection scheme based on principal component classifier. In: In IEEE Foundations and New Directions of Data Mining Workshop, in Conjunction with ICDM 2003, pp. 172–179 (2003)
Siegmund, D.: Sequential Analysis. Springer (1985)
Sperotto, A., Mandjes, M., Sadre, R., de Boer, P.T., Pras, A.: Autonomic parameter tuning of anomaly-based IDSs: an SSH case study. IEEE Transactions on Network and Service Management 9, 128–141 (2012)
Subhabrata, B.K., Krishnamurthy, E., Sen, S., Zhang, Y., Chen, Y.: Sketch-based change detection: Methods, evaluation, and applications. In: Internet Measurement Conference, pp. 234–247 (2003)
Svoboda, P., Ricciato, F., Hasenleithner, E., Pilz, R.: Composition of gprs/umts traffic: snapshots from a live network. In: 4th Intl Workshop on Internet Performance, Simulation, Monitoring and Measurement, IPS-MOME 2006, Salzburg (2006)
Tartakovsky, A., Veeravalli, V.: Changepoint detection in multi-channel and distributed systems with applications. In: Applications of Sequential Methodologies, pp. 331–363 (2004)
Thorup, M., Zhang, Y.: Tabulation based 4-universal hashing with applications to second moment estimation. In: SODA 2004: Proceedings of the Fifteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 615–624. Society for Industrial and Applied Mathematics, Philadelphia (2004)
Thottan, M., Ji, C.: Anomaly detection in IP networks. IEEE Trans. on Signal Processing 51(8) (August 2003)
Thottan, M., Liu, G., Ji, C.: Anomaly detection approaches for communication networks. In: Cormode, G., Thottan, M., Sammes, A.J. (eds.) Algorithms for Next Generation Networks. Computer Communications and Networks, pp. 239–261. Springer, London (2010)
Tolle, J., Niggemann, O.: Supporting intrusion detection by graph clustering and graph drawing. Springer (2000)
Traynor, P., McDaniel, P., La Porta, T.: On attack causality in internet-connected cellular networks. In: USENIX Security (August 2007)
Traynor, P., McDaniel, P., La Porta, T.: Security for Telecommunications Networks. Springer (2008)
Vetterli, M., Kovačevic, J.: Wavelets and subband coding. Prentice-Hall, Inc., Upper Saddle River (1995)
Wang, L., Potzelberger, K.: Boundary crossing probability for Brownian motion and general boundaries. J. Appl. Probab. 34, 54–65 (1997)
Wang, W., Battiti, R.: Identifying intrusions in computer networks with principal component analysis. In: ARES 2006: Proceedings of the First International Conference on Availability, Reliability and Security, pp. 270–279. IEEE Computer Society, Washington, DC (2006)
Wang, W., Guan, X., Zhang, X.: A Novel Intrusion Detection Method Based on Principle Component Analysis in Computer Security. In: Yin, F.-L., Wang, J., Guo, C. (eds.) ISNN 2004, Part II. LNCS, vol. 3174, pp. 657–662. Springer, Heidelberg (2004)
Yang, H., Ricciato, F., Lu, S., Zhang, L.: Securing a wireless world. Proceedings of the IEEE 94(2), 442–454 (2006)
Ye, N.: A markov chain model of temporal behavior for anomaly detection. In: Proceedings of the Workshop on Information Assurance and Security (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Callegari, C. et al. (2013). A Methodological Overview on Anomaly Detection. In: Biersack, E., Callegari, C., Matijasevic, M. (eds) Data Traffic Monitoring and Analysis. Lecture Notes in Computer Science, vol 7754. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-36784-7_7
Download citation
DOI: https://doi.org/10.1007/978-3-642-36784-7_7
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-36783-0
Online ISBN: 978-3-642-36784-7
eBook Packages: Computer ScienceComputer Science (R0)