Abstract
System states that are anomalous from the perspective of a domain expert occur with high density in some anomaly detection problems. The performance of commonly used unsupervised anomaly detection methods may suffer in that setting, because they use density as a proxy for anomaly. We propose a novel concept for anomaly detection, called relative anomaly detection. It is tailored to be robust towards anomalies that have high density, by taking into account their location relative to the most typical observations. The approaches we develop are computationally feasible even for large data sets, and they allow real-time detection. We illustrate using data sets of potential scraping attempts and Wi-Fi channel utilization, both from Google.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Aggarwal, C.C., Hinneburg, A., Keim, D.A.: On the surprising behavior of distance metrics in high dimensional space. In: Bussche, J., Vianu, V. (eds.) ICDT 2001. LNCS, vol. 1973, pp. 420–434. Springer, Heidelberg (2001). doi:10.1007/3-540-44503-X_27
Angiulli, F., Basta, S., Pizzuti, C.: Distance-based detection and prediction of outliers. IEEE Trans. Knowl. Data Eng. 18(2), 145–160 (2006)
Bonacich, P.: Factoring and weighting approaches to status scores and clique identification. J. Math. Sociol. 2(1), 113–120 (1972)
Box, G.E.P., Cox, D.R.: An analysis of transformations. J. R. Stat. Soc. Ser. B (Methodological) 26(2), 211–252 (1964)
Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. (CSUR) 41(3), 15 (2009)
Eskin, E., Arnold, A., Prerau, M., Portnoy, L., Stolfo, S.: A geometric framework for unsupervised anomaly detection. In: Barbará, D., Jajodia, S. (eds.) Applications of Data Mining in Computer Security. Advances in Information Security, vol. 6, pp. 77–101. Springer, Heidelberg (2002)
Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York (2009)
He, Z., Xiaofei, X., Deng, S.: Discovering cluster-based local outliers. Pattern Recognit. Lett. 24(9), 1641–1650 (2003)
Isaacson, D.L., Madsen, R.W.: Markov Chains, Theory and Applications, vol. 4. Wiley, New York (1976)
Moonesinghe, H.D.K., Tan, P.-N.: Outlier detection using random walks. In: 18th IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2006), pp. 532–539. IEEE (2006)
Page, L., Brin, S., Motwani, R., Winograd, T.: The pagerank citation ranking: bringing order to the web (1999)
Rahimi, A., Recht, B.: Random features for large-scale kernel machines. In: Advances in Neural Information Processing Systems, pp. 1177–1184 (2007)
Rasmussen, C.E., Williams, C.K.I.: Gaussian processes for machine learning (2006)
Schölkopf, B., Williamson, R.C., Smola, A.J., Shawe-Taylor, J., Platt, J.C., et al.: Support vector method for novelty detection. In: NIPS, vol. 12, pp. 582–588. Citeseer (1999)
Smola, A.J., Song, L., Teo, C.H., et al.: Relative novelty detection. In: AISTATS, vol. 12, pp. 536–543 (2009)
Williams, C., Seeger, M.: The effect of the input density distribution on kernel-based classifiers. In: Proceedings of the 17th International Conference on Machine Learning, number EPFL-CONF-161323, pp. 1159–1166 (2000)
Williams, C., Seeger, M.: Using the nyström method to speed up kernel machines. In: Proceedings of the 14th Annual Conference on Neural Information Processing Systems, number EPFL-CONF-161322, pp. 682–688 (2001)
Zimek, A., Schubert, E., Kriegel, H.-P.: A survey on unsupervised outlier detection in high-dimensional numerical data. Stat. Anal. Data Mining ASA Data Sci. J. 5(5), 363–387 (2012)
Acknowledgments
We thank Mitch Trott, Phil Keller and Robbie Haertel of Google as well as Lauren Hannah of Columbia University for many helpful comments, and furthermore Dave Peters and Taghrid Samak of Google for granting us access to their data sets.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Neuberg, R., Shi, Y. (2017). Detecting Relative Anomaly. In: Perner, P. (eds) Machine Learning and Data Mining in Pattern Recognition. MLDM 2017. Lecture Notes in Computer Science(), vol 10358. Springer, Cham. https://doi.org/10.1007/978-3-319-62416-7_9
Download citation
DOI: https://doi.org/10.1007/978-3-319-62416-7_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-62415-0
Online ISBN: 978-3-319-62416-7
eBook Packages: Computer ScienceComputer Science (R0)