Advertisement

AnomalyDetect: An Online Distance-Based Anomaly Detection Algorithm

  • Wunjun Huo
  • Wei WangEmail author
  • Wen Li
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11512)

Abstract

Anomaly detection is a key challenge in data mining, which refers to finding patterns in data that do not conform to expected behavior. It has a wide range of applications in many fields as diverse as finance, medicine, industry, and the Internet. In particular, intelligent operation has made great progress in recent years and has an urgent need for this technology. In this paper, we study the problem of anomaly detection in the context of intelligent operation and find the practical need for high-accuracy, online and universal anomaly detection algorithms in time series database. Based on the existing algorithms, we propose an innovative online distance-based anomaly detection algorithm. K-means and time-space trade-off mechanism are used to reduce the time complexity. Through the experiments on Yahoo! Web-scope S5 dataset we show that our algorithm can detect anomalies accurately. The comparative study of several anomaly detectors verifies the effectiveness and generality of the proposed algorithm.

Keywords

Anomaly detection Time series Online algorithm Euclidean distance Intelligent operation 

Notes

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Grant No. 61672384), Fundamental Research Funds for the Central Universities under Grants No. 0800219373.

References

  1. 1.
    Vallis, O., Hochenbaum, J., Kejariwal, A.: A novel technique for long-term anomaly detection in the cloud. In: HotCloud (2014)Google Scholar
  2. 2.
    Yahoo: S5 - A Labeled Anomaly Detection Dataset, version 1.0 (2015). http://webscope.sandbox.yahoo.com/catalog.php?datatype=s&did=70
  3. 3.
    Laptev, N., Amizadeh, S., Flint, I.: Generic and scalable framework for automated time-series anomaly detection. In: KDD, pp. 1939–1947 (2015)Google Scholar
  4. 4.
    Huang, C., Min, G., Wu, Y., Ying, Y., Pei, K., Xiang, Z.: Time series anomaly detection for trustworthy services in cloud computing systemsGoogle Scholar
  5. 5.
    Sagoolmuang, A., Sinapiromsaran, K.: Median-difference window subseries score for contextual anomaly on time series. In: IC-ICTE (2017)Google Scholar
  6. 6.
    Thill, M., Konen, W., Bäck, T.: Online anomaly detection on the webscope S5 dataset: a comparative study. In: EAIS (2017)Google Scholar
  7. 7.
    Chen, Y., Hu, B., Keogh, E., Batista, G.E.: DTW-D: time series semi-supervised learning from a single exampleGoogle Scholar
  8. 8.
    Suh, S., Chae, D.H., Kang, H.-G., Choi, S.: Echo-state conditional variational autoencoder for anomaly detection. In: International Joint Conference on Neural Networks (IJCNN) (2016)Google Scholar
  9. 9.
    Yu, Q., Jibin, L., Jiang, L.: An improved ARIMA-based traffic anomaly detection algorithm for wireless sensor networks. Int. J. Distrib. Sens. Netw. 12(1), 9653230 (2016)CrossRefGoogle Scholar
  10. 10.
    Hyndman, R.J., Wang, E., Laptev, N.: Large-scale unusual time series detectionGoogle Scholar
  11. 11.
    Wei, L., Kumar, N., Lolla, V., Keogh, E., Lonardi, S., Ann Ratanamahatana, C.: Assumption-free anomaly detection in time series. In: 17th International Conference on Scientific and Statistical (2005)Google Scholar
  12. 12.
    Berndt, D.J., Clifford, J.: Using dynamic time warping to find patterns in time series. AAAI Technical Report WS-94-03Google Scholar
  13. 13.
    Chandola, V., Cheboli, D., Kumar, V.: Detecting anomalies in a time series database. Department, University of Minnesota, Technical report 12 (2009)Google Scholar
  14. 14.
    Welford, B.P.: Note on a method for calculating corrected sums of squares and products. Technometrics 4(3), 419–420 (1962)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Watson, S.M., Tight, M., Clark, S., Redfern, E.: Detection of outliers in time series. Institute of Transport Studies, University of LeedsGoogle Scholar
  16. 16.
    Keogh, Eamonn, Lin, Jessica, Lee, Sang-Hee, Van Herle, Helga: Finding the most unusual time series subsequence: algorithms and applications. Knowl. Inf. Syst. 11(1), 1–27 (2007)CrossRefGoogle Scholar
  17. 17.
    Siffer, A., Fouque, P.A., Termier, A., Largouët, C.: Anomaly detection in streams with extreme value theory. In: Proceedings of the 23rd ACM SIGKDD International Conference (2017)Google Scholar
  18. 18.
    Lin, J., Keogh, E., Lonardi, S. Chiu, B.: A symbolic representation of time series, with implications for streaming algorithms. In: Proceedings of the 8th ACM SIGMOD Workshop on Research Issues in Data Mining and Knowledge Discovery (2003)Google Scholar
  19. 19.
    Chen, Y., Mahajan, R., Sridharan, B., Zhang, Z.-L.: A provider-side view of web search response time. In: Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM, vol. 43 (2013)Google Scholar
  20. 20.
    Yan, H.,: Argus: end-to-end service anomaly detection and localization from an ISP’s point of view. In: IEEE INFOCOM (2012)Google Scholar
  21. 21.
  22. 22.
  23. 23.
    Candès, E.J., Li, X., Ma, Y., Wright, J.: Robust principal component analysis (2011)Google Scholar
  24. 24.
  25. 25.
    Laptev, N., Amizadeh, S.: Yahoo anomaly detection dataset S5 (2015). http://webscope.sandbox.yahoo.com/catalog.php?datatype=s&did=70
  26. 26.
  27. 27.
    Zhang, S., et al.: Rapid and robust impact assessment of software changes in large internet-based services. In: CoNEXT 2015, 01–04 December 2015, Heidelberg, Germany (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringTongji UniversityShanghaiChina
  2. 2.School of Data Science and EngineeringEast China Normal UniversityShanghaiChina

Personalised recommendations