Advertisement

Computing Anomaly Score Threshold with Autoencoders Pipeline

  • Igr Alexánder Fernández-SaúcoEmail author
  • Niusvel Acosta-Mendoza
  • Andrés Gago-Alonso
  • Edel Bartolo García-Reyes
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11401)

Abstract

Autoencoders neural networks are considered an unsupervised learning algorithm which can be used for detecting anomalies on datasets. In anomaly detection systems powered by autoencoders, the evaluated samples are sorted by the reconstruction error and the anomaly score threshold is set by experts. This threshold helps to select the set of anomaly candidates. In most of the real-world scenarios, the anomaly score threshold estimation is a non-trivial task even for an expert. This paper contains a proposal for an iterative training method based on an autoencoders pipeline to automatically compute the anomaly score threshold. The proposed method achieves encouraging and consistent results collected through the experimentation over two well-known datasets. According to the network configuration, training and tuning, the estimated anomaly score threshold from the proposed method is close to the best possible for the dataset.

Keywords

Anomaly detection Autoencoders Deep neural networks 

References

  1. 1.
    Herrera-Semenets, V., Pérez García, O.A., Gago-Alonso, A., Hernández-León, R.: Classification rule-based models for malicious activity detection. Intell. Data Anal. 21, 1141–1154 (2017)CrossRefGoogle Scholar
  2. 2.
    Denning, D.E.: An intrusion-detection model. IEEE Trans. Softw. Eng. SE-13(2), 222–232 (1987)Google Scholar
  3. 3.
    Jones, A.K., Sielken, R.S.: Computer system intrusion detection: a survey. Technical report, University of Virginia, February 2001Google Scholar
  4. 4.
    Mirsky, Y., Doitshman, T., Elovici, Y., Shabtai, A.: Kitsune: an ensemble of autoencoders for online network intrusion detection. CoRR abs/1802.09089 (2018)Google Scholar
  5. 5.
    Pumsirirat, A., Yan, L.: Credit card fraud detection using deep learning based on auto-encoder and restricted Boltzmann machine. Int. J. Adv. Comput. Sci. Appl. 9, 18–25 (2018)Google Scholar
  6. 6.
    Schreyer, M., Sattarov, T., Borth, D., Dengel, A., Reimer, B.: Detection of anomalies in large scale accounting data using deep autoencoder networks. CoRR abs/1709.05254 (2017)Google Scholar
  7. 7.
    Narasimhan, M.G., Sowmya Kamath, S.: Dynamic video anomaly detection and localization using sparse denoising autoencoders. Multimed. Tools Appl. 77(11), 13173–13195 (2018)CrossRefGoogle Scholar
  8. 8.
    Xu, D., Yan, Y., Ricci, E., Sebe, N.: Detecting anomalous events in videos by learning deep representations of appearance and motion. Comput. Vis. Image Underst. 156, 117–127 (2017). Image and Video Understanding in Big DataCrossRefGoogle Scholar
  9. 9.
    Zheng, Y.J., Zhou, X.H., Sheng, W.G., Xue, Y., Chen, S.Y.: Generative adversarial network based telecom fraud detection at the receiving bank. Neural Networks 102, 78–86 (2018)CrossRefGoogle Scholar
  10. 10.
    Patterson, J., Gibson, A.: Deep Learning: A Practitioner’s Approach. O’Reilly, Beijing (2017)Google Scholar
  11. 11.
    Xu, H., et al.: Unsupervised anomaly detection via variational auto-encoder for seasonal KPIs in web applications. CoRR abs/1802.03903 (2018)Google Scholar
  12. 12.
    Hawkins, S., He, H., Williams, G., Baxter, R.: Outlier detection using replicator neural networks. In: Proceedings of the Fifth International Conference and Data Warehousing and Knowledge Discovery (DaWaK02), pp. 170–180 (2002)Google Scholar
  13. 13.
    An, J., Cho, S.: Variational Autoencoder Based Anomaly Detection Using Reconstruction Probability. Seoul National University, Seoul (2015)Google Scholar
  14. 14.
    Upton, G., Cook, I.: A Dictionary of Statistics. Oxford University Press, New York (2008)CrossRefGoogle Scholar
  15. 15.
    Yu, Y., Long, J., Cai, Z.: Network intrusion detection through stacking dilated convolutional autoencoders. Secur. Commun. Networks 2017, 1–10 (2017)CrossRefGoogle Scholar
  16. 16.
    Gower, J.C., Ross, G.J.S.: Minimum spanning trees and single linkage cluster analysis. J. Roy. Stat. Soc. Ser. C (Appl. Stat.) 18(1), 54–64 (1969)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Igr Alexánder Fernández-Saúco
    • 1
    Email author
  • Niusvel Acosta-Mendoza
    • 2
  • Andrés Gago-Alonso
    • 2
  • Edel Bartolo García-Reyes
    • 2
  1. 1.DATYS - Technological SolutionsHavanaCuba
  2. 2.Advanced Technologies Application Center (CENATAV)HavanaCuba

Personalised recommendations