Unsupervised Anomaly Thresholding from Reconstruction Errors

  • Maryleen U. NdubuakuEmail author
  • Ashiq Anjum
  • Antonio Liotta
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11874)


Internet of Things (IoT) sensors generate massive streaming data which needs to be processed in real-time for many applications. Anomaly detection is one popular way to process such data and discover nuggets of information. Various machine learning techniques for anomaly detection rely on pre-labelled data which is very expensive and not feasible for streaming scenarios. Autoencoders have been found effective for unsupervised outlier removal because of their inherent ability to better reconstruct data with higher density. Our work aims to leverage this principle to investigate approaches through which the optimal threshold for anomaly detection can be obtained in an automated and adaptive fashion for streaming scenarios. Rather than experimentally setting an optimal threshold through trial and error, we obtain the threshold from the reconstruction errors of the training data. Inspired by image processing, we investigate how thresholds set by various statistical approaches can perform in an image dataset.


Anomaly detection Anomaly thresholding Unsupervised learning 


  1. 1.
    Al-amri, S.S., Kalyankar, N.V., Khamitkar, S.D.: Image segmentation by using threshold techniques. arXiv preprint arXiv:1005.4020
  2. 2.
    Amarbayasgalan, T., Jargalsaikhan, B., Ryu, K.: Unsupervised novelty detection using deep autoencoders with density based clustering. Appl. Sci. 8(9), 1468 (2018)CrossRefGoogle Scholar
  3. 3.
    Aytekin, C., Ni, X., Cricri, F., Aksu, E.: Clustering and unsupervised anomaly detection with l\(_2\) normalized deep auto-encoder representations. In: Proceedings of the IJCNN, vol. 2018-July. IEEE, October 2018Google Scholar
  4. 4.
    Bosman, H.H., Iacca, G., Tejada, A., Wörtche, H.J., Liotta, A.: Spatial anomaly detection in sensor networks using neighborhood information. Inf. Fusion 33, 41–56 (2017)CrossRefGoogle Scholar
  5. 5.
    Cauteruccio, F., et al.: Short-long term anomaly detection in wireless sensor networks based on machine learning and multi-parameterized edit distance. Inf. Fusion 52, 13–30 (2019) CrossRefGoogle Scholar
  6. 6.
    Chong, Y.S., Tay, Y.H.: Abnormal event detection in videos using spatiotemporal autoencoder. In: Cong, F., Leung, A., Wei, Q. (eds.) ISNN 2017. LNCS, vol. 10262, pp. 189–196. Springer, Cham (2017). Scholar
  7. 7.
    Erhan, L., et al.: Analyzing objective and subjective data in social sciences: implications for smart cities. IEEE Access 7, 19890–19906 (2019)CrossRefGoogle Scholar
  8. 8.
    Ferrara, E., et al.: A pilot study mapping citizens’ interaction with urban nature. In: IEEE DASC/Piom/CyberSciTech, pp. 828–835. IEEE (2018)Google Scholar
  9. 9.
    Guo, X., Liu, X., Zhu, E., Yin, J.: Deep clustering with convolutional autoencoders. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, E.S. (eds.) ICONIP 2017. LNCS, vol. 10635, pp. 373–382. Springer, Cham (2017). Scholar
  10. 10.
    Xia, Y., Cao, X., Wen, F., Hua, G., Sun, J.: Learning discriminative reconstructions for unsupervised outlier removal. In: Proceedings of the IEEE ICCV, vol. 2015 Inter, pp. 1511–1519 (2015)Google Scholar
  11. 11.
    Yaseen, M.U., Anjum, A., Rana, O., Hill, R.: Cloud-based scalable object detection and classification in video streams. Future Gener. Comput. Syst. 80, 286–298 (2018) CrossRefGoogle Scholar
  12. 12.
    Zamani, A.R., et al.: Deadline constrained video analysis via in-transit computational environments. IEEE Trans. Serv. ComputGoogle Scholar
  13. 13.
    Zhou, C., Paffenroth, R.C.: Anomaly detection with robust deep autoencoders, pp. 665–674 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.University of DerbyDerbyUK
  2. 2.Edinburgh Napier UniversityEdinburghUK

Personalised recommendations