Abstract
Internet of Things (IoT) sensors generate massive streaming data which needs to be processed in real-time for many applications. Anomaly detection is one popular way to process such data and discover nuggets of information. Various machine learning techniques for anomaly detection rely on pre-labelled data which is very expensive and not feasible for streaming scenarios. Autoencoders have been found effective for unsupervised outlier removal because of their inherent ability to better reconstruct data with higher density. Our work aims to leverage this principle to investigate approaches through which the optimal threshold for anomaly detection can be obtained in an automated and adaptive fashion for streaming scenarios. Rather than experimentally setting an optimal threshold through trial and error, we obtain the threshold from the reconstruction errors of the training data. Inspired by image processing, we investigate how thresholds set by various statistical approaches can perform in an image dataset.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Al-amri, S.S., Kalyankar, N.V., Khamitkar, S.D.: Image segmentation by using threshold techniques. arXiv preprint arXiv:1005.4020
Amarbayasgalan, T., Jargalsaikhan, B., Ryu, K.: Unsupervised novelty detection using deep autoencoders with density based clustering. Appl. Sci. 8(9), 1468 (2018)
Aytekin, C., Ni, X., Cricri, F., Aksu, E.: Clustering and unsupervised anomaly detection with l\(_2\) normalized deep auto-encoder representations. In: Proceedings of the IJCNN, vol. 2018-July. IEEE, October 2018
Bosman, H.H., Iacca, G., Tejada, A., Wörtche, H.J., Liotta, A.: Spatial anomaly detection in sensor networks using neighborhood information. Inf. Fusion 33, 41–56 (2017)
Cauteruccio, F., et al.: Short-long term anomaly detection in wireless sensor networks based on machine learning and multi-parameterized edit distance. Inf. Fusion 52, 13–30 (2019)
Chong, Y.S., Tay, Y.H.: Abnormal event detection in videos using spatiotemporal autoencoder. In: Cong, F., Leung, A., Wei, Q. (eds.) ISNN 2017. LNCS, vol. 10262, pp. 189–196. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59081-3_23
Erhan, L., et al.: Analyzing objective and subjective data in social sciences: implications for smart cities. IEEE Access 7, 19890–19906 (2019)
Ferrara, E., et al.: A pilot study mapping citizens’ interaction with urban nature. In: IEEE DASC/Piom/CyberSciTech, pp. 828–835. IEEE (2018)
Guo, X., Liu, X., Zhu, E., Yin, J.: Deep clustering with convolutional autoencoders. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, E.S. (eds.) ICONIP 2017. LNCS, vol. 10635, pp. 373–382. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70096-0_39
Xia, Y., Cao, X., Wen, F., Hua, G., Sun, J.: Learning discriminative reconstructions for unsupervised outlier removal. In: Proceedings of the IEEE ICCV, vol. 2015 Inter, pp. 1511–1519 (2015)
Yaseen, M.U., Anjum, A., Rana, O., Hill, R.: Cloud-based scalable object detection and classification in video streams. Future Gener. Comput. Syst. 80, 286–298 (2018)
Zamani, A.R., et al.: Deadline constrained video analysis via in-transit computational environments. IEEE Trans. Serv. Comput
Zhou, C., Paffenroth, R.C.: Anomaly detection with robust deep autoencoders, pp. 665–674 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Ndubuaku, M.U., Anjum, A., Liotta, A. (2019). Unsupervised Anomaly Thresholding from Reconstruction Errors. In: Montella, R., Ciaramella, A., Fortino, G., Guerrieri, A., Liotta, A. (eds) Internet and Distributed Computing Systems . IDCS 2019. Lecture Notes in Computer Science(), vol 11874. Springer, Cham. https://doi.org/10.1007/978-3-030-34914-1_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-34914-1_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-34913-4
Online ISBN: 978-3-030-34914-1
eBook Packages: Computer ScienceComputer Science (R0)