Advertisement

Outlier Detection Based on Local Density of Vector Dot Product in Data Stream

  • Zhaoyu ShouEmail author
  • Fengbo ZouEmail author
  • Hao Tian
  • Simin Li
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 895)

Abstract

Outlier detection in data stream is an increasingly important research in many fields. To deal with the data stream with the properties of high dimension, rapid arrival in order, high cost of storing all data in memory and so on, an outlier detection algorithm based on local density of vector dot product in data stream (LDVP-OD) is proposed. LDVP-OD uses the model based on sliding window and multiple validations to decrease the false alarm rate, which divides the data stream into uniform-sized blocks. Local density of vector dot product (LDVP) is described in order to precisely evaluate the outlierness of data in data stream. Furthermore, an outlier judgment criterion based on supreme slope is introduced, which can determine the exact outliers without requiring the number of outliers or other parameters beforehand. Comparison experiments with existing algorithms on synthetic and real datasets prove the high detection rate, good stability, strong adaptability of LDVP-OD.

Keywords

Outlier detection Data stream Local density of vector dot product Multiple validations 

Notes

Acknowledgments

This work is supported by the following foundations: the national Natural Science Foundation of China (61662013, 61362021, U1501252); Natural Science Foundation of Guangxi province (2016GXNSFAA380149); Guangxi Innovation-Driven Development Project (Science and Technology Major Project) (AA17202024); the Key Laboratory of Cognitive Radio and Information Processing Ministry of Education (2011KF11); Innovation Project of GUET Graduate Education (2017YJCX34, 2018YJCX37).

References

  1. 1.
    Angelov, P.: Outside the box: an alternative data analytics framework. J. Autom. Mob. Rob. Intell. Syst. 8(2), 29–35 (2014).  https://doi.org/10.14313/JAMRIS_2-2014/16MathSciNetCrossRefGoogle Scholar
  2. 2.
    Angiulli, F., Fassetti, F.: Distance-based outlier queries in data streams: the novel task and algorithms. Data Min. Knowl. Disc. 20(2), 290–324 (2010).  https://doi.org/10.1007/s10618-009-0159-9MathSciNetCrossRefGoogle Scholar
  3. 3.
    Breunig, M.M., Kriegel, H.P., Ng, R.T., Sander, J.: LOF: identifying density-based local outliers. In: ACM Sigmod Record. vol. 29, pp. 93–104. ACM (2000).  https://doi.org/10.1145/335191.335388
  4. 4.
    Cao, L., Yang, D., Wang, Q., Yu, Y., Wang, J., Rundensteiner, E.A.: Scalable distance-based outlier detection over high-volume data streams. In: 2014 IEEE 30th International Conference on Data Engineering (ICDE), pp. 76–87. IEEE (2014).  https://doi.org/10.1109/ICDE.2014.6816641
  5. 5.
    Carcillo, F., Dal Pozzolo, A., Le Borgne, Y.A., Caelen, O., Mazzer, Y., Bontempi, G.: SCARFF: a scalable framework for streaming credit card fraud detection with spark. Inf. Fusion 41, 182–194 (2018).  https://doi.org/10.1016/j.inffus.2017.09.005CrossRefGoogle Scholar
  6. 6.
    Cremonesi, P., Koren, Y., Turrin, R.: Performance of recommender algorithms on top-n recommendation tasks. In: Proceedings of the Fourth ACM Conference on Recommender Systems. pp. 39–46. ACM (2010).  https://doi.org/10.1145/1864708.1864721
  7. 7.
    Dheeru, D., Karra Taniskidou, E.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml
  8. 8.
    Ester, M., Kriegel, H.P., Sander, J., Xu, X., et al.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: KDD, vol. 96, pp. 226–231 (1996). http://dl.acm.org/citation.cfm?id=3001460.3001507
  9. 9.
    Gao, K., Shao, F.J., Sun, R.C.: n-INCLOF: a dynamic local outlier detection algorithm for data streams. In: 2010 2nd International Conference on Signal Processing Systems (ICSPS), vol. 2, p. V2–179. IEEE (2010).  https://doi.org/10.1109/ICSPS.2010.5555276
  10. 10.
    Golab, L., Özsu, M.T.: Issues in data stream management. ACM Sigmod Rec. 32(2), 5–14 (2003).  https://doi.org/10.1145/776985.776986CrossRefGoogle Scholar
  11. 11.
    Ha, J., Seok, S., Lee, J.S.: Robust outlier detection using the instability factor. Knowl. Based Syst. 63, 15–23 (2014).  https://doi.org/10.1016/j.knosys.2014.03.001CrossRefGoogle Scholar
  12. 12.
    Huang, J.Z., Ng, M.K., Rong, H., Li, Z.: Automated variable weighting in k-means type clustering. IEEE Trans. Patt. Anal. Mach. Intell. 27(5), 657–668 (2005).  https://doi.org/10.1109/TPAMI.2005.95CrossRefGoogle Scholar
  13. 13.
    Karimian, S.H., Kelarestaghi, M., Hashemi, S.: I-incLOF: improved incremental local outlier detection for data streams. In: 2012 16th CSI International Symposium on Artificial Intelligence and Signal Processing (AISP), pp. 023–028. IEEE (2012).  https://doi.org/10.1109/AISP.2012.6313711
  14. 14.
    Knorr, E.M., Ng, R.T.: Algorithms for mining distance-based outliers in large datasets. In: Proceedings of the 24th International Conference on Very Large Data Bases, VLDB 1998. pp. 392–403. Morgan Kaufmann Publishers Inc., San Francisco (1998). http://dl.acm.org/citation.cfm?id=645924.671334
  15. 15.
    Kontaki, M., Gounaris, A., Papadopoulos, A.N., Tsichlas, K., Manolopoulos, Y.: Continuous monitoring of distance-based outliers over data streams. In: 2011 IEEE 27th International Conference on Data Engineering (ICDE), pp. 135–146. IEEE (2011).  https://doi.org/10.1109/ICDE.2011.5767923
  16. 16.
    Kriegel, H.P., Kröger, P., Schubert, E., Zimek, A.: Loop: local outlier probabilities. In: Proceedings of the 18th ACM Conference on Information and Knowledge Management. pp. 1649–1652. ACM (2009).  https://doi.org/10.1145/1645953.1646195
  17. 17.
    Kriegel, H.P., Zimek, A., et al.: Angle-based outlier detection in high-dimensional data. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 444–452. ACM (2008).  https://doi.org/10.1145/1401890.1401946
  18. 18.
    Miller, Z., Dickinson, B., Deitrick, W., Hu, W., Wang, A.H.: Twitter spammer detection using data stream clustering. Inf. Sci. 260, 64–73 (2014).  https://doi.org/10.1016/j.ins.2013.11.016CrossRefGoogle Scholar
  19. 19.
    Neeraj, K.: Anomaly-based network intrusion detection: an outlier detection techniques. In: Proceedings of the Eighth International Conference on Soft Computing and Pattern Recognition. vol. 614. Springer, Cham (2018).  https://doi.org/10.1007/978-3-319-60618-7_26
  20. 20.
    Pang, G., Cao, L., Chen, L., Liu, H.: Learning representations of ultrahigh-dimensional data for random distance-based outlier detection. arXiv preprint arXiv:1806.04808 (2018)
  21. 21.
    Pokrajac, D., Lazarevic, A., Latecki, L.J.: Incremental local outlier detection for data streams. In: IEEE Symposium on Computational Intelligence and Data Mining, CIDM 2007, pp. 504–515. IEEE (2007).  https://doi.org/10.1109/CIDM.2007.368917
  22. 22.
    Shou, Z.Y., Li, M.Y., Li, S.M.: Outlier detection based on multi-dimensional clustering and local density. J. Cent. S. Univ. 24(6), 1299–1306 (2017).  https://doi.org/10.1007/s11771-017-3535-4CrossRefGoogle Scholar
  23. 23.
    Thakran, Y., Toshniwal, D.: Unsupervised outlier detection in streaming data using weighted clustering. In: 2012 12th International Conference on Intelligent Systems Design and Applications (ISDA), pp. 947–952. IEEE (2012).  https://doi.org/10.1109/ISDA.2012.6416666
  24. 24.
    Ye, H., Kitagawa, H., Xiao, J.: Continuous angle-based outlier detection on high-dimensional data streams. In: Proceedings of the 19th International Database Engineering & Applications Symposium, pp. 162–167. ACM (2015).  https://doi.org/10.1145/2790755.2790775
  25. 25.
    Zhou, J., Kwan, C.: Anomaly detection in low quality traffic monitoring videos using optical flow. In: Pattern Recognition and Tracking XXIX, vol. 10649, p. 106490F. International Society for Optics and Photonics (2018).  https://doi.org/10.1117/12.2303651

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.School of Information and Communication EngineeringGuilin University of Electronic TechnologyGuilinChina
  2. 2.Key Laboratory of Cognitive Radio and Information ProcessingMinistry of EducationGuilinChina

Personalised recommendations