Outlier Detection Based on Local Density of Vector Dot Product in Data Stream

  • Zhaoyu ShouEmail author
  • Fengbo ZouEmail author
  • Hao Tian
  • Simin Li
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 895)


Outlier detection in data stream is an increasingly important research in many fields. To deal with the data stream with the properties of high dimension, rapid arrival in order, high cost of storing all data in memory and so on, an outlier detection algorithm based on local density of vector dot product in data stream (LDVP-OD) is proposed. LDVP-OD uses the model based on sliding window and multiple validations to decrease the false alarm rate, which divides the data stream into uniform-sized blocks. Local density of vector dot product (LDVP) is described in order to precisely evaluate the outlierness of data in data stream. Furthermore, an outlier judgment criterion based on supreme slope is introduced, which can determine the exact outliers without requiring the number of outliers or other parameters beforehand. Comparison experiments with existing algorithms on synthetic and real datasets prove the high detection rate, good stability, strong adaptability of LDVP-OD.


Outlier detection Data stream Local density of vector dot product Multiple validations 



This work is supported by the following foundations: the national Natural Science Foundation of China (61662013, 61362021, U1501252); Natural Science Foundation of Guangxi province (2016GXNSFAA380149); Guangxi Innovation-Driven Development Project (Science and Technology Major Project) (AA17202024); the Key Laboratory of Cognitive Radio and Information Processing Ministry of Education (2011KF11); Innovation Project of GUET Graduate Education (2017YJCX34, 2018YJCX37).


  1. 1.
    Angelov, P.: Outside the box: an alternative data analytics framework. J. Autom. Mob. Rob. Intell. Syst. 8(2), 29–35 (2014). Scholar
  2. 2.
    Angiulli, F., Fassetti, F.: Distance-based outlier queries in data streams: the novel task and algorithms. Data Min. Knowl. Disc. 20(2), 290–324 (2010). Scholar
  3. 3.
    Breunig, M.M., Kriegel, H.P., Ng, R.T., Sander, J.: LOF: identifying density-based local outliers. In: ACM Sigmod Record. vol. 29, pp. 93–104. ACM (2000).
  4. 4.
    Cao, L., Yang, D., Wang, Q., Yu, Y., Wang, J., Rundensteiner, E.A.: Scalable distance-based outlier detection over high-volume data streams. In: 2014 IEEE 30th International Conference on Data Engineering (ICDE), pp. 76–87. IEEE (2014).
  5. 5.
    Carcillo, F., Dal Pozzolo, A., Le Borgne, Y.A., Caelen, O., Mazzer, Y., Bontempi, G.: SCARFF: a scalable framework for streaming credit card fraud detection with spark. Inf. Fusion 41, 182–194 (2018). Scholar
  6. 6.
    Cremonesi, P., Koren, Y., Turrin, R.: Performance of recommender algorithms on top-n recommendation tasks. In: Proceedings of the Fourth ACM Conference on Recommender Systems. pp. 39–46. ACM (2010).
  7. 7.
    Dheeru, D., Karra Taniskidou, E.: UCI machine learning repository (2017).
  8. 8.
    Ester, M., Kriegel, H.P., Sander, J., Xu, X., et al.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: KDD, vol. 96, pp. 226–231 (1996).
  9. 9.
    Gao, K., Shao, F.J., Sun, R.C.: n-INCLOF: a dynamic local outlier detection algorithm for data streams. In: 2010 2nd International Conference on Signal Processing Systems (ICSPS), vol. 2, p. V2–179. IEEE (2010).
  10. 10.
    Golab, L., Özsu, M.T.: Issues in data stream management. ACM Sigmod Rec. 32(2), 5–14 (2003). Scholar
  11. 11.
    Ha, J., Seok, S., Lee, J.S.: Robust outlier detection using the instability factor. Knowl. Based Syst. 63, 15–23 (2014). Scholar
  12. 12.
    Huang, J.Z., Ng, M.K., Rong, H., Li, Z.: Automated variable weighting in k-means type clustering. IEEE Trans. Patt. Anal. Mach. Intell. 27(5), 657–668 (2005). Scholar
  13. 13.
    Karimian, S.H., Kelarestaghi, M., Hashemi, S.: I-incLOF: improved incremental local outlier detection for data streams. In: 2012 16th CSI International Symposium on Artificial Intelligence and Signal Processing (AISP), pp. 023–028. IEEE (2012).
  14. 14.
    Knorr, E.M., Ng, R.T.: Algorithms for mining distance-based outliers in large datasets. In: Proceedings of the 24th International Conference on Very Large Data Bases, VLDB 1998. pp. 392–403. Morgan Kaufmann Publishers Inc., San Francisco (1998).
  15. 15.
    Kontaki, M., Gounaris, A., Papadopoulos, A.N., Tsichlas, K., Manolopoulos, Y.: Continuous monitoring of distance-based outliers over data streams. In: 2011 IEEE 27th International Conference on Data Engineering (ICDE), pp. 135–146. IEEE (2011).
  16. 16.
    Kriegel, H.P., Kröger, P., Schubert, E., Zimek, A.: Loop: local outlier probabilities. In: Proceedings of the 18th ACM Conference on Information and Knowledge Management. pp. 1649–1652. ACM (2009).
  17. 17.
    Kriegel, H.P., Zimek, A., et al.: Angle-based outlier detection in high-dimensional data. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 444–452. ACM (2008).
  18. 18.
    Miller, Z., Dickinson, B., Deitrick, W., Hu, W., Wang, A.H.: Twitter spammer detection using data stream clustering. Inf. Sci. 260, 64–73 (2014). Scholar
  19. 19.
    Neeraj, K.: Anomaly-based network intrusion detection: an outlier detection techniques. In: Proceedings of the Eighth International Conference on Soft Computing and Pattern Recognition. vol. 614. Springer, Cham (2018).
  20. 20.
    Pang, G., Cao, L., Chen, L., Liu, H.: Learning representations of ultrahigh-dimensional data for random distance-based outlier detection. arXiv preprint arXiv:1806.04808 (2018)
  21. 21.
    Pokrajac, D., Lazarevic, A., Latecki, L.J.: Incremental local outlier detection for data streams. In: IEEE Symposium on Computational Intelligence and Data Mining, CIDM 2007, pp. 504–515. IEEE (2007).
  22. 22.
    Shou, Z.Y., Li, M.Y., Li, S.M.: Outlier detection based on multi-dimensional clustering and local density. J. Cent. S. Univ. 24(6), 1299–1306 (2017). Scholar
  23. 23.
    Thakran, Y., Toshniwal, D.: Unsupervised outlier detection in streaming data using weighted clustering. In: 2012 12th International Conference on Intelligent Systems Design and Applications (ISDA), pp. 947–952. IEEE (2012).
  24. 24.
    Ye, H., Kitagawa, H., Xiao, J.: Continuous angle-based outlier detection on high-dimensional data streams. In: Proceedings of the 19th International Database Engineering & Applications Symposium, pp. 162–167. ACM (2015).
  25. 25.
    Zhou, J., Kwan, C.: Anomaly detection in low quality traffic monitoring videos using optical flow. In: Pattern Recognition and Tracking XXIX, vol. 10649, p. 106490F. International Society for Optics and Photonics (2018).

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.School of Information and Communication EngineeringGuilin University of Electronic TechnologyGuilinChina
  2. 2.Key Laboratory of Cognitive Radio and Information ProcessingMinistry of EducationGuilinChina

Personalised recommendations