Skip to main content
SpringerLink
Log in

Outlier Detection Based on Local Density of Vector Dot Product in Data Stream

  • Conference paper
  • First Online:
Security with Intelligent Computing and Big-data Services (SICBS 2018)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 895))

Abstract

Outlier detection in data stream is an increasingly important research in many fields. To deal with the data stream with the properties of high dimension, rapid arrival in order, high cost of storing all data in memory and so on, an outlier detection algorithm based on local density of vector dot product in data stream (LDVP-OD) is proposed. LDVP-OD uses the model based on sliding window and multiple validations to decrease the false alarm rate, which divides the data stream into uniform-sized blocks. Local density of vector dot product (LDVP) is described in order to precisely evaluate the outlierness of data in data stream. Furthermore, an outlier judgment criterion based on supreme slope is introduced, which can determine the exact outliers without requiring the number of outliers or other parameters beforehand. Comparison experiments with existing algorithms on synthetic and real datasets prove the high detection rate, good stability, strong adaptability of LDVP-OD.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Angelov, P.: Outside the box: an alternative data analytics framework. J. Autom. Mob. Rob. Intell. Syst. 8(2), 29–35 (2014). https://doi.org/10.14313/JAMRIS_2-2014/16

    Article  MathSciNet  Google Scholar 

  2. Angiulli, F., Fassetti, F.: Distance-based outlier queries in data streams: the novel task and algorithms. Data Min. Knowl. Disc. 20(2), 290–324 (2010). https://doi.org/10.1007/s10618-009-0159-9

    Article  MathSciNet  Google Scholar 

  3. Breunig, M.M., Kriegel, H.P., Ng, R.T., Sander, J.: LOF: identifying density-based local outliers. In: ACM Sigmod Record. vol. 29, pp. 93–104. ACM (2000). https://doi.org/10.1145/335191.335388

  4. Cao, L., Yang, D., Wang, Q., Yu, Y., Wang, J., Rundensteiner, E.A.: Scalable distance-based outlier detection over high-volume data streams. In: 2014 IEEE 30th International Conference on Data Engineering (ICDE), pp. 76–87. IEEE (2014). https://doi.org/10.1109/ICDE.2014.6816641

  5. Carcillo, F., Dal Pozzolo, A., Le Borgne, Y.A., Caelen, O., Mazzer, Y., Bontempi, G.: SCARFF: a scalable framework for streaming credit card fraud detection with spark. Inf. Fusion 41, 182–194 (2018). https://doi.org/10.1016/j.inffus.2017.09.005

    Article  Google Scholar 

  6. Cremonesi, P., Koren, Y., Turrin, R.: Performance of recommender algorithms on top-n recommendation tasks. In: Proceedings of the Fourth ACM Conference on Recommender Systems. pp. 39–46. ACM (2010). https://doi.org/10.1145/1864708.1864721

  7. Dheeru, D., Karra Taniskidou, E.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml

  8. Ester, M., Kriegel, H.P., Sander, J., Xu, X., et al.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: KDD, vol. 96, pp. 226–231 (1996). http://dl.acm.org/citation.cfm?id=3001460.3001507

  9. Gao, K., Shao, F.J., Sun, R.C.: n-INCLOF: a dynamic local outlier detection algorithm for data streams. In: 2010 2nd International Conference on Signal Processing Systems (ICSPS), vol. 2, p. V2–179. IEEE (2010). https://doi.org/10.1109/ICSPS.2010.5555276

  10. Golab, L., Özsu, M.T.: Issues in data stream management. ACM Sigmod Rec. 32(2), 5–14 (2003). https://doi.org/10.1145/776985.776986

    Article  Google Scholar 

  11. Ha, J., Seok, S., Lee, J.S.: Robust outlier detection using the instability factor. Knowl. Based Syst. 63, 15–23 (2014). https://doi.org/10.1016/j.knosys.2014.03.001

    Article  Google Scholar 

  12. Huang, J.Z., Ng, M.K., Rong, H., Li, Z.: Automated variable weighting in k-means type clustering. IEEE Trans. Patt. Anal. Mach. Intell. 27(5), 657–668 (2005). https://doi.org/10.1109/TPAMI.2005.95

    Article  Google Scholar 

  13. Karimian, S.H., Kelarestaghi, M., Hashemi, S.: I-incLOF: improved incremental local outlier detection for data streams. In: 2012 16th CSI International Symposium on Artificial Intelligence and Signal Processing (AISP), pp. 023–028. IEEE (2012). https://doi.org/10.1109/AISP.2012.6313711

  14. Knorr, E.M., Ng, R.T.: Algorithms for mining distance-based outliers in large datasets. In: Proceedings of the 24th International Conference on Very Large Data Bases, VLDB 1998. pp. 392–403. Morgan Kaufmann Publishers Inc., San Francisco (1998). http://dl.acm.org/citation.cfm?id=645924.671334

  15. Kontaki, M., Gounaris, A., Papadopoulos, A.N., Tsichlas, K., Manolopoulos, Y.: Continuous monitoring of distance-based outliers over data streams. In: 2011 IEEE 27th International Conference on Data Engineering (ICDE), pp. 135–146. IEEE (2011). https://doi.org/10.1109/ICDE.2011.5767923

  16. Kriegel, H.P., Kröger, P., Schubert, E., Zimek, A.: Loop: local outlier probabilities. In: Proceedings of the 18th ACM Conference on Information and Knowledge Management. pp. 1649–1652. ACM (2009). https://doi.org/10.1145/1645953.1646195

  17. Kriegel, H.P., Zimek, A., et al.: Angle-based outlier detection in high-dimensional data. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 444–452. ACM (2008). https://doi.org/10.1145/1401890.1401946

  18. Miller, Z., Dickinson, B., Deitrick, W., Hu, W., Wang, A.H.: Twitter spammer detection using data stream clustering. Inf. Sci. 260, 64–73 (2014). https://doi.org/10.1016/j.ins.2013.11.016

    Article  Google Scholar 

  19. Neeraj, K.: Anomaly-based network intrusion detection: an outlier detection techniques. In: Proceedings of the Eighth International Conference on Soft Computing and Pattern Recognition. vol. 614. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-60618-7_26

  20. Pang, G., Cao, L., Chen, L., Liu, H.: Learning representations of ultrahigh-dimensional data for random distance-based outlier detection. arXiv preprint arXiv:1806.04808 (2018)

  21. Pokrajac, D., Lazarevic, A., Latecki, L.J.: Incremental local outlier detection for data streams. In: IEEE Symposium on Computational Intelligence and Data Mining, CIDM 2007, pp. 504–515. IEEE (2007). https://doi.org/10.1109/CIDM.2007.368917

  22. Shou, Z.Y., Li, M.Y., Li, S.M.: Outlier detection based on multi-dimensional clustering and local density. J. Cent. S. Univ. 24(6), 1299–1306 (2017). https://doi.org/10.1007/s11771-017-3535-4

    Article  Google Scholar 

  23. Thakran, Y., Toshniwal, D.: Unsupervised outlier detection in streaming data using weighted clustering. In: 2012 12th International Conference on Intelligent Systems Design and Applications (ISDA), pp. 947–952. IEEE (2012). https://doi.org/10.1109/ISDA.2012.6416666

  24. Ye, H., Kitagawa, H., Xiao, J.: Continuous angle-based outlier detection on high-dimensional data streams. In: Proceedings of the 19th International Database Engineering & Applications Symposium, pp. 162–167. ACM (2015). https://doi.org/10.1145/2790755.2790775

  25. Zhou, J., Kwan, C.: Anomaly detection in low quality traffic monitoring videos using optical flow. In: Pattern Recognition and Tracking XXIX, vol. 10649, p. 106490F. International Society for Optics and Photonics (2018). https://doi.org/10.1117/12.2303651

Download references

Acknowledgments

This work is supported by the following foundations: the national Natural Science Foundation of China (61662013, 61362021, U1501252); Natural Science Foundation of Guangxi province (2016GXNSFAA380149); Guangxi Innovation-Driven Development Project (Science and Technology Major Project) (AA17202024); the Key Laboratory of Cognitive Radio and Information Processing Ministry of Education (2011KF11); Innovation Project of GUET Graduate Education (2017YJCX34, 2018YJCX37).

Author information

Authors and Affiliations

  1. School of Information and Communication Engineering, Guilin University of Electronic Technology, Guilin, 541004, China

    Zhaoyu Shou, Fengbo Zou & Hao Tian

  2. Key Laboratory of Cognitive Radio and Information Processing, Ministry of Education, Guilin, 541004, China

    Simin Li

Authors
  1. Zhaoyu Shou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fengbo Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hao Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Simin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Zhaoyu Shou or Fengbo Zou .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Information Engineering, National Dong Hwa University, Hualien, Taiwan

    Ching-Nung Yang

  2. Department of Computer Science and Information Engineering, National Dong Hwa University, Hualien, Taiwan

    Sheng-Lung Peng

  3. University of Canberra, Canberra, ACT, Australia

    Lakhmi C. Jain

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shou, Z., Zou, F., Tian, H., Li, S. (2020). Outlier Detection Based on Local Density of Vector Dot Product in Data Stream. In: Yang, CN., Peng, SL., Jain, L. (eds) Security with Intelligent Computing and Big-data Services. SICBS 2018. Advances in Intelligent Systems and Computing, vol 895. Springer, Cham. https://doi.org/10.1007/978-3-030-16946-6_14

Download citation

Publish with us

Policies and ethics

Search

Navigation