Advertisement

The Journal of Supercomputing

, Volume 75, Issue 1, pp 170–188 | Cite as

A pattern-based outlier region detection method for two-dimensional arrays

  • Ki Yong Lee
  • Young-Kyoon SuhEmail author
Article
  • 35 Downloads

Abstract

Recently, with the prevalence of various sensing devices and numerical simulation software, a large amount of data is being generated in the form of a two-dimensional (2D) array. One of the important tasks for analyzing such arrays is to find anomalous or outlier regions in such a 2D array. In this article, we propose an effective method for detecting outlier regions in an arbitrary 2D array, which show a significantly different pattern from that of their surrounding regions. Unlike most existing methods that determine the outlierness of a region based on how different its average is from that of its neighboring elements, our method exploits the regression models of a region in determining its outlierness. More specifically, this method first divides the array into a number of small subarrays and then builds a regression model for each subarray. In turn, the method iteratively merges adjacent subarrays with similar regression models into larger clusters. After the clustering, the proposed method reports very small clusters as outlier regions at the final step. Lastly, we demonstrate in our experiments the effectiveness of the proposed method on synthetic and real datasets.

Keywords

Outlier detection Outlier region Two-Dimensional array 

Notes

Acknowledgements

We would like to thank anonymous reviewers for their insightful comments to improve the quality of this article. We also give thanks to Sang-Un Gu for locating and preparing for the real data sets.

References

  1. 1.
    Amidan BG, Ferryman TA, Cooley SK (2005) Data outlier detection using the Chebyshev theorem. In: Proceedings of 2005 IEEE Aerospace Conference. IEEE, pp 3814–3819Google Scholar
  2. 2.
    Bouguettaya A, Yu Q, Liu X, Zhou X, Song A (2015) Efficient agglomerative hierarchical clustering. Expert Syst Appl 42(5):2785–2797CrossRefGoogle Scholar
  3. 3.
    Chawla M, Sharma S, Sivaswamy J, Kishore LA (2009) Method for automatic detection and classification of stroke from brain CT images. In: Proceedings of the 31st Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, pp 3581–3584Google Scholar
  4. 4.
    Chawla S, Sun P (2006) SLOM: a new measure for local spatial outliers. Knowl Inf Syst 9(4):412–429CrossRefGoogle Scholar
  5. 5.
    Chen CCY, Das SK (1992) Breadth-first traversal of trees and integer sorting in parallel. Inf Proces Lett 41(1):39–49MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Franke C, Gertz M (2009) ORDEN: outlier region detection and exploration in sensor networks. In: Proceedings of the 2009 ACM SIGMOD International Conference on Management of data. ACM, pp 1075–1078Google Scholar
  7. 7.
    Friedman JH, Fisher NI (1999) Bump hunting in high-dimensional data. Stat Comput 9(2):123–143CrossRefGoogle Scholar
  8. 8.
    Janeja VP, Atluri V (2008) Random walks to identify anomalous free-form spatial scan windows. IEEE Trans Knowl Data Eng 20(10):1378–1392CrossRefGoogle Scholar
  9. 9.
    Jim J, Lee W, Song JJ, Lee SB (2017) Optimized combinatorial clustering for stochastic processes. Clust Comput 20(2):1135–1148CrossRefGoogle Scholar
  10. 10.
    Kulldorff M (1997) A spatial scan statistic. Commun Stat Theory Methods 26(6):1481–1496MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Kulldorff M, Huang L, Pickle L, Duczmal L (2006) An elliptic spatial scan statistic. Stat Med 25(22):3929–3943MathSciNetCrossRefGoogle Scholar
  12. 12.
    Kutner MH, Nachtsheim CJ, Neter J, Li W (2005) Applied linear statistical models, 5th edn. McGraw Hill, New YorkGoogle Scholar
  13. 13.
    Langfelder P, Zhang B, Horvath S (2007) Defining clusters from a hierarchical cluster tree: the dynamic tree cut package for R. Bioinformatics 24(5):719–720CrossRefGoogle Scholar
  14. 14.
    Li X, Lv J, Yi Z (2018) An efficient representation-based method for boundary point and outlier detection. IEEE Trans Neural Netw Learn Syst 29(1):51–62MathSciNetCrossRefGoogle Scholar
  15. 15.
    Lu CT, Kou Y, Zhao J, Chen L (2007) Detecting and tracking regional outliers in meteorological data. Inf Sci 177(7):1609–1632CrossRefGoogle Scholar
  16. 16.
    Lu CT, SANTOS JR RFD, Liu X, Kou Y (2011) A graph-based approach to detect abnormal spatial points and regions. Int J Artif Intell Tools 20(04):721–751CrossRefGoogle Scholar
  17. 17.
    Luts J, Laudadio T, Idema AJ, Simonetti AW, Heerschap A, Vandermeulen D, Suykens JA, Van Huffel S (2009) Nosologic imaging of the brain: segmentation and classification using MRI and MRSI. NMR Biomed 22(4):374–390CrossRefGoogle Scholar
  18. 18.
    Mao J, Wang T, Jin C, Zhou A (2017) Feature grouping-based outlier detection upon streaming trajectories. IEEE Trans Knowl Data Eng 29(12):2696–2709CrossRefGoogle Scholar
  19. 19.
    NASA (2010): New map offers a global view of health-sapping air pollution. https://www.nasa.gov/topics/earth/features/health-sapping.html. Accessed 29 Jan 2018
  20. 20.
    NASA (2018): NASA earth observation. https://neo.sci.gsfc.nasa.gov/. Accessed 29 Jan 2018
  21. 21.
    Neill DB, Moore AW, Cooper GF (2006) A Bayesian spatial scan statistic. In: Weiss Y, Schölkopf B, Platt JC (eds) Advances in Neural Information Processing Systems 18 (NIPS 2005). Neural Information Processing Systems Foundation, Inc, pp 1003–1010Google Scholar
  22. 22.
    Patil GP, Taillie C (2004) Upper level set scan statistic for detecting arbitrarily shaped hotspots. Environ Ecol Stat 11(2):183–197MathSciNetCrossRefGoogle Scholar
  23. 23.
    Prastawa M, Bullitt E, Ho S, Gerig G (2004) A brain tumor segmentation framework based on outlier detection. Med Image Anal 8(3):275–283CrossRefGoogle Scholar
  24. 24.
    Ramteke R, Monali YK (2012) Automatic Medical image classification and abnormality detection using K-nearest neighbour. Int J Adv Comput Res 2(4):190–196Google Scholar
  25. 25.
    Reed IS, Yu X (1990) Adaptive multiple-band CFAR detection of an optical pattern with unknown spectral distribution. IEEE Trans Acoust Speech Signal Proces 38(10):1760–1770CrossRefGoogle Scholar
  26. 26.
    Song JJ, Lee W (2017) Relevance maximization for high-recall retrieval problem: finding all needles in a Haystack. J Supercompu.  https://doi.org/10.1007/s11227-016-1956-8 Google Scholar
  27. 27.
    Stein DW, Beaven SG, Hoff LE, Winter EM, Schaum AP, Stocker AD (2002) Anomaly detection from hyperspectral imagery. IEEE Signal Proces Mag 19(1):58–69CrossRefGoogle Scholar
  28. 28.
    Suhail Z, Sarwar M, Murtaza K (2015) Automatic detection of abnormalities in mammograms. BMC Med Imaging 15(1):53CrossRefGoogle Scholar
  29. 29.
    Tango T, Takahashi K (2005) A flexibly shaped spatial scan statistic for detecting clusters. Int J Health Geogr 4(1):11CrossRefGoogle Scholar
  30. 30.
    Telang A, Deepak P, Joshi S, Deshpande P, Rajendran R (2014) Detecting localized homogeneous anomalies over spatio-temporal data. Data Mining Knowl Discov 28(5–6):1480–1502MathSciNetCrossRefGoogle Scholar
  31. 31.
    Tran L, Fan L, Shahabi C (2016) Distance-based outlier detection in data streams. Proc VLDB Endow 9(12):1089–1100CrossRefGoogle Scholar
  32. 32.
    You C, Robinson DP, Vidal R (2017) Provable self-representation based outlier detection in a union of subspaces. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, pp 1–10Google Scholar
  33. 33.
    Zheng G, Brantley SL, Lauvaux T, Li Z (2017) Contextual spatial outlier detection with metric learning. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 2161–2170Google Scholar
  34. 34.
    Zhu M, Aggarwal CC, Ma S, Zhang H, Huai J (2017) Outlier detection in sparse data with factorization machines. In: Proceedings of the 2017 ACM Conference on Information and Knowledge Management, pp 817–826Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Division of Computer ScienceSookmyung Women’s UniversitySeoulKorea
  2. 2.School of Computer Science and EngineeringKyungpook National UniversityDaeguKorea

Personalised recommendations