Abstract
As a widely used data mining technique, outlier detection is a process which aims at finding anomalies with good explanations. Most existing methods are designed for numeric data. They will have problems with real-life applications that contain categorical data. In this paper, we introduce a novel outlier mining method based on a hypergraph model. Since hypergraphs precisely capture the distribution characteristics in data subspaces, this method is effective in identifying anomalies in dense subspaces and presents good interpretations for the local outlierness. By selecting the most relevant subspaces, the problem of “curse of dimensionality” in very large databases can also be ameliorated. Furthermore, the connectivity property is used to replace the distance metrics, so that the distance-based computation is not needed anymore, which enhances the robustness for handling missing-value data. The fact, that connectivity computation facilitates the aggregation operations supported by most SQL-compatible database systems, makes the mining process much efficient. Finally, experiments and analysis show that our method can find outliers in categorical data with good performance and quality.
The work was partially supported by the “973” National Fundamental Research Programme of China (Grant No. G1998030414), National “863” Hi-Tech Programme of China (Grant No. 2002AA413310), grants from Ministry of Education of China, and Fok Ying Tunk Education Foundation.
The author is partially supported by Microsoft Research Fellowship.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
C. Aggarwal and P. Yu. Outlier detection for high dimensional data. In Proc. of SIGMOD’2001, pages 37–47, 2001.
R. Agrawal and R. Srikant. Fast algorithms for mining association rules in large databases. In Proc. of VLDB’94, pages 487–499, 1994.
V. Barnett and T. Lewis. Outliers In Statistical Data. John Wiley, Reading, New York, 1994.
M. Breunig, H.-P. Kriegel, R. Ng, and J. Sander. Optics-of: Identifying local outliers. In Proc. of PKDD’99, pages 262–270, 1999.
M. Breunig, H.-P. Kriegel, R. Ng, and J. Sander. Lof: Identifying density-based local outliers. In Proc. of SIGMOD’2000, pages 93–104, 2000.
M. Ester, H.-P. Kriegel, J. Sander, and X. Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proc. of KDD’96, pages 226–231, 1996.
S. Guha, R. Rastogi, and K. Shim. Cure: An efficient clustering algorithm for large databases. In Proc. of SIGMOD’98, pages 73–84, 1998.
D. Hawkins. Identification of Outliers. Chapman and Hall, Reading, London, 1980.
F. Hussain, H. Liu, C. L. Tan, and M. Dash. Discretization: An enabling technique. Technical Report TRC6/99, National University of Singapore, School of Computing, 1999.
W. Jin, A. K. Tung, and J. Han. Mining top-n local outliers in large databases. In Proc. of KDD’2001, pages 293–298, 2001.
E. Knorr and R. Ng. Algorithms for mining distance-based outliers in large datasets. In Proc. of VLDB’98, pages 392–403, 1998.
E. Knorr and R. Ng. Finding intensional knowledge of distance-based outliers. In Proc. of VLDB’99, pages 211–222, 1999.
G. Merz and P. Murphy. Uci repository of machine learning databases. Technical Report, University of California, Department of Information and Computer Science: http://www.ics.uci.edu/mlearn/MLRepository.html, 1996.
F. Preparata and M. Shamos. Computational Geometry: an Introduction. Springer-Verlag, Reading, New York, 1988.
S. Ramaswamy, R. Rastogi, and K. Shim. Efficient algorithms for mining outliers from large data sets. In Proc. of SIGMOD’2000, pages 427–438, 2000.
I. Ruts and P. Rousseeuw. Computing depth contours of bivariate point clouds. Journal of Computational Statistics and data Analysis, 23:153–168, 1996.
S. Shekhar, C.-T. Lu, and P. Zhang. Detecting graph-based spatial outliers: Algorithms and applications (a summary of results). In Proc. of KDD’2001, 2001.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wei, L., Qian, W., Zhou, A., Jin, W., Yu, J.X. (2003). HOT: Hypergraph-Based Outlier Test for Categorical Data. In: Whang, KY., Jeon, J., Shim, K., Srivastava, J. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2003. Lecture Notes in Computer Science(), vol 2637. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-36175-8_40
Download citation
DOI: https://doi.org/10.1007/3-540-36175-8_40
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-04760-5
Online ISBN: 978-3-540-36175-6
eBook Packages: Springer Book Archive