Abstract
Most commercial intrusion detection systems (IDSs) presently available are signature-based network IDSs. Organisations using these IDSs are still experiencing difficulties in detecting intrusive activity on their networks since novel new attacks are consistently being encountered, and analysts can miss legitimate alarms when reviewing large alarm logs that contain a high number of false positives. There has been research investigating the use of data mining techniques to effectively detect malicious activity in an enterprise network. The results of many of these projects have demonstrated that these techniques can be effective when trained/calibrated using labelled datasets. Labelled datasets identify and characterize normal and malicious traffic for use in training/calibrating the detection sensor. However, the creation of labelled datasets is resource intensive. It requires a significant effort by security analysts to create a data set that characterises the traffic in a specific enterprise network environment. This research simulates and analyses malicious activity on an enterprise network to explore the detection of malicious activity with data mining techniques using unlabelled datasets. Semi-discrete decomposition (SDD) is used as a clustering and outlier analysis technique to characterize network traffic as either normal or anomalous.
The original version of this chapter was revised: The copyright line was incorrect. This has been corrected. The Erratum to this chapter is available at DOI: 10.1007/978-0-387-35691-4_52
Chapter PDF
Similar content being viewed by others
References
Argus 2.0.2, http://www.gosient.com/argus/
E. Bloedorn, L. Talbot, C. Skorupka, A. Christiansen, B. Hill, J. Tivel, “Data mining Applied to Intrusion Detection: MITRE Experiences”, MITRE Technical Report MTRO I W0000103, September 2001.
L.Carosielli, “Detecting Malicious Use With Unlabelled Data Using Clustering And Outlier Analysis”, Masters Thesis, Department of Electrical and Computer Engineering, Royal Military College Of Canada, Kingston, Canada, April 2002.
S. McConnell, D.B. Skillicorn, “Outlier Detection Using Semi–Discrete Decomposition”, External Technical Report ISSN–0836–0227–2001–452, Department of Computing and Information Science Queen’s University, Kingston, Canada, November 2001.
J. Han, M. Kamber, “Data Mining: Concepts and Techniques”, Morgan Kaufmann, San Francisco, 2001.
D. Hawkins, “Identification of Outliers”, Chapman and Hall, London, 1980.
M. Iguchi, S. Goto, “Detecting Malicious Activities through Port Profiling”, IEICE Transactions on Information and Systems, E82-D(4), April 1999, 784–792.
KDD Cup 1999 Data, http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html
T.G. Kolda, D.P. O’Leary, “Computation and Uses of the Semidiscrete Matrix Decomposition”, ACM Transactions of Information Processing, 1999.
T.G. Kolda, D.P. O’Leary, “A Semi-Discrete Matrix Decomposition for Latent Semantic Indexing in Information Retrieval”, ACM Transactions on Information Systems, 16, 1997, 322–346.
W. Lee, S. Stolfo, “A Framework for Constructing Features and Models for Intrusion Detection Systems”, ACM Transactions on Information and System Security, 3 (4), 2000, 227–261.
D.P. O’Leary, S. Peleg, “Digital Image Compression by Outer Product Expansion”, IEEE Transactions on Communications, 31, 1983, 441–444.
SDDPACK, Software for the Semi-Discrete Decomposition, http://www.cs.umd.edu/users/oleary/SDDPACK/
L. Portnoy, E. Eskin, S.J. Stolfo, “Intrusion Detection With Unlabeled Data Using Clustering”, To Appear in Proceedings of ACM CSS Workshop on Data Mining Applied to Security (DMSA-2001), Philadelphia, USA, 5–8 November, 2001.
S. Zyto, A. Grama, W. Szpankowski, “Semi-Discrete Matrix Transforms (SDD) for Image and Video Compression”, Technical Report, Department of Computer Science, Purdue University, 2000.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 IFIP International Federation for Information Processing
About this paper
Cite this paper
Knight, G.S., Carosielli, L. (2003). Detecting Malicious Use with Unlabelled Data Using Clustering and Outlier Analysis. In: Gritzalis, D., De Capitani di Vimercati, S., Samarati, P., Katsikas, S. (eds) Security and Privacy in the Age of Uncertainty. SEC 2003. IFIP — The International Federation for Information Processing, vol 122. Springer, Boston, MA. https://doi.org/10.1007/978-0-387-35691-4_18
Download citation
DOI: https://doi.org/10.1007/978-0-387-35691-4_18
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4757-6489-5
Online ISBN: 978-0-387-35691-4
eBook Packages: Springer Book Archive