Abstracting Audit Data for Lightweight Intrusion Detection

  • Wei Wang
  • Xiangliang Zhang
  • Georgios Pitsilis
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6503)


High speed of processing massive audit data is crucial for an anomaly Intrusion Detection System (IDS) to achieve real-time performance during the detection. Abstracting audit data is a potential solution to improve the efficiency of data processing. In this work, we propose two strategies of data abstraction in order to build a lightweight detection model. The first strategy is exemplar extraction and the second is attribute abstraction. Two clustering algorithms, Affinity Propagation (AP) as well as traditional k-means, are employed to extract the exemplars, and Principal Component Analysis (PCA) is employed to abstract important attributes (a.k.a. features) from the audit data. Real HTTP traffic data collected in our institute as well as KDD 1999 data are used to validate the two strategies of data abstraction. The extensive test results show that the process of exemplar extraction significantly improves the detection efficiency and has a better detection performance than PCA in data abstraction.


Intrusion Detection Class Support Vector Machine Anomaly Detection Near Neighbor Intrusion Detection System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Beale, J., Baker, A.R., Caswell, B.: Snort 2.1 Intrusion Detection, 2nd edn. Syngress Press (2004)Google Scholar
  2. 2.
    Lee, W., Stolfo, S.J., Mok, K.W.: A data mining framework for building intrusion detection models. In: IEEE S&P (1999)Google Scholar
  3. 3.
    Wang, K., Stolfo, S.J.: Anomalous payload-based network intrusion detection. In: Jonsson, E., Valdes, A., Almgren, M. (eds.) RAID 2004. LNCS, vol. 3224, pp. 203–222. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  4. 4.
    Wang, K., Cretu, G.F., Stolfo, S.J.: Anomalous payload-based worm detection and signature generation. In: Valdes, A., Zamboni, D. (eds.) RAID 2005. LNCS, vol. 3858, pp. 227–246. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  5. 5.
    Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for unix processes. In: IEEE S&P (1996)Google Scholar
  6. 6.
    Guan, X., Wang, W., Zhang, X.: Fast intrusion detection based on a non-negative matrix factorization model. J. Network and f Applications 32(1), 31–44 (2009)CrossRefGoogle Scholar
  7. 7.
    Wang, W., Guan, X., Zhang, X., Yang, L.: Profiling program behavior for anomaly intrusion detection based on the transition and frequency property of computer audit data. Computers & Security 25(7), 539–550 (2006)CrossRefGoogle Scholar
  8. 8.
    Liao, Y., Vemuri, V.R.: Using text categorization techniques for intrusion detection. In: USENIX Security Symposium (2002)Google Scholar
  9. 9.
    Schonlau, M., Theus, M.: Detecting masquerades in intrusion detection based on unpopular commands. Inf. Process. Lett. 76(1-2) (2000)Google Scholar
  10. 10.
    Wang, W., Guan, X., Zhang, X.: Processing of massive audit data streams for real-time anomaly intrusion detection. Computer Communications 31(1), 58–72 (2008)CrossRefGoogle Scholar
  11. 11.
    Ingham, K.L., Inoue, H.: Comparing anomaly detection techniques for http. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 42–62. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  12. 12.
    Krügel, C., Vigna, G.: Anomaly detection of web-based attacks. In: ACM CCS (2003)Google Scholar
  13. 13.
    Song, Y., Keromytis, A.D., Stolfo, S.J.: Spectrogram: A mixture-of-markov-chains model for anomaly detection in web traffic. In: NDSS (2009)Google Scholar
  14. 14.
    Robertson, W.K., Vigna, G., Krügel, C., Kemmerer, R.A.: Using generalization and characterization techniques in the anomaly-based detection of web attacks. In: NDSSGoogle Scholar
  15. 15.
    Brauckhoff, D., Salamatian, K., May, M.: A signal-processing view on packet sampling and anomaly detection. In: INFOCOM (2010)Google Scholar
  16. 16.
    Brauckhoff, D., Tellenbach, B., Wagner, A., Lakhina, A., May, M.: Impact of packet sampling on anomaly detection metrics. In: Internet Measurement Conference, IMC (2006)Google Scholar
  17. 17.
    Frey, B.J., Dueck, D.: Clustering by passing messages between data points. Science 315(5814), 972–976 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    MacQueen, J.B.: Some methods for classification and analysis of multivariate observations. In: Proceedings of 5th Berkeley Symposium on Mathematical Statistics and Probability (1967)Google Scholar
  19. 19.
    Denning, D.E.: An intrusion-detection model. IEEE Trans. Software Eng. 13(2), 222–232 (1987)CrossRefGoogle Scholar
  20. 20.
    Smaha, S.E.: Haystack: An intrusion detection system. In: Proceedings of the IEEE Fourth Aerospace Computer Security Applications Conference (1988)Google Scholar
  21. 21.
    Cretu, G.F., Stavrou, A., Locasto, M.E., Stolfo, S.J., Keromytis, A.D.: Casting out demons: Sanitizing training data for anomaly sensors. In: IEEE S&P (2008)Google Scholar
  22. 22.
    KDD-Data: Kdd cup 1999 data (1999), (retrieved March 2009)
  23. 23.
    Shyu, M., Chen, S., Sarinnapakorn, K., Chang, L.: A novel anomaly detection scheme based on principal component classifier. In: IEEE Foundations and New Directions of Data Mining Workshop (2003)Google Scholar
  24. 24.
    Sung, A.H., Mukkamala, S.: Feature selection for intrusion detection using neural networks and support vector machines. In: 82nd Annual Meeting of the Transportation Research Board (2003)Google Scholar
  25. 25.
    Wang, W., Gombault, S., Guyet, T.: Towards fast detecting intrusions: using key attributes of network traffic. In: ICIMP (July 2008)Google Scholar
  26. 26.
    Li, Y., Lu, T.B., Guo, L., Tian, Z.H., Qi, L.: Optimizing network anomaly detection scheme using instance selection mechanism. In: Proceedings of the 28th IEEE Conference on Global Telecommunications, GLOBECOM 2009, Piscataway, NJ, USA, pp. 425–431. IEEE Press, Los Alamitos (2009)Google Scholar
  27. 27.
    Wang, W., Zhang, X., Gombault, S.: Constructing attribute weights from computer audit data for effective intrusion detection. J. Sys. and Soft. 82(12) (2009)Google Scholar
  28. 28.
    Schölkopf, B., Platt, J.C., Shawe-Taylor, J., Smola, A.J., Williamson, R.C.: Estimating the support of a high-dimensional distribution. Neural Computation 13(7), 1443–1471 (2001)CrossRefzbMATHGoogle Scholar
  29. 29.
    Liao, Y., Vemuri, V.R., Pasos, A.: Adaptive anomaly detection with evolving connectionist systems. J. Network and Computer Applications 30(1) (2007)Google Scholar
  30. 30.
    Manevitz, L.M., Yousef, M.: One-class svms for document classification. Journal of Machine Learning Research 2, 139–154 (2001)zbMATHGoogle Scholar
  31. 31.
    Jolliffe, I.T.: Principal Component Analysis, 2nd edn. Springer, Berlin (2002)zbMATHGoogle Scholar
  32. 32.
    Zhang, X., Furtlehner, C., Sebag, M.: Data streaming with affinity propagation. In: Daelemans, W., Goethals, B., Morik, K. (eds.) ECML PKDD 2008, Part II. LNCS (LNAI), vol. 5212, pp. 628–643. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  33. 33.
    McHugh, J.: Testing intrusion detection systems: a critique of the 1998 and 1999 darpa intrusion detection system evaluations as performed by lincoln laboratory. ACM Trans. Inf. Syst. Secur. 3(4), 262–294 (2000)CrossRefGoogle Scholar
  34. 34.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines (2001), Software available at

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Wei Wang
    • 1
  • Xiangliang Zhang
    • 2
  • Georgios Pitsilis
    • 3
  1. 1.Interdisciplinary Centre for Security, Reliability and Trust (SnT Centre)Université du LuxembourgLuxembourg
  2. 2.Mathematical and Computer Sciences and Engineering DivisionKing Abdullah University of Science and Technolgy (KAUST)Saudi Arabia
  3. 3.Faculty of Science, Technology and CommunicationUniversité du LuxembourgLuxembourg

Personalised recommendations