Data Collection

  • Ali A. Ghorbani
  • Wei Lu
  • Mahbod Tavallaee
Part of the Advances in Information Security book series (ADIS, volume 47)


Data collection is one of the most important steps when designing an Intrusion Detection System (IDS) and it influences the whole design and implementation process, and also the final detection result. Usually, the attacks target not only one individual computer but also aim for a group of hosts. As a result, some intrusions might show an anomalous behavior at the network layer, while others could exhibit anomawe lous behaviors at the application layer. In order to cover various network intrusions we need to monitor each layer on networks. Although ideally it is possible to design and implement an IDS that can inspect a wide range of data extracted from both network and application layer, it is infeasible in practical due to two main reasons: one is the diversity of the data, and the other one is the time and space resources that the system has to consume for collecting and interpreting the data. Intrusion detection systems collect data from many different sources, such as system log files, network packets or flows, system calls and a running code itself. The place where the data are collected decides the detection capability and scope of IDSs, i.e. a network based IDS can not detect a User-to-Root attack, while an application based IDS is not able to find a port scanning attack. In this chapter, we discuss the data collection in terms of the different locus including host-based, network-based and application-based.


Intrusion Detection System Call Network Packet Simple Network Management Protocol Network Intrusion Detection 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    TCPDUMP, Available on:, September 2008.
  2. 2.
    FlowScan,a network analysis and reporting tool, Available on: plonka/FlowScan/, October 2008.
  3. 3.
    KDD Cup 1999. Available on: 99/kddcup99.html, October 2007.
  4. 4.
    M. Almgren and U. Lindqvist, Application-integrated data collection for security monitoring, Lecture Notes in Computer Science (2001), 22–36.Google Scholar
  5. 5.
    V. Berk, G. Bakos, and R. Morris, Designing a framework for active worm detection on global networks, Proceedings of the IEEE International Workshop on Information Assurance (Darmstadt, Germany), 2003.Google Scholar
  6. 6.
    Joachim Biskup and Ulrich Flegel, Transaction-based pseudonyms in audit data for privacy respecting intrusion detection, Proceedings of Recent Advances in Intrusion Detection (RAID), Lecture Notes in Computer Science, Springer-Verlag Heidelberg, October 2000, pp. 28–48.Google Scholar
  7. 7.
    P. Dokas, L. Ertoz, V. Kumar, A. Lazarevic, J. Srivastava, and P.N. Tan, Data mining for network intrusion detection, Proceedings of the NSF Workshop on Next Generation Data Mining, 2002, pp. 21–30.Google Scholar
  8. 8.
    L. Ertoz, E. Eilertson, A. Lazarevic, P.N. Tan, P. Dokas, V. Kumar, and J. Srivastava, Detection of novel network attacks using data mining, Proceedings of the Workshop on Data Mining for Computer Security (DMSEC), 2003.Google Scholar
  9. 9.
    Chapman Flack and Mikhail J. Atallah, Better logging through formality applying formal specification techniques to improve audit logs and log consumers, Proceedings of Recent Advances in Intrusion Detection (RAID), Lecture Notes in Computer Science, Springer-Verlag Heidelberg, October 2000, pp. 1–16.Google Scholar
  10. 10.
    S. Forrest, S. Hofmeyr, A. Somayaji, and T. Longstaff, A sense of self for unix processes, Proceedings of the 1996 IEEE Symposium on Security and Privacy (Los Alamitos, CA), IEEE Computer Society Press, 1996, p. 120128.Google Scholar
  11. 11.
    Anup K. Ghosh, Christoph Michael, and Michael Schatz, A real-time intrusion detection system based on learning program behavior, Proceedings of Recent Advances in Intrusion Detection (RAID), Lecture Notes in Computer Science, Springer-Verlag Heidelberg, October 2000, pp. 93–109.Google Scholar
  12. 12.
    G. Helmer, J.S.K. Wong, V. Honavar, L. Miller, and Y. Wang, Lightweight agents for intrusion detection, The Journal of Systems & Software 67 (2003), no. 2, 109–122.CrossRefGoogle Scholar
  13. 13.
    C. Ko, System health and intrusion monitoring (shim): project summary, Proceedings of the DARPA Information Survivability Conference and Exposition II, DISCEX'03, vol. 2, April 2003, pp. 202–207.Google Scholar
  14. 14.
    C. Kruegel, D. Mutz, F. Valeur, and G. Vigna, On the detection of anomalous system call arguments, Proceedings of the 8th European Symposium on Research in Computer Security (ESORICS) (Gjovik, Norway), LNCS, Springer-Verlag, October 2003, pp. 326–343.Google Scholar
  15. 15.
    Christopher Kruegel and Giovanni Vigna, Anomaly detection of web-based attacks, Proceedings of the 10th ACM conference on Computer and communication security (Washington D.C., USA), ACM Press, October 2003, pp. 251–261.Google Scholar
  16. 16.
    Josu Kuri, Gonzalo Navarro, Ludovic M, and Laurent Heye, A pattern matching based filter for audit reduction and fast detection of potential intrusions, Proceedings of Recent Advances in Intrusion Detection (RAID), Lecture Notes in Computer Science, Springer-Verlag Heidelberg, October 2000, pp. 17–27.Google Scholar
  17. 17.
    W. Lee, S. J. Stolfo, and K. W. Mok, A data mining framework for building intrusion detection models, Proceedings of the 1999 IEEE Symposium on Security and Privacy, May 1999, pp. 120–132.Google Scholar
  18. 18.
    V.A. Mahadik, X. Wu, and D.S. Reeves, Detection of Denial-of-QoS Attacks Based On χ 2 Statistic And EWMA Control Charts, (2002).Google Scholar
  19. 19.
    J. McHugh, Intrusion and intrusion detection, International Journal of Information Security 1 (2001), no. 1, 14–35.MATHGoogle Scholar
  20. 20.
    CC Michael and A. Ghosh, Simple, state-based approaches to program-based anomaly detection, ACM Transactions on Information and System Security (TISSEC) 5 (2002), no. 3, 203–237.CrossRefGoogle Scholar
  21. 21.
    Computer Security Center (NCSC), Audit in trusted systems, July 1987, Library no. S-228 470.Google Scholar
  22. 22.
    S. Noh, C. Lee, K. Choi, and G. Jung, Detecting distributed denial of service (ddos) attacks through inductive learning, Lecture Notes in Computer Science (2003), 286–295.Google Scholar
  23. 23.
    OKENA, Stormsystem, August 2002, Cisco acquired Okena in 2003.Google Scholar
  24. 24.
    TH Ong, CP Tan, YT Tan, and C. Ting, SNMSShadow Network Management System, Proceedings of the Second International Workshop on Recent Advances in Intrusion Detection, 1999.Google Scholar
  25. 25.
    I.V. Onut, A Fuzzy Feature Evaluation Framework for Network Intrusion Detection, PhD Thesis, Faculty of Computer Science, University of New Brunswick (2008).Google Scholar
  26. 26.
    I.V. Onut and A. A. Ghorbani, A Feature Classification Scheme For Network Intrusion Detection, International Journal of Network Security 5 (2007).Google Scholar
  27. 27.
    I.V. Onut and A.A. Ghorbani, Toward a feature classification scheme for network intrusion detection, Proceedings of The Fourth Annual Conference on Communication Networks and Services Research, 2006.Google Scholar
  28. 28.
    T. Peng, C. Leckie, and R. Kotagiri, Proactively detecting ddos attack using source ip address monitoring, Proceedings of the Networking 2004 (Athens, Greece), 2004.Google Scholar
  29. 29.
    X. Qin, W. Lee, L. Lewis, and JBD Cabrera, Integrating intrusion detection and network management, IEEE/IFIP Network Operations and Management Symposium (NOMS), 2002, pp. 329–344.Google Scholar
  30. 30.
    B. Schneier and J. Kelsey, Secure audit logs to support computer forensics, ACM Transactions on Information and System Security (TISSEC) 2 (1999), no. 2, 159–176.CrossRefGoogle Scholar
  31. 31.
    Christos Siaterlis and Basil Maglaris, Towards multisensor data fusion for dos detection, Proceedings of the 2004 ACM symposium on Applied computing (Nicosia, Cyprus), 2004, pp. 439–446.Google Scholar
  32. 32.
    S. Soman, C. Krintz, and G. Vigna, Detecting malicious Java code using virtual machine auditing, Proceedings of the Twelfth USENIX Security Symposium, 2003, pp. 153–167.Google Scholar
  33. 33.
    S. Staniford, J. Hoagland, and J. McAlerney, Practical automated detection of stealthy portscans, Journal of Computer Security 10 (2002), no. 1 and 2, 105–126.Google Scholar
  34. 34.
    W.R. Stevens, TCP/IP illustrated (vol. 1): the protocols, Addison-Wesley Longman Publishing Co., Inc. Boston, MA, USA, 1993.Google Scholar
  35. 35.
    Marina Thottan and Chuanyi Ji, Anomaly detection in ip networks, IEEE Transactions on Signal Processing 51 (2003), no. 8, 148–166.CrossRefGoogle Scholar
  36. 36.
    T. Toth and C. Kruegel, Connection-history based anomaly detection, Proceedings of IEEE Workshop on Information Assurance and Security (West Point, NY), 2002.Google Scholar
  37. 37.
    G. Vigna and A. Mitchell, Mnemosyne: Designing and implementing network short-term memory, Proceedings of the 8th IEEE International Conference on Engineering of Complex Computer Systems (ICECCS) (Greenbelt, MD), IEEE Press, December 2002, pp. 91–100.Google Scholar
  38. 38.
    H. Wang, D. Zhang, and K.G. Shin, Detecting SYN flooding attacks, Proceedings of the Twenty-First Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM), vol. 3, 2002.Google Scholar
  39. 39.
    M. Welz and A. Hutchison, Interfacing trusted applications with intrusion detection systems, Lecture notes in computer science (2001), 37–53.Google Scholar
  40. 40.
    Zonghua Zhang and Hong Shen, Online training of svms for real-time intrusion detection, Proceedings of the 18th International Conference on Advanced Information Networking and Applications (AINA), vol. 1, March 2004, pp. 568–573.CrossRefGoogle Scholar
  41. 41.
    L. Zhuowei, A. Das, and S. Nandi, Utilizing statistical characteristics of N-grams for intrusion detection, Proceedings of the International Conference on Cyberworlds, 2003, pp. 486–493.Google Scholar
  42. 42.
    Cliff Changchun Zou, Lixin Gao, Weibo Gong, and Don Towsley, Monitoring and early warning for internet worms, Proceedings of the 10th ACM conference on Computer and communication security (Washington D.C., USA), ACM Press, October 2003, pp. 190–199.Google Scholar

Copyright information

© Springer-Verlag US 2010

Authors and Affiliations

  1. 1.University of New BrunswickFrederictonCanada

Personalised recommendations