Evaluation Criteria

  • Ali A. Ghorbani
  • Wei Lu
  • Mahbod Tavallaee
Part of the Advances in Information Security book series (ADIS, volume 47)


For years, the research in intrusion detection field has been primarily focused on anomaly and misuse detection techniques. The latter method is traditionally favored in commercial products due to its predictability and high accuracy. In academic research, however, anomaly detection approach is perceived as a more powerful due to its theoretically higher potential to address novel attacks in comparison to misuse based methods. While academic community proposed a wide spectrum of anomaly based intrusion techniques, adequate comparison of the strengths and limitations of these techniques that can lead to potential commercial application is challenging. In this chapter we introduce the most significant criteria which have been proposed to have a more realistic evaluation of anomaly detection systems.


False Alarm False Alarm Rate Receiver Operating Characteristic Receiver Operating Characteristic Curve Intrusion Detection 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    MIT Lincoln Labs, 1998 DARPA Intrusion Detection Evaluation. Available on:, February 2008.
  2. 2.
    KDD Cup 1999. Available on: 99/kddcup99.html, October 2007.
  3. 3.
    Nsl-kdd data set for network-based intrusion detection systems, Available on:, March 2009.
  4. 4.
    Stefan Axelsson, The base-rate fallacy and its implications for the difficulty of intrusion detection, Proceedings of the 6th ACM conference on Computer and communication security (Kent Ridge Digital Labs, Singapore), ACM Press, November 1999, pp. 1–7.Google Scholar
  5. 5.
    ——, The base-rate fallacy and the difficulty of intrusion detection, ACM Transactions on Information and System Security (TISSEC) 3 (2000), no. 3, 186–205.Google Scholar
  6. 6.
    P. Dokas, L. Ertoz, V. Kumar, A. Lazarevic, J. Srivastava, and P. Tan, Data mining for network intrusion detection, Proceedings of NSF Workshop on Next Generation Data Mining (Baltimore, MD), November 2002.Google Scholar
  7. 7.
    Mahesh V. Joshi, Ramesh C. Agarwal, and Vipin Kumar, Predicting rare classes: Can boosting make any weak lerner strong?, Proceedings of the SIG KDD (Edmonton, Alberta, Canada), 2002.Google Scholar
  8. 8.
    C. Kruegel, F. Valeur, G. Vigna, and R.A. Kemmerer, Stateful intrusion detection for high-speed networks, Proceedings of the IEEE Symposium on Security and Privacy (Oakland, CA), IEEE Press, May 2002, pp. 285–293.Google Scholar
  9. 9.
    W. Lee, W. Fan, M. Miller, s. Stolfo, and E. Zadok, Toward cost sensitive modeling for intrusion detection and response, Journal of Computer Security 10 (2002), no. 1,2, 5–22.Google Scholar
  10. 10.
    Wenke Lee, Joo B.D. Cabrera, Ashley Thomas, Niranjan Balwalli, Sunmeet Saluja, and Yi Zhang, Performance adaptation in real-time intrusion detection systems, Proceedings of Recent Advances in Intrusion Detection, 5th International Symposium, (RAID 2002) (Zurich, Switzerland) (A. Wespi, G. Vigna, and L. Deri, eds.), Lecture Notes in Computer Science, Springer-Verlag Heidelberg, October 2002, pp. 252–273.Google Scholar
  11. 11.
    R. Lippmann, D. Fried, I. Graf, J. Haines, K. Kendall, D. McClung, D. Weber, S. Webster, D. Wyschogrod, R. Cunningham, and M. Zissman, Evaluating intrusion detection systems: The 1998 darpa off-line intrusion detection evaluation, Proceedings of the 2000 DARPA Information Survivability Conference and Exposition (DISCEX-00), 2000, pp. 12–26.Google Scholar
  12. 12.
    M.V. Mahoney and P.K. Chan, An Analysis of the 1999 DARPA/Lincoln Laboratory Evaluation Data for Network Anomaly Detection, LECTURE NOTES IN COMPUTER SCIENCE (2003), 220–238.Google Scholar
  13. 13.
    John McHugh, Testing intrusion detection systems: a critique of the 1998 and 1999 darpa intrusion detection system evaluations as performed by lincoln laboratory, ACM Transactions on Information and System Security (TISSEC) 3 (2000), no. 4, 262–294.CrossRefGoogle Scholar
  14. 14.
    NIST, Technology ITL, MITLL, Peter Mell, Vincent Hu, Richard Lippmann, Josh Haines, and Marc Zissman, An overview of issues in testing intrusion detection, July 2003.Google Scholar
  15. 15.
    Bryan Pfaffenberger, Webster's new world dictionary, ninth edition ed., ch. AC-3, p. 9, Hungry Minds, 2001.Google Scholar
  16. 16.
    Foster Provost and Tom Fawcett, Robust classification for imprecise environments, Machine Learning 42 (2001), no. 3, 203–231.MATHCrossRefGoogle Scholar
  17. 17.
    T. Ptacek and T.Newsham, Insertion, evasion, and denial of service: Eluding network intrusion detection, 1998.Google Scholar
  18. 18.
    Stuart Russell and Peter Norving, Artificial intelligence a modern approach, second edition ed., ch. Uncertainty, pp. 462–491, Prentice Hall, 2003.Google Scholar
  19. 19.
    ——, Artificial intelligence a modern approach, second edition ed., ch. Probabilistic Reasoning, pp. 492–536, Prentice Hall, 2003.Google Scholar
  20. 20.
    Lambert Schaelicke, Thomas Slabach, Branden Moore, and Curt Freeland, Characterizing the performance of network intrusion detection sensors, Proceedings of Recent Advances in Intrusion Detection, 6th International Symposium, (RAID 2003) (Pittsburgh, PA, USA) (G. Vigna, E. Jonsson, and C. Kruegel, eds.), Lecture Notes in Computer Science, Springer-Verlag Heidelberg, September 2003, pp. 155–172.Google Scholar
  21. 21.
    R. Sekar, Y. Guang, S. Verma, and T. Shanbhag, A high-performance network intrusion detection system, CCS '99: Proceedings of the 6th ACM conference on Computer and communications security, ACM Press, 1999, pp. 8–17.Google Scholar
  22. 22.
    Eugene H. Spafford and Diego Zamboni, Intrusion detection using autonomous agents, Computer Networks 34 (2000), no. 4, 547–570,*411FRK9-*2/2/f818f61028e80aa2cd740fdc4a3cd696.
  23. 23.
    SJ Stolfo, W. Fan, W. Lee, A. Prodromidis, and PK Chan, Cost-based modeling for fraud and intrusion detection: results fromthe JAM project, Proceedings of the DARPA Information Survivability Conference and Exposition (DISCEX), vol. 2, 2000.Google Scholar
  24. 24.
    T. Takada and H.Koike, Nigelog: Protecting logging information by hiding multiple backups in directories, Proceedings of the International Conference on Electronic Commerece and Security, IEEE, IEEE, 1999, pp. 874–878.Google Scholar
  25. 25.
    M. Tavallaee, E. Bagheri, W. Lu, and A.A. Ghorbani, A Detailed Analysis of the KDD CUP 99 Data Set, Proceedings of the IEEE Symposium on Computational Intelligence for Security and Defense Applications (CISDA), 2009.Google Scholar
  26. 26.
    Sholom M. Weiss and Tong Zhang, The handbook of data mining, ch. Performance Alanysis and Evaluation, pp. 426–439, Lawrence Erlbaum Assoc Inc, 2003.Google Scholar
  27. 27.
    Q. Xue, J. Sun, and Z. Wei, Tjids: an intrusion detection architecture for distributed network, Proceedings of the Canadian Conference on Electrical and Computer Engineering, IEEE CCECE 2003, May 2003, pp. 709–712.Google Scholar
  28. 28.
    Dong Yu and D. Frincke, Towards survivable intrusion detection system, Proceedings of the 37th Annual Hawaii International Conference on System Sciences, January 2004, pp. 299–308.Google Scholar
  29. 29.
    D. Zamboni, Using internal sensors for computer intrusion detection, Ph.D. thesis, Purdue University, Center for Education and Research in Information Assurance and Security, August 2001.Google Scholar

Copyright information

© Springer-Verlag US 2010

Authors and Affiliations

  1. 1.University of New BrunswickFrederictonCanada

Personalised recommendations