Abstract
In 1998 (and again in 1999), the Lincoln Laboratory of MIT conducted a comparative evaluation of Intrusion Detection Systems developed under DARPA funding. While this evaluation represents a significant and monumental undertaking, there are a number of unresolved issues associated with its design and execution. Some of methodologies used in the evaluation are questionable and may have biased its results. One of the problems with the evaluation is that the evaluators have published relatively little concerning some of the more critical aspects of their work, such as validation of their test data. The purpose of this paper is to attempt to identify the shortcomings of the Lincoln Lab effort in the hope that future efforts of this kind will be placed on a sounder footing. Some of the problems that the paper points out might well be resolved if the evaluators publish a detailed description of their procedures and the rationale that led to their adoption, but other problems clearly remain.
This work was sponsored by the U.S. Department of Defense.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Stefan Axelsson. The base-rate fallacy and its implications for the difficulty of intrusion detection. In 6th ACM Conference on Computer and Communications Security, pages 1–7, 1999.
Steven M. Bellovin. Packets found on an internet. Computer Communications Review, 23(3):26–31, July 1993.
James P. Egan. Signal detection Theory and ROC Analysis. Academic Press, 1975.
Isaac Graf et al. Results of DARPA 1998 offline intrusion detection evaluation. Presentation at MIT Lincoln Laboratory PI Meeting (available at) http://ideval.ll.mit.edu/results-html-dir/, 15 December 1998.
D. A. James and S. J. Young. A fast lattice-based approach to vocabulary independent wordspotting. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 337–380, 1994.
Kristopher Kendall. A database of computer attacks for the evaluation of intrusion detection systems. BS/MS thesis, Massachusetts Institute of Technology, June 1999.
Richard P. Lippmann, Eric I. Chang, and Charles R. Jankowski. Wordspotter training using figure-of-merit back propagation. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 385–388, 1994.
Richard P. Lippmann et al. MIT Lincoln Laboratory offline component of DARPA 1998 intrusion detection evaluation. Presentation at MIT Lincoln Laboratory PI Meeting (available at) http://ideval.ll.mit.edu/intro-html-dir/, 14 December 1998.
Richard P. Lippmann et al. Evaluating intrusion detection systems: The 1998 DARPA off-line intrusion detection evaluation. In DISCEX 2000. IEEE Computer Society Press, January 2000.
Alvin Martin. Personal communications, January 2000.
Stephen L. Moshier. Personal communications, January 2000.
Vern Paxson. Bro: A system for detecting network intruders in real-time. Computer Networks, 31(23–24):2435–2463, December 1999.
Stacy J. Prowell, Carmen J. Trammell, Richard C. Linger, and Jesse H. Poore. Cleanroom Software Engineering: Technology and Process. Addison-Wesley, Reading, Mass., 1998.
John A. Swets. Measuring the accuracy of diagnostic systems. Science, 24(48):1285–1293, 3 June 1988.
Daniel Weber. A taxonomy of computer intrusions. MS thesis, Massachusetts Institute of Technology, 1998.
Q. E. Whiting-O’Keefe, Curtis Henke, and Donald W. Simborg. Choosing the correct unit of analysis in medical care experiments. Medical Care, 22(12):1101–1114, December 1984.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2000 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
McHugh, J. (2000). The 1998 Lincoln Laboratory IDS Evaluation. In: Debar, H., Mé, L., Wu, S.F. (eds) Recent Advances in Intrusion Detection. RAID 2000. Lecture Notes in Computer Science, vol 1907. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-39945-3_10
Download citation
DOI: https://doi.org/10.1007/3-540-39945-3_10
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-41085-0
Online ISBN: 978-3-540-39945-2
eBook Packages: Springer Book Archive