Abstract
Driven by the permanent search for reliable anomaly-based intrusion detection mechanisms, we investigated different options of neural network (NN) based techniques. A further improvement could be achieved by combining the best suited NN-based data mining techniques with a mechanism we call “execution chain evaluation”. This means that disassembled instruction chains are processed by the NN in order to detect malicious code. The proposed detection engine was trained and tested in various ways. Examples were taken from all publicly available polymorphic shellcode engines as well as from self-designed engines. A prototype implementation of our sensor has been realized and integrated as a plug-in into the SNORTTM[13] intrusion detection system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Metasploit project, http://www.metasploit.com (Retrieved on 15.10.2004)
AlephOne. Smashing the stack for fun and profit. Phrack Magazine 49(14) (1996), http://www.phrack.com
Bishop, C.M.: Neural networks for pattern recognition. The Clarendon Press Oxford University Press, New York (1995); With a foreword by Geoffrey Hinton
CLET team. Polymorphic shellcode engine. Phrack Magazine 61(9) (2003), http://www.phrack.com
Duda, R.O., Hart, P.E., Stork, D.G.: Pattern classification, 2nd edn. Wiley-Interscience, New York (2001)
K2. Admutate 0.8.4, http://www.ktwo.ca (Retrieved 29.03.2004)
Mathworks. Neural network toolbox, http://www.mathworks.com/products/neuralnet/ (Retrieved on 25.8.2004)
NASM SourceForge Project, http://nasm.sourceforge.net (Retrieved 11.02.2005)
Pasupulati, A.C., Levitt, J., Wu, K., Li, S.F., Kuo, S.H., Fan, J.C.: Buttercup: on network-based detection of polymorphic buffer overflow vulnerabilities. In: Network Operations and Management Symposium, 2004. NOMS 2004. IEEE/IFIP, vol. 1, pp. 235–248 (2004), http://wwwcsif.cs.ucdavis.edu/~pasupula/Buttercup-paper.doc
Roweis, S.: Levenberg-marquardt optimization, http://www.cs.toronto.edu/~roweis/notes/lm.pdf (Retrieved on 20.1.2005)
Ruiu, D.: Snort preprocessor - Multi-architecture mutated NOP sled detector, http://cansecwest.com/spp_fnord.c (Retrieved 11.02.2005)
Sedalo, M.: Polymorphic Shellcode Engine, http://www.shellcode.com.ar (Retrieved on 25.8.2004)
Snort. Open Source Network Intrusion Detection System, http://www.snort.org (Retrieved 11.02.2005)
Toth, T., Kruegel, C.: Accurate buffer overflow detection via abstract payload execution. In: Wespi, A., Vigna, G., Deri, L. (eds.) RAID 2002. LNCS, vol. 2516, pp. 274–291. Springer, Heidelberg (2002)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Payer, U., Teufl, P., Lamberger, M. (2005). Hybrid Engine for Polymorphic Shellcode Detection. In: Julisch, K., Kruegel, C. (eds) Detection of Intrusions and Malware, and Vulnerability Assessment. DIMVA 2005. Lecture Notes in Computer Science, vol 3548. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11506881_2
Download citation
DOI: https://doi.org/10.1007/11506881_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-26613-6
Online ISBN: 978-3-540-31645-9
eBook Packages: Computer ScienceComputer Science (R0)