Abstract
Network-level emulation has recently been proposed as a method for the accurate detection of previously unknown polymorphic code injection attacks. In this paper, we extend network-level emulation along two lines. First, we present an improved execution behavior heuristic that enables the detection of a certain class of non-self-contained polymorphic shellcodes that are currently missed by existing emulation-based approaches. Second, we present two generic algorithmic optimizations that improve the runtime performance of the detector. We have implemented a prototype of the proposed technique and evaluated it using off-the-shelf non-self-contained polymorphic shellcode engines and benign data. The detector achieves a modest processing throughput, which however is enough for decent runtime performance on actual deployments, while it has not produced any false positives. Finally, we report attack activity statistics from a seven-month deployment of our prototype in a production network, which demonstrate the effectiveness and practicality of our approach.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Metasploit project (2006), http://www.metasploit.com/
Akritidis, P., Markatos, E.P., Polychronakis, M., Anagnostakis, K.: STRIDE: Polymorphic sled detection through instruction sequence analysis. In: Proceedings of the 20th IFIP International Information Security Conference (IFIP/SEC) (June 2005)
Anagnostakis, K., Sidiroglou, S., Akritidis, P., Xinidis, K., Markatos, E., Keromytis, A.D.: Detecting targeted attacks using shadow honeypots. In: Proceedings of the 14th USENIX Security Symposium, August 2005, pp. 129–144 (2005)
Bania, P.: TAPiON (2005), http://pb.specialised.info/all/tapion/
Bania, P.: Windows Syscall Shellcode (2005), http://www.securityfocus.com/infocus/1844
Bellard, F.: QEMU, a fast and portable dynamic translator. In: Proceedings of the USENIX Annual Technical Conference, FREENIX Track, pp. 41–46 (2005)
Bellovin, S.M.: There be dragons. In: Proceedings of the Third USENIX UNIX Security Symposium, pp. 1–16 (1992)
Chinchani, R., Berg, E.V.D.: A fast static analysis approach to detect exploit code inside network flows. In: Valdes, A., Zamboni, D. (eds.) RAID 2005. LNCS, vol. 3858, Springer, Heidelberg (2006)
Detristan, T., Ulenspiegel, T., Malcom, Y., Underduk, M.: Polymorphic shellcode engine using spectrum analysis. Phrack 11(61) (August 2003)
Dreger, H., Kreibich, C., Paxson, V., Sommer, R.: Enhancing the accuracy of network-based intrusion detection with host-based context. In: Julisch, K., Krügel, C. (eds.) DIMVA 2005. LNCS, vol. 3548, Springer, Heidelberg (2005)
Eller, R.: Bypassing MSB Data Filters for Buffer Overflow Exploits on Intel Platforms, http://community.core-sdi.com/~juliano/bypass-msb.txt
Fogla, P., Sharif, M., Perdisci, R., Kolesnikov, O., Lee, W.: Polymorphic blending attacks. In: Proceedings of the 15th USENIX Security Symposium (2006)
K2: ADMmutate (2001), http://www.ktwo.ca/ADMmutate-0.8.4.tar.gz
Kim, H.-A., Karp, B.: Autograph: Toward automated, distributed worm signature detection. In: Proceedings of the 13th USENIX Security Symposium, pp. 271–286 (2004)
Kruegel, C., Kirda, E., Mutz, D., Robertson, W., Vigna, G.: Polymorphic worm detection using structural information of executables. In: Valdes, A., Zamboni, D. (eds.) RAID 2005. LNCS, vol. 3858, Springer, Heidelberg (2006)
Li, Z., Sanghi, M., Chen, Y., Kao, M.-Y., Chavez, B.: Hamsa: Fast signature generation for zero-day polymorphic worms with provable attack resilience. In: Proceedings of the 2006 IEEE Symposium on Security and Privacy, pp. 32–47. IEEE Computer Society Press, Los Alamitos (2006)
Ma, J., Dunagan, J., Wang, H.J., Savage, S., Voelker, G.M.: Finding diversity in remote code injection exploits. In: Proceedings of the 6th ACM SIGCOMM on Internet measurement (IMC), pp. 53–64. ACM Press, New York (2006)
Newsome, J., Karp, B., Song, D.: Polygraph: Automatically generating signatures for polymorphic worms. In: Proceedings of the IEEE Security & Privacy Symposium, May 2005, pp. 226–241. IEEE Computer Society Press, Los Alamitos (2005)
Newsome, J., Karp, B., Song, D.: Paragraph: Thwarting signature learning by training maliciously. In: Zamboni, D., Kruegel, C. (eds.) RAID 2006. LNCS, vol. 4219, Springer, Heidelberg (2006)
Obscou: Building IA32 ‘unicode-proof’ shellcodes. Phrack 11(61) (August 2003)
Payer, U., Teufl, P., Lamberger, M.: Hybrid engine for polymorphic shellcode detection. In: Julisch, K., Krügel, C. (eds.) DIMVA 2005. LNCS, vol. 3548, pp. 19–31. Springer, Heidelberg (2005)
Polychronakis, M., Markatos, E.P., Anagnostakis, K.G.: Network-level polymorphic shellcode detection using emulation. In: Büschkes, R., Laskov, P. (eds.) DIMVA 2006. LNCS, vol. 4064, pp. 54–73. Springer, Heidelberg (2006)
Rix: Writing IA32 alphanumeric shellcodes. Phrack 11(57) (August 2001)
Singh, S., Estan, C., Varghese, G., Savage, S.: Automated worm fingerprinting. In: Proc. of the 6th Symposium on Operating Systems Design & Implementation (OSDI) (December 2004)
sk: History and Advances in Windows Shellcode. Phrack 11(62) (July 2004)
Skape: Shellcode text encoding utility for 7bit shellcode, http://www.hick.org/code/skape/nologin/encode/encode.c
Skape: Implementing a Custom x86 Encoder. Uninformed 5 (September 2006)
Ször, P., Ferrie, P.: Hunting for metamorphic. In: Proceedings of the Virus Bulletin Conference, September 2001, pp. 123–144 (2001)
Toth, T., Kruegel, C.: Accurate Buffer Overflow Detection via Abstract Payload Execution. In: Wespi, A., Vigna, G., Deri, L. (eds.) RAID 2002. LNCS, vol. 2516, Springer, Heidelberg (2002)
Wang, K., Cretu, G., Stolfo, S.J.: Anomalous Payload-based Worm Detection and Signature Generation. In: Valdes, A., Zamboni, D. (eds.) RAID 2005. LNCS, vol. 3858, Springer, Heidelberg (2006)
Wang, K., Parekh, J.J., Stolfo, S.J.: Anagram: A Content Anomaly Detector Resistant to Mimicry Attack. In: Zamboni, D., Kruegel, C. (eds.) RAID 2006. LNCS, vol. 4219, Springer, Heidelberg (2006)
Wang, X., Pan, C.-C., Liu, P., Zhu, S.: Sigfree: A signature-free buffer overflow attack blocker. In: Proceedings of the USENIX Security Symposium (August 2006)
Wever, B.-J.: Alpha 2 (2004), http://www.edup.tudelft.nl/~bjwever/src/alpha2.c
Yegneswaran, V., Barford, P., Ullrich, J.: Internet intrusions: global characteristics and prevalence. In: Proceedings of the 2003 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, pp. 138–147. ACM Press, New York (2003)
Zhang, Q., Reeves, D.S., Ning, P., Lyer, S.P.: Analyzing network traffic to detect self-decrypting exploit code. In: Proceedings of the ACM Symposium on Information, Computer and Communications Security (ASIACCS), ACM Press, New York (2007)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Polychronakis, M., Anagnostakis, K.G., Markatos, E.P. (2007). Emulation-Based Detection of Non-self-contained Polymorphic Shellcode. In: Kruegel, C., Lippmann, R., Clark, A. (eds) Recent Advances in Intrusion Detection. RAID 2007. Lecture Notes in Computer Science, vol 4637. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-74320-0_5
Download citation
DOI: https://doi.org/10.1007/978-3-540-74320-0_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-74319-4
Online ISBN: 978-3-540-74320-0
eBook Packages: Computer ScienceComputer Science (R0)