Automatic Generation of Compact Alphanumeric Shellcodes for x86

Abstract

Shellcode can be viewed as machine language code that is injected in the form of string input to exploit buffer overflows. It usually contains non-ASCII values because not all machine instructions encode into ASCII values. Many applications allow arbitrary string input, even though only strings containing characters that are ASCII or a subset of ASCII are deemed valid. Thus a common defense against shellcode injection is to discard any string input containing non-ASCII characters. Alphanumeric shellcode helps attackers bypass such character restrictions. It is non-trivial to construct alphanumeric shellcodes by hand and so tools have been created to automate the process. The alphanumeric equivalent, generated by the existing tools, is much larger than the original shellcode. This paper presents two new encoding schemes to reduce the size of the alphanumeric equivalent. A smaller shellcode is better as it can fit into smaller buffers and is even more useful in case an application restricts the input size. Results show that the size reduction of the encoded shellcode is more than 20% for many shellcodes.