An Automatic Approach to Detect Anti-debugging in Malware Analysis

Abstract

Anti-debugging techniques are broadly used by malware authors to prevent security researchers from reversing engineering their created malware samples. However, the countermeasures to identify anti-debugging code patterns are insufficient, and mainly manual, which is an expensive, time-consuming, and error-prone process. There are no automatic approaches which can be used to detect anti-debugging code patterns in malware samples effectively. In this paper, we present an approach, based on instruction traces derived from dynamic malware analysis and an instruction-based pattern matching method, to detect anti-debugging tricks automatically. We evaluate this approach with a large number of malware samples collected in the wild. The experience shows that our proposed approach is effective and about 40% of malware samples in our experimental data set has been embedded anti-debugging code.