Hooking Behavior Analysis

Abstract

Installing various hooks into the victim system is an important attacking strategy employed by malware, including spyware, rootkits, stealth backdoors, and others. In order to defeat existing hook detectors, malware writers keep exploring new hooking mechanisms. However, the current malware analysis procedure is painstaking, mostly manual and error-prone. In this chapter, we propose the first systematic approach for automatically identifying hooks and extracting hooking mechanisms. We propose a unified approach, fine-grained impact analysis, to identify malware hooking behaviors. Our approach does not rely on any prior knowledge of hooking mechanisms, and thus can identify novel hooking mechanisms. Moreover, we devise a method using semantics-aware impact dependency analysis to provide a succinct and intuitive graph representation to illustrate hooking mechanisms. We have developed a prototype, HookFinder, and conducted extensive experiments using representative malware samples from various categories. We have demonstrated that HookFinder can correctly identify the hooking behaviors of all samples, and provide accurate insights about their hooking mechanisms.