Abstract
Most behavioral detectors of malware remain specific to a given language and platform, mostly executables for Windows. The objective of this paper is to define a generic approach for behavioral detection based on two layers respectively responsible for abstraction and detection. The abstraction layer is specific to a platform and a language. It interprets the collected instructions, API calls and arguments and classifies these operations, as well as the objects involved, according to their purpose in the malware lifecycle. The detection layer remains generic and interoperable with different abstraction components. It relies on parallel automata parsing attribute-grammars where semantic rules are used for object typing (object classification) and object binding (data-flow). Theoretical results are first given with respect to the grammatical constraints weighting on the signature construction as well as to the resulting complexity of the detection. For experimentation purposes, two abstraction components have then been developed: one processing system call traces and the other processing the VBScript interpreted language. Experimentations have provided promising detection rates, in particular for scripts (89%), with almost no false positives. In the case of process traces, the detection rate remains significant (51%) but could be increased by sophisticated collection tools.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Charlier, B.L., Mounji, A., Swimmer, M.: Dynamic detection and classification of computer viruses using general behaviour patterns. Virus Bulletin (1995)
Newsome, J., Song, D.: Dynamic taint analysis for automatic detection, analysis, and signature generation of exploits on commodity software. In: Proc. of the Network and Distributed System Security Symposium, NDSS (2005)
Bhatkar, S., Chaturvedi, A., Sekar, R.: Dataflow anomaly detection. In: Proc. of the IEEE Symposium on Security and Privacy (SSP), pp. 48–62 (2006)
Christodorescu, M., Jha, S., Kruegel, C.: Mining specifications of malicious behaviour. In: Proc. of the 6th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineeering, pp. 5–14 (2007)
Morales, J.A., Clarke, P.J., Deng, Y.: Identification of file infecting viruses through detection of self-reference replication. Journal in Computer Virology Online (2008)
Martignoni, L., Stinson, E., Fredrikson, M., Jha, S., Mitchell, J.C.: A layered architecture for detecting malicious behaviors. In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 78–97. Springer, Heidelberg (2008)
Knuth, D.E.: Semantics of context-free grammars. Theory of Computing Systems 2, 127–145 (1968)
Jacob, G., Filiol, E., Debar, H.: Malwares as interactive machines: A new framework for behavior modelling. Journal in Computer Virology 4(3), 235–250 (2008)
Jacob, G., Filiol, E., Debar, H.: Functional polymorphic engines: Formalisation, implementation and use cases. Journal in Computer Virology Online (2008)
US Department of Defense: “Orange Book” - Trusted Computer System Evaluation Criteria. Rainbow Series (1983)
NTInternals: The undocumented functions microsoft windows nt/2k/xp/2003, http://undocumented.ntinternals.net
Kruegel, C., Mutz, D., Valeur, F., Vigna, G.: On the detection of anomalous system call arguments. In: Snekkenes, E., Gollmann, D. (eds.) ESORICS 2003. LNCS, vol. 2808, pp. 326–343. Springer, Heidelberg (2003)
Cuppens, F., Miège, A.: Alert correlation in a cooperative intrusion detection framework. In: Proc. of the IEEE Symposium on Security and Privacy (SSP), p. 202 (2002)
Al-Mamory, S.O., Zhang, H.: Ids alerts correlation using grammar-based approach. Journal in Computer Virology Online (2008)
NtTrace: Native api tracing for windows, http://www.howzatt.demon.co.uk/NtTrace/
QEMU: Processor emulator, http://fabrice.bellard.free.fr/qemu/
Marion, J.Y., Reynaud-Plantey, D.: Practical obfuscation by interpretation. In: 3rd Workshop on the Theory of Computer Viruses, WTCV (2008)
MSDN: Vbscript language reference, http://msdn.microsoft.com/en-us/library/d1wf56tt.aspx
VXHeaven: Repository, http://vx.netlux.org/
OffensiveComputing: Repository, http://www.offensivecomputing.net/
Carrera, E.: Malware - behavior, tools, scripting and advanced analysis. In: HITBSec Conf. (2008)
Anubis: Analyzing unknown malware, http://anubis.iseclab.org/
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Jacob, G., Debar, H., Filiol, E. (2009). Malware Behavioral Detection by Attribute-Automata Using Abstraction from Platform and Language. In: Kirda, E., Jha, S., Balzarotti, D. (eds) Recent Advances in Intrusion Detection. RAID 2009. Lecture Notes in Computer Science, vol 5758. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-04342-0_5
Download citation
DOI: https://doi.org/10.1007/978-3-642-04342-0_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-04341-3
Online ISBN: 978-3-642-04342-0
eBook Packages: Computer ScienceComputer Science (R0)