Advertisement

Tutorial: An Overview of Malware Detection and Evasion Techniques

  • Fabrizio BiondiEmail author
  • Thomas Given-Wilson
  • Axel Legay
  • Cassius Puodzius
  • Jean Quilbeuf
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11244)

Abstract

This tutorial presents and motivates various malware detection tools and illustrates their usage on a clear example. We demonstrate how statically-extracted syntactic signatures can be used for quickly detecting simple variants of malware. Since such signatures can easily be obfuscated, we also present dynamically-extracted behavioral signatures which are obtained by running the malware in an isolated environment known as a sandbox. However, some malware can use sandbox detection to detect that they run in such an environment and so avoid exhibiting their malicious behavior. To counteract sandbox detection, we present concolic execution that can explore several paths of a binary. We conclude by showing how opaque predicates and JIT can be used to hinder concolic execution.

References

  1. 1.
    Agrawal, H., Bahler, L., Micallef, J., Snyder, S., Virodov, A.: Detection of global, metamorphic malware variants using control and data flow analysis. In: 31st IEEE Military Communications Conference, MILCOM 2012, Orlando, October 29 – November 1, 2012, pp. 1–6 (2012).  https://doi.org/10.1109/MILCOM.2012.6415581
  2. 2.
    Alazab, M., Venkatraman, S., Watters, P., Alazab, M.: Zero-day malware detection based on supervised learning algorithms of API call signatures. In: Proceedings of the Ninth Australasian Data Mining Conference, vol. 121, pp. 171–182. AusDM 2011, Australian Computer Society Inc., Darlinghurst (2011). http://dl.acm.org/citation.cfm?id=2483628.2483648
  3. 3.
    Canali, D., Lanzi, A., Balzarotti, D., Kruegel, C., Christodorescu, M., Kirda, E.: A quantitative study of accuracy in system call-based malware detection. In: Proceedings of the 2012 International Symposium on Software Testing and Analysis, pp. 122–132. ISSTA 2012. ACM, New York (2012).  https://doi.org/10.1145/2338965.2336768
  4. 4.
    Christodorescu, M., Jha, S., Seshia, S.A., Song, D., Bryant, R.E.: Semantics-aware malware detection. In: 2005 IEEE Symposium on Security and Privacy (SP 2005), pp. 32–46, May 2005.  https://doi.org/10.1109/SP.2005.20
  5. 5.
  6. 6.
    ClamAV: Clamav 0.99b meets yara! ClamAV blog. https://blog.clamav.net/2015/06/clamav-099b-meets-yara.html
  7. 7.
    Collberg, C., Martin, S., Myers, J., Nagra, J.: Distributed application tamper detection via continuous software updates. In: Proceedings of the 28th Annual Computer Security Applications Conference. ACSAC 2012, pp. 319–328. ACM, New York (2012).  https://doi.org/10.1145/2420950.2420997
  8. 8.
    Ehsan, F.: Detecting unknown malware: security analytics & memory forensics. Presentation at RSA 2015 Conference (2015). https://www.rsaconference.com/events/us15/agenda/sessions/1517/detecting-unknown-malware-security-analytics-memory
  9. 9.
    Godefroid, P., Klarlund, N., Sen, K.: Dart: Directed automated random testing. In: Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 213–223. PLDI 2005, ACM, New York (2005).  https://doi.org/10.1145/1065010.1065036
  10. 10.
    Goldberg, R.P.: Survey of virtual machine research. Computer 7(6), 34–45 (1974)CrossRefGoogle Scholar
  11. 11.
    Idika, N.C., Mathur, A.P.: A survey of malware detection techniques (2007)Google Scholar
  12. 12.
    Intel: Intel\({\textregistered }\) 64 and ia-32 architectures software developer’s manual combined volumes 2a, 2b, 2c, and 2d: Instruction set reference, a-z. Technical report, May 2018. https://software.intel.com/sites/default/files/managed/a4/60/325383-sdm-vol-2abcd.pdf, order Number: 325383–067US
  13. 13.
    Jung, P.: Bypassing sanboxes for fun! Presentation at hack.lu (2014). http://archive.hack.lu/2014/Bypasss_sandboxes_for_fun.pdf
  14. 14.
    Karbalaie, F., Sami, A., Ahmadi, M.: Semantic malware detection by deploying graph mining. Int. J. Comput. Sci. Issues (IJCSI) 9(1), 373 (2012)Google Scholar
  15. 15.
    Kirat, D., Vigna, G., Kruegel, C.: Barecloud: bare-metal analysis-based evasive malware detection. In: USENIX Security Symposium, pp. 287–301 (2014)Google Scholar
  16. 16.
    Kuzurin, N., Shokurov, A., Varnovsky, N., Zakharov, V.: On the concept of software obfuscation in computer security. In: Garay, J.A., Lenstra, A.K., Mambo, M., Peralta, R. (eds.) ISC 2007. LNCS, vol. 4779, pp. 281–298. Springer, Heidelberg (2007).  https://doi.org/10.1007/978-3-540-75496-1_19CrossRefzbMATHGoogle Scholar
  17. 17.
    Ligh, M.H., Case, A., Levy, J., Walters, A.: The Art of Memory Forensics: Detecting Malware and Threats in Windows, Linux, and Mac Memory. Wiley, Indianapolis (2014)Google Scholar
  18. 18.
    MissMalware: Tdsanomalpe - identifying compile time manipulation in pe headers. Miss Malware blog. http://missmalware.com/2017/02/tdsanomalpe-identifying-compile-time-manipulation-in-pe-headers/
  19. 19.
    Moser, A., Kruegel, C., Kirda, E.: Limits of static analysis for malware detection. In: Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007), pp. 421–430, December 2007.  https://doi.org/10.1109/ACSAC.2007.21
  20. 20.
    Moser, A., Kruegel, C., Kirda, E.: Limits of static analysis for malware detection. In: ACSAC, pp. 421–430. IEEE Computer Society (2007). http://dblp.uni-trier.de/db/conf/acsac/acsac2007.html#MoserKK07
  21. 21.
    Pietrek, M.: Peering inside the PE: A tour of the win32 portable executable file format. Microsoft Developer Network blog (1994). https://msdn.microsoft.com/en-us/library/ms809762.aspx
  22. 22.
    Preda, M.D., Christodorescu, M., Jha, S., Debray, S.: A semantics-based approach to malware detection. SIGPLAN Not 42(1), 377–388 (2007).  https://doi.org/10.1145/1190215.1190270,  https://doi.org/10.1145/1190215.1190270
  23. 23.
    Project, Y.: Yara documentation. https://yara.readthedocs.io/
  24. 24.
    Schwartz, M.: Oracle virtualbox multiple guest to host escape vulnerabilities. SecuriTeam Secure Disclosure blog (2018). https://blogs.securiteam.com/index.php/archives/3649
  25. 25.
    Sen, K., Marinov, D., Agha, G.: Cute: a concolic unit testing engine for c. In: Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 263–272. ESEC/FSE-13, ACM, New York, (2005).  https://doi.org/10.1145/1081706.1081750
  26. 26.
    Sharma, A., Sahay, S.K.: Evolution and detection of polymorphic and metamorphic malwares: a survey. Int. J. Comput. Appl. 90(2), 7–11 (2014)Google Scholar
  27. 27.
    Shoshitaishvili, Y., et al.: SoK: (state of) the art of war: offensive techniques in binary analysis. In: IEEE Symposium on Security and Privacy (2016)Google Scholar
  28. 28.
    Sikorski, M., Honig, A.: Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software, 1st edn. No Starch Press, San Francisco (2012)Google Scholar
  29. 29.
    Subwire, l.: throwing a tantrum, part 1: angr internals. Angr blog. http://angr.io/blog/throwing_a_tantrum_part_1
  30. 30.
    Szor, P.: The Art of Computer Virus Research and Defense. Addison-Wesley Professional, Boston (2005)Google Scholar
  31. 31.
    Vasilenko, E., Mamedov, O.: To crypt, or to mine - that is the question. post on Securelist - Kaspersky Lab’s cyberthreat research and reports (2018). https://securelist.com/to-crypt-or-to-mine-that-is-the-question/86307/
  32. 32.
  33. 33.
    Wojtczuk, R., Rutkowska, J.: Following the White Rabbit: Software attacks against Intel (R) VT-d technology (2011). http://www.invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf
  34. 34.
    Yadegari, B., Debray, S.: Symbolic execution of obfuscated code. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 732–744. ACM (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Fabrizio Biondi
    • 1
    Email author
  • Thomas Given-Wilson
    • 2
  • Axel Legay
    • 2
  • Cassius Puodzius
    • 2
  • Jean Quilbeuf
    • 2
  1. 1.CentraleSupélec/IRISARennesFrance
  2. 2.InriaRocquencourtFrance

Personalised recommendations