Skip to main content
SpringerLink
Log in

Detecting Traditional Packers, Decisively

  • Conference paper
Research in Attacks, Intrusions, and Defenses (RAID 2013)

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 8145))

Included in the following conference series:

Abstract

Many of the important decidability results in malware analysis are based Turing machine models of computation. We exhibit computational models which use more realistic assumptions about machine and attacker resources. While seminal results such as [1–5] remain true for Turing machines, we show under more realistic assumptions, important tasks are decidable instead of undecidable. Specifically, we show that detecting traditional malware unpacking behavior – in which a payload is decompressed or decrypted and subsequently executed – is decidable under our assumptions. We then examine the issue of dealing with complex but decidable problems. We look for lessons from the hardware verification community, which has been striving to meet the challenge of intractable problems for the past three decades.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Royal, P., Halpin, M., Dagon, D., Edmonds, R., Lee, W.: PolyUnpack: Automating the hidden-code extraction of unpack-executing malware. In: Annual Computer Security Applications Conference, pp. 289–300. IEEE Computer Society (2006)

    Google Scholar 

  2. Christodorescu, M., Jha, S., Seshia, S.A., Song, D.X., Bryant, R.E.: Semantics-aware malware detection. In: Security and Privacy, pp. 32–46. IEEE Computer Society Press (2005)

    Google Scholar 

  3. Oppliger, R., Rytz, R.: Does trusted computing remedy computer security problems? IEEE Security Privacy 3(2), 16–19 (2005)

    Article  Google Scholar 

  4. Brumley, D., Hartwig, C., Liang, Z., Newsome, J., Song, D.X., Yin, H.: Automatically identifying trigger-based behavior in malware. In: Botnet Detection, pp. 65–88. Springer (2008)

    Google Scholar 

  5. Newsome, J., Brumley, D., Franklin, J., Song, D.: Replayer: automatic protocol replay by binary analysis. In: ACM Conference on Computer and Communications Security, CCS 2006, pp. 311–321. ACM, New York (2006)

    Google Scholar 

  6. Bayer, U., Kirda, E., Kruegel, C.: Improving the efficiency of dynamic malware analysis. In: Proceedings of the 2010 ACM Symposium on Applied Computing, pp. 1871–1878. ACM (2010)

    Google Scholar 

  7. Guo, F., Ferrie, P., Chiueh, T.-c.: A study of the packer problem and its solutions. In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 98–115. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  8. Cook, S.A., Reckhow, R.A.: Time bounded random access machines. J. Comput. Syst. Sci. 7(4), 354–375 (1973)

    Article  MathSciNet  MATH  Google Scholar 

  9. Jang, J., Brumley, D., Venkataraman, S.: Bitshred: feature hashing malware for scalable triage and semantic analysis. In: ACM Conference on Computer and Communications Security, CCS 2011, pp. 309–320. ACM, New York (2011)

    Google Scholar 

  10. Moser, A., Kruegel, C., Kirda, E.: Limits of static analysis for malware detection. In: Computer Security Applications Conference, pp. 421–430 (2007)

    Google Scholar 

  11. Zhang, Q., Reeves, D.S.: MetaAware: Identifying metamorphic malware. In: Annual Computer Security Applications Conference, pp. 411–420. IEEE Computer Society Press (2007)

    Google Scholar 

  12. Sharif, M.I., Lanzi, A., Giffin, J.T., Lee, W.: Automatic reverse engineering of malware emulators. In: Security and Privacy, pp. 94–109. IEEE Computer Society (2009)

    Google Scholar 

  13. Kang, M.G., Poosankam, P., Yin, H.: Renovo: A hidden code extractor for packed executables. In: WORM. ACM (November 2007)

    Google Scholar 

  14. Martignoni, L., Christodorescu, M., Jha, S.: OmniUnpack: Fast, generic, and safe unpacking of malware. In: Annual Computer Security Applications Conference, pp. 431–441. IEEE Computer Society Press (2007)

    Google Scholar 

  15. Yin, H., Song, D.: Hidden code extraction. In: Automatic Malware Analysis. SpringerBriefs in Computer Science, pp. 17–26. Springer, New York (2013)

    Chapter  Google Scholar 

  16. Liu, L., Ming, J., Wang, Z., Gao, D., Jia, C.: Denial-of-service attacks on host-based generic unpackers. In: Qing, S., Mitchell, C.J., Wang, G. (eds.) ICICS 2009. LNCS, vol. 5927, pp. 241–253. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  17. Xie, P.D., Li, M.J., Wang, Y.J., Su, J.S., Lu, X.C.: Unpacking techniques and tools in malware analysis. Applied Mechanics and Materials 198–199, 343–350 (2012)

    Article  Google Scholar 

  18. Perdisci, R., Lanzi, A., Lee, W.: Classification of packed executables for accurate computer virus detection. Pattern Recognition Letters 29(14), 1941–1946 (2008)

    Article  Google Scholar 

  19. Spinellis, D.: Reliable identification of bounded-length viruses is NP-complete. IEEE Transactions on Information Theory 49(1), 280–284 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  20. Borello, J.M., Mé, L.: Code obfuscation techniques for metamorphic viruses. Journal in Computer Virology 4(3), 211–220 (2008)

    Article  Google Scholar 

  21. Katz, J., Lindell, Y.: Introduction to Modern Cryptography. Chapman & Hall (2008)

    Google Scholar 

  22. Elgot, C.C., Robinson, A.: Random-access stored-program machines, an approach to programming languages. J. ACM 11(4), 365–399 (1964)

    Article  MathSciNet  MATH  Google Scholar 

  23. Hartmanis, J.: Computational complexity of random access stored program machines. Mathematical Systems Theory 5(3), 232–245 (1971)

    Article  MathSciNet  MATH  Google Scholar 

  24. Aho, A.V., Hopcroft, J.E., Ullman, J.D.: The Design and Analysis of Computer Algorithms. Addison-Wesley (1974)

    Google Scholar 

  25. Cohen, F.: Computer Viruses. PhD thesis, University of Southern California (1986)

    Google Scholar 

  26. Cohen, F.: Computational aspects of computer viruses. Computers & Security 8(4), 297–298 (1989)

    Article  Google Scholar 

  27. Adleman, L.M.: An abstract theory of computer viruses. In: Goldwasser, S. (ed.) CRYPTO 1988. LNCS, vol. 403, pp. 354–374. Springer, Heidelberg (1990)

    Chapter  Google Scholar 

  28. Thimbleby, H., Anderson, S., Cairns, P.: A framework for modelling trojans and computer virus infection. The Computer Journal 41(7), 444–458 (1998)

    Article  MATH  Google Scholar 

  29. Chess, D.M., White, S.R.: An undetectable computer virus. In: Proceedings of Virus Bulletin Conference, vol. 5 (2000)

    Google Scholar 

  30. Filiol, E., Josse, S.: A statistical model for undecidable viral detection. Journal in Computer Virology 3(2), 65–74 (2007)

    Article  Google Scholar 

  31. Oreans Technologies, http://www.oreans.com/themida.php

  32. Sipser, M.: Introduction to the Theory of Computation, vol. 27. Thomson Course Technology, Boston (2006)

    MATH  Google Scholar 

  33. Shaw, D.E., Deneroff, M.M., Dror, R.O., Kuskin, J.S., Larson, R.H., Salmon, J.K., Young, C., Batson, B., Bowers, K.J., Chao, J.C., et al.: Anton, a special-purpose machine for molecular dynamics simulation. In: ACM SIGARCH Computer Architecture News, vol. 35, pp. 1–12. ACM (2007)

    Google Scholar 

  34. Stevens, M., Lenstra, A.K., de Weger, B.: Chosen-prefix collisions for MD5 and colliding X.509 certificates for different identities. In: Naor, M. (ed.) EUROCRYPT 2007. LNCS, vol. 4515, pp. 1–22. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  35. Garey, M.R., Johnson, D.S.: Computers and Intractability, vol. 174. Freeman, New York (1979)

    MATH  Google Scholar 

  36. Clarke, E.M., Emerson, E.A.: Design and Synthesis of Synchronization Skeletons Using Branching-Time Temporal Logic. In: Engeler, E. (ed.) Logic of Programs 1979. LNCS, vol. 125, pp. 52–71. Springer, Heidelberg (1981)

    Google Scholar 

  37. Queille, J.P., Sifakis, J.: Specification and Verification of Concurrent Systems in CESAR. In: Dezani-Ciancaglini, M., Montanari, U. (eds.) Programming 1982. LNCS, vol. 137, pp. 337–351. Springer, Heidelberg (1982)

    Chapter  Google Scholar 

  38. Clarke, E.M.: The Birth of Model Checking. In: Grumberg, O., Veith, H. (eds.) 25MC Festschrift. LNCS, vol. 5000, pp. 1–26. Springer, Heidelberg (2008)

    Google Scholar 

  39. Sistla, A.P., Clarke, E.M.: The complexity of propositional linear temporal logics. J. ACM 32(3), 733–749 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  40. Clarke, E.M., Emerson, E.A., Sistla, A.P.: Automatic Verification of Finite State Concurrent Systems Using Temporal Logic Specifications: A Practical Approach. In: POPL, pp. 117–126 (1983)

    Google Scholar 

  41. Burch, J.R., Clarke, E.M., McMillan, K.L., Dill, D.L., Hwang, L.J.: Symbolic model checking: 1020 states and beyond. In: LICS, pp. 428–439 (1990)

    Google Scholar 

  42. Marques-Silva, J.A.P., Sakallah, K.A.: GRASP-A New Search Algorithm for Satisfiability. In: Digest of IEEE International Conference on Computer-Aided Design, ICCAD, San Jose, California, pp. 220–227 (November 1996)

    Google Scholar 

  43. Marques-Silva, J.A.P., Sakallah, K.A.: GRASP: A Search Algorithm for Propositional Satisfiability. IEEE Transactions on Computers 48(5), 506–521 (1999)

    Article  MathSciNet  Google Scholar 

  44. Moskewicz, M.W., Madigan, C.F., Zhao, Y., Zhang, L., Malik, S.: Chaff: Engineering an Efficient SAT Solver. In: DAC, pp. 530–535 (2001)

    Google Scholar 

  45. Biere, A., Cimatti, A., Clarke, E.M., Zhu, Y.: Symbolic Model Checking without BDDs. In: Cleaveland, W.R. (ed.) TACAS 1999. LNCS, vol. 1579, pp. 193–207. Springer, Heidelberg (1999)

    Chapter  Google Scholar 

  46. Clarke, E., Biere, A., Raimi, R., Zhu, Y.: Bounded Model Checking Using Satisfiability Solving. Form. Methods Syst. Des. 19, 7–34 (2001)

    Article  MATH  Google Scholar 

  47. Clarke, E., Grumberg, O., Jha, S., Lu, Y., Veith, H.: Counterexample-Guided Abstraction Refinement. In: Emerson, E.A., Sistla, A.P. (eds.) CAV 2000. LNCS, vol. 1855, pp. 154–169. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  48. Clarke, E., Grumberg, O., Jha, S., Lu, Y., Veith, H.: Counterexample-Guided Abstraction Refinement for Symbolic Model Checking. J. ACM 50, 752–794 (2003)

    Article  MathSciNet  Google Scholar 

  49. Bradley, A.R., Manna, Z.: Checking Safety by Inductive Generalization of Counterexamples to Induction. In: Formal Methods in Computer Aided Design, FMCAD 2007, pp. 173–180 (November 2007)

    Google Scholar 

  50. Bradley, A.R.: SAT-Based Model Checking without Unrolling. In: Jhala, R., Schmidt, D. (eds.) VMCAI 2011. LNCS, vol. 6538, pp. 70–87. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  51. Knuth, D.E.: Art of Computer Programming, 3rd edn. Fundamental Algorithms, vol. 1. Addison-Wesley Professional (July 1997)

    Google Scholar 

  52. Amdahl, G.M.: Validity of the single processor approach to achieving large scale computing capabilities. In: Proceedings of the Spring Joint Computer Conference, AFIPS 1967 (Spring), April 18-20, pp. 483–485. ACM, New York (1967)

    Chapter  Google Scholar 

  53. Downey, R.G., Fellows, M.R., Stege, U.: Computational tractability: The view from mars. In: Bulletin of the European Association of Theoretical Computer Science, pp. 73–97

    Google Scholar 

  54. VMProtect Software, http://vmpsoft.com/

Download references

Author information

Authors and Affiliations

  1. Electrical Engineering and Computer Science Department, University of Michigan, USA

    Denis Bueno, Kevin J. Compton, Karem A. Sakallah & Michael Bailey

Authors
  1. Denis Bueno
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kevin J. Compton
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Karem A. Sakallah
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael Bailey
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer Science, Columbia University, 1214 Amsterdam Avenue, 10027, NY, USA

    Salvatore J. Stolfo

  2. Department of Computer Science, George Mason University, 22030, Fairfax, VA, USA

    Angelos Stavrou

  3. Department of Computer Science, Portland State University, 97201, Portland, OR, USA

    Charles V. Wright

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Bueno, D., Compton, K.J., Sakallah, K.A., Bailey, M. (2013). Detecting Traditional Packers, Decisively. In: Stolfo, S.J., Stavrou, A., Wright, C.V. (eds) Research in Attacks, Intrusions, and Defenses. RAID 2013. Lecture Notes in Computer Science, vol 8145. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-41284-4_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-41284-4_10

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-41283-7

  • Online ISBN: 978-3-642-41284-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation