Automated Software Vulnerability Analysis

  • Emre C. SezerEmail author
  • Chongkyung Kil
  • Peng Ning
Part of the Advances in Information Security book series (ADIS, volume 46)


Despite decades of research, software continues to have vulnerabilities. Successful exploitations of these vulnerabilities by attackers cost millions of dollars to businesses and individuals. Unfortunately, most effective defensive measures, such as patching and intrusion prevention systems, require an intimate knowledge of the vulnerabilities. Many systems for detecting attacks have been proposed. However, the analysis of the exploited vulnerabilities is left to security experts and programmers. Both the human effortinvolved and the slow analysis process are unfavorable for timely defensive measure to be deployed. The problem is exacerbated by zero-day attacks.

This chapter presents two recent research efforts, named MemSherlock and CBones, for automatically aiding experts in identifying and analyzing unknown vulnerabilities. Both methods rely on monitoring user applications during their runtime and checking for inconsistencies in their memory or memory access patterns. MemSherlock is a post-mortem analysis tool that monitors an application’s memory operations to determine malicious ones, indicative of an ongoing attack. It produces valuable information regarding the vulnerability and the attack vector. CBones takes snapshots of the memory and looks for inconsistencies by identifying invariants for an application’s memory and verifying them at runtime. Experimental evaluation shows that both methods are capable of providing critical information about vulnerabilities and attack vectors.


Structural Constraint Activation Record Return Address Memory Region Monitoring Agent 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    E.D. Berger, K.S. McKinley, R.D. Blumofe, and P.R. Wilson. Hoard: A scalable memory allocator for multithreaded applications. In Ninth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS-IX), November 2000.Google Scholar
  2. 2.
    David Brumley, James Newsome, Dawn Song, Hao Wang, and Somesh Jha. Towards automatic generation of vulnerability-based signatures. In Proceedings of the IEEE Symposium on Security and Privacy, May 2006.Google Scholar
  3. 3.
  4. 4.
  5. 5.
    S. Cesare. Shared library call redirection using elf plt infection, April 2007.
  6. 6.
    H. Chen, D. Dean, and D. Wagner. Model checking one million lines of c code. In Proceedings of the 11th Annual Network and Distributed System Security Symposium (NDSS), February 2004.Google Scholar
  7. 7.
    H. Chen and D. Wagner. MOPS: an infrastructure for examining security properties of software. In Proceedings of the 9th ACM Conference on Computer and Communications Security (CCS’02), November 2002.Google Scholar
  8. 8.
    Shou Chen, Jun Xu, and Emre C. Sezer. Non-control-data attacks are realistic threats. In Proceedings of 14th USENIX Security Symposium, 2005.Google Scholar
  9. 9.
    E. Chien and P. Szor. Blended attacks exploits, vulnerabilities and buffer-overflow techniques. In Techniques in Computer Viruses, Virus Bulletin Conference, 2002.Google Scholar
  10. 10.
    Tool Interface Standard (TIS) Committee. Executable and linking format (elf) specification, 1995.Google Scholar
  11. 11.
    J. R. Crandall and F. T. Chong. Minos: Control data attack prevention orthogonal to memory model. In Proceedings of the 37th Annual IEEE/ACM International Symposium on Microarchitecture, pages 221–232, December 2004.Google Scholar
  12. 12.
    J. R. Crandall, Z. Su, S. F. Wu, and F. T. Chong. On deriving unknown vulnerabilities from zero-day polymorphic and metamorphic worm exploits. In Proceedings of the 13th ACM Conference on Computer and Communications Security, pages 235–248, 2005.Google Scholar
  13. 13.
    H. Feng, J. Giffin, Y. Huang, S. Jha, W. Lee, and B. Miller. Formalizingsensitivity in static analysis for intrusion detection. In Proceedings of the 2004 IEEE Symposium on Security and Privacy, May 2004.Google Scholar
  14. 14.
    Dawn Song James Newsome, David Brumley. Vulnerability-specific execution filtering for exploit prevention on commodity software. In Proceedings of the 13th Annual Network and Distributed System Security Symposium (NDSS ’06), Feb 2006.Google Scholar
  15. 15.
    T. Jim, G. Morrisett, D. Grossman, M. Hicks, J. Cheney, and Y. Wang. Cyclone: A safe dialect of C. In USENIX Annual Technical Conference, June 2002.Google Scholar
  16. 16.
    Chongkyung Kil, Jinsuk Jun, Christopher Bookholt, Jun Xu, and Peng Ning. Address space layout permutation (aslp): Towards fine-grained randomization of commodity software. In Computer Security Applications Conference, 2006. ACSAC ’06. 22nd Annual, pages 339–348, Dec. 2006.Google Scholar
  17. 17.
    Chongkyung Kil, E.C. Sezer, Peng Ning, and Xiaolan Zhang. Automated security debugging using program structural constraints. In Computer Security Applications Conference, 2007. ACSAC 2007. Twenty-Third Annual, pages 453–462, Dec. 2007.Google Scholar
  18. 18.
    W. Landi. Undecidability of static analysis. ACM Letters on Programming Languages and Systems, 1(4):323–337, December 1992.CrossRefGoogle Scholar
  19. 19.
    D. Larochelle and D. Evans. Statically detecting likely buffer overflow vulnerabilities. In Proceedings of the 10th USENIX Security Symposium, August 2001.Google Scholar
  20. 20.
  21. 21.
    Z. Liang and R. Sekar. Fast and automated generation of attack signatures: a basis for building self-protecting servers. In Proceedings of the 13th ACM Conference on Computer and Communications Security, pages 213–222, 2005.Google Scholar
  22. 22.
    NIST national vulerability database.
  23. 23.
    G. Necula, S. McPeak, and W. Weimer. CCureds: Type-safe retrofitting of legacy software. In Proceedings of the 29th ACM SIGPLAN-SIGACT symposium on Principles of Programming Languages, pages 128–139, 2002.Google Scholar
  24. 24.
    Nicholas Nethercote. Dynamic binary analysis and instrumentation, 2004. Scholar
  25. 25.
    J. Newsome and D. Song. Dynamic taint analysis for automatic detection, analysis, and signature generation of exploits on commodity software. In Proceedings of The 12th Annual Network and Distributed System Security Symposium (NDSS ’05), February 2005.Google Scholar
  26. 26.
    Open group base specifications issue 6, ieee std 1003.1, 2004 edition.Google Scholar
  27. 27.
  28. 28.
  29. 29.
    G. Ramalingam. The undecidability of aliasing. ACM Transactions on Programming Languages and Systems, 16(5):1467–1471, September 1994.CrossRefGoogle Scholar
  30. 30.
    Emre C. Sezer, Peng Ning, Chongkyung Kil, and Jun Xu. Memsherlock: An automated debugger for unknown memory corruption vulnerabilities. In CCS ’07: Proceedings of the 14th ACM conference on Computer and communications security, pages 562–572, New York, NY, USA, 2007. ACM.Google Scholar
  31. 31.
    Sumus vulnerability. Common vulnerabilities and exposures (cve) 2005-1110, April 2005.
  32. 32.
    H. Wang, C. Guo, D. Simon, and A. Zugenmaier. Shield: Vulnerability-driven network filters for preventing known vulnerability exploits. In Proceedings of ACM SIGCOMM, August 2004.Google Scholar
  33. 33.
    Pin Zhou, Wei Liu, Long Fei, Shan Lu, Feng Qin, Yuanyuan Zhou, Samuel Midkiff, and Josep Torrellas. Accmon: Automatically detecting memory-related bugs via program counter-based invariants. In MICRO 37: Proceedings of the 37th annual International Symposium on Microarchitecture, pages 269–280, Washington, DC, USA, 2004. IEEE Computer Society.Google Scholar

Copyright information

© Springer-Verlag US 2010

Authors and Affiliations

  1. 1.Department of Computer ScienceNC State UniversityRaleighUSA

Personalised recommendations