Skip to main content
SpringerLink
Log in

HeapSentry: Kernel-Assisted Protection against Heap Overflows

  • Conference paper
Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA 2013)

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 7967))

Abstract

The last twenty years have witnessed the constant reaction of the security community to memory corruption attacks and the evolution of attacking techniques in order to circumvent the newly-deployed countermeasures. In this evolution, the heap of a process received little attention and thus today, the problem of heap overflows is largely unsolved.

In this paper we present HeapSentry, a system designed to detect and stop heap overflow attacks through the cooperation of the memory allocation library of a program and the operating system’s kernel. HeapSentry places unique random canaries at the end of each heap object which are later checked by the kernel, before system calls are allowed to proceed. HeapSentry operates on binaries (no source code needed) and has, by design, no false-positives. At the same time, the active involvement of the kernel provides stronger security guarantees than the current state of the art in heap protection mechanisms for a modest performance overhead.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 49.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Adobe: Security bulletins and advisories, http://www.adobe.com/support/security/

  2. Akritidis, P., Costa, M., Castro, M., Hand, S.: Baggy bounds checking: An efficient and backwards-compatible defense against out-of-bounds errors. In: Proceedings of the 18th USENIX Security Symposium, Montreal, QC (August 2009)

    Google Scholar 

  3. Aleph1: Smashing the stack for fun and profit. Phrack, 49 (1996)

    Google Scholar 

  4. Anley, C., Heasman, J., Linder, F.F., Richarte, G.: The Shellcoder’s Handbook: Discovering and Exploiting Security Holes, 2nd edn. Wiley Publishing (2007)

    Google Scholar 

  5. Berger, E.D.: Heapshield: Library-based heap overflow protection for free. UMass CS TR 06-28 (2006)

    Google Scholar 

  6. Berger, E.D., Zorn, B.G.: Diehard: Probabilistic memory safety for unsafe languages. In: Proceedings of 27th Conference on Programming Language Design and Implementation (June 2006)

    Google Scholar 

  7. Bernaschi, M., Gabrielli, E., Mancini, L.V.: Operating system enhancements to prevent the misuse of system calls. In: Proceedings of the 7th Conference on Computer and Communications Security (2000)

    Google Scholar 

  8. Bhatkar, S., DuVarney, D.C., Sekar, R.: Address obfuscation: An efficient approach to combat a broad range of memory error exploits. In: Proceedings of the 12th USENIX Security Symposium, Washington, D.C., pp. 105–120 (August 2003)

    Google Scholar 

  9. Bhatkar, S., Sekar, R.: Data space randomization. In: Zamboni, D. (ed.) DIMVA 2008. LNCS, vol. 5137, pp. 1–22. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  10. Checkoway, S., Davi, L., Dmitrienko, A., Sadeghi, A.-R., Shacham, H., Winandy, M.: Return-oriented programming without returns. In: Proceedings of CCS 2010. ACM Press (2010)

    Google Scholar 

  11. Chen, S., Xu, J., Sezer, E.C., Gauriar, P., Iyer, R.K.: Non-control-data attacks are realistic threats. In: Proceedings of the 14th USENIX Security Symposium, Baltimore, MD (August 2005)

    Google Scholar 

  12. Conover, M.: w00w00 on heap overflows, http://www.w00w00.org/files/articles/heaptut.txt

  13. Cowan, C., Pu, C., Maier, D., Hinton, H., Walpole, J., Bakke, P., Beattie, S., Grier, A., Wagle, P., Zhang, Q.: StackGuard: Automatic adaptive detection and prevention of buffer-overflow attacks. In: Proceedings of the 7th USENIX Security Symposium (1998)

    Google Scholar 

  14. Designer, S.: lpr LIBC RETURN exploit, http://insecure.org/sploits/linux.libc.return.lpr.sploit.html

  15. Dhurjati, D., Adve, V.: Backwards-compatible array bounds checking for C with very low overhead. In: Proceeding of the 28th International Conference on Software Engineering, Shanghai, China (2006)

    Google Scholar 

  16. Egele, M., Wurzinger, P., Kruegel, C., Kirda, E.: Defending browsers against drive-by downloads: Mitigating heap-spraying code injection attacks. In: Flegel, U., Bruschi, D. (eds.) DIMVA 2009. LNCS, vol. 5587, pp. 88–106. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  17. Gal, A., Eich, B., Shaver, M., Anderson, D., Mandelin, D., Haghighat, M.R., Kaplan, B., Hoare, G., Zbarsky, B., Orendorff, J., Ruderman, J., Smith, E.W., Reitmaier, R., Bebenita, M., Chang, M., Franz, M.: Trace-based just-in-time type specialization for dynamic languages. In: ACM Conference on Programming Language Design and Implementation (2009)

    Google Scholar 

  18. IBM: Gcc extension for protecting applications from stack-smashing attacks, http://www.trl.ibm.com/projects/security/ssp/

  19. Jones, R.W.M., Kelly, P.H.J.: Backwards-compatible bounds checking for arrays and pointers in C programs. In: Proceedings of the 3rd International Workshop on Automatic Debugging, Linköping, Sweden, pp. 13–26 (1997)

    Google Scholar 

  20. Kc, G.S., Keromytis, A.D.: e-NeXSh: Achieving an effectively non-executable stack and heap via system-call policing. In: Annual Computer Security Applictions Conference (2005)

    Google Scholar 

  21. Keniston, J., Panchamukhi, P.S., Hiramatsu, M.: Kernel probes (kprobes)

    Google Scholar 

  22. Lin, C., Rajagopalan, M., Baker, S., Collberg, C., Debray, S., Hartman, J.: Protecting against unexpected system calls. In: Proceedings of the 14th USENIX Security Symposium, Baltimore, Maryland. USENIX Association (August 2005)

    Google Scholar 

  23. Lvin, V.B., Novark, G., Berger, E.D., Zorn, B.G.: Archipelago: trading address space for reliability and security. In: Proceedings of the 13th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XIII. ACM (2008)

    Google Scholar 

  24. Microsoft: Security advisories, http://www.microsoft.com/technet/security/advisory/

  25. Novark, G., Berger, E.D.: Dieharder: securing the heap. In: Proceedings of the 17th ACM Conference on Computer and Communications Security, CCS 2010, pp. 573–584. ACM, New York (2010)

    Chapter  Google Scholar 

  26. National Vulnerability Database, http://nvd.nist.gov

  27. PaX: Documentation for the PaX project, http://pax.grsecurity.net/

  28. Payer, M.: I control your code. In: Proceedings of the 27th Chaos Communication Congress (27c3) (2010)

    Google Scholar 

  29. Provos, N.: Improving host security with system call policies. In: Proceedings of the 12th USENIX Security Symposium, Washington, D.C. (August 2003)

    Google Scholar 

  30. Rivner, U.: Anatomy of the rsa attack, http://blogs.rsa.com/rivner/anatomy-of-an-attack/

  31. Robertson, W., Kruegel, C., Mutz, D., Valeur, F.: Run-time detection of heap-based overflows. In: Proceedings of the 17th Large Installation Systems Administrators Conference, San Diego, CA, pp. 51–60 (October 2003)

    Google Scholar 

  32. Roglia, G.F., Martignoni, L., Paleari, R., Bruschi, D.: Surgically returning to randomized lib(c). In: 25th Annual Computer Security Applications Conference (2009)

    Google Scholar 

  33. Shacham, H.: The geometry of innocent flesh on the bone: Return-into-libc without function calls (on the x86). In: Proceedings of the 14th ACM Conference on Computer and Communications Security (2007)

    Google Scholar 

  34. Solar Designer: Non-executable user stack, http://www.openwall.com/linux/

  35. Spafford, E.H.: The internet worm program: An analysis. Computer Communication Review 19 (1988)

    Google Scholar 

  36. Strace(1): trace system calls/signals, http://linux.die.net/man/1/strace

  37. Van Acker, S., Nikiforakis, N., Philippaerts, P., Younan, Y., Piessens, F.: ValueGuard: Protection of Native Applications against Data-Only Buffer Overflows. In: Jha, S., Mathuria, A. (eds.) ICISS 2010. LNCS, vol. 6503, pp. 156–170. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  38. Wilander, J., Nikiforakis, N., Younan, Y., Kamkar, M., Joosen, W.: Ripe: Runtime intrusion prevention evaluator. In: Proceedings of the 27th Annual Computer Security Applications Conference, ACSAC (2011)

    Google Scholar 

  39. Younan, Y., Joosen, W., Piessens, F.: Efficient protection against heap-based buffer overflows without resorting to magic. In: Proceedings of the International Conference on Information and Communication Security, Raleigh, NC (December 2006)

    Google Scholar 

  40. Younan, Y., Joosen, W., Piessens, F.: Runtime countermeasures for code injection attacks against C and C++ programs. ACM Computing Surveys 44(3), 17:1–17:28 (2012)

    Google Scholar 

  41. Zeng, Q., Wu, D., Liu, P.: Cruiser: concurrent heap buffer overflow monitoring using lock-free data structures. In: Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (2011)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. iMinds-DistriNet, KU Leuven, 3001, Leuven, Belgium

    Nick Nikiforakis, Frank Piessens & Wouter Joosen

Authors
  1. Nick Nikiforakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Frank Piessens
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wouter Joosen
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Institute of Computer Science, Computer Security Group, University of Göttingen, Goldschmidtstr. 7, 37077, Göttingen, Germany

    Konrad Rieck

  2. Telekom Innovation Laboratories, Security in Telecommunications, Technische Universität Berlin, Ernst-Reuter-Platz 7, 10587, Berlin, Germany

    Patrick Stewin  & Jean-Pierre Seifert  & 

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Nikiforakis, N., Piessens, F., Joosen, W. (2013). HeapSentry: Kernel-Assisted Protection against Heap Overflows. In: Rieck, K., Stewin, P., Seifert, JP. (eds) Detection of Intrusions and Malware, and Vulnerability Assessment. DIMVA 2013. Lecture Notes in Computer Science, vol 7967. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39235-1_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-39235-1_11

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-39234-4

  • Online ISBN: 978-3-642-39235-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation