Abstract
Many software security solutions—including malware analyzers, information flow tracking systems, auditing utilities, and host-based intrusion detectors—rely on knowledge of standard system call interfaces to reason about process execution behavior. In this work, we show how a rootkit can obfuscate a commodity kernel’s system call interfaces to degrade the effectiveness of these tools. Our attack, called Illusion, allows user-level malware to invoke privileged kernel operations without requiring the malware to call the actual system calls corresponding to the operations. The Illusion interface hides system operations from user-, kernel-, and hypervisor-level monitors mediating the conventional system-call interface. Illusion alters neither static kernel code nor read-only dispatch tables, remaining elusive from tools protecting kernel memory. We then consider the problem of Illusion attacks and augment system call data with kernel-level execution information to expose the hidden kernel operations. We present a Xen-based monitoring system, Sherlock, that adds kernel execution watchpoints to the stream of system calls. Sherlock automatically adapts its sensitivity based on security requirements to remain performant on desktop systems: in normal execution, it adds 1% to 10% overhead to a variety of workloads.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Abadi, M., Budiu, M., Erlingsson, U., Ligatti, J.: Control-flow integrity: Principles, implementations, and applications. In: 12th ACM Conference on Computer and Communications Security, CCS (2005)
Baliga, A., Kamat, P., Iftode, L.: Lurking in the shadows: Identifying systemic threats to kernel data. In: IEEE Symposium on Security and Privacy (May 2007)
Blorge. Faulty drivers bypass Vistas kernel protection, http://vista.blorge.com/2007/08/02/faulty-drivers-bypass-vistas-kernel-protection/ (last accessed 15 Jan 2011)
Chew, M., Song, D.: Mitigating buffer overflows by operating system randomization. In: Technical Report CMU-CS-02-197, Carnegie Mellon University, Pittsburg (December 2002)
David, F., Chan, E., Carlyle, J., Campbell, R.: Cloaker: hardware supported rootkit concealment. In: IEEE Symposium on Security and Privacy, Oakland, CA (May 2008)
Dinaburg, A., Royal, P., Sharif, M., Lee, W.: Ether: Malware analysis via hardware virtualization extensions. In: 15th ACM Conference on Computer and Communications Security, CCS (October 2008)
Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for UNIX processes. In: IEEE Symposium on Security and Privacy (May 1996)
Ganapathy, V., Jaeger, T., Jha, S.: Automatic placement of authorization hooks in the Linux security modules framework. In: 12th ACM Conference on Computer and Communications Security (CCS), Alexandria, Virginia (November 2005)
Giffin, J.T., Jha, S., Miller, B.P.: Efficient context-sensitive intrusion detection. In: Network and Distributed System Security Symposium (NDSS), San Diego, CA (February 2004)
Hofmeyr, S.A., Forrest, S., Somayaji, A.: Intrusion detection using sequences of system calls. Journal of Computer Security 6(3), 151–180 (1998)
Jiang, X., Wang, X.: “Out-of-the-box” monitoring of VM-based high-interaction honeypots. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 198–218. Springer, Heidelberg (2007)
Jones, S.T., Arpaci-Dusseau, A.C., Arpaci-Dusseau, R.H.: Antfarm: Tracking processes in a virtual machine environment. In: USENIX Annual Technical Conference (June 2006)
Kasslin, K.: Kernel malware: The attack from within. http://www.f-secure.com/weblog/archives/kasslin_AVAR2006_KernelMalware_paper.pdf (last accessed January 15, 2011)
Krohn, M., Yip, A., Brodsky, M., Cliffer, N., Kaashoek, M.F., Kohler, E., Morris, R.: Information flow control for standard OS abstractions. In: Symposium on Operating System Principles, SOSP (October 2007)
Kruegel, C., Kirda, E., Mutz, D., Robertson, W., Vigna, G.: Automating mimicry attacks using static binary analysis. In: USENIX Security Symposium, Baltimore, MD (August 2005)
Last, J. V.: Stuxnet versus the iranian nuclear program. http://www.sfexaminer.com/opinion/op-eds/2010/12/stuxnet-versusiranian-nuclear-program (last accessed January 15, 2011)
Linn, C.M., Rajagopalan, M., Baker, S., Collberg, C., Debray, S.K., Hartman, J.H.: Protecting against unexpected system calls. In: 14th USENIX Security Symposium (August 2005)
Martignoni, L., Christodorescu, M., Jha, S.: OmniUnpack: Fast, generic, and safe unpacking of malware. In: Annual Computer Security Applications Conference, ACSAC, Miami, FL (December 2007)
Martignoni, L., Stinson, E., Fredrikson, M., Jha, S., Mitchell, J.C.: A layered architecture for detecting malicious behaviors. In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 78–97. Springer, Heidelberg (2008)
Mavinakayanahalli, A., Panchamukhi, P., Keniston, J., Keshavamurthy, A., Hiramatsu, M.: Probing the guts of kprobes. In: Linux Symposium (July 2006)
McAfee Security. System call interception, http://www.crswann.com/3-NetworkSupport/SystemCall-IinterceptionMcAfee.pdf (last accessed January 15, 2011)
Mutz, D., Robertson, W., Vigna, G., Kemmerer, R.A.: Exploiting execution context for the detection of anomalous system calls. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 1–20. Springer, Heidelberg (2007)
Onoue, K., Oyama, Y., Yonezawa, A.: Control of system calls from outside of virtual machines. In: ACM Symposium on Applied Computing (March 2008)
packetstormsecurity. Adore rootkit, http://packetstormsecurity.org/files/view/29692/adore-0.42.tgz (last accessed January 15, 2011)
packetstormsecurity. Knark rootkit, http://packetstormsecurity.org/files/view/24853/knark-2.4.3.tgz (last accessed January 15, 2011)
Payne, B.D., Carbone, M., Sharif, M., Lee, W.: Lares: An architecture for secure active monitoring using virtualization. In: IEEE Symposium on Security and Privacy (May 2008)
PCNews. Verisign working to mitigate stuxnet digital signature theft, http://pcnews.uni.cc/verisign-working-to-mitigate-stuxnet-digital-signature-theft.html (last accessed January 15, 2011)
Petroni Jr., N.L., Fraser, T., Walters, A., Arbaugh, W.A.: An architecture for specification-based detection of semantic integrity violations in kernel dynamic data. In: 15th USENIX Security Symposium (August 2006)
Petroni Jr., N.L., Hicks, M.: Automated detection of persistent kernel control-flow attacks. In: ACM Conference on Computer and Communications Security, CCS (November 2007)
Provos, N.: Improving host security with system call policies. In: 12th USENIX Security Symposium (August 2003)
Riley, R., Jiang, X., Xu, D.: Guest-transparent prevention of kernel rootkits with VMM-based memory shadowing. In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 1–20. Springer, Heidelberg (2008)
Rootkit.com. Rootkit.com, http://www.rootkit.com/ (last accessed January 15, 2011)
Sekar, R., Bendre, M., Dhurjati, D., Bollineni, P.: A fast automaton-based method for detecting anomalous program behaviors. In: IEEE Symposium on Security and Privacy (May 2001)
Seshadri, A., Luk, M., Qu, N., Perrig, A.: SecVisor: A tiny hypervisor to provide lifetime kernel code integrity for commodity OSes. In: ACM Symposium on Operating Systems Principles, SOSP (October 2007)
Sharif, M., Singh, K., Giffin, J.T., Lee, W.: Understanding precision in host based intrusion detection. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 21–41. Springer, Heidelberg (2007)
Shinagawa, T., Eiraku, H., Tanimoto, K., Omote, K., Hasegawa, S., Horie, T., Hirano, M., Kourai, K., Oyama, Y., Kawai, E., Kono, K., Chiba, S., Shinjo, Y., Kato, K.: BitVisor: A thin hypervisor for enforcing I/O device security. In: ACM VEE, Washington, DC (March 2009)
Some Observations on Rootkits. Microsoft Malware Protection Center, http://blogs.technet.com/b/mmpc/archive/2010/01/07/some-observations-on-rootkits.aspx (last accessed January 15, 2011)
Srivastava, A., Giffin, J.: Automatic discovery of parasitic malware. In: Jha, S., Sommer, R., Kreibich, C. (eds.) RAID 2010. LNCS, vol. 6307, pp. 97–117. Springer, Heidelberg (2010)
Srivastava, A., Giffin, J.: Efficient monitoring of untrusted kernel-mode execution. In: NDSS, San Diego, California (February 2011)
Sun Microsystem. Dtrace, http://wikis.sun.com/display/DTrace/DTrace (last accessed January 15, 2011)
Tan, K.M.C., Killourhy, K.S., Maxion, R.A.: Undermining an anomaly-based intrusion detection system using common exploits. In: Wespi, A., Vigna, G., Deri, L. (eds.) RAID 2002. LNCS, vol. 2516, p. 54. Springer, Heidelberg (2002)
Tan, L., Zhang, X., Ma, X., Xiong, W., Zhou, Y.: AutoISES: Automatically inferring security specifications and detecting violations. In: USENIX Security Symposium (August 2008)
Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems. In: ACM CCS (November 2002)
Wang, Z., Jiang, X., Cui, W., Ning, P.: Countering kernel rootkits with lightweight hook protection. In: ACM CCS, Chicago, IL (November 2009)
Xu, H., Du, W., Chapin, S.J.: Context sensitive anomaly monitoring of process control flow to detect mimicry attacks and impossible paths. In: Jonsson, E., Valdes, A., Almgren, M. (eds.) RAID 2004. LNCS, vol. 3224, pp. 21–38. Springer, Heidelberg (2004)
Xu, M., Jiang, X., Sandhu, R., Zhang, X.: Towards a VMM-based usage control framework for OS kernel integrity protection. In: ACM SACMAT (June 2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Srivastava, A., Lanzi, A., Giffin, J., Balzarotti, D. (2011). Operating System Interface Obfuscation and the Revealing of Hidden Operations. In: Holz, T., Bos, H. (eds) Detection of Intrusions and Malware, and Vulnerability Assessment. DIMVA 2011. Lecture Notes in Computer Science, vol 6739. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-22424-9_13
Download citation
DOI: https://doi.org/10.1007/978-3-642-22424-9_13
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-22423-2
Online ISBN: 978-3-642-22424-9
eBook Packages: Computer ScienceComputer Science (R0)