RootkitDet: Practical End-to-End Defense against Kernel Rootkits in a Cloud Environment

  • Lingchen Zhang
  • Sachin Shetty
  • Peng Liu
  • Jiwu Jing
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8713)


In cloud environments, kernel-level rootkits still pose serious security threats to guest OSes. Existing defenses against kernel-level rootkit have limitations when applied to cloud environments. In this paper, we propose RootkitDet, an end-to-end defense system capable of detecting and diagnosing rootkits in guest OSes with the intent to recover the system modifications caused by the rootkits in cloud environments. RootkitDet detects rootkits by identifying suspicious code region in the kernel space of guest OSes through the underneath hypervisor, performs diagnosis on the code of the detected rootkit to categorize it and identify modifications, and reverses the modifications if possible to eliminate the effect of rootkits. Our evaluation results show that the RootkitDet is effective on detection of kernel-level rootkits and recovery modifications with less than 1% performance overhead to the guest OSes and the computation and network overhead is linear with the quantity of the VM instances being monitored.


Hypervisor VM Kernel-level rootkit Defense Cloud 


  1. 1.
    McAfee: Rootkits, Part 1 of 3: A Growing Threat. white paper (April 2006)Google Scholar
  2. 2.
    McAfee: 2010 Threat Predictions. white paper, McAfee AVERT Labs (December 2009)Google Scholar
  3. 3.
    Baliga, A., Ganapathy, V., Iftode, L.: Detecting kernel-level rootkits using data structure invariants. IEEE Transactions on Dependable and Secure Computing 8(5), 670–684 (2011)CrossRefGoogle Scholar
  4. 4.
    Petroni Jr., N.L., Fraser, T., Walters, A., Arbaugh, W.A.: An architecture for specification-based detection of semantic integrity violations in kernel dynamic data. In: Proceedings of the 15th USENIX Security Symposium, pp. 289–304 (2006)Google Scholar
  5. 5.
    Petroni Jr., N.L., Hicks, M.: Automated detection of persistent kernel control-flow attacks. In: Proceedings of the 14th ACM Conference on Computer and Communications Security, pp. 103–115. ACM (2007)Google Scholar
  6. 6.
    Wang, Z., Jiang, X., Cui, W., Ning, P.: Countering kernel rootkits with lightweight hook protection. In: Proceedings of the 16th ACM Conference on Computer and Communications Security, pp. 545–554. ACM (2009)Google Scholar
  7. 7.
    Kruegel, C., Robertson, W., Vigna, G.: Detecting kernel-level rootkits through binary analysis. In: 20th Annual Computer Security Applications Conference 2004, pp. 91–100. IEEE (2004)Google Scholar
  8. 8.
    Seshadri, A., Luk, M., Qu, N., Perrig, A.: Secvisor: a tiny hypervisor to provide lifetime kernel code integrity for commodity oses. In: ACM SIGOPS Operating Systems Review, vol. 41, pp. 335–350. ACM (2007)Google Scholar
  9. 9.
    Riley, R., Jiang, X., Xu, D.: Guest-transparent prevention of kernel rootkits with vmm-based memory shadowing. In: Lippmann, R., Kirda, E., Trachtenberg, A. (eds.) RAID 2008. LNCS, vol. 5230, pp. 1–20. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  10. 10.
    Payne, B.D., Carbone, M., Sharif, M., Lee, W.: Lares: An architecture for secure active monitoring using virtualization. In: IEEE Symposium on Security and Privacy, SP 2008, pp. 233–247. IEEE (2008)Google Scholar
  11. 11.
    Jiang, X., Wang, X., Xu, D.: Stealthy malware detection through vmm-based out-of-the-box semantic view reconstruction. In: Proceedings of the 14th ACM Conference on Computer and Communications Security, pp. 128–138. ACM (2007)Google Scholar
  12. 12.
    Fraser, T., Evenson, M.R., Arbaugh, W.A.: Vici-virtual machine introspection for cognitive immunity. In: Annual Computer Security Applications Conference, ACSAC 2008, pp. 87–96. IEEE (2008)Google Scholar
  13. 13.
    Kemerlis, V.P., Portokalidis, G., Keromytis, A.D.: kguard: lightweight kernel protection against return-to-user attacks. In: USENIX Security Symposium (2012)Google Scholar
  14. 14.
  15. 15.
    Litty, L., Lagar-Cavilla, H.A., Lie, D.: Hypervisor support for identifying covertly executing binaries. In: Proceedings of the 17th Conference on Security Symposium, pp. 243–258 (2008)Google Scholar
  16. 16.
    Wagner, D., Dean, D.: Intrusion detection via static analysis. In: Proceedings of the 2001 IEEE Symposium on Security and Privacy, S&P 2001, pp. 156–168. IEEE (2001)Google Scholar
  17. 17.
    Baliga, A., Kamat, P., Iftode, L.: Lurking in the shadows: Identifying systemic threats to kernel data. In: IEEE Symposium on Security and Privacy, SP 2007, pp. 246–251. IEEE (2007)Google Scholar
  18. 18.
    Stealth: Announcing full functional adore-ng rootkit for 2.6 kernel,
  19. 19.
  20. 20.
    Halflife: Abuse of the Linux-kernel for Fun and Profit. Phrack Magazine 5(50) (April 1997)Google Scholar
  21. 21.
    Garfinkel, T., Rosenblum, M.: A virtual machine introspection based architecture for intrusion detection. In: Proc. Network and Distributed Systems Security Symposium (2003)Google Scholar
  22. 22.
    Hund, R., Holz, T., Freiling, F.C.: Return-oriented rootkits: Bypassing kernel code integrity protection mechanisms. In: Proceedings of the 18th USENIX Security Symposium, pp. 383–398 (2009)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Lingchen Zhang
    • 1
    • 2
    • 4
  • Sachin Shetty
    • 2
  • Peng Liu
    • 3
  • Jiwu Jing
    • 1
  1. 1.State Key Laboratory of Information Security,Institute of Information EngineeringChinese Academy of SciencesChina
  2. 2.College of EngineeringTennessee State UniversityUSA
  3. 3.College of ISTPenn State UniversityUSA
  4. 4.University of Chinese Academy of SciencesChina

Personalised recommendations