Abstract
Modern computing environments depend on extensive shared libraries. In this paper, we propose monitoring the calls between those libraries as a new source of data for host-based anomaly detection. That is, we characterize an application by its use of shared library functions and characterize each shared library function by its use of (lower-level) shared libraries. This approach to intrusion detection offers significant benefits, especially in systems such as Windows, much of which is implemented above the kernel as dynamically linked libraries (DLLs). It localizes anomalies to particular code modules, facilitating anomaly analysis and assessment and discouraging mimicry attacks. It reduces retraining after system updates and enables training concurrent with detection. The proposed approach can be used with various techniques for modeling call sequences, including N-grams, automata, and techniques that consider parameter values. To demonstrate its potential, we have studied how a DLL-level profiling IDS would detect two recent attacks on Windows systems.
Chapter PDF
Similar content being viewed by others
References
Wagner, D., Soto, P.: Mimicry Attacks on Host Based Intrusion Detection Systems. In: Proceedings of the Ninth ACM Conference on Computer and Communications Security (2002)
Allison, K.: Windows PCs Face ‘Huge’ Virus Threat. In: Financial Times (January 2, 2006)
Microsoft (TM), Microsoft Security Bulletin Ms05-053: Vulnerabilities in Graphics Rendering Engine Could Allow Code Execution (896424), http://www.microsoft.com/technet/security/bulletin/MS05-053.mspx
Forrest, S., Hofmeyr, S.A., Somajayi, A.: A Sense of Self for UNIX Processes. In: Proceedings of the IEEE Symposium on Computer Security and Privacy. IEEE Press, Los Alamitos (1996)
Hofmeyr, S.A., Forrest, S., Somayaji, A.: Intrusion Detection Using Sequences of System Calls. Journal of Computer Security 6(3), 151–180 (1998)
Debar, H., et al.: Fixed vs. Variable-Length Patterns for Detecting Suspicious Process Behavior. In: Quisquater, J.-J., Deswarte, Y., Meadows, C., Gollmann, D. (eds.) ESORICS 1998. LNCS, vol. 1485, pp. 1–15. Springer, Heidelberg (1998)
Warrender, C., Forrest, S., Pearlmutter, B.: Detecting Intrusions Using System Calls: Alternative Data Models. In: Proceedings of the IEEE Symposium on Security and Privacy, pp. 133–145 (1999)
Ghosh, A.K., Schwatzbard, A., Shatz, M.: Learning Program Behavior Profiles for Intrusion Detection. In: Proceedings of the 1st USENIX Workshop on Intrusion Detection and Network Monitoring, Santa Clara, California (1999)
Marceau, C.: Characterizing the Behavior of a Program Using Multiple-Length N-Grams. In: Proceedings of the New Security Paradigms Workshop, Ballycotton, Ireland (2000)
Pfleger, K.: On-Line Cumulative Learning of Hierarchical Sparse N-Grams. In: Proceedings of the International Conference on Development and Learning (2004)
Michael, C.C., Ghosh, A.: Simple, State-Based Approaches to Program-Based Intrusion Detection. ACM Transactions on Information and System Security 5(3), 203–237 (2002)
Balzer, R., Goldman, N.: Mediating Connectors. In: Proceedings of the ICDCS Workshop on Electronic Commerce and Web-Based Applications, Austin, TX, pp. 73–77 (1999)
Balzer, R., Goldman, N.: Mediating Connectors: A Non-Bypassable Process Wrapping Technology. In: Proceedings of the 19th IEEE International Conference on Distributed Computing Systems (1999)
Feng, H., et al.: Anomaly Detection Using Call Stack Information. In: Proceedings of the IEEE Security and Privacy, Oakland, CA, USA (2003)
Sekar, R., et al.: A Fast Automaton-Based Method for Detecting Anomalous Program Behaviors. In: Proceedings of the IEEE Symposium on Security and Privacy, Oakland, CA, pp. 144–155 (2001)
Arcata Pet, Imgviewer/32, http://www.arcatapet.net/imgv32.cfm
Ries, C.: Analysis of a Malicious JPEG Attack, http://www.vigilantminds.com/files/jpeg_attack_wp.pdf
French Security Incident Response Team (FSIRT), Windows JPEG GDI+ Overflow Administrator Exploit (Ms04-028), http://www.frsirt.com/exploits/09232004.ms04-28-admin.sh.php
Microsoft (TM) TechNet, Microsoft Security Bulletin Ms06-001: Vulnerability in Graphics Rendering Engine Could Allow Remote Code Execution (912919), http://www.microsoft.com/technet/security/bulletin/MS06-001.mspx
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Marceau, C., Stillerman, M. (2006). Modular Behavior Profiles in Systems with Shared Libraries (Short Paper). In: Ning, P., Qing, S., Li, N. (eds) Information and Communications Security. ICICS 2006. Lecture Notes in Computer Science, vol 4307. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11935308_26
Download citation
DOI: https://doi.org/10.1007/11935308_26
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-49496-6
Online ISBN: 978-3-540-49497-3
eBook Packages: Computer ScienceComputer Science (R0)