Abstract
Malware attacks which focus on exploiting an application to launch the payload have become major security threat. We present the methodology and algorithm which is able to detect anomaly in application behavior and prevent such type of attacks. Our approach is to represent the normal behavior of an application, detect deviations from this normal behavior and prevent them. We represent normal behavior using system calls made over critical resources by clustering of these system calls and then monitor the behavior of applications for any deviations from the normal behavior, by means of an enforcement algorithm. Any mismatch from the normal behavior indicates an anomaly. We provide a description of our approach. We have implemented and tested the proposed approach and the results are encouraging. As compared to previous research in this direction, we implement on Windows OS instead of Linux OS and use minifilter and registry callback techniques instead of raw system call interception which is prohibited in latest operating system versions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Idike, N., Mathur, A.P.: A Survey of Malware Detection Techniques. Technical Report, Purdue University (2007)
Warrender, C., Forrest, S., Pearlmutter, B.: Detecting Intrusion Using System Calls: Alternative Data Models. In: IEEE Computer Society Symposium on Research in Security and Privacy (1998)
Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A Sense of Self for UNIX Processes. In: IEEE Symposium on Security and Privacy (1996)
Hofmeyr, S.A., Forrest, S., Somayaji, A.: Intrusion Detection using Sequences of System Calls. Journal of Computer Security (1998)
Mazeroff, G., Cerqueira, V.D., Gregor, J., Thomason, M.G.: Probabilistic Trees and Automata for Application Behavior Modeling. In: ACM Southeast Regional Conference Proceedings (2003)
Parampalli, C., Sekar, R., Johnson, R.: A Practical Mimicry Attack Against Powerful System-Call Monitors. In: ACM Symposium on Information, Computer and Information Security (2008)
Sekar, R., Cai, Y., Segal, M.: A Specification-Based Approach for Building Survivable Systems. In: Proceedings of the NISSC (1998)
Sekar, R.: On Preventing Intrusions by Process Behavior Monitoring. In: USENIX Intrusion Detection Workshop (1999)
Sekar, R., Venkatakrishnan, V.N., Basu, S., Bhatkar, S., Daniel, DuVarney, C.: Model-Carrying Code: A Practical Approach for Safe Execution of Untrusted Applications. In: ACM Symposium on Operating system principles (2003)
Sekar, R., Bendre, M., Dhurjati, D., Bollineni, P.: A Fast Automaton-Based Method for Detecting Anomalous Program Behaviors. IEEE Symposium on Security and Privacy (2001)
Seifert, C., Steenson, R., Welch, I., Komisarczuk, P., Endicott-Popovsky, B.: Capture A behavioral analysis tool for applications and documents. In: Digital Forensics Research Conference (2007)
Rieck, K., Holz, T., Willems, C., Dussel, P., Laskov, P.: Learning and Classification of Malware Behavior. In: Conference on Detection of Intrusions and Malware and Vulnerability Assessment (2008)
Wagner, D., Soto, P.: Mimicry Attacks on Host Based Intrusion Detection Systems. In: ACM Conference on Computer and Communication Security (2002)
Xperf: http://msdn.microsoft.com/en-us/performance/cc825801.aspx
Solomon, R.: Windows Internals: Kernel Patch Protection. Microsoft Press, Redmond (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Jyostna, G., Himanshu, P., Eswari, P.R.L. (2011). Detecting Anomalous Application Behaviors Using a System Call Clustering Method over Critical Resources. In: Wyld, D.C., Wozniak, M., Chaki, N., Meghanathan, N., Nagamalai, D. (eds) Advances in Network Security and Applications. CNSA 2011. Communications in Computer and Information Science, vol 196. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-22540-6_6
Download citation
DOI: https://doi.org/10.1007/978-3-642-22540-6_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-22539-0
Online ISBN: 978-3-642-22540-6
eBook Packages: Computer ScienceComputer Science (R0)