Abstract
This paper presents the detection techniques of anomalous programs based on the analysis of their system call traces. We collect the API calls for the tested executable programs from Microsoft detour system and extract the features for our classification task using the previously established n-gram technique. We propose three different feature extraction approaches in this paper. These are frequency-based, time-based and a hybrid approach which actually combines the first two approaches. We use the well-known classifier algorithms in our experiments using WEKA interface to classify the malicious programs from the benign programs. Our empirical evidence demonstrates that the proposed feature extraction approaches can detect malicious programs over 88% which is quite promising for the contemporary similar research.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Cárdenas, A.A., Amin, S., Sastry, S.: Research Challenges for the Security of Control Systems. In: Proc. of the 3rd USENIX workshop on Hot Topics in Security, Associated with the 17th USENIX Security Symposium, San Jose, CA, USA (2008)
Protecting Critical Infrastructure SCADA Network Security Monitoring. In: Tenable Network Security Inc. (2008)
Andy, G.: America’s Hackable Backbone. Forbes (2007)
Ye, Y., Wang, D., Li, T., Ye, D., Jiang, Q.: An Intelligent PE-Malware Detection System Based on Association Mining. J. Comput. Virol. 4, 323–334 (2008)
Symantec Internet Security Threat Report Trends for 2008. vol. xiv (2009)
Forrest, S., Hofmeyr, S.A., Somayaji, A.: The Evolution of System-Call Monitoring. In: Proc. of the 24th Annual Computer Security Applications Conference, pp. 418–430 (2008)
Forrest, S., Nguyen, T.V., Weimer, W., Goues, C.L.: A Genetic Programming Approach to Automated Software Repair. In: Proc. of the Genetic and Evolutionary Computation Conference, pp. 947–954 (2009)
Kosoresow, A.P., Hofmeyr, S.A.: Intrusion Detection via System Call Traces. IEEE Software 14(5), 35–42 (1997)
Hofmeyr, S., Forrest, S., Somayaji, A.: Intrusion Detection Using Sequences of System Calls. Journal of Computer Security 6, 151–180 (1998)
Sekar, R., Uppuluri, P.: Synthesizing Fast Intrusion Prevention/Detection Systems from High-Level Specifications. In: Proc. of the 8th Conference on USENIX Security Symposium (1999)
Lane, T., Brodley, C.E.: Data Reduction Techniques for Instance-Based Learning of Human/Computer Interface Data. In: Proc. of the 17th International Conference on Machine Learning, pp. 519–526 (2000)
Endler, D.: Intrusion Detection Applying Machine Learning to Solaris Audit Data. In: Proc. of the 14th Annual Computer Security Applications Conference, p. 268 (1998)
Lee, T., Mody, J.J.: Behavioral Classification. In: Proc. of EICAR Conference (2006)
Bailey, M., Oberheide, J., Andersen, J., Mao, Z.M., Jahanian, F., Nazario, J.: Automated Classification and Analysis of Internet Malware. In: Proc. of 10th International Conference on Recent Advances in Intrusion Detection, pp. 178–197 (2007)
Jiang, X., Zhu, X.: vEye: Behavioral Footprinting for Self-Propagating Worm Detection and Profiling. Knowl. Inf. Syst. 18(2), 231–262 (2009)
Sung, A., Xu, J., Chavez, P., Mukkamala, S.: Static Analyzer of Vicious Executables (SAVE). In: Proc. of the 20th Annual Computer Security Application Conference, pp. 326–334 (2004)
Ye, Y., Wang, D., Li, T., Ye, D.: IMDS: Intelligent Malware Detection System. In: Proc. ACM Int. Conf. Knowl. Discovery and Data Mining, pp. 1043–1047 (2007)
Abou-Assaleh, T., Cercone, N., Keselj, V., Sweidan, R.: N-Gram-Based Detection of New Malicious Code. In: Proc. of the 28th Annual International Computer Software and Applications Conference-Workshops and Fast Abstracts, vol. 2, pp. 41–42. IEEE Computer Society, Washington, DC, USA (2004)
Ahmed, F., Hameed, H., Shafiq, M.Z., Farooq, M.: Using Spatio-Temporal Information in API Calls with Machine Learning Algorithms for Malware Detection. In: Proc. of the 2nd ACM Workshop on Security and Artificial Intelligence, pp. 55–62. ACM, New York (2009)
Moskovitch, R., Stopel, D., Feher, C., Nissim, N., Elovici, Y.: Unknown Malcode Detection via Text Categorization and the Imbalance Problem. In: IEEE International Conference on Intelligence and Security Informatics, pp. 156–161 (2008)
Park, Y., Reeves, D., Mulukutla, V., Sundaravel, B.: Fast Malware Classification by Automated Behavioral Graph Matching. In: Proc. of the 6th Annual Workshop on Cyber Security and Information Intelligence Research, pp. 1–4. ACM, New York (2010)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Rafiqul Islam, M., Saiful Islam, M., U. Chowdhury, M. (2011). Detecting Unknown Anomalous Program Behavior Using API System Calls. In: Abd Manaf, A., Sahibuddin, S., Ahmad, R., Mohd Daud, S., El-Qawasmeh, E. (eds) Informatics Engineering and Information Science. ICIEIS 2011. Communications in Computer and Information Science, vol 254. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-25483-3_31
Download citation
DOI: https://doi.org/10.1007/978-3-642-25483-3_31
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-25482-6
Online ISBN: 978-3-642-25483-3
eBook Packages: Computer ScienceComputer Science (R0)