Abstract
We present a new approach for malware clustering in the domain of their behavior. To this end, we use a system called tLab that offers analysis and detection of modern complex malware including user-oriented and targeted attacks. Due to technologies used, tLab identifies and describes malware behavior at various levels of semantics, which makes it very instrumental in cluster analysis.
Technically, the system employs secure containers enabling user-dependent execution environment in malicious activity analysis. To provide effective malware detection, tLab has a technology for deep dynamic inspection of system-wide behavior, which allows for structural analysis and construction of so-called activity trees defined in the domain of system functionalities. Modified Hierarchical Colored Petri Nets are used for run-time recognition of system functionalities including obfuscated and distributed ones.
In this paper, we perform cluster analysis at the level of activity trees, which provide highly semantic representation of malicious behavior. Our clustering approach is evaluated with corpus of real malware families. Results demonstrated that the used activity tree domain enables excellent behavior clustering and provides better and more consistent results compared to antiviruses vendors.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Malware Statistics Report by AV-Test Institute. https://www.av-test.org/en/statistics/malware/
Cohen, F.: Computer Viruses Theory and Experiments, Computers and Security, v. 6 (1987)
The Increased Use of PowerShell in Attacks. Whitepaper by Semantic Corporation (2016). https://www.symantec.com/content/dam/symantec/docs/security-center/white-papers/
Tokhtabayev, A.G., Skormin, V.A., Dolgikh, A.M.: Expressive, efficient and obfuscation resilient behavior based IDS. In: Gritzalis, D., Preneel, B., Theoharidou, M. (eds.) ESORICS 2010. LNCS, vol. 6345, pp. 698–715. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15497-3_42
tLab (Version 1.5) [computer software], T&T Security LLP, Astana, Kazakhstan (2017)
Tokhtabayev, A., Skormin, V., Dolgikh, A.: Detection of worm propagation engines in the system call domain using colored petri nets. In: Proceedings of the IEEE IPCCC ’07, USA, December 2008
Jensen, K.: Coloured Petri nets (2nd ed.): basic concepts, analysis methods and practical use, vol. 1, Springer, Berlin (1996)
Zhang, K., Shasha, D.: Simple fast algorithms for the editing distance between trees and related problems. SIAM J. Comput. 18(6), 1245–1262 (1989)
Pawlik, M., Augsten, N.: RTED: a robust algorithm for the tree edit distance. Proc. VLDB Endow. 5(4), 334–345 (2011)
Bailey, M., Oberheide, J., Andersen, J., Morley Mao, Z., Jahanian, F., Nazario, J.: Automated Classification and Analysis of Internet Malware (2007)
Bayer, U., Comparetti, P.M., Hlauschek, C., Kruegel, C., Kirda, E.: Scalable, behavior-based malware clustering. In: NDSS (2009)
Egele, M., Scholte, T., Kirda, E., Kruegel, C.: A survey on automated dynamic malware-analysis techniques and tools. ACM Comput. Surv. 44(2), 6:1–6:42 (2008)
Falliere, N., Murchu, L.O., Chien, E.: W32.stuxnet dossier (2011). www.symantec.com White paper 2011
Gusfield, D.: Algorithms on Strings, Trees, and Sequences - Computer Science and Computational Biology. Cambridge University Press (1997)
Reversal and Analysis of Zeus and SpyEye Banking Trojans. Technical report, IOActive (2012)
Jacob, G., Debar, H., Filiol, E.: Behavioral detection of malware: from a survey towards an established taxonomy. J. Comput. Virol. 4, 251–266 (2008)
Jang, J., Brumley, D., Venkataraman, S.: Bitshred: feature hashing malware for scalable triage and semantic analysis. In: Proceedings of the 18th ACM Conference on Computer and Communications Security, pp. 309–320. ACM (2011)
The flame: Questions and answers, May 2012. www.securelist.com
New malware classification system. www.securelist.com. Accessed June 2012
Rules for naming detected objects. www.securelist.com. Accessed 2012
Kirillov, I., Beck, D., Chase, P., Martin, R.: Malware attribute enumeration and characterization, MITRE (2011)
Langfelder, P., Zhang, B., Horvath, S.: Defining clusters from a hierarchical cluster tree: the dynamic tree cut package for r. Bioinformatics 24(5), 719–720 (2008)
Li, P., Liu, L., Gao, D., Reiter, M.K.: On challenges in evaluating malware clustering. In: Jha, S., Sommer, R., Kreibich, C. (eds.) RAID 2010. LNCS, vol. 6307, pp. 238–255. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15512-3_13
Rieck, K., Trinius, P., Willems, C., Holz, T.: Automatic analysis of malware behavior using machine learning. J. Comput. Secur. 19(4), 639–668 (2011)
RSA. The Current State of Cybercrime and What to Expect in 2012. Technical report, RSA (2012)
Trinius, P., Holz, T., Gobel, J., Freiling, F.C.: Visual analysis of malware behavior using tree maps and thread graphs. In: 2009 6th International Workshop on Visualization for Cyber Security, pp. 33–38 (2009)
Ukkonen, E.: Constructing suffix trees on-line in linear time. In: IFIP Congress, pp. 484–492 (1992)
Wagener, G., State, R., Dulaunoy, A.: Malware behaviour analysis. J. Comput. Virology 4(4), 279–287 (2007)
Ye, Y., Li, T., Chen, Y., Jiang, Y.: Automatic malware categorization using cluster ensemble. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and data mining, KDD 2010, pp. 95–104. ACM, New York (2010)
Acknowledgment
The tLab system was fully funded and developed by T&T Security LLP. The malware behavioral clustering research effort is funded by T&T Security LLP and is partially supported by scientific projects of L.N. Gumilyov Eurasian National University, Kazakhstan managed by the authors of this paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Kopeikin, A., Tokhtabayev, A., Tashatov, N., Satybaldina, D. (2017). tLab: A System Enabling Malware Clustering Based on Suspicious Activity Trees. In: Rak, J., Bay, J., Kotenko, I., Popyack, L., Skormin, V., Szczypiorski, K. (eds) Computer Network Security. MMM-ACNS 2017. Lecture Notes in Computer Science(), vol 10446. Springer, Cham. https://doi.org/10.1007/978-3-319-65127-9_16
Download citation
DOI: https://doi.org/10.1007/978-3-319-65127-9_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-65126-2
Online ISBN: 978-3-319-65127-9
eBook Packages: Computer ScienceComputer Science (R0)