A Three-Way Decision Making Approach to Malware Analysis

  • Mohammad Nauman
  • Nouman AzamEmail author
  • JingTao Yao
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9436)


Malware analysis techniques generally classify software behaviors as malicious (i.e., harmful) or benign (i.e., not harmful). Due to ambiguous nature of application behavior, there are cases where it may not be possible to confidently reach two-way conclusions. This may result in higher classification errors which in turn affect users trust on malware analysis outcomes. In this paper, we investigate a three-way decision making approach based on probabilistic rough set models, such as, information-theoretic rough sets and game-theoretic rough sets, for malware analysis. The essential idea is to add a third option of deferment or delaying a decision whenever the available information is not sufficient to reach certain conclusions. We demonstrate the applicability of the proposed approach with an example from system call sequences of a vulnerable Linux application.



This work was partially supported by a discovery grant from NSERC canada.


  1. 1.
    Azam, N., Yao, J.T.: Analyzing uncertainties of probabilistic rough set regions with game-theoretic rough sets. Int. J. Approx. Reasoning 55(1), 142–155 (2014)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Biddle, R., van Oorschot, P.C., Patrick, A.S., Sobey, J., Whalen, T.: Browser interfaces and extended validation ssl certificates: an empirical study. In: Proceedings of the 2009 ACM Workshop on Cloud Computing Security, pp. 19–30 (2009)Google Scholar
  3. 3.
    Demchenko, Y., Ngo, C., de Laat, C., Membrey, P., Gordijenko, D.: Big security for big data: addressing security challenges for the big data infrastructure. In: Jonker, W., Petković, M. (eds.) SDM 2013. LNCS, vol. 8425, pp. 76–94. Springer, Heidelberg (2014) CrossRefGoogle Scholar
  4. 4.
    Deng, X., Yao, Y.: An information-theoretic interpretation of thresholds in probabilistic rough sets. In: Li, T., Nguyen, H.S., Wang, G., Grzymala-Busse, J., Janicki, R., Hassanien, A.E., Yu, H. (eds.) RSKT 2012. LNCS, vol. 7414, pp. 369–378. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  5. 5.
    Egele, M., Scholte, T., Kirda, E., Kruegel, C.: A survey on automated dynamic malware-analysis techniques and tools. ACM Comput. Surv. 44(2), 6 (2012)CrossRefGoogle Scholar
  6. 6.
    Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: UNM dataset. Accessed 6 April 2015
  7. 7.
    Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for unix processes. In: IEEE Symposium on Security and Privacy, pp. 120–128 (1996)Google Scholar
  8. 8.
    Gandotra, E., Bansal, D., Sofat, S.: Malware analysis and classification: a survey. J. Inf. Secur. 5, 56–64 (2014)Google Scholar
  9. 9.
    Herbert, J.P., Yao, J.T.: Game-theoretic rough sets. Fundamenta Informaticae 108(3–4), 267–286 (2011)MathSciNetzbMATHGoogle Scholar
  10. 10.
    Leyton-Brown, K., Shoham, Y.: Essentials of Game Theory: A Concise Multidisciplinary Introduction. Morgan & Claypool Publishers, San Rafael (2008) zbMATHGoogle Scholar
  11. 11.
    Liu, D., Yao, Y.Y., Li, T.R.: Three-way investment decisions with decision-theoretic rough sets. Int. J. Comput. Intel. Syst. 4(1), 66–74 (2011)CrossRefGoogle Scholar
  12. 12.
    Mehdi, B., Ahmed, F., Khayyam, S.A., Farooq, M.: Towards a theory of generalizing system call representation for in-execution malware detection. In: IEEE International Conference on Communications (ICC), pp. 1–5 (2010)Google Scholar
  13. 13.
    Pawlak, Z.: Rough sets. Int. J. Comput. Inf. Sci. 11, 241–256 (1982)CrossRefGoogle Scholar
  14. 14.
    Symantec: Internet security threat report, vol. 19. Accessed 12 April 2015
  15. 15.
    Tanenbaum, A.S., Woodhull, A.S.: Operating systems: design and implementation, vol. 2. Prentice-Hall, Englewood Cliffs (1987) Google Scholar
  16. 16.
    Wright, C., Cowan, C., Morris, J., Smalley, S., Kroah-Hartman, G.: Linux security module framework. In: Ottawa Linux Symposium, vol. 8032, pp. 6–16 (2002)Google Scholar
  17. 17.
    Yang, H., Li, T., Hu, X., Wang, F., Zou, Y.: A survey of artificial immune system based intrusion detection. The Scientific World Journal (2014)Google Scholar
  18. 18.
    Yao, Y.Y.: Probabilistic rough set approximations. Int. J. Approx. Reasoning 49(2), 255–271 (2008)CrossRefGoogle Scholar
  19. 19.
    Yao, Y.Y.: Three-way decisions with probabilistic rough sets. Inf. Sci. 180(3), 341–353 (2010)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Yao, Y.Y.: Two semantic issues in a probabilistic rough set model. Fundamenta Informaticae 108(3–4), 249–265 (2011)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 2.5 International License (, which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Authors and Affiliations

  1. 1.National University of Computer and Emerging SciencesPeshawarPakistan
  2. 2.Department of Computer ScienceUniversity of ReginaReginaCanada

Personalised recommendations