Applied Intelligence

, Volume 49, Issue 7, pp 2641–2658 | Cite as

RAMD: registry-based anomaly malware detection using one-class ensemble classifiers

  • Asghar Tajoddin
  • Mahdi AbadiEmail author


Malware is continuously evolving and becoming more sophisticated to avoid detection. Traditionally, the Windows operating system has been the most popular target for malware writers because of its dominance in the market of desktop operating systems. However, despite a large volume of new Windows malware samples that are collected daily, there is relatively little research focusing on Windows malware. The Windows Registry, or simply the registry, is very heavily used by programs in Windows, making it a good source for detecting malicious behavior. In this paper, we present RAMD, a novel approach that uses an ensemble classifier consisting of multiple one-class classifiers to detect known and especially unknown malware abusing registry keys and values for malicious intent. RAMD builds a model of registry behavior of benign programs and then uses this model to detect malware by looking for anomalous registry accesses. In detail, it constructs an initial ensemble classifier by training multiple one-class classifiers and then applies a novel swarm intelligence pruning algorithm, called memetic firefly-based ensemble classifier pruning (MFECP), on the ensemble classifier to reduce its size by selecting only a subset of one-class classifiers that are highly accurate and have diversity in their outputs. To combine the outputs of one-class classifiers in the pruned ensemble classifier, RAMD uses a specific aggregation operator, called Fibonacci-based superincreasing ordered weighted averaging (FSOWA). The results of our experiments performed on a dataset of benign and malware samples show that RAMD can achieve about 98.52% detection rate, 2.19% false alarm rate, and 98.43% accuracy.


Windows malware Registry-based malware detection Ensemble classifier One-class classification Pruning algorithm Memetic firefly algorithm Aggregation operator Superincreasing ordered weighted averaging 



  1. 1.
    Abbas H, Yasin M, Ahmed F, Sajid A, Khan FA, Ashfaq RAR, Haldar NAH (2016) Forensic artifacts modeling for social media client applications to enhance investigatory learning mechanisms. J Intell Fuzzy Syst 31(5):2645–2658. Google Scholar
  2. 2.
    Alazab M (2015) Profiling and classifying the behavior of malicious codes. J Syst Softw 100:91–102. Google Scholar
  3. 3.
    Apap F, Honig A, Hershkop S, Eskin E, Stolfo SJ (2002) Detecting malicious software by monitoring anomalous Windows Registry accesses. In: Proceedings of the 5th International Symposium on Recent Advances in Intrusion Detection (RAID’02), pp 36-53. Springer, Berlin
  4. 4.
  5. 5.
    Brown G, Wyatt J, Harris R, Yao X (2005) Diversity creation methods: a survey and categorisation. Inf Fusion 6(1):5–20. Google Scholar
  6. 6.
    Carvey H (2016) Windows Registry Forensics: Advanced Digital Forensic Analysis of the Windows Registry, 2nd edn. Syngress, AmsterdamGoogle Scholar
  7. 7.
    Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: a survey. ACM Comput Surv 41(3):15:1–15:58. Google Scholar
  8. 8.
    Christodorescu M, Jha S (2003) Static analysis of executables to detect malicious patterns. In: Proceedings of the 12th USENIX Security Symposium (Security’03), pp 169-186, USENIX Association, Berkeley, CA, USAGoogle Scholar
  9. 9.
    Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30MathSciNetzbMATHGoogle Scholar
  10. 10.
    Ding Y, Xia X, Chen S, Li Y (2018) A malware detection method based on family behavior graph. Comput Secur 73:73–86. Google Scholar
  11. 11.
    Ding Y, Yuan X, Tang K, Xiao X, Zhang Y (2013) A fast malware detection algorithm based on objective-oriented association mining. Comput Secur 39:315–324. Google Scholar
  12. 12.
    Duin RPW, Tax DMJ (2000) Experiments with classifier combining rules. In: Proceedings of the 1st International Workshop on Multiple Classifier Systems (MCS’00). Springer, Berlin, pp 16–29
  13. 13.
    Eskin E (2002) Probabilistic anomaly detection over discrete records using inconsistency checks. Technical report, Department of Computer Science Columbia UniversityGoogle Scholar
  14. 14.
    Fattori A, Lanzi A, Balzarotti D, Kirda E (2015) Hypervisor-based malware protection with AccessMiner. Comput Secur 52:33–50. Google Scholar
  15. 15.
    Galal HS, Mahdy YB, Atiea MA (2016) Behavior-based features model for malware detection. J Comput Virol Hacking Techniques 12(2):59–67. Google Scholar
  16. 16.
    Gautam C, Tiwari A, Leng Q (2017) On the construction of extreme learning machine for online and offline one-class classification–an expanded toolbox. Neurocomputing 261:126–143. Google Scholar
  17. 17.
    Ghaffari F, Abadi M, Tajoddin A (2017) AMD-EC: anomaly-based android malware detection using ensemble classifiers. In: Proceedings of the 2017 25th Iranian Conference on Electrical Engineering (ICEE’17), pp 2247-2252. IEEE, Piscataway
  18. 18.
    Guo X, Yin Y, Dong C, Yang G, Zhou G (2008) On the class imbalance problem. In: Proceedings of the 2008 4th International Conference on Natural Computation (ICNC’08), pp 192-201. IEEE, Piscataway
  19. 19.
    Gupta S, Kumar P (2015) An immediate system call sequence based approach for detecting malicious program executions in cloud environment. Wirel Pers Commun 81(1):405–425. Google Scholar
  20. 20.
    Halsey M, Bettany A (2015) Windows Registry troubleshooting. Apress, New York. Google Scholar
  21. 21.
    Heller KA, Svore KM, Keromytis AD, Stolfo SJ (2003) One class support vector machines for detecting anomalous Windows Registry accesses. In: Proceedings of the 2003 ICDM Workshop on Data Mining for Computer Security (DMSEC’03), pp 1–8.
  22. 22.
    Ho TK (1998) The random subspace method for constructing decision forests. IEEE Trans Pattern Anal Mach Intell 20(8):832–844. Google Scholar
  23. 23.
    Hollander M, Wolfe DA, Chicken E (2014) Nonparametric statistical methods, 3rd edn. Wiley, HobokenzbMATHGoogle Scholar
  24. 24.
    Hosseini Bamakan SM, Wang H, Shi Y (2017) Ramp loss K-support vector classification-regression: a robust and sparse multi-class approach to the intrusion detection problem. Knowl-Based Syst 126:113–126. Google Scholar
  25. 25.
    Jodavi M, Abadi M (2015) JSObfusDetector: a binary PSO-based one-class classifier ensemble to detect obfuscated JavaScript code. In: Proceedings of the 2015 International Symposium on Artificial Intelligence and Signal Processing (AISP’15), pp 322-327. IEEE, Piscataway
  26. 26.
    Jodavi M, Abadi M, Parhizkar E (2015) DbDHunter: an ensemble-based anomaly detection approach to detect drive-by download attacks. In: Proceedings of the 2015 5th International Conference on Computer and Knowledge Engineering (ICCKE’15), pp 273-278. IEEE, Piscataway
  27. 27.
    Juszczak P, Tax DMJ, Pekalska E, Duin RPW (2009) Minimum spanning tree based one-class classifier. Neurocomputing 72(7–9):1859–1869. Google Scholar
  28. 28.
    Karaboga D, Gorkemli B, Ozturk C, Karaboga N (2014) A comprehensive survey: artificial bee colony (ABC) algorithm and applications. Artif Intell Rev 42(1):21–57. Google Scholar
  29. 29.
    Kazem A, Sharifi E, Hussain FK, Saberi M, Hussain OK (2013) Support vector regression with chaos-based firefly algorithm for stock market price forecasting. Appl Soft Comput 13(2):947–958. Google Scholar
  30. 30.
    Khan SS, Madden MG (2014) One-class classification: taxonomy of study and review of techniques. Knowl Eng Rev 29(3):345–374. Google Scholar
  31. 31.
    Khatri Y (2015) Forensic implications of System Resource Usage Monitor (SRUM) data in Windows 8. Digit Investig 12:53–65. Google Scholar
  32. 32.
    Khreich W, Murtaza SS, Hamou-Lhadj A, Talhi C (2018) Combining heterogeneous anomaly detectors for improved software security. J Syst Softw 137:415–429. Google Scholar
  33. 33.
    Kirat D, Vigna G (2015) MalGene: Automatic extraction of malware analysis evasion signature. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS’15), pp 769-780. ACM, New York
  34. 34.
    Kirat D, Vigna G, Kruegel C (2014) BareCloud: bare-metal analysis-based evasive malware detection. In: Proceedings of the 23rd USENIX Security Symposium (Security’14), pp 287-301, USENIX Association, Berkeley, CA, USAGoogle Scholar
  35. 35.
    Kolbitsch C, Comparetti PM, Kruegel C, Kirda E, Zhou X, Wang X (2009) Effective and efficient malware detection at the end host. In: Proceedings of the 18th USENIX Security Symposium (Security’09), pp 351-366, USENIX Association, Berkeley, CA, USAGoogle Scholar
  36. 36.
    Kramer O (2017) Genetic algorithm essentials. Springer international publishing. Cham, Switzerland. Google Scholar
  37. 37.
    Krawczyk B, Woźniak M (2016) Dynamic classifier selection for one-class classification. Knowl-Based Syst 107:43–53. Google Scholar
  38. 38.
    Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51(2):181–207. zbMATHGoogle Scholar
  39. 39.
    Lei B, Xu G, Feng M, Zou Y, van der Heijden F, de Ridder D, Tax DMJ (2017) Classification, parameter estimation and state estimation: an engineering approach using MATLAB, 2nd edn. Wiley, HobokenGoogle Scholar
  40. 40.
    Liu J, Miao Q, Sun Y, Song J, Quan Y (2016) Fast structural ensemble for one-class classification. Pattern Recogn Lett 80:179–187. Google Scholar
  41. 41.
    Long NC, Meesad P, Unger H (2015) A highly accurate firefly based algorithm for heart disease prediction. Expert Syst Appl 42(21):8221–8231. Google Scholar
  42. 42.
    Luo L, Ming J, Wu D, Liu P, Zhu S (2017) Semantics-based obfuscation-resilient binary code similarity comparison with applications to software and algorithm plagiarism detection. IEEE Trans Softw Eng 43(12):1157–1177. Google Scholar
  43. 43.
    Mandayam Comar P, Liu L, Saha S, Tan PN, Nucci A (2013) Combining supervised and unsupervised learning for zero-day malware detection. In: Proceedings of the 32nd IEEE International Conference on Computer Communications (INFOCOM’13), pp 2022-2030. IEEE, Piscataway
  44. 44.
    Miao Q, Liu J, Cao Y, Song J (2016) Malware detection using bilayer behavior abstraction and improved one-class support vector machines. Int J Inf Secur 15(4):361–379. Google Scholar
  45. 45.
    Miller RG Jr (1997) Beyond ANOVA: basics of applied statistics. Chapman and Hall/CRC, LondonzbMATHGoogle Scholar
  46. 46.
    Naval S, Laxmi V, Rajarajan M, Gaur MS, Conti M (2015) Employing program semantics for malware detection. IEEE Trans Inf Forensics Secur 10(12):2591–2604. Google Scholar
  47. 47.
    Neri F, Cotta C (2012) Memetic algorithms and memetic computing optimization: a literature review. Swarm Evol Comput 2:1–14. Google Scholar
  48. 48.
    Nissim N, Lapidot Y, Cohen A, Elovici Y (2018) Trusted system-calls analysis methodology aimed at detection of compromised virtual machines using sequential mining. Knowl-Based Syst 153:147–175. Google Scholar
  49. 49.
    O’Kane P, Sezer S, Mclaughlin K (2011) Obfuscation: the hidden malware. IEEE Secur Priv 9(5):41–47. Google Scholar
  50. 50.
    Parhizkar E, Abadi M (2015) BeeOWA: a novel approach based on ABC algorithm and induced OWA operators for constructing one-class classifier ensembles. Neurocomputing 166:367–381. Google Scholar
  51. 51.
    Reformat M, Yager RR (2008) Building ensemble classifiers using belief functions and OWA operators. Soft Comput 12(6):543–558. zbMATHGoogle Scholar
  52. 52.
    Rokach L (2010) Ensemble-based classifiers. Artif Intell Rev 33(1):1–39. MathSciNetGoogle Scholar
  53. 53.
    Rudd EM, Rozsa A, Günther M, Boult TE (2017) A survey of stealth malware: attacks, mitigation measures, and steps toward autonomous open world solutions. IEEE Commun Surv Tutorials 19(2):1145–1172. Google Scholar
  54. 54.
    Sengupta S, Das AK (2016) An approach to development of an ensemble classification system. In: Proceedings of the 2016 2nd International Conference on Research in Computational Intelligence and Communication Networks (ICRCICN’16), pp 218-223. IEEE, Piscataway
  55. 55.
    Shen YD, Zhang Z, Yang Q (2002) Objective-oriented utility-based association mining. In: Proceedings of the 2002 IEEE International Conference on Data Mining (ICDM’02), pp 426-433. IEEE, Piscataway
  56. 56.
    Stolfo SJ, Apap F, Eskin E, Heller KA, Hershkop S, Honig A, Svore KM (2005) A comparative evaluation of two algorithms for Windows Registry anomaly detection. J Comput Secur 13(4):659–693. Google Scholar
  57. 57.
    Su H, Cai Y, Du Q (2017) Firefly-algorithm-inspired framework with band selection and extreme learning machine for hyperspectral image classification. IEEE J Sel Topics Appl Earth Observations Remote Sens 10(1):309–320. Google Scholar
  58. 58.
    Symantec (2016) Internet security threat report (ISTR)
  59. 59.
    Tax DMJ (2018) DDTools, the data description toolbox for MATLAB. Version 2.1.3Google Scholar
  60. 60.
    Wasikowski M, Chen XW (2010) Combating the small sample class imbalance problem using feature selection. IEEE Trans Knowl Data Eng 22(10):1388–1400. Google Scholar
  61. 61.
    Xing HJ, Ji M (2018) Robust one-class support vector machine with rescaled hinge loss function. Pattern Recogn 84:152–164. Google Scholar
  62. 62.
    Xing HJ, Wang XZ (2017) Selective ensemble of SVDDs with Renyi entropy based diversity measure. Pattern Recogn 61:185–196. Google Scholar
  63. 63.
    Yager RR (1988) On ordered weighted averaging aggregation operators in multicriteria decisionmaking. IEEE Trans Syst Man Cybern 18(1):183–190. MathSciNetzbMATHGoogle Scholar
  64. 64.
    Yager RR (1993) Families of OWA operators. Fuzzy Sets Syst 59(2):125–148. MathSciNetzbMATHGoogle Scholar
  65. 65.
    Yager RR, Grichnik AJ, Yager RL (2014) A soft computing approach to controlling emissions under imperfect sensors. IEEE Trans Syst Man Cybern 44(6):687–691. Google Scholar
  66. 66.
    Yahyazadeh M, Abadi M (2015) BotGrab: a negative reputation system for botnet detection. Comput Electr Eng 41:68–85. Google Scholar
  67. 67.
    Yang XS (2010) Firefly algorithm, stochastic test functions and design optimisation. Int J Bio-Inspired Comput 2(2):78–84. Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of Electrical and Computer EngineeringTarbiat Modares UniversityTehranIran

Personalised recommendations