Malware Detection Using Machine Learning and Deep Learning

  • Hemant RathoreEmail author
  • Swati Agarwal
  • Sanjay K. Sahay
  • Mohit Sewak
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11297)


Research shows that over the last decade, malware have been growing exponentially, causing substantial financial losses to various organizations. Different anti-malware companies have been proposing solutions to defend attacks from these malware. The velocity, volume, and the complexity of malware are posing new challenges to the anti-malware community. Current state-of-the-art research shows that recently, researchers and anti-virus organizations started applying machine learning and deep learning methods for malware analysis and detection. We have used opcode frequency as a feature vector and applied unsupervised learning in addition to supervised learning for malware classification. The focus of this tutorial is to present our work on detecting malware with (1) various machine learning algorithms and (2) deep learning models. Our results show that the Random Forest outperforms Deep Neural Network with opcode frequency as a feature. Also in feature reduction, Deep Auto-Encoders are overkill for the dataset, and elementary function like Variance Threshold perform better than others. In addition to the proposed methodologies, we will also discuss the additional issues and the unique challenges in the domain, open research problems, limitations, and future directions.


Auto-encoders Cyber security Deep learning Machine learning Malware detection 


  1. 1.
  2. 2.
    David, O.E., Netanyahu, N.S.: Deepsign: deep learning for automatic malware signature generation and classification. In: 2015 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2015)Google Scholar
  3. 3.
    Firdausi, I., Erwin, A., Nugroho, A.S., et al.: Analysis of machine learning techniques used in behavior-based malware detection. In: 2010 Second International Conference on Advances in Computing, Control and Telecommunication Technologies (ACT), pp. 201–203. IEEE (2010)Google Scholar
  4. 4.
    Hardy, W., Chen, L., Hou, S., Ye, Y., Li, X.: Dl4md: a deep learning framework for intelligent malware detection. In: Proceedings of the International Conference on Data Mining (DMIN), p. 61. The Steering Committee of The World Congress in Computer Science, Computer Engineering and Applied Computing (2016)Google Scholar
  5. 5.
    He, H., Bai, Y., Garcia, E.A., Li, S.: Adasyn: adaptive synthetic sampling approach for imbalanced learning. In: IEEE International Joint Conference on Neural Networks IJCNN 2008. (IEEE World Congress on Computational Intelligence), pp. 1322–1328. IEEE (2008)Google Scholar
  6. 6.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  7. 7.
    Masud, M.M., et al.: Cloud-based malware detection for evolving data streams. ACM Trans. Manage. Inf. Syst. (TMIS) 2(3), 16 (2011)Google Scholar
  8. 8.
    Moskovitch, R., et al.: Unknown malcode detection using OPCODE representation. In: Ortiz-Arroyo, D., Larsen, H.L., Zeng, D.D., Hicks, D., Wagner, G. (eds.) EuroIsI 2008. LNCS, vol. 5376, pp. 204–215. Springer, Heidelberg (2008). Scholar
  9. 9.
    Nappa, A., Rafique, M.Z., Caballero, J.: Driving in the cloud: an analysis of drive-by download operations and abuse reporting. In: Rieck, K., Stewin, P., Seifert, J.-P. (eds.) DIMVA 2013. LNCS, vol. 7967, pp. 1–20. Springer, Heidelberg (2013). Scholar
  10. 10.
    Sahay, S.K., Sharma, A.: Grouping the executables to detect malwares with high accuracy. Procedia Comput. Sci. 78, 667–674 (2016)CrossRefGoogle Scholar
  11. 11.
    Santos, I., Brezo, F., Ugarte-Pedrero, X., Bringas, P.G.: Opcode sequences as representation of executables for data-mining-based unknown malware detection. IET Inf. Sci. 231, 64–82 (2013)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Sewak, M., Sahay, S.K., Rathore, H.: Comparison of deep learning and the classical machine learning algorithm for the malware detection. In: 2018 19th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), pp. 293–296. IEEE (2018)Google Scholar
  13. 13.
    Sewak, M., Sahay, S.K., Rathore, H.: An investigation of a deep learning based malware detection system. In: Proceedings of the 13th International Conference on Availability, Reliability and Security, p. 26. ACM (2018)Google Scholar
  14. 14.
    Sharma, A., Sahay, S.K.: An effective approach for classification of advanced malware with high accuracy. arXiv preprint arXiv:1606.06897 (2016)
  15. 15.
    Ye, Y., Li, T., Adjeroh, D., Iyengar, S.S.: A survey on malware detection using data mining techniques. ACM Comput. Surv. (CSUR) 50(3), 41 (2017)CrossRefGoogle Scholar
  16. 16.
    Ye, Y., Li, T., Chen, Y., Jiang, Q.: Automatic malware categorization using cluster ensemble. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 95–104. ACM (2010)Google Scholar
  17. 17.
    Ye, Y., Wang, D., Li, T., Ye, D.: IMDS: intelligent malware detection system. In: Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1043–1047. ACM (2007)Google Scholar
  18. 18.
    Ye, Y., Wang, D., Li, T., Ye, D., Jiang, Q.: An intelligent pe-malware detection system based on association mining. J. Comput. Virol. 4(4), 323–334 (2008)CrossRefGoogle Scholar
  19. 19.
    Yousefi-Azar, M., Varadharajan, V., Hamey, L., Tupakula, U.: Autoencoder-based feature learning for cyber security applications. In: 2017 International Joint Conference on Neural Networks (IJCNN), pp. 3854–3861. IEEE (2017)Google Scholar
  20. 20.
    Zak, R., Raff, E., Nicholas, C.: What can n-grams learn for malware detection? In: 2017 12th International Conference on Malicious and Unwanted Software (MALWARE), pp. 109–118. IEEE (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Hemant Rathore
    • 1
    Email author
  • Swati Agarwal
    • 1
  • Sanjay K. Sahay
    • 1
  • Mohit Sewak
    • 1
  1. 1.Department of CS and ISBITS PilaniGoaIndia

Personalised recommendations