Automatic Detection of Various Malicious Traffic Using Side Channel Features on TCP Packets

  • George Stergiopoulos
  • Alexander Talavari
  • Evangelos Bitsikas
  • Dimitris GritzalisEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11098)


Modern intrusion detection systems struggle to detect advanced, custom attacks against most vectors; from web application injections to malware reverse connections with encrypted traffic. Current solutions mostly utilize complex patterns or behavioral analytics on software, user actions and services historical data together with traffic analysis, in an effort to detect specific types of attacks. Still, false positives and negatives plague such systems. Behavioral-based security solutions provides good results but need large amounts of time and data to train (often spanning months or even years of surveillance) - especially when encryption comes into play. In this paper, we present a network traffic monitoring system that implements a detection method using machine learning over side channel characteristics of TCP/IP packets and not deep packet inspection, user analytics or binary analysis. We were able to efficiently distinguish normal from malicious traffic over a wide range of attacks with a true positive detection rate of about 94%. Few similar efforts have been made for the classification of malicious traffic but existing methods rely on complex feature selection and deep packet analysis to achieve similar (or worse) detection rates. Most focus on encrypted malware traffic. We manage to distinguish malicious from normal traffic in a wide range of different types of attacks (e.g. unencrypted and encrypted malware traffic and/or shellcode connections, website defacing attacks, ransomware downloaded cryptolocker attacks, etc.) using only few side channel packet characteristics and we achieve similar or better overall detection rates from similar detection systems. We compare seven different machine learning algorithms on multiple traffic sets to produce the best possible results. We use less features than other proposed solutions and thus require less data and achieve short times during training and classification.


Malware traffic Malware detection Machine learning Defacement SVR Neural networks CART Botnet Reverse shells Trojan 


  1. 1.
    Biondi, P.: Scapy (2011)Google Scholar
  2. 2.
    McKinney, W.: PyData development team. Pandas: Powerful Python Data Analy. Toolkit 1625 (2015)Google Scholar
  3. 3.
    Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011). Scholar
  4. 4.
    MariaDB database server. Accessed 1 Jan 2018
  5. 5., Hands-on Network Forensics - Training PCAP dataset from FIRST 2015.
  6. 6.
    Milicenso, Ponmocup Malware dataset (Update 2012-10-07, Accessed 1 Jan 2018)
  7. 7.
    CTU-13 dataset, CTU University, Czech Republic, 2011,
  8. 8.
    Livadas, C., Walsh, B., Lapsley, D., Strayer, T.: Using machine learning techniques to identify botnet traffic. In: Proceedings of the IEEE LCN Workshop on Network Security (2006)Google Scholar
  9. 9.
    Cooke, E., Jahanian, F., McPherson, D.: The zombie roundup: understanding, detecting, and disrupting botnets. In: Proceedings of the Workshop on Steps to Reducing Unwanted Traffic on the Internet (2005)Google Scholar
  10. 10.
    Binkley, J., Singh, S.: An algorithm for anomaly-based Botnet detection. In: Proceedings of the Workshop on Steps to Reducing Unwanted Traffic on the Internet (2006)Google Scholar
  11. 11.
    Gu, G., Porras, P., Yegneswaran, V., Fong, M.W., Lee, W.: BotHunter: detecting malware. Infection through IDS-Driven Dialog Correlation. In: Proceedings of the USENIX Security Symposium (2007)Google Scholar
  12. 12.
    Timofeev, R.: Classification and regression trees (cart) theory and applications. Humboldt University, Berlin (2004)Google Scholar
  13. 13.
    Střasák, F.: Detection of HTTPS malware Traffic (Detekce Malware v HTTPS komunikaci). BSC thesis. České vysoké učení technické v Praze. Vypočetní a informační centrum (2017)Google Scholar
  14. 14.
    Taylor, C., Alves-Foss, J.: NATE - network analysis of anomalous traffic events, a low-cost approach. In: Proceedings of the New Security Paradigms Workshop (2001)Google Scholar
  15. 15.
    Lakhina, A., Papagiannaki, K., Crovella, M.: Structural analysis of network traffic flows. In: Proceedings of ACM SIGMETRICS/Performance (2004)Google Scholar
  16. 16.
    Terrell, J., et al.: Multivariate SVD analyses for network anomaly detection. In: (Poster) Proceeding of ACM SIGCOMM (2005)Google Scholar
  17. 17.
    Yen, T.-F., Reiter, M.K.: Traffic aggregation for malware detection. In: Zamboni, D. (ed.) DIMVA 2008. LNCS, vol. 5137, pp. 207–227. Springer, Heidelberg (2008). Scholar
  18. 18.
    Kohout, J., Pevny, T.: Automatic discovery of web servers hosting similar applications. In: Proceedings of the IFIP/IEEE International Symposium on Integrated Network Management, IEEE, pp. 1310–1315 (2015)Google Scholar
  19. 19.
    Lokoč, J., Kohout, J., Čech, P., Skopal, T., Pevný, T.: k-NN classification of Malware in HTTPS traffic using the metric space approach. In: Chau, M., Wang, G.A., Chen, H. (eds.) PAISI 2016. LNCS, vol. 9650, pp. 131–145. Springer, Cham (2016). Scholar
  20. 20.
    Crotti, M., et al.: Traffic classification through simple statistical fingerprinting. ACM SIG-COMM Comput. Commun. Rev. 37(1), 5–16 (2007)CrossRefGoogle Scholar
  21. 21.
    Prasse, P., et al.: Malware Detection by HTTPS Traffic Analysis (2017)Google Scholar
  22. 22.
    Chari, S., et al.: A platform and analytics for usage and entitlement analytics. IBM J. Res. Dev. 60(4), 7-1 (2016)CrossRefGoogle Scholar
  23. 23.
    Combs, G.: “Wireshark.” (2007). Accessed 12 Feb
  24. 24.
    Roesch, M.: Snort: lightweight intrusion detection for networks. In: Lisa, Vol. 99, no. 1 (1999)Google Scholar
  25. 25.
    Liu, J., et al.: Effective and real-time in-app activity analysis in encrypted internet traffic streams. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM (2017)Google Scholar
  26. 26.
    Chen, S., et al.: Side-channel leaks in web applications: a reality today, a challenge tomorrow. In: IEEE Symposium on 2010 Security and Privacy, IEEE (2010)Google Scholar
  27. 27.
  28. 28.
    Bro, I.: (2008).
  29. 29.
    Beale, J., Baker, A., Esler, J.: Snort: IDS and IPS toolkit. Syngress (2007)Google Scholar
  30. 30.
    Suricata, I.D.S.: open-source IDS. IPS/NSM engine (2014). (
  31. 31.
    Sommer, R., Paxson, V.: Outside the closed world: on using machine learning for network intrusion detection. In: IEEE Symposium on 2010 Security and Privacy (SP), IEEE (2010)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • George Stergiopoulos
    • 1
  • Alexander Talavari
    • 1
  • Evangelos Bitsikas
    • 1
  • Dimitris Gritzalis
    • 1
    Email author
  1. 1.Information Security and Critical Infrastructure Protection (INFOSEC) Laboratory, Department of InformaticsAthens University of Economics and BusinessAthensGreece

Personalised recommendations