Error-Sensor: Mining Information from HTTP Error Traffic for Malware Intelligence

  • Jialong Zhang
  • Jiyong JangEmail author
  • Guofei Gu
  • Marc Ph. Stoecklin
  • Xin Hu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11050)


Malware often encounters network failures when it launches malicious activities, such as connecting to compromised servers that have been already taken down, connecting to malicious servers that are blocked based on access control policies in enterprise networks, or scanning/exploiting vulnerable web pages. To overcome such failures and improve the resilience in light of such failures, malware authors have employed various strategies, e.g., connecting to multiple backup servers or connecting to benign servers for initial network connectivity checks. These network failures and recovery strategies lead to distinguishing traits, which are newly discovered and thoroughly studied in this paper. We note that network failures caused by malware are quite different from the failures caused by benign users/software in terms of their failure patterns and recovery behavior patterns.

In this paper, we present the results of the first large-scale measurement study investigating the different network behaviors of both benign user/software and malware in light of HTTP errors. By inspecting over 1 million HTTP logs generated by over 16,000 clients, we identify strong indicators of malicious activities derived from error provenance patterns, error generation patterns, and error recovery patterns. Based on the insights, we design a new system, Error-Sensor, to automatically detect traffic caused by malware from only HTTP errors and their surrounding successful requests. We evaluate Error-Sensor on a large scale of real-world web traces collected in an enterprise network. Error-Sensor achieves a detection rate of 99.79% at a false positive rate of 0.005% to identify HTTP errors generated by malware, and further, spots surreptitious malicious traffic (e.g., malware backup behavior) that was not caught by existing deployed intrusion detection systems.



This material is based upon work supported in part by the National Science Foundation (NSF) under Grant no. 1314823. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of NSF.


  1. 1.
    Domain Generation Algorithms (DGA) in Stealthy Malware.
  2. 2.
    Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content (RFC 7231).
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
    Story of the Cutwail/Pushdo hidden C&C server (2013).
  8. 8.
    VirusTotal (2017).
  9. 9.
    Agrawal, R., Imielinski, T., Swami, A.: Mining association rules between sets of items in large databases. In: SIGMOD (1993)Google Scholar
  10. 10.
    Antonakakis, M., Perdisci, R., Dagon, D., Lee, W., Feamster, N.: Building a dynamic reputation system for DNS. In: USENIX Security (2010)Google Scholar
  11. 11.
    Antonakakis, M., Perdisci, R., Lee, W., Vasiloglou, N., Dagon, D.: Detecting malware domains at the upper DNS hierarchy. In: USENIX Security (2011)Google Scholar
  12. 12.
    Antonakakis, M., et al.: From throw-away traffic to bots: detecting the rise of DGA-based malware. In: USENIX Security (2012)Google Scholar
  13. 13.
    Beverly, R., Sollins, K.: Exploiting transport-level characteristics of spam. In: Proceedings of the Fifth Conference on Email and Anti-Spam (CEAS 2008) (2008)Google Scholar
  14. 14.
    Bilge, L., Kirda, E., Kruegel, C., Balduzzi, M.: Exposure: finding malicious domains using passive DNS analysis. In: NDSS (2011)Google Scholar
  15. 15.
    Carroll, T., Grosu, D.: A game theoretic investigation of deception in network security. In: ICCCN (2009)Google Scholar
  16. 16.
    Wagener, G., State, R., Dulaunoy, A., Engel, T.: Self adaptive high interaction honeypots driven by game theory. In: Proceedings of the 11th International Symposium on Stabilization, Safety, and Security of Distributed Systems (SSS 2009) (2009)Google Scholar
  17. 17.
    Gao, H., et al.: An empirical reexamination of global DNS behavior. In: ACM SIGCOMM (2013)Google Scholar
  18. 18.
    Gu, G., Perdisci, R., Zhang, J., Lee, W.: Botminer: clustering analysis of network traffic for protocol-and structure-independent botnet detection. In: USENIX Security (2008)Google Scholar
  19. 19.
    Gu, G., Zhang, J., Lee, W.: Botsniffer: detecting botnet command and control channels in network traffic. In: NDSS (2008)Google Scholar
  20. 20.
    Hu, X., et al.: BAYWATCH: robust beaconing detection to identify infected hosts in large-scale enterprise networks. In: DSN (2016)Google Scholar
  21. 21.
    Jiang, N., Cao, J., Jin, Y., Li, L., Zhang, Z.: Identifying suspicious activities through DNS failure graph analysis. In: IEEE ICNP (2010)Google Scholar
  22. 22.
    Kwon, B.J., Srinivas, V., Deshpande, A., Dumitras, T.: Catching worms, trojan horses and PUPs: unsupervised detection of silent delivery campaigns. In: NDSS (2017)Google Scholar
  23. 23.
    Moser, A., Kirda, E., Kruegel, C.: Exploring multiple execution paths for malware analysis. In: IEEE Security and Privacy (2007)Google Scholar
  24. 24.
    Nadji, Y., Antonakakis, M., Perdisci, R., Lee, W.: Understanding the prevalence and use of alternative plans in malware with network games. In: Proceedings of the Annual Computer Security Applications Conference (ACSAC 2011) (2011)Google Scholar
  25. 25.
    Nelms, T., Perdisci, R., Ahamad, M.: Execscent: mining for new c&c domains in live networks with adaptive control protocol templates. In: USENIX Security (2013)Google Scholar
  26. 26.
    Neugschwandtner, M., Comparetti, P.M., Platzer, C.: Detecting malware’s failover c&c strategies with squeeze. In: ACSAC (2011)Google Scholar
  27. 27.
    Perdisci, R., Lee, W., Feamster, N.: Behavioral clustering of HTTP-based malware and signature generation using malicious network traces. In: NSDI (2010)Google Scholar
  28. 28.
    Porras, P., Saidi, H., Yegneswaran, V.: A Foray into Conficker’s logic and rendezvous points. In: USENIX Workshop on Large-Scale Exploits and Emergent Threats (LEET 2009) (2009)Google Scholar
  29. 29.
    Rusakov, V., Golovanov, S.: TDSS (2010).
  30. 30.
    Salusky, W., Danford, R.: Know your enemy: fast-flux service networks. In: The Honeynet Project (2008)Google Scholar
  31. 31.
    Stringhini, G., Kruegel, C., Vigna, G.: Shady paths: leveraging surfing crowds to detect malicious web pages. In: ACM CCS (2013)Google Scholar
  32. 32.
    Thomas, M., Mohaisen, A.: Kindred domains: detecting and clustering botnet domains using DNS traffic. In: WWW (2014)Google Scholar
  33. 33.
    Wang, T., Hu, X., Jang, J., Ji, S., Stoecklin, M., Taylor, T.: BotMeter: charting DGA-botnet landscapes in large networks. In: ICDCS (2016)Google Scholar
  34. 34.
    Wilhelm, J., Chiueh, T.: A forced sampled execution approach to kernel rootkit identification. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 219–235. Springer, Heidelberg (2007). Scholar
  35. 35.
    Yadav, S., Reddy, A.L.N.: Winning with DNS failures: strategies for faster botnet detection. In: Rajarajan, M., Piper, F., Wang, H., Kesidis, G. (eds.) SecureComm 2011. LNICST, vol. 96, pp. 446–459. Springer, Heidelberg (2012). Scholar
  36. 36.
    Yen, T., et al.: Beehive: large-scale log analysis for detecting suspicious activity in enterprise network. In: ACSAC (2013)Google Scholar
  37. 37.
    Yen, T.-F., Reiter, M.K.: Traffic aggregation for malware detection. In: Zamboni, D. (ed.) DIMVA 2008. LNCS, vol. 5137, pp. 207–227. Springer, Heidelberg (2008). Scholar
  38. 38.
    Zhang, H., Yao, D., Ramakrishnan, N.: Detection of stealthy malware activities with traffic causality and scalable triggering relation discovery. In: ASIACCS (2014)Google Scholar
  39. 39.
    Zhang, J., Hu, X., Jang, J., Wang, T., Gu, G., Stoecklin, M.: Hunting for invisibility: characterizing and detecting malicious web infrastructures through server visibility analysis. In: IEEE INFOCOM (2016)Google Scholar
  40. 40.
    Zhang, J., Saha, S., Gu, G., Lee, S., Mellia, M.: Systematic mining of associated server herds for malware campaign discovery. In: ICDCS (2015)Google Scholar
  41. 41.
    Zhu, Z., Yegneswaran, V., Chen, Y.: Using failure information analysis to detect enterprise zombies. In: Chen, Y., Dimitriou, T.D., Zhou, J. (eds.) SecureComm 2009. LNICST, vol. 19, pp. 185–206. Springer, Heidelberg (2009). Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Jialong Zhang
    • 1
  • Jiyong Jang
    • 1
    Email author
  • Guofei Gu
    • 2
  • Marc Ph. Stoecklin
    • 1
  • Xin Hu
    • 3
  1. 1.IBM ResearchYorktown HeightsUSA
  2. 2.Texas A&M UniversityCollege StationUSA
  3. 3.PinterestSan FranciscoUSA

Personalised recommendations