Abstract
Internet-borne threats have evolved from easy to detect denial of service attacks to zero-day exploits used for targeted exfiltration of data. Current intrusion detection systems cannot always keep-up with zero-day attacks and it is often the case that valuable data have already been communicated to an external party over an encrypted or plain text connection before the intrusion is detected.
In this paper, we present a scalable approach called Network Interrogator (NetGator) to detect network-based malware that attempts to exfiltrate data over open ports and protocols. NetGator operates as a transparent proxy using protocol analysis to first identify the declared client application using known network flow signatures.Then we craft packets that “challenge” the application by exercising functionality present in legitimate applications but too complex or intricate to be present in malware. When the application is unable to correctly solve and respond to the challenge, NetGator flags the flow as potential malware. Our approach is seamless and requires no interaction from the user and no changes on the commodity application software. NetGator introduces a minimal traffic latency (0.35 seconds on average) to normal network communication while it can expose a wide-range of existing malware threats.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Anti-Phishing Working Group, ADWG 2011 Trends Report, http://apwg.org/reports/apwg_trends_report_h1_2011.pdf
Greasyspoon, http://greasyspoon.sourceforge.net/
Javascript encryption, http://javascript.about.com/library/blencrypt.html
Planetlab, http://planet-lab.org/
Squid, http://www.squid-cache.org/
Tcpflow, http://afflib.org/software/tcpflow
Wireshark, http://www.wireshark.org/
Androulidakis, G., Chatzigiannakis, V., Papavassiliou, S.: Network anomaly detection and classification via opportunistic sampling. IEEE Network 23(1), 6–12 (2009)
AsSadhan, B., Moura, J., Lapsley, D., Jones, C., Strayer, W.: Detecting Botnets Using Command and Control Traffic. In: Proceedings of the 2009 Eighth IEEE International Symposium on Network Computing and Applications, pp. 156–162. IEEE Computer Society (2009)
Bailey, M., Cooke, E., Jahanian, F., Xu, Y., Karir, M.: A survey of botnet technology and defenses. In: Proceedings of the 2009 Cybersecurity Applications & Technology Conference for Homeland Security, pp. 299–304. IEEE Computer Society (2009)
Cyveillance. Malware detection rates for leading av solutions (2010), http://www.cyveillance.com/web/docs/WP_MalwareDetectionRates.pdf
Eckersley, P.: How Unique Is Your Web Browser? In: Atallah, M.J., Hopper, N.J. (eds.) PETS 2010. LNCS, vol. 6205, pp. 1–18. Springer, Heidelberg (2010)
Garcia-Teodoro, P., Diaz-Verdejo, J., Macia-Fernandez, G., Vazquez, E.: Anomaly-based network intrusion detection: Techniques, systems and challenges. Computers & Security 28(1-2), 18–28 (2009)
Gu, G., Perdisci, R., Zhang, J., Lee, W.: BotMiner: Clustering analysis of network traffic for protocol-and structure-independent botnet detection. In: Proceedings of the 17th Conference on Security Symposium, pp. 139–154. USENIX Association (2008)
Gu, G., Porras, P., Yegneswaran, V., Fong, M., Lee, W.: Bothunter: detecting malware infection through ids-driven dialog correlation. In: SS 2007: Proceedings of 16th USENIX Security Symposium on USENIX Security Symposium, pp. 1–16. USENIX Association, Berkeley (2007)
Gu, G., Yegneswaran, V., Porras, P., Stoll, J., Lee, W.: Active botnet probing to identify obscure command and control channels. In: Computer Security Applications Conference, ACSAC (2009)
Gu, G., Zhang, J., Lee, W.: BotSniffer: Detecting botnet command and control channels in network traffic. In: Proceedings of the 15th Annual Network and Distributed System Security Symposium, NDSS 2008. Citeseer (2008)
Inoue, D., Yoshioka, K., Eto, M., Hoshizawa, Y., Nakao, K.: Malware Behavior Analysis in Isolated Miniature Network for Revealing Malware’s Network Activity. In: IEEE International Conference on Communications, ICC 2008, pp. 1715–1721. IEEE (2008)
Karasaridis, A., Rexroad, B., Hoeflin, D.: Wide-scale botnet detection and characterization. In: Proceedings of the First Conference on First Workshop on Hot Topics in Understanding Botnets, p. 7. USENIX Association (2007)
Lin, Z., Jiang, X., Xu, D., Zhang, X.: Automatic protocol format reverse engineering through conectect-aware monitored execution. In: 15th Symposium on Network and Distributed System Security, NDSS (2008)
Lyon, G.: Nmap security scanner (2010)
McKinley, K.: Cleaning Up After Cookies (2008)
Microsoft Developer Network: How to: Detect browser types and browser capabilities in asp.net web pages (2010)
Perdisci, R., Lee, W., Feamster, N.: Behavioral clustering of http-based malware and signature generation using malicious network traces. In: Proceedings of the 7th USENIX Conference on Networked Systems Design and Implementation, p. 26. USENIX Association (2010)
Polychronakis, M., Anagnostakis, K., Markatos, E.: Network-level polymorphic shellcode detection using emulation. Journal in Computer Virology 2, 257–274 (2007) 10.1007/s11416-006-0031-z
Schools, W.: Javascript browser detection (2010), http://www.w3schools.com/js/js_browser.asp
Shon, T., Moon, J.: A hybrid machine learning approach to network anomaly detection. Information Sciences 177(18), 3799–3821 (2007)
Thorat, S., Khandelwal, A., Bruhadeshwar, B., Kishore, K.: Payload content based network anomaly detection. In: First International Conference on the Applications of Digital Information and Web Technologies, ICADIWT 2008, pp. 127–132 (August 2008)
Wondracek, G., Comparetti, P.M., Kruegel, C., Kirda, E.: Automatic network protocol analysis. In: 15th Symposium on Network and Distributed System Security, NDSS (2008)
Yen, T.-F., Huang, X., Monrose, F., Reiter, M.K.: Browser Fingerprinting from Coarse Traffic Summaries: Techniques and Implications. In: Flegel, U., Bruschi, D. (eds.) DIMVA 2009. LNCS, vol. 5587, pp. 157–175. Springer, Heidelberg (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Schulte, B., Andrianakis, H., Sun, K., Stavrou, A. (2013). NetGator: Malware Detection Using Program Interactive Challenges. In: Flegel, U., Markatos, E., Robertson, W. (eds) Detection of Intrusions and Malware, and Vulnerability Assessment. DIMVA 2012. Lecture Notes in Computer Science, vol 7591. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-37300-8_10
Download citation
DOI: https://doi.org/10.1007/978-3-642-37300-8_10
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-37299-5
Online ISBN: 978-3-642-37300-8
eBook Packages: Computer ScienceComputer Science (R0)