Detection of Application-Layer Tunnels with Rules and Machine Learning

  • Huaqing Lin
  • Gao Liu
  • Zheng YanEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11611)


Application-layer tunnels are often used to construct covert channels in order to transmit secret data, which is often applied to raise network threats in recent years. Detection of application-layer tunnels can assist identifying a variety of network threats, thus has high research significance. In this paper, we explore application-layer tunnel detection and propose a generic detection method by applying both rules and machine learning. Our detection method mainly consists of two parts: rule-based domain name filtering for Domain Generation Algorithm (DGA) based on a trigram model and a machine learning model based on our proposed generic feature extraction framework for tunnel detection. The rule-based DGA domain name filtering can eliminate some obvious tunnels in order to reduce the amount of data processed by machine learning-based detection, thereby, the detection efficiency can be improved. The generic feature extraction framework comprehensively integrates previous research results by combining multiple detection methods, supporting multiple layers and performing multiple feature extraction. We take the three most common application-layer tunnels, i.e., DNS tunnel, HTTP tunnel and HTTPS tunnel as examples to analyze and test our detection method. The experimental results show that the proposed method is generic and efficient, compared with other existing approaches.


Application-layer tunnels detection Machine learning DGA domain name Feature extraction 



This work is sponsored by the National Key Research and Development Program of China (Grant 2016YFB0800700), the National Natural Science Foundation of China (Grants 61672410 and U1536202), the Academy of Finland (Grants 308087 and 314203), the open grant of the Tactical Data Link Lab of the 20th Research Institute of China Electronics Technology Group Corporation (grant No. CLDL-20182119), the Key Lab of Information Network Security, Ministry of Public Security (Grant C18614).


  1. 1.
    Nuojua, V., David, G., Hämäläinen, T.: DNS tunneling detection techniques – classification, and theoretical comparison in case of a real APT campaign. In: Galinina, O., Andreev, S., Balandin, S., Koucheryavy, Y. (eds.) NEW2AN/ruSMART/NsCC 2017. LNCS, vol. 10531, pp. 280–291. Springer, Cham (2017). Scholar
  2. 2.
    Borders, K., Prakash, A.: Web tap: detecting covert web traffic. In: Proceedings of the 11th ACM Conference on Computer and Communications Security, pp. 110–120. ACM, New York (2004)Google Scholar
  3. 3.
    Crotti, M., Dusi, M., Gringoli, F., Salgarelli, L.: Detecting http tunnels with statistical mechanisms. In: 2007 IEEE International Conference on Communications, Glasgow, pp. 6162–6168. IEEE (2007)Google Scholar
  4. 4.
    Dusi, M., Crotti, M., Gringoli, F., Salgarelli, L.: Tunnel hunter: detecting application-layer tunnels with statistical fingerprinting. Comput. Netw. 53(1), 81–97 (2009)CrossRefGoogle Scholar
  5. 5.
    Do, V.T., Engelstad, P., Feng, B., van Do, T.: Detection of DNS tunneling in mobile networks using machine learning. In: Kim, K., Joukov, N. (eds.) ICISA 2017. LNEE, vol. 424, pp. 221–230. Springer, Singapore (2017). Scholar
  6. 6.
    Almusawi, A., Amintoosi, H.: DNS Tunneling detection method based on multilabel support vector machine. In: Security and Communication Networks 2018 (2018)Google Scholar
  7. 7.
    Qi, C., Chen, X., Xu, C., Shi, J., Liu, P.: A bigram based real time DNS tunnel detection approach. Procedia Comput. Sci. 17, 852–860 (2013)CrossRefGoogle Scholar
  8. 8.
    Aiello, M., Mongelli, M., Papaleo, G.: DNS tunneling detection through statistical fingerprints of protocol messages and machine learning. Int. J. Commun. Syst. 28(14), 1987–2002 (2015)CrossRefGoogle Scholar
  9. 9.
    Liu, J., Li, S., Zhang, Y., Xiao, J., Chang, P., Peng, C.: Detecting DNS tunnel through binary-classification based on behavior features. In: IEEE Trustcom/BigDataSE/ICESS, Sydney, pp. 339–346. IEEE (2017)Google Scholar
  10. 10.
    Ding, Y.J., Cai, W.D.: A method for HTTP-tunnel detection based on statistical features of traffic. In: 2011 IEEE 3rd International Conference on Communication Software and Networks, Xi’an, pp. 247–250. IEEE (2011)Google Scholar
  11. 11.
    Piraisoody, G., Huang, C., Nandy, B., Seddigh, N.: Classification of applications in HTTP tunnels. In: 2013 IEEE 2nd International Conference on Cloud Networking (CloudNet), San Francisco, pp. 67–74. IEEE (2013)Google Scholar
  12. 12.
    Li, S., Yun, X., Zhang, Y.: Anomaly-based model for detecting HTTP-tunnel traffic using network behavior analysis. High Technol. Lett. 20(1), 63–69 (2014)Google Scholar
  13. 13.
    Mujtaba, G., Parish, D.J.: Detection of applications within encrypted tunnels using packet size distributions. In: 2009 International Conference for Internet Technology and Secured Transactions (ICITST), London, pp. 1–6. IEEE (2009)Google Scholar
  14. 14.
    Wang, F., Huang, L., Chen, Z., Miao, H., Yang, W.: A novel web tunnel detection method based on protocol behaviors. In: Zia, T., Zomaya, A., Varadharajan, V., Mao, M. (eds.) SecureComm 2013. LNICST, vol. 127, pp. 234–251. Springer, Cham (2013). Scholar
  15. 15.
    Allard, F., Dubois, R., Gompel, P., Morel, M.: Tunneling activities detection using machine learning techniques. J. Telecommun. Inf. Technol. 2011(1), 37–42 (2011)Google Scholar
  16. 16.
    Buczak, A.L., Guven, E.: A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Commun. Surv. Tutor. 18(2), 1153–1176 (2016)CrossRefGoogle Scholar
  17. 17.
    Mishra, P., Varadharajan, V., Tupakula, U., Pilli, E.S.: A detailed investigation and analysis of using machine learning techniques for intrusion detection. IEEE Commun. Surv. Tutor. 21(1), 686–728 (2018)CrossRefGoogle Scholar
  18. 18.
    Wang, T.S., Lin, H.T., Cheng, W.T., Chen, C.Y.: DBod: Clustering and detecting DGA-based botnets using DNS traffic analysis. Comput. Secur. 64, 1–15 (2017)CrossRefGoogle Scholar
  19. 19.
    Khehra, G., Sofat, S.: BotScoop: scalable detection of DGA based botnets using DNS traffic. In: 2018 9th International Conference on Computing, Communication and Networking Technologies (ICCCNT), Bangalore, pp. 1–6. IEEE (2018)Google Scholar
  20. 20.
    Alexa Top 1 Million Sites. Accessed 20 Jan 2019
  21. 21.
    360 Netlab Open Data DGA. Accessed 20 Jan 2019
  22. 22.
    Jing, X., Yan, Z., Pedrycz, W.: Security data collection and data analytics in the Internet: a survey. IEEE Commun. Surv. Tutor. 21(1), 586–618 (2019)CrossRefGoogle Scholar
  23. 23.
    Lin, H., Yan, Z., Fu, Y.: Adaptive security-related data collection with context awareness. J. Netw. Comput. Appl. 126, 88–103 (2019)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.State Key Lab of ISN, School of Cyber EngineeringXidian UniversityXi’anChina
  2. 2.Department of Communications and NetworkingAalto UniversityEspooFinland

Personalised recommendations