Skip to main content
SpringerLink
Log in

An interactive visual analytics approach for network anomaly detection through smart labeling

  • Regular Paper
  • Published:
Journal of Visualization Aims and scope Submit manuscript

Abstract

Network anomaly detection is an important means for safeguarding network security. On account of the difficulties encountered in traditional automatic detection methods such as lack of labeled data, expensive retraining costs for new data and non-explanation, we propose a novel smart labeling method, which combines active learning and visual interaction, to detect network anomalies through the iterative labeling process of the users. The algorithms and the visual interfaces are tightly integrated. The network behavior patterns are first learned by using the self-organizing incremental neural network. Then, the model uses a Fuzzy c-means-based algorithm to do classification on the basis of user feedback. After that, the visual interfaces are updated to present the improved results of the model, which can help users to choose meaningful candidates, judge anomalies and understand the model results. The experiments show that compared to labeling without our visualizations, our method can achieve a high accuracy rate of anomaly detection with fewer labeled samples.

Graphic abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • Ahmed M, Mahmood AN, Hu J (2016) A survey of network anomaly detection techniques. J Netw Comput Appl 60:19–31

    Article  Google Scholar 

  • Aljawarneh S, Aldwairi M, Yassein MB (2018) Anomaly-based intrusion detection system through feature selection analysis and building hybrid efficient model. J Comput Sci 25:152–160

    Article  Google Scholar 

  • Almgren M, Jonsson E (2004) Using active learning in intrusion detection. In: Proceedings. 17th IEEE Computer Security Foundations Workshop, 2004. IEEE, pp 88–98

  • Bernard J, Sessler D, Bannach A, May T, Kohlhammer J (2015) A visual active learning system for the assessment of patient well-being in prostate cancer research. In: Proceedings of the 2015 workshop on visual analytics in healthcare. ACM, p 1

  • Bernard J, Hutter M, Zeppelzauer M, Fellner D, Sedlmair M (2018a) Comparing visual-interactive labeling with active learning: an experimental study. IEEE Trans Vis Comput Graph 24(1):298–308

    Article  Google Scholar 

  • Bernard J, Zeppelzauer M, Sedlmair M, Aigner W (2018b) VIAL: a unified process for visual interactive labeling. Vis Comput 34(9):1189–1207

    Article  Google Scholar 

  • Bhuyan MH, Bhattacharyya DK, Kalita JK (2014) Network anomaly detection: methods, systems and tools. IEEE Commun Surv Tutor 16(1):303–336

    Article  Google Scholar 

  • Bruns-Smith D, Baskaran MM, Ezick J, Henretty T, Lethin R (2016) Cyber security through multidimensional data decompositions. In: Cybersecurity symposium (CYBERSEC), 2016. IEEE, pp 59–67

  • Buczak AL, Guven E (2016) A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Commun Surv Tutor 18(2):1153–1176

    Article  Google Scholar 

  • Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: a survey. ACM Comput Surv (CSUR) 41(3):15

    Article  Google Scholar 

  • Chen W, Kong F, Mei F, Yuan G, Li B (2017) A novel unsupervised anomaly detection approach for intrusion detection system. In: Big data security on cloud (BigDataSecurity), IEEE international conference on high performance and smart computing (HPSC), and IEEE international conference on intelligent data and security (IDS). IEEE, pp 69–73

  • Culotta A, Mccallum A (2005) Reducing labeling effort for structured prediction tasks. In: National conference on artificial intelligence, pp 746–751

  • Dunn JC (1973) A fuzzy relative of the isodata process and its use in detecting compact well-separated clusters. J Cybern 3(3):32–57

    Article  MathSciNet  Google Scholar 

  • Fischer F, Mansmann F, Keim DA, Pietzko S, Waldvogel M (2008) Large-scale network monitoring for visual analysis of attacks. In: International workshop on visualization for computer security. Springer, Berlin, pp 111–118

  • Fritzke B (1995) A growing neural gas network learns topologies. In: International conference on neural information processing systems, pp 625–632

  • Furao S, Hasegawa O (2006) An incremental network for on-line unsupervised classification and topology learning. Neural Netw 19(1):90–106

    Article  Google Scholar 

  • Görnitz N, Kloft M, Rieck K, Brefeld U (2009) Active learning for network intrusion detection. In: Proceedings of the 2nd ACM workshop on security and artificial intelligence. ACM, New York, pp 47–54

  • Heimerl F, Koch S, Bosch H, Ertl T (2012) Visual classifier training for text document retrieval. IEEE Trans Vis Comput Graph 12:2839–2848

    Article  Google Scholar 

  • Höferlin B, Netzel R, Höferlin M, Weiskopf D, Heidemann G (2012) Inter-active learning of ad-hoc classifiers for video visual analytics. In: Visual analytics science and technology (VAST). IEEE, pp 23–32

  • Kohonen T (1990) The self-organizing map. Proc IEEE 78(9):1464–1480

    Article  Google Scholar 

  • Lin H, Gao S, Gotz D, Du F, He J, Cao N (2018) Rclens: interactive rare category exploration and identification. IEEE Trans Vis Comput Graph 24(7):2223–2237

    Article  Google Scholar 

  • Livnat Y, Agutter J, Moon S, Foresti S (2005) Visual correlation for situational awareness. In: IEEE Symposium on information visualization, 2005. INFOVIS 2005. IEEE, pp 95–102

  • Lvd M, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9(Nov):2579–2605

    MATH  Google Scholar 

  • Mansmann F, Keim DA, North SC, Rexroad B, Sheleheda D (2007) Visual analysis of network traffic for resource planning, interactive monitoring, and interpretation of security threats. IEEE Trans Vis Comput Graph 13(6):1105–1112

    Article  Google Scholar 

  • McPherson J, Ma K-L, Krystosk P, Bartoletti T, Christensen M (2004) Portvis: a tool for port-based detection of security events. In:Proceedings of the 2004 ACM workshop on Visualization and data mining for computer security. ACM, New York, pp 73–81

  • Mukherjee B, Heberlein LT, Levitt KN (1994) Network intrusion detection. IEEE Netw 8(3):26–41

    Article  Google Scholar 

  • Scheffer T, Decomain C, Wrobel S (2001) Active hidden Markov models for information extraction. In: International symposium on intelligent data analysis. Springer, Berlin, pp 309–318

    Chapter  Google Scholar 

  • Settles B (2009) Active learning literature survey. Technical report, University of Wisconsin-Madison Department of Computer Sciences

  • Settles B, Craven M (2008) An analysis of active learning strategies for sequence labeling tasks. In: Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, pp 1070–1079

  • Shi R, Yang M, Zhao Y, Zhou F, Huang W, Zhang S (2015) A matrix-based visualization system for network traffic forensics. IEEE Syst J 10(4):1350–1360

    Article  Google Scholar 

  • Shi Y, Zhao Y, Zhou F, Shi R, Zhang Y (2018) A novel radial visualization of intrusion detection alerts. IEEE Comput Graph Appl 38(6):83–95

    Article  Google Scholar 

  • Shiravi H, Shiravi A, Ghorbani AA (2012) A survey of visualization systems for network security. IEEE Trans Vis Comput Graph 18(8):1313–1329

    Article  Google Scholar 

  • Sommer R, Paxson V (2010) Outside the closed world: on using machine learning for network intrusion detection. In: 2010 IEEE symposium on security and privacy (SP). IEEE, pp 305–316

  • Sudo A, Sato A, Hasegawa O (2009) Associative memory for online learning in noisy environments using self-organizing incremental neural network. IEEE Trans Neural Netw 20(6):964–972

    Article  Google Scholar 

  • Velea R, Ciobanu C, Gurzau F, Patriciu V-V (2017) Feature extraction and visualization for network PCAPNG traces. In: 2017 21st international conference on control systems and computer science (CSCS). IEEE, pp 311–316

  • Yousefi-Azar M, Varadharajan V, Hamey L, Tupakula U (2017) Autoencoder-based feature learning for cyber security applications. In: 2017 international joint conference on neural networks (IJCNN). IEEE, pp 3854–3861

  • Zhang S, Fung C, Huang S, Luan Z, Qian D (2017) Psom: periodic self-organizing maps for unsupervised anomaly detection in periodic time series. In: 2017 IEEE/ACM 25th international symposium on quality of service (IWQoS). IEEE, pp 1–6

  • Zhao Y, Liang X, Fan X, Wang Y, Yang M, Zhou F (2014) MVSec: multi-perspective and deductive visual analytics on heterogeneous network security data. J Vis 17(3):181–196

    Article  Google Scholar 

  • Zhao Y, Luo F, Chen M, Wang Y, Xia J, Zhou F, Wang Y, Chen Y, Chen W (2018) Evaluating multi-dimensional visualizations for understanding fuzzy clusters. IEEE Trans Vis Comput Graph 25(1):12–21

    Article  Google Scholar 

Download references

Acknowledgements

Supported by National Key Research and Development Program of China (Grant Nos. 2016QY02D0304, 2017YFB0701900), National Nature Science Foundation of China (Grant No. 61100053), and Key Laboratory of Machine Perception in Peking University (Grant No. K-2019-09).

Author information

Authors and Affiliations

  1. BASICS, Department of Computer Science and Engineering, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China

    Xin Fan, Chenlu Li & Xiaoju Dong

  2. Key Laboratory of Machine Perception (Ministry of Education), and National Engineering Laboratory for Big Data Analysis and Application, Peking University, Beijing, China

    Xiaoru Yuan

  3. University of Technology Sydney, Ultimo, Australia

    Jie Liang

Authors
  1. Xin Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chenlu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoru Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoju Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jie Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoju Dong.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, X., Li, C., Yuan, X. et al. An interactive visual analytics approach for network anomaly detection through smart labeling. J Vis 22, 955–971 (2019). https://doi.org/10.1007/s12650-019-00580-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12650-019-00580-7

Keywords

Search

Navigation