Intrusion Detection in High-Speed Big Data Networks: A Comprehensive Approach

  • Kamran Siddique
  • Zahid Akhtar
  • Yangwoo KimEmail author
Conference paper
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 474)


In network intrusion detection research, two characteristics are generally considered vital to build efficient intrusion detection systems (IDSs) namely, optimal feature selection technique and robust classification schemes. However, an emergence of sophisticated network attacks and the advent of big data concepts in anomaly detection domain require the need to address two more significant aspects. They are concerned with employing appropriate big data computing framework and utilizing contemporary dataset to deal with ongoing advancements. Based on this need, we present a comprehensive approach to build an efficient IDS with the aim to strengthen academic anomaly detection research in real-world operational environments. The proposed system is a representative of the following four characteristics: It (i) performs optimal feature selection using branch-and-bound algorithm; (ii) employs logistic regression for classification; (iii) introduces bulk synchronous parallel processing to handle computational requirements of large-scale networks; and (iv) utilizes real-time contemporary dataset named ISCX-UNB to validate its efficacy.


Anomaly detection Network intrusion detection systems Bulk synchronous parallel BSP Big data ISCX-UNB dataset Darpa KDD Cup ’99 


  1. 1.
    Buczak, A.L., Guven, E.: A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Commun. Surv. Tutorials 18(2), 1153–1176 (2016)CrossRefGoogle Scholar
  2. 2.
    Suthaharan, S.: Big data classification: Problems and challenges in network intrusion prediction with machine learning. ACM SIGMETRICS Perform. Eval. Rev. 41(4), 70–73 (2014)CrossRefGoogle Scholar
  3. 3.
    Grahn, K., Westerlund, M., Pulkkis, G.: Analytics for network security: a survey and taxonomy. In: Information Fusion for Cyber-Security Analytics, pp. 175–193. Springer (2017)Google Scholar
  4. 4.
    Manzoor, M.A., Morgan, Y.: Network intrusion detection system using apache storm. Adv. Sci. Technol. Eng. Syst. J. 2(3), 812–818 (2017)CrossRefGoogle Scholar
  5. 5.
    Rathore, M.M., Ahmad, A., Paul, A.: Real time intrusion detection system for ultra-high-speed big data environments. J. Supercomputing 72(9), 3489–3510 (2016)CrossRefGoogle Scholar
  6. 6.
    Anderson, J.P.: Computer security threat monitoring and surveillance. vol. 17. Technical report, James P. Anderson Company, Fort Washington, Pennsylvania (1980)Google Scholar
  7. 7.
    Shiravi, A., et al.: Toward developing a systematic approach to generate benchmark datasets for intrusion detection. Comput. Secur. 31(3), 357–374 (2012)CrossRefGoogle Scholar
  8. 8.
    Liu, H.: Instance Selection and Construction for Data Mining (2010)Google Scholar
  9. 9.
    Hosmer Jr., D.W., Lemeshow, S., Sturdivant, R.X.: Applied Logistic Regression, vol. 398. Wiley, New York (2013)CrossRefGoogle Scholar
  10. 10.
    Sokolova, M., Lapalme, G.: A systematic analysis of performance measures for classification tasks. Inf. Process. Manag. 45(4), 427–437 (2009)CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  1. 1.Dongguk UniversitySeoulRepublic of Korea
  2. 2.INRS-EMTUniversity of QuebecQuebec CityCanada

Personalised recommendations