Feature Extraction in Security Analytics: Reducing Data Complexity with Apache Spark

  • Dimitrios Sisiaridis
  • Olivier Markowitch
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 733)


Feature extraction is the first task of pre-processing input logs in order to detect cybersecurity threats and attacks while utilizing machine learning. When it comes to the analysis of heterogeneous data derived from different sources, this task is found to be time-consuming and difficult to be managed efficiently. In this paper we present an approach for handling feature extraction for security analytics of heterogeneous data derived from different network sensors. The approach is implemented in Apache Spark, using its python API, named pyspark.


Machine learning Feature extraction Security analytics Apache Spark 


  1. 1.
    Bird, S., Klein, E., Loper, E.: Natural Language Processing with Python. O’ Reilly Media Inc. (2009)Google Scholar
  2. 2.
    Hutchins, E.M., Cloppert, M.J., Amin, R.M.: Intelligence-driven computer network defense informed by analysis of adversary campaigns and intrusion kill chains. In: Ryan, J. (ed.) Leading Issues in Information Warfare and Security Research, vol. 1, p. 80. Academic Publishing International Ltd., Reading (2011)Google Scholar
  3. 3.
    Kalyan, V., Ignacio, A., Alfredo, C.-I., Vamsi, K., Costas, B., Ke, L.: AI2: Training a big data machine to defend. In: IEEE International Conference on Big Data Security, New York, NY, USA, June 2016Google Scholar
  4. 4.
    Shyu, M.-L., Huang, Z., Luo, H.: Efficient mining and detection of sequential intrusion patterns for network intrusion detection systems. In: Yu, P.S., Tsai, J.J.P. (eds.) Machine Learning in Cyber Trust, pp. 133–154. Springer, Boston (2009)CrossRefGoogle Scholar
  5. 5.
    Sisiaridis, D., Carcillo, F., Markowitch, O.: A framework for threat detection in communication systems. In: Proceedings of the 20th Pan-Hellenic Conference on Informatics, pp. 68:1–68:6. ACM (2016)Google Scholar
  6. 6.
    Sisiaridis, D., Kuchta, V., Markowitch, O.: A categorical approach in handling event-ordering in distributed systems. In: Parallel and Distributed Systems (ICPADS), pp. 1145–1150. IEEE (2016)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.QualSec Group, Departement d’ InformatiqueUniversité Libre de BruxellesBrusselsBelgium

Personalised recommendations