Encyclopedia of Big Data Technologies

Living Edition
| Editors: Sherif Sakr, Albert Zomaya

Big Data in Computer Network Monitoring

  • Idilio Drago
  • Marco Mellia
  • Alessandro D’Alconzo
Living reference work entry
DOI: https://doi.org/10.1007/978-3-319-63962-8_26-1



Network monitoring applications (e.g., anomaly detection and traffic classification) are among the first sources of big data. With the advent of algorithms and frameworks able to handle datasets of unprecedented scales, researchers and practitioners have the opportunity to face network monitoring problems with novel data-driven approaches. This section summarizes the state of the art on the use of big data approaches for network monitoring. It describes why network monitoring is a big data problem and how the big data approaches are assisting on network monitoring tasks. Open research directions are then highlighted.

Network Monitoring: Goals and Challenges

Monitoring and managing the Internet is more fundamental than ever, since the critical services that rely on the Internet to operate are growing day by day. Monitoring helps administrators to guarantee that the network is working as expected as well as...

This is a preview of subscription content, log in to check access


  1. Akidau T, Bradshaw R, Chambers C, Chernyak S, Fernández-Moctezuma RJ, Lax R, McVeety S, Mills D, Perry F, Schmidt E, Whittle S (2015) The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. Proc VLDB Endow 8(12):1792–1803CrossRefGoogle Scholar
  2. Apache Beam (2017) Apache Beam: an advanced unified programming model. https://beam.apache.org/
  3. Apache Spot (2017) A community approach to fighting cyber threats. http://spot.incubator.apache.org/
  4. Bär A, Finamore A, Casas P, Golab L, Mellia M (2014) Large-scale network traffic monitoring with DBStream, a system for rolling big data analysis. In: Proceedings of the BigData, pp 165–170Google Scholar
  5. Bhuyan MH, Bhattacharyya DK, Kalita JK (2013) Network anomaly detection: methods, systems and tools. Commun Surv Tutorials 16(1):303–336CrossRefGoogle Scholar
  6. Callado A, Kamienski C, Szabó G, Gero BP, Kelner J, Fernandes S, Sadok D (2009) A survey on internet traffic identification. Commun Surv Tutorials 11(3): 37–52CrossRefGoogle Scholar
  7. Casas P, D’Alconzo A, Zseby T, Mellia M (2016) Big-DAMA: big data analytics for network traffic monitoring and analysis. In: Proceedings of the LANCOMM, pp 1–3Google Scholar
  8. Čermák M, Jirsík T, Laštovička M (2016) Real-time analysis of NetFlow data for generating network traffic statistics using Apache Spark. In: Proceedings of the NOMS, pp 1019–1020Google Scholar
  9. Dean J, Ghemawat S (2004) MapReduce: simplified data processing on large clusters. In: Proceedings of the OSDI, pp 10–10Google Scholar
  10. Fayyad UM, Piatetsky-Shapiro G, Smyth P (1996) From data mining to knowledge discovery: an overview. AI Mag 17(3):37–54Google Scholar
  11. Fontugne R, Mazel J, Fukuda K (2014) Hashdoop: a mapreduce framework for network anomaly detection. In: Proceedings of the INFOCOM WKSHPS, pp 494–499Google Scholar
  12. García-Teodoro P, Díaz-Verdejo J, Maciá-Fernández G, Vázquez E (2009) Anomaly-based network intrusion detection: techniques, systems and challenges. Comput Secur 28:18–28CrossRefGoogle Scholar
  13. Hofstede R, Čeleda P, Trammell B, Drago I, Sadre R, Sperotto A, Pras A (2014) Flow monitoring explained: from packet capture to data analysis with NetFlow and IPFIX. Commun Surv Tutorials 16(4):2037–2064CrossRefGoogle Scholar
  14. Lee Y, Lee Y (2013) Toward scalable internet traffic measurement and analysis with hadoop. SIGCOMM Comput Commun Rev 43(1):5–13CrossRefGoogle Scholar
  15. Liu J, Liu F, Ansari N (2014) Monitoring and analyzing big traffic data of a large-scale cellular network with hadoop. IEEE Netw 28(4):32–39CrossRefGoogle Scholar
  16. Marchal S, Jiang X, State R, Engel T (2014) A big data architecture for large scale security monitoring. In: Proceedings of the BIGDATACONGRESS, pp 56–63Google Scholar
  17. Nguyen TT, Armitage G (2008) A survey of techniques for internet traffic classification using machine learning. Commun Surv Tutorials 10(4):56–76CrossRefGoogle Scholar
  18. Orsini C, King A, Giordano D, Giotsas V, Dainotti A (2016) BGPStream: a software framework for live and historical BGP data analysis. In: Proceedings of the IMC, pp 429–444Google Scholar
  19. Sperotto A, Schaffrath G, Sadre R, Morariu C, Pras A, Stiller B (2010) An overview of IP flow-based intrusion detection. Commun Surv Tutorials 12(3):343–356CrossRefGoogle Scholar
  20. Trevisan M, Finamore A, Mellia M, Munafo M, Rossi D (2017) Traffic analysis with off-the-shelf hardware: challenges and lessons learned. IEEE Commun Mag 55(3):163–169CrossRefGoogle Scholar
  21. Valenti S, Rossi D, Dainotti A, Pescapè A, Finamore A, Mellia M (2013) Reviewing traffic classification. In: Data traffic monitoring and analysis – from measurement, classification, and anomaly detection to quality of experience, 1st edn. Springer, HeidelbergGoogle Scholar
  22. Vanerio J, Casas P (2017) Ensemble-learning approaches for network security and anomaly detection. In: Proceedings of the Big-DAMA, pp 1–6Google Scholar
  23. Wang Y, Ke W, Tao X (2016) A feature selection method for large-scale network traffic classification based on spark. Information 7(1):6CrossRefGoogle Scholar
  24. Wullink M, Moura GCM, Müller M, Hesselman C (2016) ENTRADA: a high-performance network traffic data stream. In: Proceedings of the NOMS, pp 913–918Google Scholar
  25. Zaharia M, Chowdhury M, Franklin MJ, Shenker S, Stoica I (2010) Spark: cluster computing with working sets. In: Proceedings of the HotCloud, pp 10–10Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Idilio Drago
    • 1
  • Marco Mellia
    • 1
  • Alessandro D’Alconzo
    • 2
  1. 1.Politecnico di TorinoTurinItaly
  2. 2.Austrian Institute of TechnologyViennaAustria

Section editors and affiliations

  • Kamran Munir
    • 1
  • Antonio Pescapè
    • 2
  1. 1.Computer Science and Creative TechnologiesUniversity of the West of EnglandBristolUnited Kingdom
  2. 2.Department of Electrical Engineering and Information TechnologyUniversity of Napoli Federico IINapoliItaly