Count Me In: Viable Distributed Summary Statistics for Securing High-Speed Networks

Abstract

Summary statistics represent a key primitive for profiling and protecting operational networks. Many network operators routinely measure properties such as throughput, traffic mix, and heavy hitters. Likewise, security monitoring often deploys statistical anomaly detectors that trigger, e.g., when a source scans the local IP address range, or exceeds a threshold of failed login attempts. Traditionally, a diverse set of tools is used for such computations, each typically hard-coding either the features it operates on or the specific calculations it performs, or both. In this work we present a novel framework for calculating a wide array of summary statistics in real-time, independent of the underlying data, and potentially aggregated from independent monitoring points. We focus on providing a transparent, extensible, easy-to-use interface and implement our design on top of an open-source network monitoring system. We demonstrate a set of example applications for profiling and statistical anomaly detection that would traditionally require significant effort and different tools to compute. We have released our implementation under BSD license and report experiences from real-world deployments in large-scale network environments.