Abstract
Counting the number of flows present in network traffic is not trivial, given that the naive approach of using a hash table to track the active flows is too slow for the current backbone network speeds. Several algorithms have been proposed in the recent literature that can calculate an approximate count using small amount of memory and few memory accesses per packet. Fewer works have addressed the more complex problem of counting flows over sliding windows, where the main challenge is to continuously expire old information. One of the existing proposals is a straightforward adaptation of the direct bitmaps technique to the sliding window model. We present an algorithm called Countdown Vector that also builds upon the direct bitmaps technique. Our algorithm, however, obtains significant cost reductions both in terms of memory and CPU, by introducing an extra approximation in the mechanism in charge of the expiration of old information.
Chapter PDF
Similar content being viewed by others
References
Estan, C., Varghese, G., Fisk, M.: Bitmap algorithms for counting active flows on high speed links. In: Proc. of ACM SIGCOMM Internet Measurement Conf. (October 2003)
Fusy, E., Giroire, F.: Estimating the number of Active Flows in a Data Stream over a Sliding Window. In: Proc. of SIAM Workshop on Analytic Algorithmics and Combinatorics (January 2007)
Kim, H., O’Hallaron, D.: Counting network flows in real time. In: Proc. of IEEE GLOBECOM (December 2003)
Fang, W., Peterson, L.: Inter-AS traffic patterns and their implications. In: Proc. of IEEE GLOBECOM (December 1999)
Cisco Systems: NetFlow services and applications. White Paper (2000)
Barlet-Ros, P., Iannaccone, G., Sanjuàs-Cuxart, J., Amores-López, D., Solé-Pareta, J.: Load shedding in network monitoring applications. In: Proc. of USENIX Annual Technical Conf. (June 2007)
Duffield, N., Lund, C., Thorup, M.: Properties and prediction of flow statistics from sampled packet streams. In: Proc. of ACM SIGCOMM Internet Measurement Workshop (November 2002)
Whang, K.Y., Vander-Zanden, B.T., Taylor, H.M.: A linear-time probabilistic counting algorithm for database applications. ACM Trans. Database Syst. 15(2) (June 1990)
Durand, M., Flajolet, P.: Loglog Counting of Large Cardinalities. In: Proc. of Annual European Symposium on Algorithms (September 2003)
Giroire, F.: Order statistics and estimating cardinalities of massive data sets. In: Proc. of Intl. Conf. on Analysis of Algorithms (June 2005)
Metwally, A., Agrawal, D., El Abbadi, A.: Why go logarithmic if we can go linear?: Towards effective distinct counting of search traffic. In: Proc. of Intl. Conf. on Extending Database Technology: Advances in Database Technology (March 2008)
Babcock, B., Babu, S., Datar, M., Motwani, R., Widom, J.: Models and issues in data stream systems. In: Proc. of ACM SIGMOD-SIGACT-SIGART Symp. on Principles of Database Systems (June 2002)
Golab, L., Özsu, T.M.: Issues in data stream management. SIGMOD Record 32 (June 2003)
Cranor, C., Johnson, T., Spataschek, O., Shkapenyuk, V.: Gigascope: A stream database for network applications. In: Proc. of ACM SIGMOD (June 2003)
Iannaccone, G.: Fast prototyping of network data mining applications. In: Proc. of Passive and Active Measurement Conf. (March 2006)
Reiss, F., Hellerstein, J.M.: Declarative network monitoring with an underprovisioned query processor. In: Proc. of IEEE Intl. Conf. on Data Engineering (April 2006)
Muthukrishnan, S.: Data Streams: Algorithms And Applications. Now Publishers Inc. (2005)
Hayes, B.: The Britney Spears Problem, http://www.americanscientist.org/issues/pub/the-britney-spears-problem
Datar, M., Gionis, A., Indyk, P., Motwani, R.: Maintaining stream statistics over sliding windows. In: Proc. of ACM-SIAM Symp. on Discrete Algorithms (January 2002)
Carter, J.L., Wegman, M.N.: Universal classes of hash functions. J. Comput. Syst. Sci. 18 (April 1979)
Jacobson, V., Leres, C., McCanne, S. (libpcap) Lawrence Berkeley Laboratory, Berkeley, CA. Initial public release (June 1994), http://www.tcpdump.org
Endace: DAG network monitoring cards, http://www.endace.com
Golab, L., DeHaan, D., Demaine, E., Lopez-Ortiz, A., Munro, J.: Identifying frequent items in sliding windows over on-line packet streams. In: Proc. of ACM SIGCOMM Internet Measurement Conf. (October 2003)
Sanjuàs-Cuxart, J., Barlet-Ros, P., Solé-Pareta, J.: Counting network flows over sliding windows in high-speed networks. UPC Technical Report, http://loadshedding.ccaba.upc.edu/papers/counting-network-flows.techrep2008.pdf
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 IFIP International Federation for Information Processing
About this paper
Cite this paper
Sanjuàs-Cuxart, J., Barlet-Ros, P., Solé-Pareta, J. (2009). Counting Flows over Sliding Windows in High Speed Networks. In: Fratta, L., Schulzrinne, H., Takahashi, Y., Spaniol, O. (eds) NETWORKING 2009. NETWORKING 2009. Lecture Notes in Computer Science, vol 5550. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-01399-7_7
Download citation
DOI: https://doi.org/10.1007/978-3-642-01399-7_7
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-01398-0
Online ISBN: 978-3-642-01399-7
eBook Packages: Computer ScienceComputer Science (R0)