Approximate Counting with a Floating-Point Counter

Abstract

When many objects are counted simultaneously in large data streams, as in the course of network traffic monitoring, or Webgraph and molecular sequence analyses, memory becomes a limiting factor. Robert Morris [Communications of the ACM, 21:840–842, 1978] proposed a probabilistic technique for approximate counting that is extremely economical. The basic idea is to increment a counter containing the value X with probability 2^− X. As a result, the counter contains an approximation of $\lg n$ after n probabilistic updates, stored in $\lg\lg n$ bits. Here we revisit the original idea of Morris. We introduce a binary floating-point counter that combines a d-bit significand with a binary exponent, stored together on $d+\lg\lg n$ bits. The counter yields a simple formula for an unbiased estimation of n with a standard deviation of about 0.6·n2^− d/2.

We analyze the floating-point counter’s performance in a general framework that applies to any probabilistic counter. In that framework, we provide practical formulas to construct unbiased estimates, and to assess the asymptotic accuracy of any counter.