Estimating Frequency Moments of Data Streams Using Random Linear Combinations

Abstract

The problem of estimating the k ^th frequency moment F _k for any non-negative k, over a data stream by looking at the items exactly once as they arrive, was considered in a seminal paper by Alon, Matias and Szegedy [1,2]. The space complexity of their algorithm is $\tilde{O}(n^{1-\frac{1}{k}})$. For k > 2, their technique does not apply to data streams with arbitrary insertions and deletions. In this paper, we present an algorithm for estimating F _k for k > 2, over general update streams whose space complexity is $\tilde{O}(n^{1-\frac{1}{k-1}})$ and time complexity of processing each stream update is $\tilde{O}(1)$.

Recently, an algorithm for estimating F _k over general update streams with similar space complexity has been published by Coppersmith and Kumar [7]. Our technique is, (a) basically different from the technique used by [7], (b) is simpler and symmetric, and, (c) is more efficient in terms of the time required to process a stream update $(\tilde{O}(1)$ compared with $\tilde{O}(n^{1-\frac{1}{k-1}})$).