Finding a Duplicate and a Missing Item in a Stream

Abstract

We consider the following problem in a stream model: Given a sequence $a= \langle a_1,a_2,\cdot,a_m \langle$ wich each a _i  ∈ [n] = {1,...,n} and m > n, find a duplicate in the sequence, i.e., find some d = a _i  = a _l with i ≠ l by using limited s bits of memory and r passes over the input sequence. In one pass an algorithm reads the input sequence a in the order a ₁, a ₂, ..., a _m . Since m > n, a duplicate exists by the pigeonhole principle. Muthukrishnan [Mu05a], [Mu05b] has posed the following question for the case where m = n + 1: For s = O(logn), is there a solution with a constant number of passes? We have described the problem generalizing Muthukrishnan’s question by taking the sequence length m as a parameter. We give a negative answer to the original question by showing the following: Assume that m = n + 1. A streaming algorithm with O(logn) space requires Ω(logn/loglogn) passes; a k-pass streaming algorithm requires Ω(n ^{1/(2k − 1)}) space. We also consider the following problem of finding a missing item: Assuming that n < m, find x ∈ [m] such that x ≠ a _j for 1 ≤ j ≤ n. The same lower bound applies for the missing-item finding problem. The proof is a simple reduction to the communication complexity of a relation. We also consider one-pass algorithms and exactly determine the minimum space required. Interesting open questions such as the following remain. For the number of passes of algorithms using O(logn) space, show an ω(1) lower bound (or an O(1) upper bound) for: (1) duplicate finding for m = 2n, (2) missing-item finding for m = 2n, and (3) the case where we allow Las-Vegas type randomization for m = n + 1.