Optimal Bounds for Estimating Entropy with PMF Queries

Abstract

Let p be an unknown probability distribution on $[n] := \{1, 2, \dots n\}$ that we can access via two kinds of queries: A SAMP query takes no input and returns $x \in [n]$ with probability p[x]; a PMF query takes as input $x \in [n]$ and returns the value p[x]. We consider the task of estimating the entropy of p to within $\pm \varDelta$ (with high probability). For the usual Shannon entropy H(p), we show that $\varOmega (\log ^2 n/\varDelta ^2)$ queries are necessary, matching a recent upper bound of Canonne and Rubinfeld. For the Rényi entropy $H_\alpha$(p), where $\alpha >1$, we show that $\varTheta (n^{1-1/\alpha })$ queries are necessary and sufficient. This complements recent work of Acharya et al. in the $\mathsf SAMP$-only model that showed $O(n^{1-1/\alpha })$ queries suffice when $\alpha$ is an integer, but $\widetilde{\varOmega }(n)$ queries are necessary when $\alpha$ is a noninteger. All of our lower bounds also easily extend to the model where CDF queries (given x, return $\sum _{y \le x}$ p[y]) are allowed.

R. O’Donnell—Work performed while the author was at the Boğaziçi University Computer Engineering Department, supported by Marie Curie International Incoming Fellowship project number 626373.