On Algorithmic Statistics for Space-Bounded Algorithms

Abstract

Algorithmic statistics studies explanations of observed data that are good in the algorithmic sense: an explanation should be simple i.e. should have small Kolmogorov complexity and capture all the algorithmically discoverable regularities in the data. However this idea can not be used in practice because Kolmogorov complexity is not computable.

In this paper we develop algorithmic statistics using space-bounded Kolmogorov complexity. We prove an analogue of one of the main result of ‘classic’ algorithmic statistics (about the connection between optimality and randomness deficiences). The main tool of our proof is the Nisan-Wigderson generator.

Appendix

Symmetry of Information

Define ${{\mathrm{\mathrm {CD}}}}^m(A,B)$ as the minimal length of a program that inputs a pair of strings (a, b) and outputs a pair of boolean values $(a \in A, b \in B)$ using space at most m for every input.

Lemma 4

(Symmetry of information). Assume $A, B \subseteq \{0,1\}^n$. Then

$$ (a) \forall m \text { } {{\mathrm{\mathrm {CD}}}}^p(A, B) \le {{\mathrm{\mathrm {CD}}}}^m(A) + {{\mathrm{\mathrm {CD}}}}^m(B |A) + O(\log ({{\mathrm{\mathrm {CD}}}}^m(A,B)+m + n))$$

for $p = m + \text {poly}(n + {{\mathrm{\mathrm {CD}}}}^m(A,B))$.

$$ (b) \forall m \text { } {{\mathrm{\mathrm {CD}}}}^p(A) + {{\mathrm{\mathrm {CD}}}}^p(B |A) \le {{\mathrm{\mathrm {CD}}}}^m(A, B) + O(\log ({{\mathrm{\mathrm {CD}}}}^m(A,B)+m + n) )$$

for $p = 2m + \text {poly}(n + {{\mathrm{\mathrm {CD}}}}^m(A,B))$.

Proof

(of Lemma 4(a)). The proof is similar to the proof of Theorem 4(a).

Proof

(of Lemma 4(b)). Let $k:= {{\mathrm{\mathrm {CD}}}}^m(A, B)$. Denote by $\mathcal {D}$ the family of sets (U, V) such that ${{\mathrm{\mathrm {CD}}}}^m(U,V) \le k$ and $U,V \subseteq \{0,1\}^n$. It is clear that $|\mathcal {D}| < 2^{k+1}$. Denote by $\mathcal {D}_{A}$ the pairs of $\mathcal {D}$ whose the first element is equal to A. Let t satisfy the inequalities $2^t \le |\mathcal {D}_{A}| < 2^{t+1}$.

Let us prove that

• ${{\mathrm{\mathrm {CD}}}}^p(B |A)$ does not exceed t significantly;

• ${{\mathrm{\mathrm {CD}}}}^p(A)$ does not exceed $k - t$ significantly.

Here $p=m + O(n)$.

We start with the first statement. There exists a program that enumerates all sets from $\mathcal {D}_{A}$ using A as an oracle and that works in space $2m + O(n)$. Indeed, such enumeration can be done in the following way: enumerate all programs of length k and verify the following condition for every pair of n-bit strings. First, a program uses at most m space on this input. Second, if a second n-bit string belongs to A then the program outputs 1, and 0 otherwise. Since some program loops we need additional $m + O(n)$ space to take it into account.

Append to this program the ordinal number of a program that distinguishes (A, B). This number is not greater than $t+1$. Therefore we have ${{\mathrm{\mathrm {CD}}}}^p(B |A) \le t + O(\log ({{\mathrm{\mathrm {CD}}}}^m(A,B) + m + n))$.

Now let us prove the second statement. Note that there exist at most $2^{k-t +1}$ sets U such that $|\mathcal {D}_U| \ge 2^t$ (including A). Hence, if we construct a program that enumerates all sets with such property (and does not use much space) then we will win—the set A can be described by the ordinal number of this enumeration.

Let us construct such a program. It works as follows:

enumerate all sets U that are the first elements from $\mathcal {D}$, i.e. we enumerate programs that distinguish the corresponding sets (say, lexicographically). We go to the next step if the following properties holds. First, $|\mathcal {D}_U| \ge 2^t$, and second: we did not meet set U earlier (i.e. every program whose the lexicographical number is smaller does not distinguish U or is not the first element from a set from $\mathcal {D}$).

This program works in $2m + \text {poly}(n + {{\mathrm{\mathrm {CD}}}}^m(A,B))$ space (that we want) and has length $O(\log ({{\mathrm{\mathrm {CD}}}}^m(A)+ n +m))$.

Proof

(of Lemma 3 ). Let us show that $\mathcal {B}$ satisfies property $(1)^*$ with probability at most $2^{-n}$. Since $\mathcal {B}$ satisfies property (2) with probability at most $\frac{1}{4}$ (see the proof of Lemma 2) it would be enough for us.

For this let us show that every part is ‘bad’ (i.e. has at least $(n + k)^2 + 1$ sets from $\mathcal {B}$) with probability at most $2^{-2n}$. The probability of such event is equal to the probability of the following event: a binomial random variable with parameters $(2^k, 2^{-k}(n + 2)\ln 2)$ is greater than $(n + k)^2$. To get the needed upper bound for this probability is not difficult however the correspondent formulas are cumbersome. Take $w:=2^k$, $p:=2^{-k}(n + 2)\ln 2$ and $v:=(n + k)^2$. We need to estimate

$$\sum _{i=v}^{w} {{w}\atopwithdelims (){i}} p^i(1-p)^{w-i}< w \cdot {{w}\atopwithdelims (){v}} p^v(1-p)^{w-v}< w \cdot {{w}\atopwithdelims (){v}} p^v < w \frac{(wp)^{v}}{v!}.$$

The first inequality holds since $wp = (n+2) \ln 2 \le (n+k)^2 = v$. Now note that $wp= (n+2) \ln 2 < 10 n$. So

$$w \frac{(wp)^{v}}{v!} < \frac{2^k (10n)^{(n+k)^2}}{((n+k)^2)!} \ll 2^{-2n}. $$