An information-theoretic approach to curiosity-driven reinforcement learning

Abstract

We provide a fresh look at the problem of exploration in reinforcement learning, drawing on ideas from information theory. First, we show that Boltzmann-style exploration, one of the main exploration methods used in reinforcement learning, is optimal from an information-theoretic point of view, in that it optimally trades expected return for the coding cost of the policy. Second, we address the problem of curiosity-driven learning. We propose that, in addition to maximizing the expected return, a learner should choose a policy that also maximizes the learner’s predictive power. This makes the world both interesting and exploitable. Optimal policies then have the form of Boltzmann-style exploration with a bonus, containing a novel exploration–exploitation trade-off which emerges naturally from the proposed optimization principle. Importantly, this exploration–exploitation trade-off persists in the optimal deterministic policy, i.e., when there is no exploration due to randomness. As a result, exploration is understood as an emerging behavior that optimizes information gain, rather than being modeled as pure randomization of action choices.

Appendix

Clever random policy

There are two world states, $x \in \{0,1\}$ and a continuous action set, $a \in [ 0,1 ].$ The value of the action sets how strongly the agent tries to stay in or leave a state, and $p({\bar x}|x,a) = a.$ The interest in reward is switched off ($\alpha = 0$), so that the optimal action becomes the one that maximizes only the predictive power.

• Policies that maximize $I[X_{t+1}, \{{X_t,A_t}\}]$

For brevity of notation, we drop the index t for the current state and action.

$$ I [ X_{t+1}, \{X,A\} ] = H [ X_{t+1} ] - H [ X_{t+1}|X,A ] $$

(26)

The second term in (24) is minimized and equal to zero for all policies that result in deterministic world transitions. Those are all policies for which $\pi(\tilde{a}|x) = 0$ for all $\tilde{a} \notin \{0, 1\}.$ This limits the agent to using only two (the most extreme) actions: $a \in \{ 0, 1\}.$ Since we have only two states, policies in this class are determined by two probabilities, for example the flip probabilities π(A = 0|X = 1) and π(A = 1|X = 0).

The first term in Eq. (26) is maximized for p(X _t+1 = 1) = p(X _t+1 = 0) = 1/2. Setting p(X _t+1 = 1) to 1/2 yields

$$ \pi(A=0|X=1) p(X=1) + \pi(A=1|X=0) p(X=0) = \frac{1}{2}. $$

(27)

We assume that p(X = 0) is estimated by the learner. Equation 27 is true for all values of p(X = 0), if π(A = 0|X = 1) = π(A = 1|X = 0) = 1/2. We call this the “clever random” policy (π _R ). The agent uses only those actions that make the world transitions deterministic, and uses them at random, i.e. it explores within the subspace of actions that make the world deterministic. This policy maximizes $I [ X_{t+1}, \{X,A\} ],$ independent of the estimated value of p(X = 0).

However, when stationarity holds, p(X = 0) = p(X = 1) = 1/2, then all policies for which

$$ \pi(A=0|X=1) = \pi(A=0|X=0) $$

(28)

maximize I[X _t+1, {X, A}]. Those include “STAY-STAY”, and “FLIP-FLIP”.

• Self-consistent policies.

Since α = 0, the term in the exponent of Eq. (21), for a given state x and action a, is:

$$ {\cal D}^{\pi}(x,a)= -H [ a ] + a \log\left[\frac{p(X_{t+1} = x)}{p(X_{t+1} = \bar{x})}\right] - \log [ p(X_{t+1} = x) ] $$

(29)

with $\bar{x}$ being the opposite state, and $H [ a ] = - (a\log(a) + (1-a)\log(1-a)).$ Note that H [0] = H [1] = 0. The clever random policy π _R is self-consistent, because under this policy, for all x, both actions, STAY (a = 0) and FLIP (a = 1) are equally likely. This is due to the fact that $p(X_{t+1} = x) = p(X_{t+1} = \bar{x}) =1/2,$ hence $D^{\pi_R} (x,0) = D^{\pi_R} (x,1), \forall x.$ If stationarity holds, p(X = 0) = 1/2, and no policy which uses only actions $a \in \{ 0, 1\}$ other than policy π _R is self consistent. This is because under other such policies we also have that $p(X_{t+1} = x) = p(X_{t+1} = \bar{x}) = 1/2,$ and we have H[0] = H[1] = 0, and therefore $D^{\pi} (x,0) - D^{\pi} (x,1) = 0.$ This means that the algorithm gets to π _R after one iteration. We can conclude that π _R is the unique optimal self-consistent solution.

A reliable and an unreliable state

There are two possible actions, STAY (s) or FLIP (f), and two world states, $x \in \{0,1\},$ distinguished by the transitions: p(X _t+1 = 0|X _t  = 0, A _t  = s) = p(X _t+1 = 1|X _t  = 0, A _t  = f) = 1, while $p(X_{t+1}=x|X_t=1,a) = 1/2, \forall x, \forall a.$ In other words, state 0 is fully reliable, and state 1 is fully unreliable, in terms of the action effects. There is no uncertainty when we start in the reliable state, and the uncertainty when starting in the unreliable state is exactly one bit. The predictive power is then given by

$$ I [ X_{t+1}, \{X,A\} ] = - \sum_{x \in \{0,1\}} p(X_{t+1} = x)\log_2 [ p(X_{t+1} = x) ] - p(X_t = 1) $$

(30)

Starting with a fixed value for p(X _t  = 1) which is estimated from past experiences, the maximum is reached by a policy that results in equiprobable futures, i.e., p(X _t+1 = 1) = 1/2. We have $p(X_{t+1}=0) = \pi(A=s|X=0) p(X=0) + \frac{1} {2} p(X=1).$ Therefore, this implies that π(A = s|X = 0) = 1/2, which, in turn, implies that after some time p(X _t  = 1) = 1/2, and thus I[X _t+1, {X, A}] = 1/2. However, asymptotically, p(X _t  = 0) = p(X _t+1 = 0), and the information is given by $-(p(X=0) \log_2 [ p(X=0)/(1-p(X=0)) ] - \log_2 [ 1-p(X=0) ] ) + p(X=0) - 1.$ Setting the first derivative, $1-\log_2 [ p(X=0)/(1-p(X=0)) ] ,$ to zero implies that the extremum lies at p(X = 0) = 2/3, where the information reaches $\log_2(3) - 1/3 \simeq 5/4$ bits. Now, p(X _t+1 = 0) = 2/3 implies that π(A = s|X = 0) = 3/4. Asymptotically, the optimal strategy is to stay in the reliable state with probability 3/4. We conclude that the agent starts with the random strategy in state 0, i.e., π(A = s|X = 0) = 1/2, and asymptotically finds the strategy π(A = s|X = 0) = 3/4. This asymptotic strategy still allows for exploration, but it results in a more controlled environment than the purely random strategy. Note that the optimal policy in state 1 is obviously random, i.e. π(A|1) = 1/2, because $D_{{\rm KL}} [ p(X_{t+1}|X_t=1, A_t=s) || p(X_{t+1}) ] = D_{{\rm KL}} [ p(X_{t+1}|X_t=1, A_t=f) || p(X_{t+1}) ]$.