Differentially Private Reinforcement Learning

Abstract

With remarkable performance and extensive applications, reinforcement learning is becoming one of the most popular learning techniques. Often, the policy $\pi ^*$ released by reinforcement learning model may contain sensitive information, and an adversary can infer demographic information through observing the output of the environment. In this paper, we formulate differential privacy in reinforcement learning contexts, design mechanisms for $\epsilon$-greedy and Softmax in the K-armed bandit problem to achieve differentially private guarantees. Our implementation and experiments illustrate that the output policies are under good privacy guarantees with a tolerable utility cost.

Appendices

Appendices

A. Proof of Algorithm 1

(20)

B. Analysis on Total Time Steps

We analyze the total time steps n needed to get a accurate approximation of $q_*(a)=\mathbb {E}_t[R_t\mid A_t=a]$ for every action a. The analysis is presented in two aspects. The first aspect is to consider how many times m we need to select action a to get an accurate approximation of $q_*(a)$. The second aspect is to analyze the value of n needed to guarantee m times sampling of a. We start with the first aspect.

Recall that w.l.o.g. $R_t$ is assumed to be within $[0,\varLambda ]$. Consider the Doob martingale $B_i=\mathbb {E}[\tfrac{1}{m}(X_1+X_2+\cdots +X_m)\mid X_1,X_2,\dotsc ,X_i]$, where $X_i$ is the numerical reward received when we select action a. Note that $X_i$’s are i.i.d. The stochastic process $B_0,B_1,\dotsc$ is a martingale w.r.t. $X_i$ as $\mathbb {E}(|B_j|)\le \varLambda <\infty$ and

$$\begin{aligned} \begin{aligned}&\mathbb {E}[B_{j+1}\mid X_1,\dotsc ,X_j]\\ =~&\mathbb {E}\left[ \mathbb {E}\left( \frac{1}{m}(X_1+\cdots +X_m)\mid X_1,\cdots ,X_{j+1}\right) \mid X_1,\cdots ,X_j\right] \\ =~&\mathbb {E}\left[ \frac{1}{m}(X_1+X_2+\cdots +X_m)\mid X_1,\dotsc ,X_j\right] \\ =~&B_j \end{aligned} \end{aligned}$$

(21)

holds. Note that $B_0=\mathbb {E}[\tfrac{1}{m}(X_1+X_2+\cdots +X_m)]=q_*(a)$ and $B_m=\tfrac{1}{m}(X_1+X_2+\cdots +X_m)$, which is just Q(a) in the algorithm. We also have $|B_{j+1}-B_j|\le \tfrac{\varLambda }{m}$ as $X_i$’s are independent and

$$\begin{aligned} \begin{aligned}&\left| {\frac{1}{m}(x_1+\cdots +x_j+\cdots +x_m)} \right. \\&\left. { -\frac{1}{m}(x_1+\cdots +x'_j+\cdots +x_m) } \right| \le \frac{\varLambda }{m} \end{aligned} \end{aligned}$$

(22)

holds. According to the Azuma-Hoeffding inequality, we then have $P(|Q(a)-q_*(a)|\ge \lambda \varLambda )\le 2\exp {(-\tfrac{(\lambda \varLambda )^2}{2\varLambda ^2/m})}=2\exp {(-\tfrac{\lambda ^2m}{2})}$, where $\lambda \in (0,1)$.

Now we consider the second aspect. Instead of counting the number of times a particular action a is selected, we consider the number of times selecting a when $rand()<\epsilon _{rl}$, which servers as a lower bound of the actual counting. Denote the latter as $Y_i$. So in every time step, action a is selected with probability $\tfrac{\epsilon _{rl}}{K}$. Applying Chernoff bound, if the total time steps $n=\tfrac{2mK}{\epsilon _{rl}}$ we have $P(Y_i<m)\le \exp {(-\tfrac{m}{4})}$. Applying the union bound we have the probability that every action is selected at least m times is at least $1-K\exp {(-\tfrac{m}{4})}$. With conditional probability the final probability that every estimate Q(a) is within $\lambda \varLambda$ of $q_*(a)$ is

$$\begin{aligned} \begin{aligned}&\left( 1-K\cdot 2 e^{-\frac{\lambda ^2 m}{2}}\right) \left( 1-K e^{-\frac{m}{4}}\right) \\ \ge \,\,&1-K\left( 2e^{-\frac{\lambda ^2 m}{2}}+e^{-\frac{m}{4}}\right) \\ \ge \,\,&1-\frac{3}{K^{c'}}\ge 1-\frac{1}{K^c} \end{aligned} \end{aligned}$$

(23)

if we choose $m=\tfrac{2}{\lambda ^2}(c'+1)\ln K$, where $c'$, c are constants. In a word if the total time steps $n\ge \tfrac{4K}{\epsilon _{rl}\lambda ^2}(c'+1)\ln K$ then w.h.p. every action value estimate is within a preferable range of the true action value.

C. Traverse in Algorithm 2