Differential Privacy Analysis of Data Processing Workflows

Abstract

Differential privacy is an established paradigm to measure and control private information leakages occurring as a result of disclosures of derivatives of sensitive data sources. The bulk of differential privacy research has focused on designing mechanisms to ensure that the output of a program or query is $\epsilon$-differentially private with respect to its input. In an enterprise environment however, data processing generally occurs in the context of business processes consisting of chains of tasks performed by multiple IT system components, which disclose outputs to multiple parties along the way. Ensuring privacy in this setting requires us to reason in terms of series of disclosures of intermediate and final outputs, derived from multiple data sources. This paper proposes a method to quantify the amount of private information leakage from each sensitive data source vis-a-vis of each party involved in a business process. The method relies on generalized composition rules for sensitivity and differential privacy, which are applicable to chained compositions of tasks, where each task may have multiple inputs and outputs of different types, and such that a differentially private output of a task may be taken as input by other tasks.

A Lifting Distances to Probability Distributions

To interpret a privacy-enhanced DP-workflow $(W,\mathcal {E},\mathcal {C})$ (where $W=(D,P,F)$), we have to give metrics $\mathsf {d}_{d}$ on $\mathcal {D}({X_{d}})$ for each $d\in D$. Moreover, for the interpretation to be matched by the annotations, the mappings $\overline{f}_{{p}\rightarrow {d}}$ between these probability distributions must have the sensitivities given by $\mathcal {E}$ and $\mathcal {C}$. It may be more natural to assume that the interpretation gives us metrics on $X_{d}$, not on $\mathcal {D}({X_{d}})$. It is also more natural to require the mappings $f_{{p}\rightarrow {d}}$ to have a certain sensitivity.

We thus define that a pre-interpretation consists of sets $X_{d}$ for each $d\in D$ together with a metric $\mathsf {d}^\flat _{d}$ on it, as well as the mappings $f_{{p}\rightarrow {d}}$ for each $p\in P$ and $d\in p\bullet$. We have to specify what kind of interpretation it generates, and when to the annotations $\mathcal {E},\mathcal {C}$ match the pre-interpretation. The key for this is to specify the metric $\mathsf {d}_{d}$ on $\mathcal {D}({X_{d}})$.

Let X be a set and $d_X$ a metric on it. It turns out that the following definition of a metric $d^\#_X$ on $\mathcal {D}({X})$ is a suitable one. Let $\chi ,\chi '\in \mathcal {D}({X})$. Then

$$\begin{aligned} d^\#_X(\chi ,\chi ')=\inf _{\psi \in \chi \otimes \chi '}\sup _{(x,x')\in \mathrm {supp}({\psi })} d_X(x,x'). \end{aligned}$$

(A.1)

The proposed metric $d^\#_X$ can be seen as a kind of “worst-case” earth mover’s distance (or Wasserstein metric). In the “usual” earth mover’s distance, one would take the average over $\psi$, not the supremum over $\mathrm {supp}({\psi })$.

The suitability of the construction (A.1) is given by the following two propositions. Note that the first of them would not hold for the “usual” earth mover’s distance.

Proposition 3

Let $f:X\rightarrow \mathcal {D}({Y})$ be $\varepsilon$-sensitive according to the distance $d_X$ on X and distance $d_\mathrm {dp}$ on $\mathcal {D}({Y})$. Then the lifting $\overline{f}:\mathcal {D}({X})\rightarrow \mathcal {D}({Y})$ is $\varepsilon$-sensitive according to the distance $d^\#_X$ on $\mathcal {D}({X})$ and $d_\mathrm {dp}$ on $\mathcal {D}({Y})$.

Proof

Let $\chi ,\chi '\in \mathcal {D}({X})$, $\psi \in \chi \otimes \chi '$ and $y\in Y$. Then

$$\begin{aligned}&\mathrm {Pr}[\overline{f}(\chi )=y] = \sum _{x\in X}\chi (x)\cdot \mathrm {Pr}[f(x)=y] = \sum _{x,x'\in X}\psi (x,x')\cdot \mathrm {Pr}[f(x)=y] \le \\&\qquad \qquad \qquad \quad \sum _{x,x'\in X}\psi (x,x')\cdot e^{\varepsilon \cdot d_X(x,x')}\mathrm {Pr}[f(x')=y] \le \\&\qquad \quad \quad \sum _{x,x'\in X}\psi (x,x')\cdot e^{\sup _{x\in \mathrm {supp}({\psi (\cdot ,x')})}\varepsilon \cdot d_X(x,x')}\mathrm {Pr}[f(x')=y] = \\&\qquad \qquad \quad \sum _{x'\in X}\chi '(x')\cdot e^{\sup _{x\in \mathrm {supp}({\psi (\cdot ,x')})}\varepsilon \cdot d_X(x,x')}\mathrm {Pr}[f(x')=y] \le \\&\qquad \qquad \quad e^{\sup _{x,x'\in \mathrm {supp}({\psi })}\varepsilon \cdot d_X(x,x')}\cdot \sum _{x'\in X}\chi '(x')\cdot \mathrm {Pr}[f(x')=y] = \\&\qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad e^{\sup _{x,x'\in \mathrm {supp}({\psi })}\varepsilon \cdot d_X(x,x')}\cdot \mathrm {Pr}[\overline{f}(\chi ')=y], \end{aligned}$$

where $\mathrm {supp}({\psi (\cdot ,x')})$ denotes the set of all $x\in X$, such that $\psi (x,x')>0$. We obtain

$$\begin{aligned}&d_\mathrm {dp}(\overline{f}(\chi ),\overline{f}(\chi '))=\sup _{y\in Y}\left| \ln \frac{\mathrm {Pr}[\overline{f}(\chi ')=y]}{\mathrm {Pr}[\overline{f}(\chi )=y]}\right| \le \\&\qquad \qquad \qquad \qquad \qquad \qquad \inf _{\psi \in \chi \otimes \chi '}\sup _{x,x'\in \mathrm {supp}({\psi })}\varepsilon \cdot d_X(x,x')=\varepsilon \cdot d^\#_X(\chi ,\chi '). \end{aligned}$$

Proposition 4

Let $f:X\rightarrow \mathcal {D}({Y})$ be c-sensitive according to the distance $d_X$ on X and distance $d^\#_Y$ on $\mathcal {D}({Y})$, where $d^\#_Y$ is constructed from some distance $d_Y$ on Y according to (A.1). Then $\overline{f}:\mathcal {D}({X})\rightarrow \mathcal {D}({Y})$ is c-sensitive according to the distance $d^\#_X$ on $\mathcal {D}({X})$ and $d^\#_Y$ on $\mathcal {D}({Y})$.

Proof

Let $\chi ,\chi '\in \mathcal {D}({X})$. Define $\mathbf {F}$ as the following set of mappings of type $X\times X\rightarrow \mathcal {D}({Y\times Y})$:

$$ \mathbf {F}=\{\xi \,|\,\forall x,x'\in X: \xi (x,x')\in f(x)\otimes f(x')\}. $$

Also consider the set $\varPhi \subseteq \mathcal {D}({Y\times Y})$, defined as follows:

$$ \varPhi =\{\sum _{x,x'\in X} \psi (x,x')\cdot \xi (x,x')\,|\,\psi \in \chi \otimes \chi ', \xi \in \mathbf {F}\}. $$

In the definition of $\varPhi$, we take the averages over $\xi (x,x')$ with the weights given by $\psi (x,x')$. We have $\varPhi \subseteq \overline{f}(\chi )\otimes \overline{f}(\chi ')$ because the first [resp. second] projection of any element of $\varPhi$ is $\overline{f}(\chi )$ [resp. $\overline{f}(\chi ')$]. We now have

$$\begin{aligned}&\quad d^\#_Y(\overline{f}(\chi ),\overline{f}(\chi '))= \inf _{\phi \in \overline{f}(\chi )\otimes \overline{f}(\chi ')} \sup _{(y,y')\in \mathrm {supp}({\phi })} d_Y(y,y') \le \\&\inf _{\phi \in \varPhi } \sup _{(y,y')\in \mathrm {supp}({\phi })} d_Y(y,y') = \inf _{\psi \in \chi \otimes \chi '}\inf _{\xi \in \mathbf {F}} \sup _{(x,x')\in \mathrm {supp}({\psi })} \sup _{(y,y')\in \mathrm {supp}({\xi (x,x')})} d_Y(y,y') = \\&\qquad \quad \quad \quad \inf _{\psi \in \chi \otimes \chi '}\sup _{(x,x')\in \mathrm {supp}({\psi })}\inf _{\phi \in f(x)\otimes f(x')} \sup _{(y,y')\in \mathrm {supp}({\phi })} d_Y(y,y') =\\&\inf _{\psi \in \chi \otimes \chi '}\sup _{(x,x')\in \mathrm {supp}({\psi })} d^\#_Y(f(x),f(x')) \le \inf _{\psi \in \chi \otimes \chi '}\sup _{(x,x')\in \mathrm {supp}({\psi })} c\cdot d_X(x,x') = c\cdot d^\#_X(\chi ,\chi ') \end{aligned}$$

These two propositions tells us how to turn a pre-interpretation of a privacy-enhanced DP-workflow into an interpretation. We define $\mathsf {d}_{d}=(\mathsf {d}^\flat _{d})^\#$ for each $d\in D$. The annotations $\mathcal {E},\mathcal {C}$ match the pre-interpretation if for all $p\in P$, $d'\in \bullet p$ and $d\in p\bullet$:

• the sensitivity of $f_{{p}\rightarrow {d}}$ in its argument “$d'$” is $c_p[d',d]$ with respect to the distances $\mathsf {d}^\flat _{d'}$ on $X_{d'}$ and $\mathsf {d}_{d}$ on $\mathcal {D}({X_{d}})$;

• the sensitivity of $f_{{p}\rightarrow {d}}$ in its argument “$d'$” is $\epsilon _p[d',d]$ with respect to the distances $\mathsf {d}^\flat _{d'}$ on $X_{d'}$ and $d_\mathrm {dp}$ on $\mathcal {D}({X_{d}})$.

In this way, the corresponding interpretation is also matched by the annotations.