Evaluation Methods of Cause-Effect Pairs

Abstract

This chapter addresses the problem of benchmarking causal models or validating particular putative causal relationships, in the limited setting of cause-effect pairs, when empirical “observational” data are available. We do not address experimental validations e.g. via randomized controlled trials. Our goal is to compare methods, which provide a score C(X, Y ), called causation coefficient, rating a pair of variable (X, Y ) for being in a potential causal relationship X → Y . Causation coefficients may be used for various purposes, including to prioritize experiments, which may be costly or risky, or guiding decision makers in domains in which experiments are infeasible or unethical. We provide a methodology to evaluate their reliability. We take three points of views: (1) that of algorithm developers who must justify the soundness of their method, particularly with respect to identifiability and consistency, (2) that of practitioners who seek to understand on what basis algorithms make their decisions and evaluate their statistical significance, and (3) that of benchmark organizers who desire to make fair evaluations to compare methods. We adopt the framework of pattern recognition in which pairs of variable (X, Y ) and their ground truth causal graph are drawn i.i.d. from a “mother distribution”. This leads us to define new notions of probabilistic identifiability, Bayes optimal causation coefficients, and multi-part statistical tests. These new notions are evaluated on the data of the first cause-effect pair challenge. We also compile a list of resources, including datasets of real or synthetic pairs, and data generative models.

Appendices

Appendix 1: Derivation of Bayes Optimal Causation Coefficients

We derive the Bayes optimal causation coefficients introduced in Sect. 2.3.2.

The Bayes optimal decision rule prescribes the following:

$$\displaystyle \begin{aligned} \begin{array}{rll} \text{Classify pair }(X_\pi, Y_\pi)\text{ as }[X \rightarrow Y] & iff~& \\ P_{\mathscr{M}}\Big( G = [X \rightarrow Y]~|~(X_\pi, Y_\pi)\Big) & > & P_{\mathscr{M}}\Big( G \neq [X \rightarrow Y]~|~(X_\pi, Y_\pi)\Big). \end{array} \end{aligned} $$

(2.28)

Given a classification problem of patterns z (in our case z = (X _π, Y _π) pairs), with ground truth g ₁ and g ₀ (in our case g ₁ = [X → Y ] and g ₀ = ¬[X → Y ]), we define a “discriminant function” as a function g(z) taking values in $\mathbb {R}$ such that we predict class g ₁ if g(z) > θ and class g ₀ otherwise. Important: with our definition, θ is a real number not necessarily equal to 0 (as is commonly used).

In this context,

$$\displaystyle \begin{aligned} P_{\mathscr{M}}\Big(G = [X \rightarrow Y]~|~(X_\pi, Y_\pi) \Big)- P_{\mathscr{M}}\Big(G \neq [X \rightarrow Y]~|~(X_\pi, Y_\pi)\Big) \end{aligned}$$

is a Bayes optimal discriminant function and because:

$$\displaystyle \begin{aligned} P_{\mathscr{M}}\Big(G = [X \rightarrow Y]~|~(X_\pi, Y_\pi)\Big) = 1- P_{\mathscr{M}}\Big(G \neq [X \rightarrow Y]~|~(X_\pi, Y_\pi)\Big), \end{aligned}$$

the following is also a Bayes optimal discriminant function:

$$\displaystyle \begin{aligned} 2~P_{\mathscr{M}}\Big(G = [X \rightarrow Y]~|~(X_\pi, Y_\pi)\Big)-1 \end{aligned}$$

and therefore, so is:

$$\displaystyle \begin{aligned} P_{\mathscr{M}} \Big(G = [X \rightarrow Y]~|~(X_\pi, Y_\pi) \Big) . \end{aligned}$$

Also note that we could have considered the symmetric problem of classifying Y → X vs. all other cases. If we assume that the mother distribution is perfectly symmetrical, i.e. that for each pair (X, Y ) labeled X → Y we have the symmetric pair (Y, X) labeled Y → X and for all pairs (X, Y ) labeled X ↔ Y , we have the same pair labeled Y ↔ X and for all pairs (X, Y ) labeled X ⊥ Y , we have the same pair labeled Y ⊥ X, then, the ranking of all pairs (X _π, Y _π) by sorting according to $P_{\mathscr {M}} \Big (G = [X \rightarrow Y]~|~(X_\pi , Y_\pi ) \Big )$ should be in reverse order as the ranking with $P_{\mathscr {M}} \Big (G = [X \leftarrow Y]~|~(X_\pi , Y_\pi ) \Big )$. Consequently we obtain the same ranking with:

(2.29)

where Φ(.) is any strictly monotonically increasing function.

It is important to note that, because we consider four possible truth values for G, G ∈{X → Y, X ← Y, X ↔ Y, X ⊥ Y }, $P_{\mathscr {M}}\Big (G = [X \rightarrow Y]~|~(X_\pi , Y_\pi ) \Big )$ is NOT equal to $(1- P_{\mathscr {M}}\Big (G = [X \leftarrow Y]~|~(X_\pi , Y_\pi ) \Big )$.

It may also be convenient to define an Bayes optimal causation coefficient in terms of data generative model. To that end, notice that if a − b is a discriminant function, so is a∕b − 1. Therefore,

$$\displaystyle \begin{aligned} P_{\mathscr{M}}\Big(G=[X \rightarrow Y]~|~(X_\pi, Y_\pi) \Big) / P_{\mathscr{M}}\Big(G=[X \leftarrow Y]~|~(X_\pi, Y_\pi) \Big) - 1 \end{aligned}$$

is also a Bayes optimal discriminant function. Using Bayes’ rule again,

$$\displaystyle \begin{aligned} \begin{array}{ll} &P_{\mathscr{M}}\Big(G=[X \rightarrow Y]~|~(X_\pi, Y_\pi) \Big) = \\ &~~~~~~~~P_{\mathscr{M}}\Big((X_\pi, Y_\pi)~|~G=[X \rightarrow Y] \Big) P_{\mathscr{M}}\Big(G=[X \rightarrow Y]\Big)/P_{\mathscr{M}}\Big((X_\pi, Y_\pi) \Big) \end{array} \end{aligned}$$

and further assuming that the mother distribution is not biased towards a particular causal direction:

$$\displaystyle \begin{aligned} P_{\mathscr{M}}\Big(G=[X \rightarrow Y]\Big) = P_{\mathscr{M}}\Big(G=[Y \rightarrow X] \Big) \end{aligned}$$

the following is also a Bayes optimal discriminant function:

$$\displaystyle \begin{aligned} P_{\mathscr{M}}\Big((X_\pi, Y_\pi)~|~G=[X \rightarrow Y]\Big) /P_{\mathscr{M}}\Big((X_\pi, Y_\pi)~|~G=[Y \rightarrow X]\Big) - 1 \end{aligned}$$

and so is, for any strictly monotonically increasing function Φ(.):

(2.30)

Appendix 2: Proof of Theorem 2.1: $\mathscr {B}$-Identifiability Implies (α, β)-Identifiability

Given the hypotheses of the theorem (symmetrical mother distribution), we can choose $C_{\mathscr {B} 2}$ as causation coefficient (Eq. (2.8)) and apply it to $\mathcal {B}_\varPi (X, Y)$:

$$\displaystyle \begin{aligned} C_{\mathscr{B} 2}(X_\pi, Y_\pi) =& P_{\mathscr{M}}\Big(\mathcal{B}_\varPi(X, Y)~|~G=[X \rightarrow Y]\Big)\\ &- P_{\mathscr{M}}\Big(\mathcal{B}_\varPi(X, Y)~|~G=[Y \rightarrow X]\Big). \end{aligned} $$

Imposing α = β = θ = 0, as per the definition of (α, β)-identifiability, the causal direction is (α, β)-identifiable for $P_{\mathscr {M}} \Big ( \mathcal {B}_\varPi (X, Y), G \Big )$ iff:

$$\displaystyle \begin{aligned} \begin{array}{rcl} Pr(C_{\mathscr{B} 2}(X_\pi, Y_\pi)>0~|~G = [X \leftarrow Y])= 0. &\displaystyle &\displaystyle \text{ (Type I errors) } \\ Pr(C_{\mathscr{B} 2}(X_\pi, Y_\pi)<0~|~G = [X \rightarrow Y])= 0. &\displaystyle &\displaystyle \text{ (Type II errors) } \end{array} \end{aligned} $$

In other words, (α, β)-identifiability with α = β = θ = 0 for $\mathcal {B}_\varPi (X, Y)$ is equivalent to:

$$\displaystyle \begin{aligned} G &= [X \rightarrow Y]\\ &\quad \Rightarrow P_{\mathscr{M}}\Big(\mathcal{B}_\varPi(X, Y)~|~G=[X \rightarrow Y]\Big) > P_{\mathscr{M}}\Big(\mathcal{B}_\varPi(X, Y)~|~G=[Y \rightarrow X]\Big) . {} \end{aligned} $$

(2.31)

and vice versa if we invert the roles of X and Y . If $\mathscr {B}$-identifiability holds, then:

$$\displaystyle \begin{aligned} G = [X \rightarrow Y] \Rightarrow \left \{ \begin{array}{lll} \exists \left(f\in\mathcal{F} \wedge P(N) \in \mathcal{N}\right) & \text{s.t.} & Y:=f(X,N) \\ \nexists \left(f\in\mathcal{F} \wedge P(N) \in \mathcal{N} \right) & \text{s.t.} & X:=f(Y,N). \end{array} \right. {} \end{aligned} $$

(2.32)

which can be equivalently re-written, for the given pair (X _π, Y _π), as:

$$\displaystyle \begin{aligned} G = [X \rightarrow Y] \Rightarrow \left \{ \begin{array}{lll} P_{\mathscr{M}}\Big(\mathcal{B}_\varPi(X, Y)~|~G=[X \rightarrow Y]\Big) > 0 \\ P_{\mathscr{M}}\Big(\mathcal{B}_\varPi(X, Y)~|~G=[Y \rightarrow X]\Big) = 0. \end{array} \right. {} \end{aligned} $$

(2.33)

It can easily be seen that if Eq. (2.33) is satisfied then Eq. (2.31) holds. Thus we have proved that $\mathscr {B}$-identifiability implies (α, β)-identifiability with α = β = θ = 0 for $\mathcal {B}_\varPi (X, Y)$. Let us prove now the reciprocal statement.

Starting from Eq. (2.31), if we swap the role of X and Y , we obtain: $G = [Y \rightarrow X] \Rightarrow P_{\mathscr {M}}\Big (\mathcal {B}_\varPi (X, Y)~|~G=[Y \rightarrow X]\Big ) > P_{\mathscr {M}}\Big (\mathcal {B}_\varPi (X, Y)~|~G=[X \rightarrow Y]\Big ) ;$and if we contrapose Eq. (2.31) we obtain: $P_{\mathscr {M}}\Big (\mathcal {B}_\varPi (X, Y)~|~G=[Y \rightarrow X]\Big ) > P_{\mathscr {M}}\Big (\mathcal {B}_\varPi (X, Y)~|~G=[X \rightarrow Y]\Big ) \Rightarrow G = [Y \rightarrow X] ,$ since when data are generated with a binary process ¬(G = [X → Y ]) is equivalent to G = [Y → X].

Thus we have an equivalence, both for the previous formula and for Eq. (2.31):

$$\displaystyle \begin{aligned} G &= [X \rightarrow Y] \Leftrightarrow P_{\mathscr{M}}\Big(\mathcal{B}_\varPi(X, Y)~|~G=[X \rightarrow Y]\Big)\\ &\quad > P_{\mathscr{M}}\Big(\mathcal{B}_\varPi(X, Y)~|~G=[Y \rightarrow X]\Big) \geq 0. {} \end{aligned} $$

(2.34)

In this last formula, if $P_{\mathscr {M}}\Big (\mathcal {B}_\varPi (X, Y)~|~G=[Y \rightarrow X]\Big ) \neq 0$, then ∃(X, Y ) s.t. G = [Y → X]. This would contradicts that G = [X → Y ]. Hence we must have $P_{\mathscr {M}}\Big (\mathcal {B}_\varPi (X, Y)~|~G=[Y \rightarrow X]\Big ) = 0$. Therefore, if Eq. (2.34) holds, then Eq. (2.33) holds too, or equivalently Eq. (2.32). Thus, we have proved that (α, β)-identifiability with α = β = θ = 0 implies $\mathscr {B}$-identifiability. $\square$

Appendix 3: Examples of Cause-Effect Pairs

In this section, we should graphically how the some of the causation coefficient filters are computed. The examples are drawn from Tables 2.1 and 2.2. It can be observed that, depending on the type of data generative model, one or the other assumption is violated. Hence the causation coefficient filters generally disagree on the causal direction. Even though the small decision tree of Fig. 7.7 performs relatively well on the challenge data, it is very easy to construct pairs to make it fail. In the following chapters, we will see more advanced method (Figs. 2.15, 2.16, 2.17, 2.18, 2.19, 2.20, 2.21, 2.22, 2.23, 2.24, 2.25 and 2.26).

Fig. 2.15

Linear with additive Gaussian noise. Pair 1 in Table 2.1. This is a well-known non identifiable pair. However, due to the finite sample size, wrong decisions might be made

Fig. 2.16

Linear with additive uniform noise. Pair 2 in Table 2.1. Unlike the previous pair, this one is identifiable with the Additive Noise Model (ANM): IR _Y > IR _X, there is a better independence between the input and the residual in the correct direction. However the R ² is better in the wrong direction!

Fig. 2.17

Linear with multiplicative uniform noise. Pair 3 in Table 2.1. The ANM has difficulties, but the CDA works

Fig. 2.18

S-shaped function with Gaussian input violating the independence of input density and function. Pair 4 in Table 2.1. The IGCI entropy criterion fails, but the IGCI slope criterion works

Fig. 2.19

Parabola with very small noise. Pair 5 in Table 2.1. The IGCI slope criterion fails, because the function is ntn invertible

Fig. 2.20

Square root with binary noise. Pair 6 in Table 2.1. The regression fit and residual criteria fail as well as the CDS. But the IGCI criteria work

Fig. 2.21

Altitude vs. temperature of German cities. Pair 1 in Table 2.2. The IGCI slope criterion fails, because the function is not invertible. More surprisingly, CDS fails too, probably because of outliers

Fig. 2.22

Simulated altitude vs. temperature. Pair 2 in Table 2.2. Only the IGCI slope criterion fails, because the function is not invertible

Fig. 2.23

Real weight vs. age pair. Pair 3 in Table 2.2. Only H really works on that pair (IR is neutral)

Fig. 2.24

Simulated weight vs. age pair. Pair 4 in Table 2.2. Only R ² and H work on that pair

Fig. 2.25

Real pair “hill shade at 3 pm” vs. aspect. Pair 5 in Table 2.2. R ² fails because of the multiplicative noise and S fails because the function is not invertible

Fig. 2.26

Simulated pair “hill shade at 3 pm” vs. aspect. Pair 6 in Table 2.2. The only pair in which all diagnoses agree between real and synthetic data