Multiple hypothesis testing in experimental economics

Abstract

The analysis of data from experiments in economics routinely involves testing multiple null hypotheses simultaneously. These different null hypotheses arise naturally in this setting for at least three different reasons: when there are multiple outcomes of interest and it is desired to determine on which of these outcomes a treatment has an effect; when the effect of a treatment may be heterogeneous in that it varies across subgroups defined by observed characteristics and it is desired to determine for which of these subgroups a treatment has an effect; and finally when there are multiple treatments of interest and it is desired to determine which treatments have an effect relative to either the control or relative to each of the other treatments. In this paper, we provide a bootstrap-based procedure for testing these null hypotheses simultaneously using experimental data in which simple random sampling is used to assign treatment status to units. Using the general results in Romano and Wolf (Ann Stat 38:598–633, 2010), we show under weak assumptions that our procedure (1) asymptotically controls the familywise error rate—the probability of one or more false rejections—and (2) is asymptotically balanced in that the marginal probability of rejecting any true null hypothesis is approximately equal in large samples. Importantly, by incorporating information about dependence ignored in classical multiple testing procedures, such as the Bonferroni and Holm corrections, our procedure has much greater ability to detect truly false null hypotheses. In the presence of multiple treatments, we additionally show how to exploit logical restrictions across null hypotheses to further improve power. We illustrate our methodology by revisiting the study by Karlan and List (Am Econ Rev 97(5):1774–1793, 2007) of why people give to charitable causes.

Appendix

1.1 Proof of Theorem 3.1

First note that under Assumption 2.1, $Q\in \omega _{s}$ if and only if $P\in {\tilde{\omega }}_{s}$, where

$${\tilde{\omega }}_{s}=\{P(Q):Q\in \varOmega ,E_{P}[Y_{i,k}|D_{i}=d,Z_{i}=z]=E_{P}[Y_{i,k}|D_{i}=d',Z_{i}=z]\}.$$

The proof of this result now follows by verifying the conditions of Corollary 5.1 in Romano and Wolf (2010). In particular, we verify Assumptions B.1–B.4 in Romano and Wolf (2010).

In order to verify Assumption B.1 in Romano and Wolf (2010), let

$$T_{s,n}^{*}(P)=\sqrt{n}\left( \frac{1}{n_{d,z}}\sum _{1\le i\le n:D_{i}=d,Z_{i}=z}(Y_{i,k}-{\tilde{\mu }}_{k|d,z}(P))-\frac{1}{n_{d',z}}\sum _{1\le i\le n:D_{i}=d',Z_{i}=z}(Y_{i,k}-{\tilde{\mu }}_{k|d',z}(P))\right),$$

and note that

$$T_{n}^{*}(P)=(T_{s,n}^{*}(P):s\in {\mathcal {S}})=f(A_{n}(P),B_{n}),$$

where

$$A_{n}(P)=\frac{1}{\sqrt{n}}\sum _{1\le i\le n}A_{n,i}(P),$$

with $A_{n,i}(P)$ equal to the $2|{\mathcal {S}}|$-dimensional vector formed by stacking vertically for $s\in {\mathcal {S}}$ the terms

$$\left( \begin{array}{c} (Y_{i,k}-{\tilde{\mu }}_{k|d,z}(P))I\{D_{i}=d,Z_{i}=z\}\\ (Y_{i,k}-{\tilde{\mu }}_{k|d',z}(P))I\{D_{i}=d',Z_{i}=z\} \end{array}\right),$$

(10)

and $B_{n}$ is the $2|{\mathcal {S}}|$-dimensional vector formed by stacking vertically for $s\in {\mathcal {S}}$ the terms

$$\left( \begin{array}{c} \frac{1}{\frac{1}{n}\sum _{1\le i\le n}I\{D_{i}=d,Z_{i}=z\}}\\ -\frac{1}{\frac{1}{n}\sum _{1\le i\le n}I\{D_{i}=d',Z_{i}=z\}} \end{array}\right).$$

(11)

and $f:{\mathbf {R}}^{2|{\mathcal {S}}|}\times {\mathbf {R}}^{2|{\mathcal {S}}|}\rightarrow {\mathbf {R}}^{2|{\mathcal {S}}|}$ is the function of $A_{n}(P)$ and $B_{n}$ whose sth argument for $s\in {\mathcal {S}}$ is given by the inner product of the sth pair of terms in $A_{n}(P)$ and the sth pair of terms in $B_{n}$, i.e., the inner product of (10) and (11). The weak law of large numbers and central limit theorem imply that

$$B_{n}{\mathop {\rightarrow }\limits ^{P}}B(P),$$

where B(P) is the $2|{\mathcal {S}}|$-dimensional vector formed by stacking vertically for $s\in {\mathcal {S}}$ the terms

$$\left( \begin{array}{c} \frac{1}{P\{D_{i}=d,Z_{i}=z\}}\\ -\frac{1}{P\{D_{i}=d',Z_{i}=z\}} \end{array}\right).$$

Next, note that $E_{P}[A_{n,i}(P)]=0$. Assumption 2.3 and the central limit theorem therefore imply that

$$A_{n}(P){\mathop {\rightarrow }\limits ^{d}}N(0,V_{A}(P))$$

for an appropriate choice of $V_{A}(P)$. In particular, the diagonal elements of $V_{A}(P)$ are of the form

$${\tilde{\sigma }}_{k|d,z}^{2}(P)P\{D_{i}=d,Z_{i}=z\}.$$

The continuous mapping theorem thus implies that

$$T_{n}^{*}(P){\mathop {\rightarrow }\limits ^{d}}N(0,V(P))$$

for an appropriate variance matrix V(P). In particular, the sth diagonal element of V(P) is given by

$$\frac{{\tilde{\sigma }}_{k|d,z}^{2}(P)}{P\{D_{i}=d,Z_{i}=z\}}+\frac{{\tilde{\sigma }}_{k|d',z}^{2}(P)}{P\{D_{i}=d',Z_{i}=z\}}.$$

(12)

In order to verify Assumptions B.2–B.3 in Romano and Wolf (2010), it suffices to note that (12) is strictly greater than zero under our assumptions. Note that it is not required that V(P) be non-singular for these assumptions to be satisfied.

In order to verify Assumption B.4 in Romano and Wolf (2010), we first argue that

$$T_{n}^{*}(P_{n}){\mathop {\rightarrow }\limits ^{d}}N(0,V(P))$$

(13)

under $P_{n}$ for an appropriate sequence of distributions $P_{n}$ for $(Y_{i},D_{i},Z_{i})$. To this end, assume that

• (a) $P_{n}{\mathop {\rightarrow }\limits ^{d}}P$.

• (b) ${\tilde{\mu }}_{k|d,z}(P_{n})\rightarrow {\tilde{\mu }}_{k|d,z}(P)$.

• (c) $B_{n}{\mathop {\rightarrow }\limits ^{P_{n}}}B(P)$.

• (d) $\text {Var}_{P_{n}}[A_{n,i}(P_{n})]\rightarrow \text {Var}_{P}[A_{n,i}(P)]$.

Under (a) and (b), it follows that $A_{n,i}(P_{n}){\mathop {\rightarrow }\limits ^{d}}A_{n,i}(P)$ under $P_{n}$. By arguing as in Theorem 15.4.3 in Lehmann and Romano (2006) and using (d), it follows from the Lindeberg–Feller central limit theorem that

$$A_{n}(P_{n}){\mathop {\rightarrow }\limits ^{d}}N(0,V_{A}(P))$$

under $P_{n}$. It thus follows from (c) and the continuous mapping theorem that (13) holds under $P_{n}$. Assumption B.4 in Romano and Wolf (2010) now follows simply by nothing that the Glivenko-Cantelli theorem, strong law of large numbers and continuous mapping theorem ensure that ${\hat{P}}_{n}$ satisfies (a)–(d) with probability one under P.

Table 1 Multiple outcomes

Table 2 Multiple subgroups

Table 3 Multiple treatments (Comparing multiple treatments with a control)

Table 4 Multiple treatments (All pairwise comparisons across multiple treatments and a control)

Table 5 Multiple outcomes, subgroups, and treatments