Meta-analysis as Judgment Aggregation

Abstract

For several decades now, a new inductive method, meta-analysis, is all the rage in social and medical sciences. Meta-analyses, that is, statistical reviews of the results of primary studies concerning a test hypothesis, set new standards of excellence on what counts as strong evidence. In the current prevailing mood in medical and behavioural sciences, it is only a properly conducted, up-to-date meta-analysis that licenses detachment of hypotheses from the host of evidential claims made in individual studies, which claims may be inconclusive or contradictory with each other. My goal in this chapter is to see the extent to which judgment aggregation methods subsume meta-analytic ones. To this end, I derive a generalized version of the classical Condorcet Jury Theorem, and I contend that one can model at least some meta-analytic procedures using this theorem.

This research was funded by BU research funds BAP6021.

Appendix

Proof of (A). This result amounts to showing that $\frac{{S(n)}}{n} - r$ converges to 0 in probability as n tends to infinity, given H. One may show it by applying a general weak law of large numbers, such as Theorem 7.2.1 in (Resnick 1998, 205), to the special setup of this result. One may also prove it as follows, noting that

$$\begin{array}{c}\displaystyle \frac{{S(n)}}{n} - r = \left( {\frac{{S(n)}}{n} - \frac{{(r_1 + \ldots + r_n)}}{n}} \right) + \left( {\frac{{(r_1 + \ldots + r_n) }}{n} - r} \right) \\ \displaystyle = \frac{1}{n}\sum\limits_{i = 1}^n {(S_i - r_i } ) + \left( {\frac{{(r_1 + \ldots + r_n) }}{n} - r} \right) \\ \end{array}$$

Since we assume that $r = \lim _{n \to \infty } \frac{{r_1 + \ldots + r_n }}{n}$ exists, the second term of the above sum converges to 0 as n goes to infinity. The first term can be shown to converge in probability to zero, since using Chebychev’s inequality, we have:

$$P\left[ {\left| {\frac{1}{n}\sum\limits_{i = 1}^n {(S_i - r_i )} } \right| > \varepsilon |{\textrm{H}}} \right] \le \frac{1}{{\varepsilon ^2 }}Var\left( {\frac{1}{n}\sum\limits_{i = 1}^n {(S_i - r_i )} } \right) = \frac{1}{{\varepsilon ^2 n^2 }}\sum\limits_{i = 1}^n {Var(S_i - r_i )}$$

Here we have used the assumption that S_i’s are independent from each other, as well as the fact that this implies that the (S_i–r_i)’s are also independent from each other, given H. Now, it is easy to see that for each i, ${\textrm{Var(S}}_{\textrm{i}} - {\textrm{r}}_{\textrm{i}} {)} = {\textrm{r}}_{\textrm{i}} (1 - {\textrm{r}}_{\textrm{i}} ) \le 1$, so that the above probability is less than or equal to $\frac{n}{{\varepsilon ^2 n^2 }} = \frac{1}{{\varepsilon ^2 n}}$, and this quantity approaches to 0 as tends to infinity. This shows that $\frac{{S(n)}}{n} - r$ can be written as the sum of two random variables each of which converges to 0 in probability as n tends to infinity, proving the same for the sum.

Proof of (i): Note that when c < r, we have the following:

$$\begin{array}{ll} P(A_n = 1|H) &= P\left(\frac{{S(n)}}{n} >\left. c\right|H\right) \\&= P\left(r - \frac{{S(n)}}{n} < r - \left.c\right|H\right) \\ &\ge P\left(c - r < r - \frac{{S(n)}}{n} < r - \left.c\right|H\right) \\ &= P\left(\left| {\frac{{S(n)}}{n} - r} \right| < r - \left.c\right|H\right) \end{array}$$

When in addition (1) holds, then the result (A), the weak law of large numbers with $\varepsilon = r - c > 0$ applies, and thus the last expression converges to 1 when n tends to infinity. This establishes the same for ${\textrm{P(A}}_{\textrm{n}} {\textrm{ = 1|H)}}$.

The proofs of the remaining clauses are very similar.