Applications of Sequential Methods in Multiple Hypothesis Testing

Abstract

One of the main computational burdens in genome-wide statistical applications is the evaluation of large scale multiple hypothesis tests. Such tests are often implemented using replication-based methods, such as the permutation test or bootstrap procedure. While such methods are widely applicable, they place a practical limit on the computational complexity of the underlying test procedure. In such cases it would seem natural to apply sequential procedures. For example, suppose we observe the first ten replications of an upper-tailed statistic under a null distribution generated by random permutations, and of those ten, five exceed the observed value. It would seem reasonable to conclude that the P-value will not be small enough to be of interest, and further replications should not be needed.

While such methods have been proposed in the literature, for example by Hall in 1983, by Besag and Clifford in 1991 and by Lock in 1991, they have not been widely applied in multiple testing applications generated by high dimensional data sets, where they would likely be of some benefit. In this article related methods will first be reviewed. It will then be shown how commonly used multiple testing procedures may be modified so as to introduce sequential procedures while preserving the validity of reported error rates. A number of examples will show how such procedures can reduce computation time by an order of magnitude with little loss in power.