Applications of Sequential Methods in Multiple Hypothesis Testing

  • Anthony AlmudevarEmail author


One of the main computational burdens in genome-wide statistical applications is the evaluation of large scale multiple hypothesis tests. Such tests are often implemented using replication-based methods, such as the permutation test or bootstrap procedure. While such methods are widely applicable, they place a practical limit on the computational complexity of the underlying test procedure. In such cases it would seem natural to apply sequential procedures. For example, suppose we observe the first ten replications of an upper-tailed statistic under a null distribution generated by random permutations, and of those ten, five exceed the observed value. It would seem reasonable to conclude that the P-value will not be small enough to be of interest, and further replications should not be needed.

While such methods have been proposed in the literature, for example by Hall in 1983, by Besag and Clifford in 1991 and by Lock in 1991, they have not been widely applied in multiple testing applications generated by high dimensional data sets, where they would likely be of some benefit. In this article related methods will first be reviewed. It will then be shown how commonly used multiple testing procedures may be modified so as to introduce sequential procedures while preserving the validity of reported error rates. A number of examples will show how such procedures can reduce computation time by an order of magnitude with little loss in power.


Multiple hypothesis testing Sequential hypothesis testing Gene expression analysis 



This work was supported by NIH grant R21HG004648.


  1. 1.
    Almudevar, A. (2000). Exact confidence regions for species assignment based on DNA markers. The Canadian Journal of Statistics, 28, 81–95.MathSciNetCrossRefGoogle Scholar
  2. 2.
    Almudevar, A. (2010). A hypothesis test for equality of Bayesian network models. EURASIP Journal on Bioinformatics and Systems Biology, 2010, 10.CrossRefGoogle Scholar
  3. 3.
    Benjamini, Y., & Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 29, 1165–1188.MathSciNetCrossRefGoogle Scholar
  4. 4.
    Besag, J., & Clifford, P. (1989). Generalized Monte Carlo significance tests. Biometrika, 76, 633–642.MathSciNetCrossRefGoogle Scholar
  5. 5.
    Besag, J., & Clifford, P. (1991). Sequential Monte Carlo p-values. Biometrika, 78, 301–304.MathSciNetCrossRefGoogle Scholar
  6. 6.
    Dudoit, S., Shaffer, J. P., & Boldrick, J. C. (2003). Multiple hypothesis testing in microarray experiments. Statistical Science, 18, 71–103.MathSciNetCrossRefGoogle Scholar
  7. 7.
    Dudoit, S., & van der Laan, M. J. (2008). Multiple testing procedures with applications to genomics. New York: Springer.CrossRefGoogle Scholar
  8. 8.
    Fay, M. P., & Follmann, D. A. (2002). Designing Monte Carlo implementations of permutation or bootstrap hypothesis tests. The American Statistician, 56, 63–70.MathSciNetCrossRefGoogle Scholar
  9. 9.
    Hall, W. J. (1983). Some sequential tests for matched pairs: A sequential permutation test. In P. K. Sen (ed.), Contributions to statistics: essays in honour of Norman L. Johnson, (pp. 211–228). Amsterdam: North-Holland.Google Scholar
  10. 10.
    Ljung, L. (2007). Strong convergence of a stochastic approximation algorithm. The American Statistician, 6, 680–696.MathSciNetzbMATHGoogle Scholar
  11. 11.
    Lock, R. H. (1991). A sequential approximation to a permutation test. Communications in Statistics: Simulation and Computation, 20, 341–363.MathSciNetCrossRefGoogle Scholar
  12. 12.
    Medland, S., Schmitt, J., Webb, B., Kuo, P.-H., & Neale, M. (2009). Efficient calculation of empirical P-values for genome-wide linkage analysis through weighted permutation. Behavior Genetics, 39, 91–100.CrossRefGoogle Scholar
  13. 13.
    Mootha, V. K., Lindgren, C. M., Eriksson, K. F., Subramanian, A., Sihag, S., Lehar, J., et al. (2003). PGC-1 α-responsive genes involved in oxidative phosphorylation are coordinately downregulated in human diabetes. Nature Genetics, 100, 605–610.Google Scholar
  14. 14.
    Robbins, H., & Monro, S. (1951). A stochastic approximation method. The Annals of Mathematical Statistics, 22, 400–407.MathSciNetCrossRefGoogle Scholar
  15. 15.
    Siegmund, D. (1985). Sequential analysis: tests and confidence intervals. New York: Springer-Verlag.CrossRefGoogle Scholar
  16. 16.
    Subramanian, A., Tamayo, P., Mootha, V. K., Mukherjee, S., Ebert, B. L., Gillette, M. A., et al. (2005). Gene set enrichment analysis: A knowledge-based approach for interpreting genome-wide expression profiles. Proceedings of the National Academy of Sciences of the United States of America, 102, 15545–15550.Google Scholar
  17. 17.
    Wald, A. (1947). Sequential analysis. New York: John Wiley and Sons.zbMATHGoogle Scholar
  18. 18.
    Wald, A. (1948). Optimum character of the sequential probability ratio test. The Annals of Mathematical Statistics, 19, 326–339.MathSciNetCrossRefGoogle Scholar
  19. 19.
    Yang, H. Y., & Speed, T. (2003). Design and analysis of comparative microarray experiments. In T. Speed (ed.) Statistical analysis of gene expression microarray data (pp. 35–92). Boca Raton, FL: Chapman and Hall.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of Biostatistics and Computational BiologyUniversity of RochesterRochesterUSA

Personalised recommendations