The New Palgrave Dictionary of Economics

2018 Edition
| Editors: Macmillan Publishers Ltd

Multiple Testing

  • Joseph P. Romano
  • Azeem M. Shaikh
  • Michael Wolf
Reference work entry


Multiple testing refers to any instance that involves the simultaneous testing of more than one hypothesis. If decisions about the individual hypotheses are based on the unadjusted marginal p-values, then there is typically a large probability that some of the true null hypotheses will be rejected. Unfortunately, such a course of action is still common. In this article, we describe the problem of multiple testing more formally and discuss methods which account for the multiplicity issue. In particular, recent developments based on resampling result in an improved ability to reject false hypotheses compared to classical methods such as Bonferroni.


Multiple testing Familywise error rate Real estate finance Resampling 

JEL Classifications

This is a preview of subscription content, log in to check access.


  1. Davison, A.C., and D.V. Hinkley. 1997. Bootstrap Methods and their Application. Cambridge: University Press, Cambridge.CrossRefGoogle Scholar
  2. Hansen, P.R. 2005. A test for superior predictive ability. Journal of Business and Economics Statistics 23: 365–80.CrossRefGoogle Scholar
  3. Hochberg, Y., and A. Tamhane. 1987. Multiple comparison procedures. New York: Wiley.CrossRefGoogle Scholar
  4. Holm, S. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6: 65–70.Google Scholar
  5. Lahiri, S.N. 2003. Resampling methods for dependent data. New York: Springer.CrossRefGoogle Scholar
  6. Romano, J.P., and M. Wolf. 2005. Exact and approximate stepdown methods for multiple hypothesis testing. Journal of the American Statistical Association 100: 94–108.CrossRefGoogle Scholar
  7. Romano, J.P., A.M. Shaikh, and M. Wolf. 2008a. Control of the false discovery rate under dependence using the bootstrap and subsampling (with discussion). Test 17: 417–42.CrossRefGoogle Scholar
  8. Romano, J.P., A.M. Shaikh, and M. Wolf. 2008b. Formalized data snooping based on generalized error rates. Econometric Theory 24: 404–47.CrossRefGoogle Scholar
  9. Sarkar, S.K. 1998. Some probability inequalities for ordered M T P2 random variables: A proof of simes conjecture. Annals of Statistics 26: 494–504.CrossRefGoogle Scholar
  10. Westfall, P.H., and S.S. Young. 1993. Resampling-based multiple testing: Examples and methods for p-value adjustment. New York: Wiley.Google Scholar
  11. White, H.L. 2000. A reality check for data snooping. Econometrica 68: 1097–126.CrossRefGoogle Scholar

Copyright information

© Macmillan Publishers Ltd. 2018

Authors and Affiliations

  • Joseph P. Romano
    • 1
  • Azeem M. Shaikh
    • 1
  • Michael Wolf
    • 1
  1. 1.