The Perils of P-Values

Abstract

The chapter gives an extensive discussion of p-values. It begins with a metaphorical example of a convention in which an arbitrary cutoff is used to dichotomize numerical information. The ritualistic use of p-values as a basis for constructing tests of hypotheses and computing sample sizes will be presented and discussed. This will be followed by a discussion of the use and misuse of p-values to establish “statistical significance” as a basis for making inferences, and practical problems with how p-values are computed. Bayes Factors will be presented as an alternative to p-values. The way that the hypothesis testing paradigm often is manipulated to obtain a desired sample size will be described, including an example of the power curve of a test as a more honest representation of the test’s properties. An example will be given to show that a p-value should not be used to quantify strength of evidence. An example from the published literature will be given that illustrates how the conventional comparison of a p-value to the conventional cutoff 0.05 may be misleading and harmful. The problem of dealing with false positive conclusions in multiple testing will be discussed. Type S error and the use of Bayesian posterior probabilities will be given as alternative methods. The chapter will close with an account of the ongoing P-value war in the scientific community.

A significant p-value is like obscenity. I can’t define it, but I know it when I see it.

Anonymous