Statistical Power and Sample Size

Abstract

Figure 6.1 shows two graphs of t-distributions. The lower graph (H1) could be a probability distribution of a sample of data or of a sample of paired differences between two observations. N  =  20 and so 95% of the observations is within 2.901  ±  2.101 standard errors of the mean (SEMs) on the x-axis (usually called z-axis in statistics). The upper graph is identical, but centers around 0 instead of 2.901. It is called the null-hypothesis H0, and represents the data of our sample if the mean results were not different from zero. However, our mean result is 2.901 SEMs distant from zero. If we had many samples obtained by similar trials under the same null-hypothesis, the chance of finding a mean value of more than 2.101 is <5%, because the area under the curve (AUC) of H0 right from 2.101 <5% of total AUC. We, therefore, reject the assumption that our results indicate a difference just by chance and decide that we have demonstrated a true difference. What is the power of this test. The power has as prior assumption that there is a difference from zero in our data. What is the chance of demonstrating a difference if there is one. If our experiment would be performed many times, the distribution of obtained mean values of those many experiments would center around 2.901, and about 70% of the AUC of H1 would be larger than 2.101. When smaller than 2.101, our statistical analysis would not be able to reject the null-hypothesis of no difference, when larger, it would rightly be able to reject the null-hypothesis of no difference. So, in fact 100%  –  70%  =  30% of the many trials would erroneously be unable to reject the null-hypothesis of no difference, even when a true difference is in the data. We say the power of this experiment  =  1  −  0.3  =  0.7 (70%), otherwise called the chance of finding a difference when there is one (area under curve (1  −  β)  ×  100%). β is also called the chance of making a type II error  =  chance of finding no difference when there is one. Another chance is the chance of finding a difference where there is none, otherwise called the type I error (area under the curve (2x α/2)  ×  100%). This type of error is usually set to be 0.05 (5%).