Variability of Statistics in Generalizability Theory

Abstract

Estimates of variance components, error variances, generalizability coefficients, and so on, like all statistics, are subject to sampling variability. Traditionally, such variability is quantified through estimated standard errors and/or confidence intervals. Cronbach et al. (1972) recognized the importance of this topic for generalizability analyses and gave it more than passing attention, although at that time statistical methodologies for addressing the topic were limited. Subsequently, in the generalizability theory literature, Smith (1978, 1982) considered standard errors of estimated variance components, Brennan (1992a) summarized some procedures for establishing standard errors and confidence intervals, Brennan et al. (1987) considered bootstrap and jackknife procedures, Betebenner (1998) examined a relatively new procedure for establishing confidence intervals, Wiley (2000) studied the bootstrap, and Gao and Brennan (2001) provided examples from the performance assessment literature.