Abstract
The two-sample problem for Cronbach’s coefficient \(\alpha _C\), as an estimate of test or composite score reliability, has attracted little attention compared to the extensive treatment of the one-sample case. It is necessary to compare the reliability of a test for different subgroups, for different tests or the short and long forms of a test. In this paper, we study statistical procedures of comparing two coefficients \(\alpha _{C,1}\) and \(\alpha _{C,2}\). The null hypothesis of interest is \(H_0 : \alpha _{C,1} = \alpha _{C,2}\), which we test against one-or two-sided alternatives. For this purpose, resampling-based permutation and bootstrap tests are proposed for two-group multivariate non-normal models under the general asymptotically distribution-free (ADF) setting. These statistical tests ensure a better control of the type-I error, in finite or very small sample sizes, when the state-of-affairs ADF large-sample test may fail to properly attain the nominal significance level. By proper choice of a studentized test statistic, the resampling tests are modified in order to be valid asymptotically even in non-exchangeable data frameworks. Moreover, extensions of this approach to other designs and reliability measures are discussed as well. Finally, the usefulness of the proposed resampling-based testing strategies is demonstrated in an extensive simulation study and illustrated by real data applications.
Similar content being viewed by others
Notes
\(\alpha \) without any indices denotes the significance level of the corresponding statistical test, not Cronbach’s alpha.
References
Bader, K., Hänny, C., Schäfer, V., Neuckel, A., & Kuhl, C. (2009). Childhood Trauma Questionnaire—Psychometrische Eigenschaften einer deutschsprachigen Version. Zeitschrift für Klinische Psychologie und Psychotherapie, 38(4), 223–230.
Bentler, P. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74(1), 137.
Bernstein, D. P., et al. (2003). Development and validation of a brief screening version of the childhood trauma questionnaire. Child Abuse & Neglect, 27(2), 169–190.
Bonett, D. G. (2003). Sample size requirements for comparing two alpha coefficients. Applied Psychological Measurement, 27(1), 72–74.
Bonett, D. G., & Wright, T. A. (2015). Cronbach’s alpha reliability: Interval estimation, hypothesis testing, and sample size planning. Journal of Organizational Behavior, 36(1), 3–15.
Bradbury, I. (1987). Analysis of variance versus randomization tests—A comparison. British Journal of Mathematical and Statistical Psychology, 40(2), 177–187.
Brennan, R. L. (2001). Generalizability Theory. New York: Springer.
Chung, E., & Romano, J. P. (2013). Exact and asymptotically robust permutation tests. The Annals of Statistics, 41(2), 484–507.
Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78(1), 98.
Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Princeton: ERIC.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334.
D’Zurilla, T. J., Nezu, A. M., & Maydeu-Olivares, A. (2002). Social problem-solving inventory—Revised (SPSI-R). North Tonawanda: Multi Health Systems.
Furr, R. M., & Bacharach, V. R. (2013). Psychometrics: An introduction. Thousand Oaks: Sage.
Green, S. B., & Yang, Y. (2009a). Commentary on coefficient alpha: A cautionary tale. Psychometrika, 74(1), 121–135.
Green, S. B., & Yang, Y. (2009b). Reliability on summed item scores using structural equation modeling: An alternative to coefficient alpha. Psychometrika, 74(1), 155–167.
Gulliksen, H. (2013). Theory of mental tests. Abingdon: Routledge.
Guttmann, L. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10(4), 255–282.
Hogan, T. P., Benjamin, A., & Brezinski, K. L. (2000). Reliability methods: A note on the frequency of use of various types. Educational and Psychological Measurement, 60(4), 523–531.
Janssen, A. (1997). Studentized permutation tests for non-iid hypotheses and the generalized Behrens–Fisher problem. Statistics & Probability Letters, 36(1), 9–21.
Kim, S., & Feldt, L. S. (2008). A comparison of tests for equality of two or more independent alpha coefficients. Journal of Educational Measurement, 45(2), 179–193.
Konietschke, F., Bathke, A. C., Harrar, S. W., & Pauly, M. (2015). Parametric and non-parametric bootstrap methods for general MANOVA. Journal of Multivariate Analysis, 140, 291–301.
Konietschke, F., & Pauly, M. (2014). Bootstrapping and permuting paired t-test type statistics. Statistics and Computing, 24(3), 283–296.
Kuijpers, R. E., Ark, L. A., & Croon, M. A. (2013). Testing hypotheses involving Cronbach’s alpha using marginal models. British Journal of Mathematical and Statistical Psychology, 66(3), 503–520.
Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading: Addison-Wesley.
Maydeu-Olivares, A., Coffman, D. L., & Hartmann, W. M. (2007). Asymptotically distribution-free (ADF) interval estimation of coefficient alpha. Psychological Methods, 12(2), 157.
Maydeu-Olivares, A., Coffman, D. L., García-Forero, C., & Gallardo-Pujol, D. (2010). Hypothesis testing for coefficient alpha: An SEM approach. Behavior research methods, 42(2), 618–625.
McDonald, R. P. (1978). Generalizability in factorable domains: domain validity and generalizability 1. Educational and Psychological Measurement, 38(1), 75–79.
McDonald, R. P. (1999). Test theory: A unified treatment. New York: Psychology Press.
Mellenbergh, G. J. (1996). Measurement precision in test score and item response models. Psychological Methods, 1(3), 293.
Molenaar, I. W., & Sijtsma, K. (1988). Mokken’s approach to reliability estimation extended to multicategory items. Kwantitatieve Methoden, 9(28), 115–126.
Muthén, B., & Kaplan, D. (1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology, 38(2), 171–189.
Muthén, B., & Kaplan, D. (1992). A comparison of some methodologies for the factor analysis of non-normal Likert variables: A note on the size of the model. British Journal of Mathematical and Statistical Psychology, 45(1), 19–30.
Novick, M. R., & Lewis, C. (1966). Coefficient alpha and the reliability of composite measurements. Psychometrika, 32(1), 1–13.
Nunnally, J. C., & Bernstein, I. H. (1978). Psychometric theory. New York: McGraw-Hill.
Omelka, M., & Pauly, M. (2012). Testing equality of correlation coefficients in two populations via permutation methods. Journal of Statistical Planning and Inference, 142(6), 1396–1406.
Padilla, M. A., Divers, J., & Newton, M. (2012). Coefficient alpha bootstrap confidence interval under nonnormality. Applied Psychological Measurement, 36(5), 331–348.
Pauly, M. (2011). Discussion about the quality of F-ratio resampling tests for comparing variances. Test, 20(1), 163–179.
Pauly, M., Brunner, E., & Konietschke, F. (2015). Asymptotic permutation tests in general factorial designs. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 77(2), 461–473.
Peterson, R. A. (1994). A meta-analysis of Cronbach’s coefficient alpha. Journal of Consumer Research, 21(2), 381–391.
Prelog, A. J., Berry, K. J., & Mielke, P. W, Jr. (2009). Resampling permutation probability values for Cronbach’s alpha. Perceptual and Motor Skills, 108(2), 431–438.
R Core Team (2016). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.
Rao, C. R., & Sinharay, S. (2006). Handbook of statistics: Psychometrics (Vol. 26). New York: Elsevier.
Revelle, W., & Zinbarg, R. E. (2009). Coefficient alpha, beta, omega, and the glb: Comments on Sijtsma. Psychometrika, 74(1), 145–154.
Schury, K., Zimmermann, J., Umlauft, M., Hulbert, A. L., Guendel, H., Ziegenhain, U., et al. (2017). Childhood maltreatment, postnatal distress and the protective role of social support. Child Abuse & Neglect, 67, 228–239.
Sijtsma, K. (2009a). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107–120.
Sijtsma, K. (2009b). Reliability beyond theory and into practice. Psychometrika, 74(1), 169–173.
Sijtsma, K., & Molenaar, I. W. (1987). Reliability of test score in nonparametric item response theory. Psychometrika, 52, 79–97.
Still, A., & White, A. (1981). The approximate randomization test as an alternative to the F-test in analysis of variance. British Journal of Mathematical and Statistical Psychology, 34(2), 243–252.
Ten Berge, J. M., & Soc̆an, G. (2004). The greatest lower bound to the reliability of a test and the hypothesis of unidimensionality. Psychometrika, 69(4), 613–625.
Umlauft, M., Konietschke, F., & Pauly, M. (2017). Rank-based permutation approaches for nonparametric factorial designs. British Journal of Mathematical and Statistical Psychology, 70(3), 368–390.
van der Ark, L. A. (2012). New developments in Mokken scale analysis in R. Journal of Statistical Software, 48(5), 1–27.
van der Ark, L. A., van der Palm, D. W., & Sijtsma, K. (2011). A latent class approach to estimating test-score reliability. Applied Psychological Measurement, 35, 380–392.
van Zyl, J. M., Neudecker, H., & Nel, D. (2000). On the distribution of the maximum likelihood estimator of Cronbach’s alpha. Psychometrika, 65(3), 271–280.
Yuan, K.-H., Guarnaccia, C. A., & Hayslip, B. (2003). A study of the distribution of sample coefficient alpha with the Hopkins symptom checklist: Bootstrap versus asymptotics. Educational and Psychological Measurement, 63(1), 5–23.
Zinbarg, R. E., Revelle, W., Yovel, I., & Li, W. (2005). Cronbachs \(\alpha \), Revelles \(\beta \), and McDonalds \(\omega _h\): Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70(1), 123–133.
Author information
Authors and Affiliations
Corresponding author
Additional information
The work of Markus Pauly and Maria Umlauft was supported by the German Research Foundation project DFG-PA 2409/3-1.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Pauly, M., Umlauft, M. & Ünlü, A. Resampling-Based Inference Methods for Comparing Two Coefficients Alpha. Psychometrika 83, 203–222 (2018). https://doi.org/10.1007/s11336-017-9601-x
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11336-017-9601-x