Hypothesis Testing of the Q-matrix
The recent surge of interests in cognitive assessment has led to the development of cognitive diagnosis models. Central to many such models is a specification of the Q-matrix, which relates items to latent attributes that have natural interpretations. In practice, the Q-matrix is usually constructed subjectively by the test designers. This could lead to misspecification, which could result in lack of fit of the underlying statistical model. To test possible misspecification of the Q-matrix, traditional goodness of fit tests, such as the Chi-square test and the likelihood ratio test, may not be applied straightforwardly due to the large number of possible response patterns. To address this problem, this paper proposes a new statistical method to test the goodness fit of the Q-matrix, by constructing test statistics that measure the consistency between a provisional Q-matrix and the observed data for a general family of cognitive diagnosis models. Limiting distributions of the test statistics are derived under the null hypothesis that can be used for obtaining the test p-values. Simulation studies as well as a real data example are presented to demonstrate the usefulness of the proposed method.
KeywordsQ-matrix diagnostic classification models hypothesis testing
The authors thank the Editor, the Associate Editor, and four reviewers for many helpful and constructive comments. This work is partially supported by National Science Foundation (Grant No. SES-1659328, DMS-1712717, IIS-1633360, MMS-1826540), Institute of Education Sciences (Grant No. R305D160010), and Army Grant (Grant No. W911NF-15-1-0159).
- Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via EM algorithm. Journal of the Royal Statistical Society Series B-Methodological, 39(1), 1–38.Google Scholar
- DiBello, L., Stout, W., & Roussos, L. (1995). Unified cognitive psychometric assessment likelihood-based classification techniques. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 361–390). Hillsdale, NJ: Erlbaum.Google Scholar
- Gu, Y., & Xu, G. (2018). Partial identifiability of restricted latent class models. arXiv preprint arXiv:1803.04353.
- Hartz, S. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Doctoral Dissertation, University of Illinois, Urbana-Champaign.Google Scholar
- Henson, R., & Templin, J. (2005). Hierarchical log-linear modeling of the skill joint distribution. Technical report, External Diagnostic Research Group.Google Scholar
- Lehmann, E. L., & Romano, J. P. (2006). Testing statistical hypotheses. Berlin: Springer.Google Scholar
- Rupp, A. (2002). Feature selection for choosing and assembling measurement models: A building-block-based organization. Psychometrika, 2, 311–360.Google Scholar
- Rupp, A., & Templin, J. (2008b). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement: Interdisciplinary Research and Perspective, 6, 219–262.Google Scholar
- Rupp, A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York City: Guilford Press.Google Scholar
- Tatsuoka, K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition, (pp. 453–488).Google Scholar
- Tatsuoka, C. (2002). Data-analytic methods for latent partially ordered classification models. Applied Statistics (JRSS-C), 51, 337–350.Google Scholar
- Tatsuoka, K. (2009). Cognitive assessment: An introduction to the rule space method. Boca Raton: CRC Press.Google Scholar
- Templin, J. (2006). CDM: Cognitive diagnosis modeling with Mplus . Available from http://jtemplin.myweb.uga.edu/cdm/cdm.html.
- Templin, J., He, X., Roussos, L., & Stout, W. (2003). The pseudo-item method: A simple technique for analysis of polytomous data with the fusion model. Technical report, External Diagnostic Research Group.Google Scholar
- Van der Vaart, A. W. (2000). Asymptotic statistics (Vol. 3). Cambridge: Cambridge university press.Google Scholar
- von Davier, M. (2005). A general diagnosis model applied to language testing data. Research report, Educational Testing Service.Google Scholar
- Xu, G., & Shang, Z. (2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association. https://doi.org/10.1080/01621459.2017.1340889.
- Zhang, S. S., DeCarlo, L. T., & Ying, Z. (2013). Non-identifiability, equivalence classes, and attribute-specific classification in Q-matrix based cognitive diagnosis models. ArXiv e-prints.Google Scholar