Abstract
The Q-matrix is usually unknown for many existing tests. If the Q-matrix is specified by subject matter experts but contains a large amount of misspecification, it will be difficult for the recovery of a high-quality Q-matrix through a validation method, because the performance of the validation method relies on the quality of a provisional Q-matrix. Under these two situations above, an exploratory technique is necessary. The purpose of this study is to explore a simple method for Q-matrix specification, called a discretized factor loading (DFL) method, in which exploratory factor analysis regarding latent attributes as latent factors is used to estimate a factor loading matrix after which a discretization process is employed on the factor loading matrix to obtain a binary Q-matrix. A series of simulation studies were conducted to investigate the performance of the DFL method under various conditions. The simulation results showed that the DFL method can provide a high-quality provisional Q-matrix.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Buck, G., VanEssen, T., Tatsuoka, K. K., Kostin, I., Lutz, D., & Phelps, M. (1998). Development, selection and validation of a set of cognitive and linguistic attributes for the SAT I verbal sentence completion section (RR-98-23). Princeton, NJ: Educational Testing Services.
Castellan, N. J. (1966). On the estimation of the tetrachoric correlation coefficient. Psychometrika, 31(1), 67–73.
Chang, H.-H. (2015). Psychometrics behind computerized adaptive testing. Psychometrika, 80(1), 1–20.
Chang, H.-H., & Wang, W. Y. (2016). “Internet Plus” measurement and evaluation: A new way for adaptive learning. Journal of Jiangxi Normal University (Natural Science), 40(5), 441–455.
Cheng, Y. (2009). When cognitive diagnosis meets computerized adaptive testing: CD-CAT. Psychometrika, 74(4), 619–632.
Chiu, C.-Y. (2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37(8), 598–618.
Chiu, C.-Y., Douglas, J. A., & Li, X.-D. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74(4), 633–665.
Close, C. N. (2012). An exploratory technique for finding the Q-matrix for the DINA model in cognitive diagnostic assessment: Combining theory with data. Unpublished doctoral dissertation, University of Minnesota, Educational Psychology.
Cui, Y., Gierl, M. J., & Chang, H.-H. (2012). Estimating classification consistency and accuracy for cognitive diagnostic assessment. Journal of Educational Measurement, 49(1), 19–38.
de la Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343–362.
de la Torre, J., & Chiu, C.-Y. (2010, April). A general method of empirical Q-matrix validation. Paper presented at the meeting of the National Council on Measurement in Education, Denver, CO.
de la Torre, J., & Chiu, C.-Y. (2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253–273.
DeCarlo, L. T. (2012). Recognizing uncertainty in the Q-matrix via a bayesian extension of the DINA model. Applied Psychological Measurement, 36(6), 447–468.
Ding, S. L., Yang, S. Q., & Wang, W. Y. (2010). The importance of reachability matrix in constructing cognitively diagnostic testing. Journal of Jiangxi Normal University (Natural Science), 34(5), 490–495.
Embretson, S. E. (1984). A general latent trait model for response processes. Psychometrika, 49(2), 175–186.
Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26(4), 301–321.
Hartz, S. M. (2002). A bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign.
Henson, R., & Douglas, J. (2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29(4), 262–277.
Huo, Y., & de la Torre, J. (2013, April). Data-driven Q-matrix specification for subsequent test forms. Paper presented at the meeting of the National Council on Measurement in Education, San Francisco, CA.
Im, S. (2007). Statistical consequences of attribute misspecfication of the rule space model. Unpublished doctoral dissertation, Columbia University.
Im, S., & Corter, J. E. (2011). Statistical consequences of attribute misspecification in the rule space method. Educational and Psychological Measurement, 71(4), 712–731.
Jang, E. E. (2009). Cognitive diagnostic assessment of L2 reading comprehension ability: Validity arguments for Fusion model application to language assessment. Language Testing, 26(1), 31–73.
Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272.
Kaiser, H. F. (1958). The varimax criterion for analytic rotation in factor analysis. Psychometrika, 23(3), 187–200.
Leighton, J. P., & Gierl, M. J. (Eds.). (2007). Cognitive diagnostic assessment for education: Theory and applications. New York: Cambridge University Press.
Liu, J., Xu, G., & Ying, Z. (2012). Data-driven learning of Q-matrix. Applied Psychological Measurement, 36(7), 548–564.
Liu, J., Xu, G., & Ying, Z. (2013). Theory of self-learning Q-matrix. Bernoulli, 19(5A), 1790–1817.
Liu, R., Huggins-Manley, A. C., & Bradshaw, L. (2016). The impact of Q-matrix designs on diagnostic classification accuracy in the presence of attribute hierarchies. Educational and Psychological Measurement, 77(2), 220–240.
Ma, W., Iaconangelo, C., & de la Torre, J. (2016). Model similarity, model selection, and attribute classification. Applied Psychological Measurement, 40(3), 200–217.
McGlohen, M. K. (2004). The application of cognitive diagnosis and computerized adaptive testing to a large-scale assessment. Unpublished doctorial dissertation, University of Texas at Austin.
McGlohen, M. K., & Chang, H.-H. (2008). Combining computer adaptive testing technology with cognitively diagnostic assessment. Behavior Research Methods, 40(3), 808–821.
Rupp, A. A., & Templin, J. (2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78–96.
Rupp, A. A., Templin, J. L., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York: The Guilford Press.
Tatsuoka, K. K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In N. Frederiksen, R. L. Glaser, A. M. Lesgold, & M. G. Safto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 453–488). Hillsdale, NJ: Erlbaum.
Tatsuoka, K. K. (1995). Architecture of knowledge structures and cognitive diagnosis: A statistical pattern classification approach. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.), Cognitively diagnostic assessments (pp. 327–359). Hillsdale: Erlbaum.
Tatsuoka, K. K. (2009). Cognitive assessment: An introduction to the rule space method. New York: Taylor & Francis Group.
Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287–305.
Tu, D.-B., Cai, Y., & Dai, H.-Q. (2012). A new method of Q-matrix validation based on DINA model. Acta Psychologica Sinica, 44(4), 558–568.
Xing, D., & Hambleton, R. K. (2004). Test design, item quality, and item bank size on the psychometric properties of computer-based credentialing examinations. Educational and Psychological Measurement, 64(1), 5–21.
Acknowledgements
This research is supported by the National Natural Science Foundation of China (Grant No. 31500909, 31360237, and 31160203), the Key Project of National Education Science “Twelfth Five Year Plan” of Ministry of Education of China (Grant No. DHA150285), the National Social Science Fund of China (Grant No. 16BYY096), the Humanities and Social Sciences Research Foundation of Ministry of Education of China (Grant No. 12YJA740057), the National Natural Science Foundation of Jiangxi (Grant No. 20161BAB212044), the Education Science Foundation of Jiangxi (Grant No. 13YB032), the Social Science Foundation of Jiangxi (Grant No. 17JY10), and the Youth Growth Fund and the Doctoral Starting up Foundation of Jiangxi Normal University. The authors would like to thank the editor Dylan Molenaar for the valuable comments on our submitted manuscript.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Wang, W., Song, L., Ding, S. (2018). An Exploratory Discrete Factor Loading Method for Q-Matrix Specification in Cognitive Diagnostic Models. In: Wiberg, M., Culpepper, S., Janssen, R., González, J., Molenaar, D. (eds) Quantitative Psychology. IMPS 2017. Springer Proceedings in Mathematics & Statistics, vol 233. Springer, Cham. https://doi.org/10.1007/978-3-319-77249-3_29
Download citation
DOI: https://doi.org/10.1007/978-3-319-77249-3_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-77248-6
Online ISBN: 978-3-319-77249-3
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)