Advertisement

Adjusting Person Fit Index for Skewness in Cognitive Diagnosis Modeling

  • Kevin Carl P. SantosEmail author
  • Jimmy de la Torre
  • Matthias von Davier
Article
  • 14 Downloads

Abstract

Because the validity of diagnostic information generated by cognitive diagnosis models (CDMs) depends on the appropriateness of the estimated attribute profiles, it is imperative to ensure the accurate measurement of students’ test performance by conducting person fit (PF) evaluation to avoid flawed remediation measures. The standardized log-likelihood statistic lZ has been extended to the CDM framework. However, its null distribution is found to be negatively skewed. To address this issue, this study applies different methods of adjusting the skewness of lZ that have been proposed in the item response theory context, namely, χ2-approximation, Cornish-Fisher expansion, and Edgeworth expansion to bring its null distribution closer to the standard normal distribution. The skewness-corrected PF statistics are investigated by calculating their type I error and detection rates using a simulation study. Fraction-subtraction data are also used to illustrate the application of these PF statistics.

Keywords

Cognitive diagnosis models Person fit Aberrant response patterns χ2-approximation Cornish-Fisher expansion Edgeworth expansion 

Notes

Funding information

This research was funded by the Philippine Commission on Higher Education, Philippine Social Science Council, and University of the Philippines-Diliman. Moreover, this research was carried out in part using the CoARE Facility of the DOST-Advance Science and Technology Institute and the Computing and Archiving Research Environment (CoARE) Project.

References

  1. Bedrick, E. J. (1997). Approximating the conditional distribution of person fit indexes for checking the Rasch model. Psychometrika, 62, 191–199.CrossRefzbMATHGoogle Scholar
  2. Conjin, J. M. (2013). Detecting and explaining person misfit in non-cognitive measurement. Ridderkerk: Ridderprint.Google Scholar
  3. Cui, Y., & Leighton, J. O. (2009). The hierarchy consistency index: evaluating person fit for cognitive diagnostic assessment. Journal of Educational Measurement, 46, 429–449.CrossRefGoogle Scholar
  4. Cui, Y., & Li, J. (2015). Evaluating person fit for cognitive diagnostic assessment. Applied Psychological Measurement, 39, 223–238.CrossRefGoogle Scholar
  5. Cui, Y., & Roberts, M. R. (2013). Validating student score inferences with person-fit statistic and verbal reports: a person-fit study for cognitive diagnostic assessment. Educational Measurement: Issues and Practice, 32, 34–42.CrossRefGoogle Scholar
  6. Cui, Y., Gierl, M. J., & Chang, H. H. (2012). Estimating classification consistency and accuracy for cognitive diagnostic assessment. Journal of Educational Measurement, 49, 19–38.CrossRefGoogle Scholar
  7. de la Torre, J. (2009). DINA model and parameter estimation: a didactic. Journal of Educational and Behavioral Statistics, 34, 115–130.CrossRefGoogle Scholar
  8. de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76, 179–199.MathSciNetCrossRefzbMATHGoogle Scholar
  9. de la Torre, J., & Deng, W. (2008). Improving person fit assessment by correcting the ability estimate and its reference distribution. Journal of Educational Measurement, 45, 159–177.CrossRefGoogle Scholar
  10. de la Torre, J., & Minchen, N. (2014). Cognitively diagnostic assessments and the cognitive diagnosis model framework. Psicologa Educativa, 20, 89–97.CrossRefGoogle Scholar
  11. Drasgow, F., Levine, M. V., & Williams, E. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67–86.CrossRefGoogle Scholar
  12. Glas, C. A. W., & Dagohoy, A. V. T. (2007). A person fit test for IRT models for polytomous items. Psychometrika, 72, 159–180.MathSciNetCrossRefzbMATHGoogle Scholar
  13. Haertel, E. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 301–321.CrossRefGoogle Scholar
  14. Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: blending theory with practicality, unpublished doctoral dissertation, University of Illinois at Urbana–Champaign, Urbana–Champaign, IL.Google Scholar
  15. Henson, R. A., Templin, J. L., & Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191–210.MathSciNetCrossRefzbMATHGoogle Scholar
  16. Hulin, C. L., Drasgow, F., & Parsons, C. K. (1983). Item response theory: application to psychological measurement. Homewood: Dow Jones-Irwin.Google Scholar
  17. Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.MathSciNetCrossRefGoogle Scholar
  18. Levine, M. V., & Drasgow, F. (1983). Appropriateness measurement: validating studies and variable ability models. In D. J. Weiss (Ed.), New horizons in testing: latent trait test theory and computerized adaptive testing (pp. 109–131). New York: Academic Press.Google Scholar
  19. Levine, M. V., & Rubin, D. B. (1979). Measuring the appropriateness of multiple choice test scores. Journal of Educational Statistics, 4, 269–290.CrossRefGoogle Scholar
  20. Liu, Y., Douglas, J., & Henson, R. (2009). Testing person fit in cognitive diagnosis. Applied Psychological Measurement, 33, 579–598.CrossRefGoogle Scholar
  21. Ma, W., & de la Torre, J (2017). GDINA: the generalized DINA model framework. R package version 1.4.2, Available at https://CRAN.R-project.org/package=GDINA.
  22. Ma, W., Iaconangelo, C., & de la Torre, J. (2016). Model similarity, model selection, and attribute classification. Applied Psychological Measurement, 40, 200–217.CrossRefGoogle Scholar
  23. Mccullagh, E. (1986). The conditional distribution of goodness-of-fit statistics for discrete data. Journal of the American Statistical Association, 81, 104–107.CrossRefGoogle Scholar
  24. Meijer, R., & Nering, M. L. (1997). Trait level estimation for nonfitting response vectors. Applied Psychological Measurement, 21, 321–336.CrossRefGoogle Scholar
  25. Molenaar, I. W., & Hoijtink, H. (1990). The many null distributions of person fit indices. Psychometrika, 55, 75–106.MathSciNetCrossRefGoogle Scholar
  26. Nering, M. L. (1995). The distribution of person fit using true and estimated person parameters. Applied Psychological Measurement, 19, 121–129.CrossRefGoogle Scholar
  27. Nering, M. L. (1997). The distribution of indexes of person-fit within the computerized adaptive testing environment. Applied Psychological Measurement, 21, 115–127.CrossRefGoogle Scholar
  28. Reise, S. P. (1995). Scoring method and the detection of person misfit in a personality assessment context. Applied Psychological Measurement, 19, 213–229.CrossRefGoogle Scholar
  29. Rupp, A. A., & Templin, J. (2008). Unique characteristics of diagnostic classification models: a comprehensive review of the current state-of-the-art. Measurement: Interdisciplinary Research and Perspectives, 6, 219–262.Google Scholar
  30. Sinharay, S. (2015). Assessing person fit using l Z* and the posterior predictive model checking method for dichotomous item response theory models. International Journal of Quantitative Research in Education, 2, 265–284.CrossRefGoogle Scholar
  31. Sinharay, S. (2016). Assessment of person fit using resampling-based approaches. Journal of Educational Measurement, 53, 63–85.CrossRefGoogle Scholar
  32. Snijders, T. (2001). Asymptotic distribution of person-fit statistics with estimated person parameter. Psychometrika, 66, 331–342.MathSciNetCrossRefzbMATHGoogle Scholar
  33. Tatsuoka, K. K. (1983). Rule space: an approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345–354.CrossRefGoogle Scholar
  34. Tatsuoka, K. K. (1984). Caution indices based on item response theory. Psychometrika, 49, 95–110.CrossRefGoogle Scholar
  35. Tatsuoka, K. K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Safto (Eds.), Monitoring skills and knowledge acquisition (pp. 453–488). Hillsdale: Erlbaum.Google Scholar
  36. Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305.CrossRefGoogle Scholar
  37. Von Davier, M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287–307.MathSciNetCrossRefGoogle Scholar
  38. Von Davier, M., & Molenaar, I. (2003). A person-fit index for polytomous rasch models, latent class models, and their mixture generalizations. Psychometrika, 68, 213–228.MathSciNetCrossRefzbMATHGoogle Scholar
  39. Wright, B. D. (1977). Solving measurement problems with the Rasch model. Journal of Educational Measurement, 14, 97–115.CrossRefGoogle Scholar

Copyright information

© The Classification Society 2019

Authors and Affiliations

  1. 1.School of StatisticsUniversity of the Philippines-DilimanQuezon CityPhilippines
  2. 2.The University of Hong Kong, Pok Fu LamHong Kong
  3. 3.National Board of Medical ExaminersPhiladelphiaUSA

Personalised recommendations