Advertisement

Recent Developments in Cognitive Diagnostic Computerized Adaptive Testing (CD-CAT): A Comprehensive Review

  • Xiaofeng Yu
  • Ying ChengEmail author
  • Hua-Hua Chang
Chapter
Part of the Methodology of Educational Measurement and Assessment book series (MEMA)

Abstract

In this chapter, we provide a comprehensive and up-to-date review of cognitive diagnosis computer adaptive testing (CD-CAT). Similar to Cheng and Keng (Computerized adaptive testing in criterion-referenced testing. In Smith E, Stone G (eds) Applications of Rasch measurement in criterion-reference testing: practive analysis to score reporting. JAM Press, Maple Grove, 2009), which provided a flowchart for a typical CAT, we provide a typical CD-CAT flowchart. Compared to regular CAT, a key distinction is that in CD-CAT the goal is to obtain the latent mastery profile for each respondent in an efficient fashion, or alternatively to obtain both the latent mastery profile (formative) and the latent ability (summative) simultaneously. The former is referred to as single-purpose CD-CAT, and the latter dual-purpose CD-CAT. We discuss the main components of CD-CAT in this chapter. These components will be covered in the following order: starting rule, item selection strategies, stopping rule, scoring rule, and item bank development and more specifically online calibration.

References

  1. Baker, F. B., & Kim, S. H. (2004). Item response theory: Parameter estimation techniques (2nd ed.). New York, NY: Marcel Dekker.CrossRefGoogle Scholar
  2. Ban, J. C., Hanson, B. A., Wang, T. Y., & Harris, D. J. (2001). A comparative study of on-line pretest item–calibration/scaling methods in computerized adaptive testing. Journal of Educational Measurement, 38(3), 191–212.CrossRefGoogle Scholar
  3. Chang, H. H. (2015). Psychometrics behind computerized adaptive testing. Psychometrika, 80(1), 1–20.CrossRefGoogle Scholar
  4. Chang, H. H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20(3), 213–229.CrossRefGoogle Scholar
  5. Chang, H. H., & Ying, Z. (1999). a-Stratified multistage computerized adaptive testing. Applied Psychological Measurement, 23(3), 211–222.CrossRefGoogle Scholar
  6. Chen, P., Xin, T., Wang, C., & Chang, H. H. (2012). Online calibration methods for the DINA model with independent attributes in CD-CAT. Psychometrika, 77(2), 201–222.CrossRefGoogle Scholar
  7. Chen, Y. X., Liu, J. C., & Ying, Z. L. (2015). Online item calibration for Q-Matrix in CD-CAT. Applied Psychological Measurement, 39(1), 5–15.CrossRefGoogle Scholar
  8. Cheng, Y. (2008). Computerized adaptive testing: New development and applications. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign.Google Scholar
  9. Cheng, Y. (2009). When cognitive diagnosis meets computerized adaptive testing: CD-CAT. Psychometrika, 74(4), 619–632.CrossRefGoogle Scholar
  10. Cheng, Y. (2010). Improving cognitive diagnostic computerized adaptive testing by balancing attribute coverge: The modified maximum global discrimination index method. Educational and Psychological Measurement, 70(6), 902–913.CrossRefGoogle Scholar
  11. Cheng, Y., & Chang, H. H. (2007a). Dual information method in cognitive diagnostic computerized adaptive testing. Paper presented at the the meeting of the National Council on Measurement in Education, Chicago, IL.Google Scholar
  12. Cheng, Y., & Chang, H. H. (2007b). The modified maximum global discrimination index method for cognitive diagnostic computerized adaptive testing. In the Proceedings of the 2007 GMAC Conference on Computerized Adaptive Testing.Google Scholar
  13. Cheng, Y., & Chang, H. H. (2009). The maximum priority index method for severely constrained item selection in computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 62(2), 369–383.CrossRefGoogle Scholar
  14. Cheng, Y., Chang, H. H., & Yi, Q. (2007). Two-phase item selection procedure for flexible content balancing in CAT. Applied Psychological Measurement, 31(6), 467–482.CrossRefGoogle Scholar
  15. Cheng, Y., & Keng, L. (2009). Computerized adaptive testing in criterion-referenced testing. In E. Smith & G. Stone (Eds.), Applications of Rasch measurement in criterion-reference testing: Practive analysis to score reporting. Maple Grove, MN: JAM Press.Google Scholar
  16. Chung, M. T. (2014). Estimating the Q-matrix for cognitive diangnosis models in a Bayesian framework. Unpublished doctoral dissertation, Columbia University.Google Scholar
  17. Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. New York: Wiley.CrossRefGoogle Scholar
  18. Dai, B. Y., Zhang, M. Q., & Li, G. M. (2016). Exploration of item selection in dual-purpose cognitive diagnostic computerized adaptive testing: Based on the RRUM. Applied Psychological Measurement, 40(8), 625–640.CrossRefGoogle Scholar
  19. de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199.CrossRefGoogle Scholar
  20. de la Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343–362.CrossRefGoogle Scholar
  21. de la Torre, J., & Chiu, C. Y. (2016). A general method of empirical Q-Matrix validation. Psychometrika, 81(2), 253–273.CrossRefGoogle Scholar
  22. Eggen, T. J. H. M., & Straetmans, G. J. J. M. (2000). Computerized adaptive testing for classifying examinees into three categories. Educational and Psychological Measurement, 60(5), 713–734.CrossRefGoogle Scholar
  23. Embretson, S. E. (2001). The second century of ability testing: Some predictions and speculations. Retrieved from http://www.ets.org/Media/Research/pdf/PICANG7.pdf
  24. Georgiadou, E., Triantafillou, E., & Econimides, A. (2007). A review of item exposure control strategies for computerized adaptive testing. Journal of Technology, Learning, and Assessment, 5(8), 4–38.Google Scholar
  25. Gierl, M. J., & Zhou, J. W. (2008). Computer adaptive-attribute testing. Journal of Psychology, 216(1), 29–39.Google Scholar
  26. He, W., & Reckase, M. D. (2014). Item pool design for an operational variable-length computerized adaptive test. Educational and Psychological Measurement, 74(3), 473–494.CrossRefGoogle Scholar
  27. Henson, R. A., & Douglas, J. (2005). Test construction for cognitive diagnostics. Applied Psychological Measurement, 29(4), 262–277.CrossRefGoogle Scholar
  28. Hsu, C. L., Wang, W. C., & Chen, S. Y. (2013). Variable-length computerized adaptive testing based on cognitive diagnosis models. Applied Psychological Measurement, 37(7), 563–582.CrossRefGoogle Scholar
  29. Huebner, A. (2010). An overview of recent development in cognitive diagnostic computer adaptive assessments. Practical Assessment, Research & Evaluation, 15(3), 1–7.Google Scholar
  30. Huebner, A., & Wang, C. (2011). A note on comparing examinee classification methods for cognitive diagnosis models. Educational and Psychological Measurement, 71(2), 407–419.CrossRefGoogle Scholar
  31. Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272.CrossRefGoogle Scholar
  32. Kang, H. A., Zhang, S. S., & Chang, H. H. (2017). Dual-objective item selection criteria in cognitive diagnostic computerized adaptive testing. Journal of Educational Measurement, 54(2), 165–183.CrossRefGoogle Scholar
  33. Kaplan, M., de la Torre, J., & Barrada, J. R. (2015). New item selection methods for cognitive diagnosis computerized adaptive testing. Applied Psychological Measurement, 39(3), 167–188.CrossRefGoogle Scholar
  34. Liu, H. Y., You, X. F., Wang, W. Y., Ding, S. L., & Chang, H. H. (2013). The development of computerized adaptive testing with cognitive diagnosis for english achievement test in China. Journal of Classification, 30, 152–172.CrossRefGoogle Scholar
  35. Liu, J. C., Xu, G. J., & Ying, Z. L. (2012). Data driven learning of Q matrix. Applied Psychological Measurement, 36(7), 548–564.CrossRefGoogle Scholar
  36. Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum.Google Scholar
  37. McGlohen, M., & Chang, H. H. (2008). Combining computer adaptive testing technology with cognitively diagnostic assessment. Behavior Research Methods, 40(3), 808–821.CrossRefGoogle Scholar
  38. Reckase, M. D. (1985). The difficulty of test items that measure more than one ability. Applied Psychological Measurement, 9(4), 401–412.CrossRefGoogle Scholar
  39. Reckase, M. D. (1997). The past and future of multidimensional item response theory. Applied Psychological Measurement, 21(1), 25–36.CrossRefGoogle Scholar
  40. Reckase, M. D. (2010). Designing item pools to optimized the functioning of a computerized adaptive test. Psychological Test and Assessment Modeling, 52(2), 127–141.Google Scholar
  41. Revuelta, J., & Ponsoda, V. (1998). A comparison of item exposure control methods in computerized adaptive testing. Journal of Educational Measurement, 35(4), 311–327.CrossRefGoogle Scholar
  42. Riley, B. B., Conrad, K. J., Bezruczko, N., & Dennis, M. L. (2007). Relative precision, efficiency and construct validity of different starting and stopping rules for a computerized adaptive test: The GAIN substance problem scale. Journal of Applied Measurement, 8(1), 48–64.Google Scholar
  43. Rupp, A. A., Templin, J., & Henson, R. (2010). Diagnostic measurement: Theory, methods and application. New York: Guilford.Google Scholar
  44. Shannon, C. E. (1948). A mathematical theory of communication. The Bell System Technical Journal, 27(3), 379–423.CrossRefGoogle Scholar
  45. Stocking, M. L. (1988). Scale drift in on-line calibration (RR-88-28-ONR). Princeton, NJ: Educational Testing Service.CrossRefGoogle Scholar
  46. Tatsuoka, C. (2002). Data analytic methods for latent parially ordered classification models. Journal of the Royal Statistical Society: Series C (Applied Statistics), 51(3), 337–350.CrossRefGoogle Scholar
  47. Thissen, D., & Mislevy, R. J. (2000). Testing algorithm. In H. Wainer & N. J. Dorans (Eds.), Computerized adaptive testing: A primer. Hillsdale, NJ: Erlbarm.Google Scholar
  48. van der Linden, W. J. (1998). Bayesian item-selection criteria for adaptive testing. Psychometrika, 63(2), 201–216.CrossRefGoogle Scholar
  49. van der Linden, W. J. (2000). Constrained adaptive testing with shadow tests. In W. J. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 27–52). Boston: Kluwer.CrossRefGoogle Scholar
  50. van der Linden, W. J., & Chang, H. H. (2003). Implementing content constraints in a stratified adaptive testing using a shadow test approach. Applied Psychological Measurement, 27(2), 107–120.CrossRefGoogle Scholar
  51. van der Linden, W. J., & Glas, C. A. W. (2000). Capitalization on item calibration error in adaptive testing. Applied Measurement in Education, 13(1), 35–53.CrossRefGoogle Scholar
  52. Veldkamp, B. P., & van der Linden, W. J. (2000). Designing item pools for computerized adaptive testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 149–162). Dordrecht, The Netherlands: Kluwer Academic Publishers.CrossRefGoogle Scholar
  53. von Davier, M., & Cheng, Y. (2014). Multistage testing using diagnostic models. In D. L. Yan, A. A. von Davier, & C. Lewis (Eds.), Computerized multistage testing: Theory and applications (pp. 219–227). New York, NY: CRC Press.Google Scholar
  54. Wainer, H., & Mislevy, R. J. (2000). Item response theory, item calibration, and proficiency estimation. In H. Wainer & N. J. Dorans (Eds.), Computerized adaptive testing: A primer (2nd ed., pp. 61–100). Hillsdale, NJ: Lawrence Erlbaum.CrossRefGoogle Scholar
  55. Wang, C. (2013). Mutual information item selection method in cognitive diagnostic computerized adaptive testing with short test length. Educational and Psychological Measurement, 73(6), 1017–1035.CrossRefGoogle Scholar
  56. Wang, C., Chang, H. H., & Douglas, J. (2012). Combining CAT with cognitive diagnosis: A weighted item selection approach. Behavior Research Methods, 44(1), 95–109.CrossRefGoogle Scholar
  57. Wang, C., Chang, H. H., & Huebner, A. (2011). Restrictive stochastic item selection methods in cognitive diagnostic CAT. Journal of Educational Measurement, 48(3), 255–273.CrossRefGoogle Scholar
  58. Wang, C., Zheng, C. J., & Chang, H. H. (2014). An enhanced approach to combine item response theory with cognitive diagnosis in adaptive testing. Journal of Educational Measurement, 51(4), 358–380.CrossRefGoogle Scholar
  59. Wang, S. Y., Lin, H. Y., Chang, H. H., & Douglas, J. (2016). Hybrid computerized adaptive testing: From group sequential design to fully sequential design. Journal of Educational Measurement, 53(1), 45–62.CrossRefGoogle Scholar
  60. Xiang, R. (2013). Nonlinear penalized estimation of true Q-Matrix in cognitive diagnostic models. Unpublished doctoral dissertation, Columbia University.Google Scholar
  61. Xu, X. L., Chang, H. H., & Douglas, J. (2003). A simulation study to compare CAT strategies for cognitive diagnosis. Paper presented at the the Annual Meeting of American Educational Research Association, Chicago, IL.Google Scholar
  62. Yan, D. L., Lewis, C., & von Davier, A. A. (2014). Overview of computerized multistage tests. In D. L. Yan, A. A. von Davier, & C. Lewis (Eds.), Computerized multistage testing: Theory and applicatioins (pp. 3–20): CRC Press Boca Raton, FL.Google Scholar
  63. Yan, D. L., von Davier, A. A., & Lewis, C. (Eds.). (2014). Computerized multistage testing: Theory and applications. CRC Press: Boca Raton, FL.Google Scholar
  64. Yi, Q., & Chang, H. H. (2003). a-Stratified CAT design with content blocking. British Journal of Mathematical and Statistical Psychology, 56(2), 359–378.CrossRefGoogle Scholar
  65. Zheng, C. J., & Chang, H. H. (2016). High-efficiency response distribution-based item selection algorithms for short-length cognitive diagnostc computerized adaptive testing. Applied Psychological Measurement, 40(6), 608–624.CrossRefGoogle Scholar
  66. Zheng, C. J., & Wang, C. (2017). Application of binary searching for item exposure control in cognitive diagnostic computerized adaptive testing. Applied Psychological Measurement, 41(7), 561–576.CrossRefGoogle Scholar
  67. Zheng, Y. (2014). New methods of online calibration for item bank replenishment. Unpublished doctoral dissertation. University of Illinois at Urbana-Champaign.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of PsychologyUniversity of Notre DameNotre DameUSA
  2. 2.Jiangxi Normal UniversityNanchangChina
  3. 3.Department of Educational StudiesPurdue UniversityUrbana-ChampaignUSA

Personalised recommendations