Advertisement

New Item-Selection Methods for Balancing Test Efficiency Against Item-Bank Usage Efficiency in CD-CAT

  • Wenyi Wang
  • Shuliang Ding
  • Lihong SongEmail author
Part of the Springer Proceedings in Mathematics & Statistics book series (PROMS, volume 89)

Abstract

Cognitive diagnostic computerized adaptive testing (CD-CAT) is a popular mode of online testing for cognitive diagnostic assessment (CDA). A key issue in CD-CAT programs is item-selection methods. Existing popular methods can achieve high measurement efficiencies but fail to yield balanced item-bank usage. Diagnostic tests often have low stakes, so item overexposure may not be a major concern. However, item underexposure leads to wasted time and money on item development, and high test overlap leads to intense practice effects, which in turn threaten test validity. The question is how to improve item-bank usage without sacrificing too much measurement precision (i.e., the correct recovery of knowledge states) in CD-CAT, which is the major purpose of this study. We have developed several item-selection methods that successfully meet this goal. In addition, we have investigated the Kullback–Leibler expected discrimination (KL-ED) method that considers only measurement precision except for item-bank usage.

Keywords

Posterior Distribution Item Parameter Knowledge State Item Bank Computerize Adaptive Testing 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgments

This work is partially supported by the National Natural Science Foundation of China (30860084, 31160203, 31100756,31360237), the Ministry of Education of Humanities and Social Planning Project of China (13YJC880060), the Specialized Research Fund for the Doctoral Program of Higher Education (20103604110001, 20103604110002, 20113604110001), the Jiangxi Provincial Social Science Planning Project (12JY07), the Jiangxi Provincial Education Planning Project (13YB032), the Jiangxi Provincial Department of Education Science and Technology Project (GJJ11385, GJJ10238, GJJ13207, GJJ13226), and the Jiangxi Normal University Youth Growth Fund. All opinions and conclusions are solely those of the authors. The authors are indebted to the editor and reviewers for their constructive suggestions and comments on the earlier manuscript.

References

  1. Breithaupt K, Ariel AA, Hare DR (2010) Assembling an inventory of multistage adaptive testing systems. In: van der Linden WJ, Glas CAW (eds) Elements of adaptive testing. Springer, New York, pp 247–266Google Scholar
  2. Chang H-H, Zhang J (2002) Hypergeometric family and item overlap rates in computerized adaptive testing. Psychometrika 67(3):387–398CrossRefzbMATHMathSciNetGoogle Scholar
  3. Chang H-H, Ying Z-L (1996) A global information approach to computerized adaptive testing. Appl Psychol Meas 20(3):213–229CrossRefGoogle Scholar
  4. Chang H-H, Ying Z-L (1999) a-stratified multistage computerized adaptive testing. Appl Psychol Meas 23(3):211–222Google Scholar
  5. Chang Y-CI, Lu H-Y (2010) Online calibration via variable length computerized adaptive testing. Psychometrika 75(1):140–157CrossRefzbMATHMathSciNetGoogle Scholar
  6. Chen P, Xin T, Wang C, Chang H-H (2012) On-line calibration methods for the DINA model with independent attributes in CA-CAT. Psychometrika 77(2):201–222CrossRefzbMATHMathSciNetGoogle Scholar
  7. Cheng Y (2009) When cognitive diagnosis meets computerized adaptive testing: CD-CAT. Psychometrika 74(4):619–632CrossRefzbMATHMathSciNetGoogle Scholar
  8. Cheng Y (2010) Improving cognitive diagnostic computerized adaptive testing by balancing attribute coverage: the modified maximum global discrimination index method. Educ Psychol Meas 70(6):902–913CrossRefGoogle Scholar
  9. Collins JA, Greer JE, Huang SX (1993) Adaptive assessment using granularity hierarchies and Bayesian nets. Paper presented at the 3rd international conference intelligent tutoring systemsGoogle Scholar
  10. de la Torre J, Douglas J (2004) Higher-order latent trait models for cognitive diagnosis. Psychometrika 69:333–353CrossRefzbMATHMathSciNetGoogle Scholar
  11. Embretson SE (1984) A general latent trait model for response processes. Psychometrika 49(2):175–186CrossRefMathSciNetGoogle Scholar
  12. Haertel EH (1989) Using restricted latent class models to map the skill structure of achievement items. J Educ Meas 26(4):301–321CrossRefGoogle Scholar
  13. Henson R (2005) Test construction for cognitive diagnosis. Appl Psychol Meas 29(4):262–277CrossRefMathSciNetGoogle Scholar
  14. Hsu CL, Wang WC, Chen SY (2013) Variable-length computerized adaptive testing based on cognitive diagnosis models. Appl Psychol Meas 37(7):563–582CrossRefGoogle Scholar
  15. Huebner A (2010) An overview of recent developments in cognitive diagnostic computer adaptive assessments. Pract Assess Res Eval 15(3):1–7. Available online: http://pareonline.net/getvn.asp?v=15&n=13
  16. Junker BW, Sijtsma K (2001) Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Appl Psychol Meas 25:258–272CrossRefMathSciNetGoogle Scholar
  17. Leighton JP, Gierl MJ (2007) Why cognitive diagnostic assessment? In: Leighton JP, Gierl MJ (eds) Cognitive diagnostic assessment for education: theory and applications. Cambridge University Press, New York, pp 3–18CrossRefGoogle Scholar
  18. Leighton JP, Gierl MJ, Hunka SM (2004) The attribute hierarchy method for cognitive assessment: a variation on Tatsuoka’s rule-space approach. J Educ Meas 41(3):205–237CrossRefGoogle Scholar
  19. Lin H-J, Ding S-L (2007) An exploration and realization of computerized adaptive testing with cognitive diagnosis. Acta Psychologica Sinica 39:747–753Google Scholar
  20. Liu H-Y, You X-F, Wang W-Y, Ding S-L, Chang H-H (2013) The development of computerized adaptive testing with cognitive diagnosis for an English achievement test in China. J Clas 30:152–172Google Scholar
  21. McGlohen MK (2004) The application of cognitive diagnosis and computerized adaptive testing to a large-scale assessment. Unpublished Doctorial Dissertation, University of Texas at AustinGoogle Scholar
  22. McGlohen MK, Chang H-H (2008) Combining computer adaptive testing technology with cognitively diagnostic assessment. Behav Res Methods 40(3):808–821CrossRefGoogle Scholar
  23. Millán E, Pérez-de-la-Cruz JL (2002) A Bayesian diagnostic algorithm for student modeling and its evaluation. User Model User-adapt Interact 12:281–330CrossRefzbMATHGoogle Scholar
  24. Quellmalz ES, Pellegrino JW (2009) Technology and testing. Science 323(2):75–79CrossRefGoogle Scholar
  25. Representatives, U. S. H. o. (2001) Text of the ‘No Child Left Behind Act’. Public Law No. 107–110, 115 Stat. 1425Google Scholar
  26. Revuelta J, Ponsoda V (1998) A comparison of item exposure control methods in computerized adaptive testing. J Educ Meas 35(4):311–327CrossRefGoogle Scholar
  27. Shang Z-Y, Ding S-L (2011) The exploration of item selection strategy of computerized adaptive testing for cognitive diagnosis. J Jiangxi Norm Univ (Nat Sci) 35(4):418–421Google Scholar
  28. Tatsuoka C (2002) Data analytic methods for latent partially ordered classification models. J R Stat Soc: Ser C: Appl Stat 51:337–350CrossRefzbMATHMathSciNetGoogle Scholar
  29. Tatsuoka C, Ferguson T (2003) Sequential classification on partially ordered sets. J R Stat Soc Ser B (Stat Methodol) 65(1):143–157CrossRefzbMATHMathSciNetGoogle Scholar
  30. Tatsuoka KK (1995) Architecture of knowledge structures and cognitive diagnosis: a statistical pattern classification approach. In: Nichols PD, Chipman SF, Brennan RL (eds) Cognitively diagnostic assessments. Erlbaum, Hillsdale, pp 327–359Google Scholar
  31. Tatsuoka KK (2009) Cognitive assessment: an introduction to the rule space method. Taylor & Francis Group, New YorkGoogle Scholar
  32. Veldkamp BP, van der Linden WJ (2010) Designing item pools for adaptive testing. In: van der Linden WJ, Glas CAW (eds) Elements of adaptive testing. Springer, New York, pp 231–245Google Scholar
  33. Wang C (2013) Mutual information item selection method in cognitive diagnostic computerized adaptive testing with short test length. Educ Psychol Meas 73(6):1017–1035CrossRefGoogle Scholar
  34. Wang C, Chang H-H, Douglas J (2012) Combining CAT with cognitive diagnosis: a weighted item selection approach. Behav Res Methods 44:95–109CrossRefGoogle Scholar
  35. Wang C, Chang H-H, Huebner A (2011) Restrictive stochastic item selection methods in cognitive diagnostic computerized adaptive testing. J Educ Meas 48(3):255–273CrossRefGoogle Scholar
  36. Wu H-M, Kuo B-C, Yang J-M (2006) Evaluating knowledge structure-based adaptive testing algorithms and system development. Educ Technol Soc 15:73–88Google Scholar
  37. Xu XL, Chang HH, Douglas J (2003) A simulation study to compare CAT strategies for cognitive diagnosis. Paper presented at the annual meeting of the American Educational Research Association, ChicagoGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.College of Computer Information Engineering, Jiangxi Normal UniversityNanchangChina
  2. 2.Elementary Educational College, Jiangxi Normal UniversityNanchangChina

Personalised recommendations