Abstract
Although the field of multidimensional item response theory (MIRT) has enjoyed tremendous growth over recent years, solutions to some problems remain to be studied. One case in point is the estimate of classification accuracy and consistency indices. There have been a few research studies focusing on these indices based on total scores under MIRT. The purposes of this study are to extend Rudner-based index for MIRT under complex decision rules and to compare it with the Guo-based index and the Lee-based index. The Rudner-based index assumes that an ability estimation error follows a multivariate normal distribution around each examinee’s ability estimate, and a simple Monte Carlo method is used to estimate accuracy and consistency indices. The simulation results showed that the Rudner-based index worked well under various conditions. Finally, conclusions are described along with thoughts for future research.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
T.A. Ackerman, Full-information factor analysis for polytomous item responses. Appl. Psychol. Meas. 18(3), 257–275 (1994)
R.J. Adams, M. Wilson, W.-C. Wang, The multidimensional random coefficients multinomial logit model. Appl. Psychol. Meas. 21(1), 1–23 (1997)
D.M. Bolt, V.F. Lall, Estimation of compensatory and noncompensatory multidimensional item response models using Markov chain Monte Carlo. Appl. Psychol. Meas. 27(6), 395–414 (2003)
L. Cai, High-dimensional exploratory item factor analysis by a metropolis–Hastings Robbins–Monro algorithm. Psychometrika 75(1), 33–57 (2010)
L. Cai, D. Thissen, S.H.C. du Toit, IRTPRO: Flexible, Multidimensional, Multiple Categorical IRT Modeling [Computer software] (Scientific Software International, Lincolnwood, IL, 2011)
R.P. Chalmers, Mirt: a multidimensional item response theory package for the R environment. J. Stat. Softw. 48(6), 1–29 (2012)
H.-H. Chang, The asymptotic posterior normality of the latent trait for polytomous IRT models. Psychometrika 61(3), 445–463 (1996)
H.-H. Chang, W. Stout, The asymptotic posterior normality of the latent trait in an IRT model. Psychometrika 58(1), 37–52 (1993)
H.-H. Chang, W. Wang, Internet plus measurement and evaluation: a new way for adaptive learning. J. Jiangxi Norm. Univ. (Nat. Sci.) 40(5), 441–455 (2016)
D. Debeer, J. Buchholz, J. Hartig, R. Janssen, Student, school, and country differences in sustained test-taking effort in the 2009 PISA reading assessment. J. Educ. Behav. Stat. 39(6), 502–523 (2014)
K.M. Douglas, R.J. Mislevy, Estimating classification accuracy for complex decision rules based on multiple scores. J. Educ. Behav. Stat. 35(3), 280–306 (2010)
A. Grima, L. H. Yao, in Classification consistency and accuracy for test of mixed item types: unidimensional versus multidimensional IRT procedures. Paper presented at the annual meeting of National Council on Measurement in Education, New Orleans, LA, (2011)
F. Guo, Expected classification accuracy using the latent distribution. Pract. Assessm. Res. Eval. 11(6), 1–6 (2006)
H. Huynh, Computation and statistical inference for decision consistency indexes based on the Rasch model. J. Educ. Stat. 15(4), 353–368 (1990)
U. Kroehne, F. Goldhammer, I. Partchev, Constrained multidimensional adaptive testing without intermixing items from different dimensions. Psychol. Test Assess. Modeling 56(4), 348–367 (2014)
L.J. LaFond, Decision Consistency and Accuracy Indices for the Bifactor and Testlet Response Theory Models Detecting Heterogeneity in Logistic Regression Models (University of Iowa, Iowa City, IA, 2014.) (Unpublished doctoral dissertation)
Q.N. Lathrop, Y. Cheng, Two approaches to estimation of classification accuracy rate under item response theory. Appl. Psychol. Meas. 37(3), 226–241 (2013)
W.-C. Lee, Classification consistency and accuracy for complex assessments using item response theory. J. Educ. Meas. 47(1), 1–17 (2010)
G. Makransky, E.L. Mortensen, C.A.W. Glas, Improving personality facet scores with multidimensional computer adaptive testing: an illustration with the Neo Pi-R. Assessment 20(1), 3–13 (2012)
M.D. Reckase, Multidimensional Item Response Theory (Springer, New York, NY, 2009)
F. Rijmen, M. Jeon, M. von Davier, S. Rabe-Hesketh, A third-order item response theory model for modeling the effects of domains and subdomains in large-scale educational assessment surveys. J. Educ. Behav. Stat. 39(4), 235–256 (2014)
L.M. Rudner, Computing the expected proportions of misclassified examinees. Pract. Assess. Res. Eval. 7(14), 1–8 (2001)
L.M. Rudner, Expected classification accuracy. Pract. Assess. Res. Eval. 10(13), 1–4 (2005)
F. Samejima, Estimation of Latent Ability Using a Response Pattern of Graded Scores (Psychometric Monograph No. 17) (Psychometric Society, Richmond, VA, 1969)
E.M. Schulz, M.J. Kolen, W.A. Nicewander, A rationale for defining achievement levels using IRT-estimated domain scores. Appl. Psychol. Meas. 23(4), 347–362 (1999)
C. Wang, On latent trait estimation in multidimensional compensatory item response models. Psychometrika 80(2), 428–449 (2015)
C. Wang, S. Nydick, Comparing two algorithms for calibrating the restricted non-compensatory multidimensional IRT model. Appl. Psychol. Meas. 39(2), 119–134 (2015)
T. Wang, M.J. Kolen, D.J. Harris, Psychometric properties of scale scores and performance levels for performance assessments using polytomous IRT. J. Educ. Meas. 37(2), 141–162 (2000)
W. Wang, L. Song, S. Ding, Y. Meng, in Quantitative Psychology Research: The 80th Annual Meeting of the Psychometric Society, ed. by L. A. van der Ark, D. M. Bolt, W.-C. Wang, J. A. Douglas, M. Wiberg. Estimating classification accuracy and consistency indices for multidimensional latent ability (Springer International Publishing, Cham, 2016), pp. 89–103
A.E. Wyse, S. Hao, An evaluation of item response theory classification accuracy and consistency indices. Appl. Psychol. Meas. 36(7), 602–624 (2012)
L. Yao, BMIRT: Bayesian Multivariate Item Response Theory [Computer Software] (CTB/McGraw-Hill, Monterey, 2003)
L. Yao, Multidimensional CAT item selection methods for domain scores and composite scores: theory and applications. Psychometrika 77(3), 495–523 (2012)
L. Yao, Multidimensional CAT item selection methods for domain scores and composite scores with item exposure control and content constraints. J. Educ. Meas. 51(1), 18–38 (2014)
L. Yao, The BMIRT toolkit. Retrieved March 1, 2016., from http://www.bmirt.com/media/f5abb5352d553d5fffff807cffffd524.pdf
L. Yao, K.A. Boughton, A multidimensional item response modeling approach for improving subscale proficiency estimation and classification. Appl. Psychol. Meas. 31(2), 1–23 (2007)
L. Yao, R.D. Schwarz, A multidimensional partial credit model with associated item and test statistics: an application to mixed-format tests. Appl. Psychol. Meas. 30(6), 469–492 (2006)
J. Zhang, Calibration of response data using MIRT models with simple and mixed structures. Appl. Psychol. Meas. 36(5), 375–398 (2012)
Acknowledgments
This research is supported by the National Natural Science Foundation of China (Grant Nos. 31500909, 31360237, and 31160203), the Key Project of National Education Science “Twelfth Five Year Plan” of Ministry of Education of China (Grant No. DHA150285), the National Social Science Fund of China (Grant No. 16BYY096), the Humanities and Social Sciences Research Foundation of Ministry of Education of China (Grant Nos. 13YJC880060 and 12YJA740057), the National Natural Science Foundation of Jiangxi Province (Grant No. 20161BAB212044), Jiangxi Education Science Foundation (Grant No. 13YB032), the Science and Technology Research Foundation of Education Department of Jiangxi Province (Grant No. GJJ13207), the China Scholarship Council (CSC No. 201509470001), and the Youth Growth Fund and the Doctoral Starting up Foundation of Jiangxi Normal University. The authors would like to thank Prof. Hua-Hua Chang for his kind support and Prof. Wen-Chung Wang for his valuable comments.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Wang, W., Song, L., Ding, S. (2017). An Extension of Rudner-Based Consistency and Accuracy Indices for Multidimensional Item Response Theory. In: van der Ark, L.A., Wiberg, M., Culpepper, S.A., Douglas, J.A., Wang, WC. (eds) Quantitative Psychology. IMPS 2016. Springer Proceedings in Mathematics & Statistics, vol 196. Springer, Cham. https://doi.org/10.1007/978-3-319-56294-0_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-56294-0_5
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-56293-3
Online ISBN: 978-3-319-56294-0
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)