Advertisement

The Effect of Using Principal Components to Create Plausible Values

  • Tom BentonEmail author
Conference paper
Part of the Springer Proceedings in Mathematics & Statistics book series (PROMS, volume 265)

Abstract

In all large scale educational surveys such as PISA and TIMSS the distribution of student abilities is estimated using the method of plausible values. This method treats student abilities within each country as missing variables that should be imputed based upon both student responses to cognitive items and a conditioning model using background information from questionnaires. Previous research has shown that, in contrast to creating single estimates of ability for each individual student, this technique will lead to unbiased population parameters in any subsequent analyses, provided the conditioning model is correctly specified (Wu in Studies in Educational Evaluation 31:114–128, 2005). More recent research has shown that, even if the conditioning model is incorrectly specified, the approach will provide a good approximation to population parameters as long as sufficient cognitive items are answered by each student (Marsman, Maris, Bechger, & Glas in Psychometrika 81:274–289, 2016). However, given the very large amount of background information collected in studies such as PISA, background variables are not all individually included in the conditioning model, and a smaller number of principal components are used instead. Furthermore, since no individual student answers cognitive items from every dimension of ability, we cannot rely on sufficient items having been answered to ignore possible resulting misspecification in the conditioning model. This article uses a simple simulation to illustrate how relying upon principal components within the conditioning model could potentially lead to bias in later estimates. A real example of this issue is provided based upon analysis of regional differences in performance in PISA 2015 within the UK.

Keywords

Plausible values Principal components PISA 

References

  1. Braun, H., & von Davier, M. (2017). The use of test scores from large-scale assessment surveys: Psychometric and statistical considerations. Large-scale Assessments in Education, 5(1), 17.CrossRefGoogle Scholar
  2. Chalmers, R. P. (2012). Mirt: A Multidimensional Item Response Theory Package for the R Environment. Journal of Statistical Software, 48(6), 1–29.CrossRefGoogle Scholar
  3. Jerrim, J., & Shure, N. (2016a). Achievement of 15-Year-Olds in England: PISA 2015 National Report (DFE-RR630). London, England: Department for Education.Google Scholar
  4. Jerrim, J., & Shure, N. (2016b). Achievement of 15-Year-Olds in Northern Ireland: PISA 2015 National Report. London, England: UCL.Google Scholar
  5. Laukaityte, I., & Wiberg, M. (2017). Using plausible values in secondary analysis in large-scale assessments. Communication in Statistics—Theory and Methods, 46(22), 11341–11357.MathSciNetCrossRefGoogle Scholar
  6. Marsman, M., Maris, G., Bechger, T., & Glas, C. (2016). What can we learn from Plausible Values? Psychometrika, 81(2), 274–289.MathSciNetCrossRefGoogle Scholar
  7. OECD. (2013). PISA 2012 results: Excellence through equity: Giving every student the chance to succeed (Vol. II). Paris: OECD.CrossRefGoogle Scholar
  8. OECD. (2014). PISA 2012 technical report. Paris: OECD.Google Scholar
  9. OECD. (2017). PISA 2015 technical report. Paris: OECD.Google Scholar
  10. Robitzsch, A., Kiefer, T., & Wu, M. (2017). TAM: Test analysis modules (Version 2.8–21) [Computer software]. Retrieved from http://CRAN.R-project.org/package=TAM.
  11. Von Davier, M., Gonzalez, E., & Mislevy, R. (2009). What are plausible values and why are they useful. IERI monograph series: Vol. 2, (pp. 9–36).Google Scholar
  12. Wu, M. (2005). The role of plausible values in large-scale surveys. Studies in Educational Evaluation, 31(2–3), 114–128.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Cambridge AssessmentCambridgeUK

Personalised recommendations