Advertisement

Behavior Research Methods

, Volume 51, Issue 1, pp 332–341 | Cite as

Measuring growth in students’ proficiency in MOOCs: Two component dynamic extensions for the Rasch model

  • Dmitry AbbakumovEmail author
  • Piet Desmet
  • Wim Van den Noortgate
Article

Abstract

Massive open online courses (MOOCs) are increasingly popular among students of various ages and at universities around the world. The main aim of a MOOC is growth in students’ proficiency. That is why students, professors, and universities are interested in the accurate measurement of growth. Traditional psychometric approaches based on item response theory (IRT) assume that a student’s proficiency is constant over time, and therefore are not well suited for measuring growth. In this study we sought to go beyond this assumption, by (a) proposing to measure two components of growth in proficiency in MOOCs; (b) applying this idea in two dynamic extensions of the most common IRT model, the Rasch model; (c) illustrating these extensions through analyses of logged data from three MOOCs; and (d) checking the quality of the extensions using a cross-validation procedure. We found that proficiency grows both across whole courses and within learning objectives. In addition, our dynamic extensions fit the data better than does the original Rasch model, and both extensions performed well, with an average accuracy of .763 in predicting students’ responses from real MOOCs.

Keywords

Psychometrics Item response theory Cross-classification multilevel logistic model Learning effects 

References

  1. Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716–723.  https://doi.org/10.1109/TAC.1974.1100705 CrossRefGoogle Scholar
  2. Andersen, E. B. (1985). Estimating latent correlations between repeated testings. Psychometrika, 50, 3–16.CrossRefGoogle Scholar
  3. Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1–48.  https://doi.org/10.18637/jss.v067.i01 CrossRefGoogle Scholar
  4. Coursera. (n.d.). Producing engaging video lectures. Retrieved from Coursera Partner Resource Center: https://partner.coursera.help/hc/en-us/articles/203525739-Producing-Engaging-Video-Lectures
  5. Davis, F. B. (1964). Educational measurements and their interpretation. Belmont, CA: Wadsworth.Google Scholar
  6. De Boeck, P., Bakker, M., Zwister, R., Nivard, M., Hofman, A., Tuerlinckx, F., & Partchev, I. (2011). The estimation of item response models with the lmer function from the lme4 package in R. Journal of Statistical Software, 39, 1–28.CrossRefGoogle Scholar
  7. Drasgow, F., & Lissak, R. (1983). Modified parallel analysis: A procedure for examining the latent dimensionality of dichotomously scored item responses. Journal of Applied Psychology, 68, 363–373.CrossRefGoogle Scholar
  8. Ekanadham, C., & Karklin, Y. (2015, July). T-SKIRT: Online estimation of student proficiency in an adaptive learning system. Paper presented at the 31st International Conference on Machine Learning, Lille, France.Google Scholar
  9. Elo, A. (1978). The rating of chessplayers, past and present. New York, NY: Arco.Google Scholar
  10. Fisher, G. H. (1976). Some probabilistic models for measuring change. In D. N. De Gruijter, & L. J. Van der Kamp (Eds.), Advances in psychological and educational measurement (pp. 97–110). New York, NY: Wiley.Google Scholar
  11. Fisher, G. H. (1995). Linear logistic models for change. In G. H. Fischer, & I. W. Molenaar (Eds.), Rasch models: Foundations, recent developments, and applications (pp. 157–180). New York, NY: Springer.CrossRefGoogle Scholar
  12. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.Google Scholar
  13. Higher School of Economics. (n.d.-a). Economics for non-economists. Retrieved from Coursera: https://www.coursera.org/learn/ekonomika-dlya-neekonomistov
  14. Higher School of Economics. (n.d.-b). Game theory. Retrieved from Coursera: https://www.coursera.org/learn/game-theory
  15. Higher School of Economics. (n.d.-c). Introduction to neuroeconomics: How the brain makes decisions. Retrieved from Coursera: https://www.coursera.org/learn/neuroeconomics
  16. Kadengye, D. T., Ceulemans, E., & Van den Noortgate, W. (2014). A generalized longitudinal mixture IRT model for measuring differential growth in learning environments. Behavior Research Methods, 46, 823–840.  https://doi.org/10.3758/s13428-013-0413-3 PubMedGoogle Scholar
  17. Kadengye, D. T., Ceulemans, E., & Van den Noortgate, W. (2015). Modeling growth in electronic learning environments using a longitudinal random item response model. Journal of Experimental Education, 83, 175–202.CrossRefGoogle Scholar
  18. Klinkenberg, S., Straatemeier, M., & van der Maas, H. L. (2011). Computer adaptive practice of Maths ability using a new item response model for on the fly ability and difficulty estimation. Computers & Education, 57, 1813–1824.CrossRefGoogle Scholar
  19. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison Wesley.Google Scholar
  20. Molenaar, I. W. (1995). Some background for Item Response Theory and the Rasch model. In G. H. Fischer, & I. W. Molenaar (Eds.), Rasch models: Foundations, recent developments, and applications (pp. 3–14). New York, NY: Springer.CrossRefGoogle Scholar
  21. R Core Team. (2013). R: A language and environment for statistical computing (R Foundation for Statistical Computing) Retrieved from http://www.R-project.org/
  22. Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Danish Institute for Educational Research.Google Scholar
  23. Rizopoulos, D. (2006). ltm: An R package for latent variable modelling and item response theory analyses. Journal of Statistical Software, 17, 1–25.CrossRefGoogle Scholar
  24. Shah, D. (2016). Monetization over massiveness: Breaking down MOOCs by the numbers in 2016. Retrieved from EdSurge: https://www.edsurge.com/news/2016-12-29-monetization-over-massiveness-breaking-down-moocs-by-the-numbers-in-2016
  25. Van den Noortgate, W., De Boeck, P., & Meulders, M. (2003). Cross-classification multilevel logistic models in psychometrics. Journal of Educational and Behavioral Statistics, 28, 369–386.CrossRefGoogle Scholar
  26. Verguts, T., & De Boeck, P. (2000). A Rasch model for detecting learning while solving an intelligence test. Applied Psychological Measurement, 24, 151–162.CrossRefGoogle Scholar
  27. Verhelst, N. D., & Glas, C. A. (1993). A dynamic generalization of the Rasch model. Psychometrika, 58, 395–415.CrossRefGoogle Scholar
  28. Verhelst, N. D., & Glas, C. A. (1995). Dynamic generalizations of the Rasch model. In G. H. Fischer, & I. W. Molenaar (Eds.), Rasch models: Foundations, recent developments, and applications (pp. 181–201). New York, NY: Springer.CrossRefGoogle Scholar
  29. von Davier, A. A. (2017). Computational psychometrics in support of collaborative educational assessments. Journal of Educational Measurement, 54, 3–11.CrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2018

Authors and Affiliations

  • Dmitry Abbakumov
    • 1
    • 2
    • 3
    Email author
  • Piet Desmet
    • 1
    • 4
  • Wim Van den Noortgate
    • 1
    • 2
  1. 1.ITEC imecLeuvenBelgium
  2. 2.Faculty of Psychology and Educational SciencesKU LeuvenLeuvenBelgium
  3. 3.eLearning OfficeNational Research University Higher School of EconomicsMoscowRussia
  4. 4.Faculty of ArtsKU LeuvenLeuvenBelgium

Personalised recommendations