Abstract
Matrix factorization techniques are widely used to build collaborative filtering recommender systems. These recommenders aim at discovering latent variables or attributes that are supposed to explain and ultimately predict the interest of users. In cognitive modeling, skills and competencies are considered as key latent attributes to understand and assess student learning. For this purpose, Tatsuoka introduced the concept of Q-matrix to represent the mapping between skills and test items. In this paper we evaluate how predictive expert-created Q-matrices can be when used as a decomposition factor in a matrix factorization recommender. To this end, we developed an evaluation method using cross validation and the weighted least squares algorithm that measures the predictive accuracy of Q-matrices. Results show that expert-made Q-matrices can be reasonably accurate at predicting users success in specific circumstances that are discussed at the end of this paper.
This is a preview of subscription content, log in via an institution to check access.
Notes
- 1.
Weighted least squares method:http://en.wikipedia.org/wiki/Least_squares#Weighted_least_squares.
- 2.
References
Barnes, T.: The Q-matrix method: Mining student response data for knowledge. In: AAAI Educational Data Mining workshop. p. 39 (2005)
Beheshti, B., Desmarais, M.C.: Predictive performance of prevailing approaches to skills assessment techniques: Insights from real vs. synthetic data sets (2014)
Birenbaum, M., Kelly, A.E., Tatsuoka, K.K.: Diagnosing Knowledge States in Algebra Using the Rule Space Model. ETS research report, Educational Testing Service, Educational Testing Service Princeton, NJ (1992)
DeCarlo, L.T.: On the analysis of fraction subtraction data: the DINA model, classification, latent class sizes, and the Q-matrix. Appl. Psychol. Meas. 35(1), 8–26 (2011)
Desmarais, M.C., Beheshti, B., Naceur, R.: Item to skills mapping: deriving a conjunctive Q-matrix from data. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 454–463. Springer, Heidelberg (2012)
Desmarais, M.C., Naceur, R.: A matrix factorization method for mapping items to skills and for enhancing expert-based Q-Matrices. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS, vol. 7926, pp. 441–450. Springer, Heidelberg (2013)
Gierl, M.J., Leighton, J.P., Hunka, S.M.: Exploring the logic of Tatsuoka’s rule-space model for test development and analysis. Educ. Meas. Issues Pract. 19(3), 34–44 (2000). http://dx.doi.org/10.1111/j.1745-3992.2000.tb00036.x
Junker, B.W., Sijtsma, K.: Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Appl. Psychol. Meas. 25(3), 258–272 (2001). http://apm.sagepub.com/content/25/3/258.abstract
Koedinger, K.R., Baker, R.S.J.D., Cunningham, K., Skogsholm, A., Leber, B., Stamper, J., Ventura, S., Pechenizkiy, M., Baker, R.S.J.D.: A data repository for the EDM community: The PSLC DataShop. In: Romero, C. (ed.) Handbook of Educational Data Mining, pp. 43–56. CRC Press, Boca Raton (2010)
Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender systems. Computer 42(8), 30–37 (2009). http://dx.doi.org/10.1109/MC.2009.263
Lan, A., Studer, C., Baraniuk, R.: Quantized matrix completion for personalized learning. In: 7th International Conference on Educational Data Mining, pp. 280–283 (2014)
Liu, J., Xu, G., Ying, Z.: Data-driven learning of Q-matrix. Appl. Psychol. Meas. 36(7), 548–564 (2012)
Mislevy, R.J.: Test theory reconceived. J. Educ. Meas. 33(4), 379–416 (1996). http://dx.doi.org/10.1111/j.1745-3984.1996.tb00498.x
Nelder, J.A., Mead, R.: A simplex method for functional minimization. Comput. J. 7, 308–313 (1965)
Ritter, S., Anderson, J., Koedinger, K., Corbett, A.: Cognitive tutor: applied research in mathematics education. Psychon. Bull. Rev. 14(2), 249–255 (2007). http://dx.doi.org/10.3758/BF03194060
Rupp, A.A., Templin, J.: The effects of q-matrix misspecification on parameter estimates and classification accuracy in the dina model. Educ. Psychol. Meas. 68(1), 78–96 (2008). http://epm.sagepub.com/content/68/1/78.abstract
Stamper, J., Ritter, S.: Cog model discovery experiment fall 2011. dataset 605 in datashop (2013). https://pslcdatashop.web.cmu.edu/datasetinfo?datasetid=605. Accessed
Stamper, J.C., Koedinger, K.R.: Human-machine student model discovery and improvement using DataShop. In: Biswas, G., Bull, S., Kay, J., Mitrovic, A. (eds.) AIED 2011. LNCS, vol. 6738, pp. 353–360. Springer, Heidelberg (2011)
Su, Y.L., Choi, K.M., Lee, W.C., Choi, T., McAninch, M.: Hierarchical cognitive diagnostic analysis for TIMSS] 2003 mathematics. Technical report 35, Center for Advanced Studies in Measurement and Assessment (CASMA), University of Iowa (2013)
Sun, Y., Ye, S., Inoue, S., Sun, Y.: Alternating recursive method for Q-matrix learning. In: 7th International Conference on Educational Data Mining, pp. 14–20 (2014)
Tatsuoka, K.K.: Rule space: an approach for dealing with misconceptions based on item response theory. J. Educ. Meas. 20(4), 345–354 (1983)
Tatsuoka, K.K.: Analysis of errors in fraction addition and subtraction problems. Final report, Computer-based Education Research Laboratory, University of Illinois at Urbana-Champaign (1984)
Tatsuoka, K.K.: Toward an integration of item-response theory and cognitive error diagnosis. In: Frederiksen, N., et al. (eds.) Diagnostic monitoring of skill and knowledge acquisition, pp. 453–488. Lawrence Erlbaum Associates, Hillsdale, NJ (1990)
Tatsuoka, K.K.: Item Construction and Psychometric Models Appropriate for Constructed Responses. ETS research report, Educational Testing Service, Educational Testing Service Princeton, NJ (1991)
Tatsuoka, K.K.: Architecture of knowledge structures and cognitive diagnosis: a statistical pattern recognition and classification approach. In: Nichols, P.D., Chipman, S.F., Brennan, R. (eds.) Cognitively Diagnostic Assessment, pp. 327–359. Lawrence Erlbaum Associates, Hillsdale (1995)
The MathWorks, I.: Matlab 7, function reference. In: Matlab 7. Natick, Massachusetts (2008)
de la Torre, J.: An empirically based method of q-matrix validation for the dina model: development and applications. J. Educ. Meas. 45(4), 343–362 (2008). http://dx.doi.org/10.1111/j.1745-3984.2008.00069.x
de la Torre, J.: DINA model and parameter estimation: a didactic. J. Educ. Behav. Stat. 34(1), 115–130 (2009)
de la Torre, J., Douglas, J.: Model evaluation and multiple strategies in cognitive diagnosis: an analysis of fraction subtraction data. Psychometrika 73(4), 595–624 (2008)
de la Torre, J., Douglas, J.A.: Higher-order latent trait models for cognitive diagnosis. Psychometrika 69(3), 333–353 (2004)
Acknowledgments
This work is part of the National Research Council Canada program Learning and Performance Support Systems (LPSS), which addresses training, development and performance support in all industry sectors, including education, oil and gas, policing, military and medical devices.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Durand, G., Belacel, N., Goutte, C. (2015). Evaluation of Expert-Based Q-Matrices Predictive Quality in Matrix Factorization Models. In: Conole, G., Klobučar, T., Rensing, C., Konert, J., Lavoué, E. (eds) Design for Teaching and Learning in a Networked World. EC-TEL 2015. Lecture Notes in Computer Science(), vol 9307. Springer, Cham. https://doi.org/10.1007/978-3-319-24258-3_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-24258-3_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-24257-6
Online ISBN: 978-3-319-24258-3
eBook Packages: Computer ScienceComputer Science (R0)