Abstract
This paper reviews challenges and opportunities of using deep knowledge tracing (DKT) in a production hybrid instructional and assessment system (HIAS) for measuring learner skills. An empirical data analysis using K-12 Mathematics practice data from an operational HIAS showed that the DKT estimates in the training set were well calibrated, although the estimates in a holdout set overestimated student skill mastery at lower ability levels and underestimated mastery at higher ability levels. Examination of expected vs. observed plots across ability levels provided a means for evaluating the generalizability of the predictions to new students and suggested a stratified sampling approach may be needed for creating training and validation sets that better represented the full practice data. Estimation accuracy was further examined in a parameter recovery study, with response data simulated from patterns in the empirical data. The study showed that in general, the DKT procedure had high estimation accuracy, with an absolute mean difference of approximately 0.03 units on the [0, 1] probability scale and a Pearson correlation of +0.89 between true and estimated values. To examine uncertainty in the estimates, 80% and 95% confidence intervals were constructed around the mean estimate for each true ability level. As expected, results showed less uncertainty at ability levels where more response data were generated, indicating the DKT approach is sensitive to ability level information in the training data. Finally, the DKT approach was able to detect learning across attempts within a skill, suggesting this approach may be preferred over traditional measurement approaches in scenarios where ability levels change over time.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Reference
Corbett, A.T., Anderson, J.R.: Knowledge tracing: modeling the acquisition of procedural knowledge. User Model. User-Adap. Inter. 4(4), 253–278 (1994). https://doi.org/10.1007/BF01099821
Donghua, L., Yanming, J., Jian, Z., Wufeng, W., Ning, X.: Deep knowledge tracing based on Bayesian neural network. In: Xhafa, F., Patnaik, S., Tavana, M. (eds.) IISA 2019. AISC, vol. 1084, pp. 29–37. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-34387-3_4
Gal, Y., Ghahramani, Z.: A theoretically grounded application of dropout in recurrent neural networks. Adv. Neural Inf. Process. Syst. 29, 1019–1027 (2016)
Graves, A., Mohamed, A.R., Hinton, G.: Speech recognition with deep recurrent neural networks. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. pp. 6645–6649. IEEE (2013)
Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: International Conference on Machine Learning, pp. 1321–1330. PMLR (2017)
King, D.R.: Production implementation of recurrent neural networks in adaptive instructional systems. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2020. LNCS, vol. 12214, pp. 350–361. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50788-6_25
Kuleshov, V., Fenner, N., Ermon, S.: Accurate uncertainties for deep learning using calibrated regression. In: International Conference on Machine Learning, pp. 2796–2804. PMLR (2018)
Lord, F.M.: Applications of Item Response Theory to Practical Testing Problems. Lawrence Eribaum Associates Inc., Hillsdale (1980)
Mao, Y.: Deep learning vs. Bayesian knowledge tracing: student models for interventions. J. Educ. Data Min. 10(2) (2018)
Narkhede, S.: Understanding AUC-ROC curve. Towards Data Sci. 26, 220–227 (2018)
Piech, C., et al.: Deep knowledge tracing. arXiv preprint arXiv:1506.05908 (2015)
Yeung, C.K., Yeung, D.Y.: Addressing two problems in deep knowledge tracing via prediction-consistent regularization. In: Proceedings of the Fifth Annual ACM Conference on Learning at Scale, pp. 1–10 (2018)
Zhang, L., Xiong, X., Zhao, S., Botelho, A., Heffernan, N.T.: Incorporating rich features into deep knowledge tracing. In: Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale, pp. 169–172 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
King, D.R., Zhou, Z., Therior, W. (2021). Uncertainty of Skill Estimates in Operational Deep Knowledge Tracing. In: Stephanidis, C., et al. HCI International 2021 - Late Breaking Papers: Cognition, Inclusion, Learning, and Culture. HCII 2021. Lecture Notes in Computer Science(), vol 13096. Springer, Cham. https://doi.org/10.1007/978-3-030-90328-2_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-90328-2_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-90327-5
Online ISBN: 978-3-030-90328-2
eBook Packages: Computer ScienceComputer Science (R0)