Abstract
This paper is intended as guidance for those who are familiar with user modeling field but are less fluent in statistical methods. It addresses potential problems with user model selection and evaluation, that are often clear to expert modelers, but are not obvious for others. These problems are frequently a result of a falsely straightforward application of statistics to user modeling (e.g. over-reliance on model fit metrics). In such cases, absolute trust in arguably shallow model accuracy measures could lead to selecting models that are hard-to-interpret, less meaningful, over-fit, and less generalizable. We offer a list of questions to consider in order to avoid these modeling pitfalls. Each of the listed questions is backed by an illustrative example based on the user modeling approach called Performance Factors Analysis (PFA) [9].
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Baayen, R.H., Davidson, D.J., Bates, D.M.: Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language 59(4), 390–412 (2008)
Cen, H., Koedinger, K.R., Junker, B.: Comparing Two IRT Models for Conjunctive Skills. In: Woolf, B.P., Aïmeur, E., Nkambou, R., Lajoie, S. (eds.) ITS 2008. LNCS, vol. 5091, pp. 796–798. Springer, Heidelberg (2008)
Clark, H.H.: The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior 12, 335–359 (1973)
Corbett, A.T., Anderson, J.R.: Locus of feedback control in computer-based tutoring: impact on learning rate, achievement and attitudes. In: Proceedings of CHI 2002, Human Factors in Computing Systems, Seattle, WA, USA, March 31-April 5, pp. 245–252. ACM, New York (2001)
Cummins, D.D., Kintsch, W., Reusser, K., Weimer, R.: The role of understanding in solving algebra word problems. Cognitive Psychology 20, 405–438 (1988)
Koedinger, K.R., Nathan, M.J.: The real story behind story problems: Effects of representation on quantitative reasoning. Journal of the Learning Sciences 13, 129–164 (2004)
van der Linden, W.J., Hambleton, R.K. (eds.): Handbook of Modern Item Response Theory. Springer, New York (1997)
Pavlik, P.I., Cen, H., Koedinger, K.R.: Learning factors transfer analysis: Using learning curve analysis to automatically generate domain models. In: Barnes, T., Desmarais, M., Romero, C., Ventura, S. (eds.) Proceedings of The 2nd International Conference on Educational Data Mining, Cordoba, Spain, pp. 121–130 (2009)
Pavlik Jr., P.I., Cen, H., Koedinger, K.R.: Performance factors analysis – A new alternative to knowledge tracing. In: Dimitrova, V., Mizoguchi, R. (eds.) Proceedings of the 14th International Conference on Artificial Intelligence in Education, Brighton, England (2009)
Pitt, M.A., Myung, I.J., Zhang, S.: Toward a method of selecting among computational models of cognition. Psychological Review 109(3), 472–491 (2002)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Yudelson, M., Pavlik, P.I., Koedinger, K.R. (2011). User Modeling – A Notoriously Black Art. In: Konstan, J.A., Conejo, R., Marzo, J.L., Oliver, N. (eds) User Modeling, Adaption and Personalization. UMAP 2011. Lecture Notes in Computer Science, vol 6787. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-22362-4_27
Download citation
DOI: https://doi.org/10.1007/978-3-642-22362-4_27
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-22361-7
Online ISBN: 978-3-642-22362-4
eBook Packages: Computer ScienceComputer Science (R0)