Abstract
Personalised environments such as adaptive educational systems can be evaluated and compared using performance curves. Such summative studies are useful for determining whether or not new modifications enhance or degrade performance. Performance curves also have the potential to be utilised in formative studies that can shape adaptive model design at a much finer level of granularity. We describe the use of learning curves for evaluating personalised educational systems and outline some of the potential pitfalls and how they may be overcome. We then describe three studies in which we demonstrate how learning curves can be used to drive changes in the user model. First, we show how using learning curves for subsets of the domain model can yield insight into the appropriateness of the model’s structure. In the second study we use this method to experiment with model granularity. Finally, we use learning curves to analyse a large volume of user data to explore the feasibility of using them as a reliable method for fine-tuning a system’s model. The results of these experiments demonstrate the successful use of performance curves in formative studies of adaptive educational systems.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ainsworth S.E., Grimshaw S.: Evaluating the REDEEM authoring tool: can teachers create effective learning environments?. Int. J. Artif. Intell. Educ. 14(3), 279–312 (2004)
Anderson J.R.: Rules of the Mind. Lawrence Erlbaum Associates, Hillsdale, NJ (1993)
Anderson J.R., Corbett A.T., Koedinger K.R., Pelletier R.: Cognitive tutors: lessons learned. J. Learn. Sci. 4(2), 167–207 (1995)
Baker, R.S.J.D., Habgood, M.P.J., Ainsworth, S.E., Corbett, A.T.: Modeling the acquisition of fluent skill in educational action games. In: Conati, C., McCoy, K., Paliouras, G. (eds.) UM2007, vol. 4511, pp. 17–26. LNCS, Corfu (2007)
Cen H., Koedinger K.R., Junker B.: Learning factors analysis: a general method for cognitive model evaluation and improvement. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds) Proceedings of the 8th International Conference on Intelligent Tutoring Systems, vol. 4053, pp. 164–175. LNCS, Jhongli, Taiwan (2006)
Eagle M., Barnes T.: Intelligent tutoring systems, educational data mining, and the design and evaluation of video games. In: Aleven, V., Kay, J., Mostow, J. (eds) ITS2010, vol. 6094, pp. 215–217. LNCS, Pittsburgh, USA (2010)
Heathcote A., Brown S., Mewhort D.J.: The power law repealed: the case for an exponential law of practice. Psychon. Bull. Rev. 7(2), 185–207 (2000)
Holt P., Dubs S., Jones M., Greer J.: The state of student modeling. In: Greer, J., McCalla, G. (eds) Student Modeling: The Key to Individualized Knowledge-Based Instruction, pp. 3–39. Springer, New York (1994)
Koedinger, K.R., Mathan, S.: Distinguishing qualitatively different kinds of learning using log files and learning curves. In: ITS 2004 Log Analysis Workshop, Maceio, Brazil, pp. 39–46, (2004)
Koedinger, K.R., Corbett, A.C., Perfetti, C.: The Knowledge-Learning-Instruction (KLI) framework: toward bridging the science-practice chasm to enhance robust student learning. CMU-HCII Tech Report 10–102. http://reports-archive.adm.cs.cmu.edu/hcii.html (2010)
Martin B.: Constraint-based modelling: representing student knowledge. N. Z. J. Comput. 7(2), 30–38 (1999)
Martin, B., Mitrovic, A.: Authoring web-based tutoring systems with WETAS. In: International Conference on Computers in Education, Auckland, pp. 183–187, (2002a)
Martin, B., Mitrovic, A.: Automatic problem generation in constraint-based tutors. In: Sixth International Conference on Intelligent Tutoring Systems, Biarritz, pp. 388–398, (2002b)
Martin, B., Mitrovic, A.: ITS domain modelling: art or science? In: International Conference on Artificial Intelligence in Education, AIED2003, Sydney, Australia, pp. 183–190, (2003)
Martin B., Mitrovic A.: The effect of adapting feedback generality in ITS. In: Wade, V., Ashman, H., Smyth, B. (eds) AH2006, vol. 4018, pp. 192–202. LNCS, Dublin, Ireland (2006)
Martin, B., Koedinger, K.R., Mitrovic, A., Mathan, S.: On using learning curves to evaluate ITS. In: AIED 2005, Amsterdam, pp. 419–426, (2005)
Mathan, S.: Recasting the feedback debate: benefits of tutoring error detection and correction skills. PhD thesis, School of Computer Science, Human–Computer Interaction Institute. Pittsburgh, PA, Carnegie Mellon University, 130 pp (2003)
Mitrovic A.: An intelligent SQL Tutor on the web. Int. J. Artif. Intell. Educ. 13(2–4), 173–197 (2003)
Mitrovic A., Ohlsson S.: Evaluation of a constraint-based tutor for a database language. Int. J. Artif. Intell. Educ. 10, 238–256 (1999)
Mitrovic A., Martin B., Mayo M.: Using evaluation to shape ITS design: Results and experiences with SQL-Tutor. User Model. User Adapt. Interact. 12(2-3), 243–279 (2002)
Mizoguchi R., Bourdeau J.: Using ontological engineering to overcome common AI-ED problems. Int. J. Artif. Intell. Educ. 11, 107–121 (2000)
Munro A., Johnson M.C., Pizzini Q.A., Surmon D.S., Towne D.M., Wogulis J.L.: Authoring simulation-centred tutors with RIDES. Int. J. Artif. Intell. Educ. 8, 284–316 (1997)
Newell A., Rosenbloom P.S.: Mechanisms of skill acquisition and the law of practice. In: Anderson, J.R. (ed.) Cognitive Skills and Their Acquisition, pp. 1–56. Lawrence Erlbaum Associates, Hillsdale, NJ (1981)
Nwaigwe, A., Koedinger, K. R., VanLehn, K., Hausmann, R., Weinstein, A.: Exploring alternative methods for error attribution in learning curves analysis in intelligent tutoring systems. In: AIED2007, Los Angeles, pp. 246–253 (2007)
Ohlsson S.: Constraint-based student modeling. In: Greer, J., McCalla, G. (eds) Student Modeling: The Key to Individualized Knowledge-Based Instruction, pp. 167–189. Springer, New York (1994)
Paramythis A., Weibelzahl S.: A decomposition model for the layered evaluation of interactive adaptive systems. In: Ardissono, L., Brna, P., Mitrovic, A. (eds) 10th International Conference on User Modeling (UM2005), vol. 3538, pp. 438–442. LNCS, Edinburgh, Scotland (2005)
Pavlik P.I. Jr., Cen H., Koedinger K.R.: Learning factors transfer analysis: using learning curve analysis to automatically generate domain models. In: Barnes, T., Desmarais, M., Romero, C., Ventura, S. (eds) Proceedings of the 2nd International Conference on Educational Data Mining, pp. 121–130. Universidad de Cordoba, Cordoba, Spain (2009)
Snoddy G.S.: Learning and stability. J. Appl. Psychol. 10, 1–36 (1926)
Stevens J.C., Savin H.B.: On the form of learning curves. J. Exp. Anal. Behav. 5(1), 15–18 (1962)
Suraweera P., Mitrovic A.: An intelligent tutoring system for entity relationship modelling. Int. J. Artif. Intell. Educ. 14(3), 375–417 (2004)
Suraweera P., Mitrovic A., Martin B.: The role of domain ontology in knowledge acquisition for ITS. In: Lester, J.C., Vicari, R., Paraguacu, F. (eds) Seventh International Conference on Intelligent Tutoring Systems, vol. 3220, pp. 207–216. LNCS, Maceio, Brazil (2004)
Uresti J., Du Boulay B.: Expertise, motivation and teaching in learning companion systems. Int. J. Artif. Intell. Educ. 14, 67–106 (2004)
Walker A., Recker M., Lawless K., Wiley D.: Collaborative information filtering: a review and an educational application. Int. J. Artif. Intell. Educ. 14(1), 3–28 (2004)
Witten I.H., Frank E.: Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufman, Seattle (2005)
Wright T.P.: Factors affecting the cost of airplanes. J. Aeronaut. Sci. 3, 122–128 (1936)
Zapata-Rivera J.D., Greer J.E.: Interacting with inspectable Bayesian student models. Int. J. Artif. Intell. Educ. 14(2), 127–163 (2004)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Martin, B., Mitrovic, A., Koedinger, K.R. et al. Evaluating and improving adaptive educational systems with learning curves. User Model User-Adap Inter 21, 249–283 (2011). https://doi.org/10.1007/s11257-010-9084-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11257-010-9084-2