It is pleasing to see that the 25th Anniversary of IJAIED also brings a certain maturity to the field. In the last 5 years there have been a number useful and positive analyses of the effectiveness of AIED systems designed to tutor one-to-one. For example, VanLehn (2011) found that the effectiveness of the “step-based” intelligent tutoring systems he surveyed were nearly as effective as average human tutors, and that neither of these were as effective as expert human tutors, but certainly both better than providing the domain content without tutoring. “Step-based” means that the tutor evaluates and reacts to each step that the learner makes in a problem-solving session as opposed to evaluating only the final answer given by the learner.
In a meta-analysis, Ma et al. (2014) found similar positive results for step-based ITSs both when compared to a no tutoring condition and when compared to large group human teacher led-instruction, but no differences when compared to small group human tutoring or to one–to–one tutoring. The same authors analysed systems for teaching programming and also found a “a significant advantage of ITSs over teacher-led classroom instruction and non-ITS computer-based instruction” (Nesbit et al. 2014). Likewise Kulik and Fletcher (2015) found similar sized improvements but distinguished between studies that used standardised tests and those where the tests were more specifically tuned to the system providing tuition. Smaller effect sizes were found by Steenbergen-Hu and Cooper (2013) in their meta-analysis of pupils using ITSs in a School setting. They also noted that lower-achievers seemed to do worse with ITSs than did the broad spectrum of school pupils. In a parallel study of university students, Steenbergen-Hu and Cooper (2014) found more positive effects for ITSs as compared to conventional instruction.
Finally in a large-scale study in the USA of the Cognitive Tutors, (Pane et al. 2014) found only limited evidence of the relative effectiveness of these tutors over conventional teaching, though we note that how the tutors were actually used in the classrooms was not strongly controlled. While not specifically addressing ITS effectiveness, Schroeder et al. (2013) found that online pedagogical agents “produced a small but significant effect on learning”.
References
Kulik, J. A., & Fletcher, J. D. (2015). Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review. Review of Educational Research, X(X), 1–37. doi: 10.3102/0034654315581420
Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: a meta-analysis. Journal of Educational Psychology, 106(4), 901–918. doi:10.1037/a0037123.
Nesbit, J. C., Adesope, O. O., Liu, Q., & Ma, W. (2014). How Effective are Intelligent Tutoring Systems in Computer Science Education? Paper presented at the IEEE 14th International Conference on Advanced Learning Technologies (ICALT), Athens, Greece.
Pane, J. F., Griffin, B. A., McCaffrey, D. F., & Karam, R. (2014). Effectiveness of cognitive tutor Algebra I at scale. Educational Evaluation and Policy Analysis, 36(2), 127–144. doi:10.3102/0162373713507480.
Schroeder, N. L., Adesope, O. O., & Gilbert, R. B. (2013). How effective are pedagogical agents for learning? A meta-analytic review. Journal of Educational Computing Research, 49(1), 1–39. doi:10.2190/EC.49.1.a.
Steenbergen-Hu, S., & Cooper, H. (2013). A meta-analysis of the effectiveness of intelligent tutoring systems on K–12 students’ mathematical learning. Journal of Educational Psychology, 105(4), 970–987. doi:10.1037/a0032447.
Steenbergen-Hu, S., & Cooper, H. (2014). A meta-analysis of the effectiveness of intelligent tutoring systems on college students’ academic learning. Journal of Educational Psychology, 106(2), 331–347. doi:10.1037/a0034752.
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221. doi:10.1080/00461520.2011.611369.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
du Boulay, B. Recent Meta-reviews and Meta–analyses of AIED Systems. Int J Artif Intell Educ 26, 536–537 (2016). https://doi.org/10.1007/s40593-015-0060-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40593-015-0060-1