Commentary on: “Toward Computer-Based Support of MetaCognitive Skills: a Computational Framework to Coach Self Explanation”
This paper is a commentary on “Toward Computer-Based Support of Meta-Cognitive Skills: a Computational Framework to Coach Self-Explanation”, by Cristina Conati and Kurt Vanlehn, published in the IJAED in 2000 (Conati and VanLehn 2010). This work was one of the first examples of Intelligent Learning Environments (ILE) that target activities beyond problem solving doing so by providing adaptive support to apply relevant domain independent meta-cognitive skills, as opposed to targeting domain-dependent knowledge. We provide an overview of the SE-Coach, the ILE presented in (Conati and Vanlehn 2000), in terms of underlying motivation, objectives, appeach and contibutions. This overview is followed by a discussion of subsequent developments in research on providing intelligent, student-adaptive support to meta-cognition, as well as of remaining open issues and avenues for future research.
KeywordsIntelligent tutoring for meta-cognition Student modeling Self-explanation
One of the goals of AI in education research is to devise computer-based environments that provide learning experiences that is student-adaptive, i.e., tailored to the needs of each student (Intelligent Learning Environments, or ILE, from now on). To date, research in this field has successfully delivered techniques and systems that provide adaptive support for student problem solving in a variety of domains. There are, however, other educational activities that can benefit from individualized computer-based support, such as studying examples, exploring interactive simulations and playing educational games. Providing individualized support for these activities poses unique challenges, because it requires having an ILE that can model and adapt to student behaviors, skills and mental states often not as structured and well-defined as those involved in problem solving. For instance, an ILE that provides support to learning from worked-out examples needs to understand, at some level, both what it means to study an example effectively and how to detect when a student is not doing it. This is arguably more difficult than monitoring a student’s problem solving progress because when studying an example there are no overt naturally occurring cues of poor outcome as there are in problem solving (e.g., incorrect answers).
The SE-Coach system described in Conati and Vanlehn (2000) is one of the first examples of research in AI and Education addressing these unique challenges, and it was innovative as an ILE in two ways. First, the SE-Coach was the first attempt at understanding how to provide computer-based support to learning from examples that was student-adaptive, i.e., individualized to the perceived needs of each individual students. Second, this individualized support is designed to help students acquire a domain independent meta-cognitive skill needed to study examples effectively, as opposed to focusing on domain-depended cognitive skills as many ILE do. The meta-cognitive skill targeted by the SE-Coach is self-explanation, namely one’s tendency to explain and elaborate the studied instructional material to one-self in light of one’s existing domain knowledge (Chi et al. 1989). In the rest of this commentary, we first summarize the rationale, methodology, and contributions of the research in Conati and Vanlehn (2000). Next, we discuss some of the progress made in the last 15 years on the themes addressed by this work, as well as future needs for tackling the remaining open issues.
SE-Coach: Why, How and What We Learned
Motivation and General Approach
The SE-Coach is an ILE designed to help students self-explain worked-out example solutions in introductory physics. The rationale underlying the design of the SE-Coach is grounded in Cognitive Science findings showing that students can greatly benefit from studying examples after receiving theoretical instruction on a domain and before starting to solve problems in that domain. The effectiveness of an example-studying activity, however, is mediated by how well students process the available example solutions. Students who engage in self-explanation to better understand the examples show the greatest benefit during problem solving while students who study the examples more superficially don’t necessarily learn from this activity. Cognitive science studies have further shown that prompting for self-explanation can help those students who tend to not self-explain spontaneously (see Chi 2000 for an overview). The goal of the SE-Coach is to automate the provision of this prompting by both monitoring students as they study examples and providing adaptive interventions to help students self-explain when they don’t do so spontaneously but self-explanations can help them improve their domain knowledge.
The notion that the SE-Coach’s prompts should be student-adaptive, i.e., tailored to student individual differences both at the cognitive level (e.g., existing knowledge) and meta-cognitive level (e.g., tendency to self-explain) was the key innovative point of this research. In other research that investigated prompting for self-explanation during problem solving, prompting occurred in a less individualized fashion, for instance by asking students to self-explain every possible problem solving step (Aleven and Koedinger 2002) or every new or incorrect problem solving step (Mitrovic 2003) regardless of whether students may self-explain even if not prompted and whether they may already understand the necessary justifications. In contrast, the SE-Coach provides individualized support to help a student better understand a given example based on its on-going assessment of the student’s knowledge, reading patterns, and explanations that the student can generate via dedicated interface tools. Based on this assessment, the SE-Coach guides the student to more carefully explain only parts of the example that may not be fully understood.
An explicit representation of what it means to understand an example solution. This representation is based on the solution graph, a dependency network that models how each solution step derives from previous steps and physics knowledge leveraged also by Andes (Conati et al. 2002). The SE-Coach sees the task of learning from one of its examples as understanding all the dependencies between solution steps and physics knowledge encoded by the corresponding solution graph. Thus it scaffolds the students in generating self-explanations for these dependencies.
Interface tools to scaffold the target self-explanations. The philosophy underlying the design of these tools is to provide guidance to self-explanation incrementally, in order to interfere as little as possible with a student’s spontaneous behaviors. This incremental guidance is delivered via three levels of scaffolding: (i) a “masking interface” that requires students to mouse over each solution step in the example in order to see it. The fact that not all solution steps are visible at once helps students focus their attention and reflect on individual steps; (ii) explicit prompts to self-explain that appear next to each solution step when it is uncovered; (iii) menu-based tools designed to provide constructive but controllable ways to generate the different types of explanations relevant for a solution step (e.g., which physics rule the step was derived from or what role the step plays in the overall solution), to help those students who would be unable to properly self-explain if left to their own devices. The SE-Coach evaluates the correctness of explanations generated via the menu-based tools based on the information contained in the solution graph, and provides feedback for correctness as needed.
A probabilistic student model. This model leverages the solution graph and the observed student’s interface actions to assess which parts of the example have been explained correctly and how the explanations impacts a student’s physics knowledge.
Initially, self-explanation is voluntary. The interface tools described above are designed to provide incremental scaffolding for self-explanation that students can access at their own discretion to adhere to our goal of providing an environment that stimulates as much as possible the student’s initiative in the learning process. However, to help those students who are not receptive to interface scaffolding because of their low tendency for self-explanation, the SE-Coach can also provide more direct tutorial interventions, targeting specific limitations in a student’s self-explanation behaviour. To identify these limitations, the SE-Coach’s student model keeps track of each student’s progress through an example, including how much time the student looked at each solution line (through the masking interface), what she chose to self-explain via the interface tools and whether or not the self-explanations were correct. This information is collected in the SE-Coach’s student model as evidence to assess which example lines may benefit from further self-explanation. Then, when the student tries to close the example, the SE-Coach generates prompts to make the student self-explain those parts.
The SE-Coach was evaluated in a controlled study with 56 college students who were taking an Introductory Physics course and had recently covered Newton’s Laws (the topic of the examples used in the study). The rational for this selection criterion was to have participants who could understand the topic of the examples, but would not be so knowledgeable that they would find the examples not worthy of attention. The students came from 4 different colleges.
Log data analysis revealed that the SE-Coach’s interface is easy to use and explicit tutorial interventions are quite successful at stimulating self-explanation.
The SE-Coach’s multiple levels of scaffolding during example studying improved1 students’ problem solving performance when students were in the early stage of learning the relevant physics knowledge. Namely, students who had just started covering Newton’s Laws learned more with the adaptive version of the tutor than with the control version. The adaptive version was not as effective for students who had started covering Newton’s laws earlier in the term. These students didn’t have higher initial physics knowledge than the students in the early stage of learning (as measured by the pre-test), however, it appears that for these students the milder form of scaffolding provided by the control version of the SE-Coach (i.e., the masking interface and untailored reminders) was sufficient to trigger effective self-explanation, whereas they tended to ignore the more elaborated guidance provided by the SE-Coach in the experimental condition.
Thus, this research provided encouraging preliminary evidence that (i) it is possible and useful (if done at the right time during the curriculum) to give students individualized guidance to studying examples effectively before moving to problem solving; (ii) this guidance can target a meta-cognitive skill (iii); the guidance can improve student learning of the target domain knowledge from the activity during which they received meta-cognitive support. It also opened several avenues for future investigation which we address in the subsequent sections.
Providing Computer Based Support to Meta-Cognitive Skills
Since the appearance of Conati and VanLehn (2000), increasing AIED research has been devoted to designing ILE that can explicitly support student meta-cognitive skills. Some of this work has targeted self-explanation in a variety of instructional activities, including studying textual example solutions (e.g., Crippen and Boyd 2007), exploring interactive simulations (Conati and Merten 2007), viewing examples as Flash videos (McLaren et al. 2008), engaging in problem solving (e.g., Aleven and Koedinger 2002; Mitrovic 2003) and in problem solving with multiple representations (Rau et al. 2009).
Researchers have also started investigating support for other meta-cognitive skills, ranging from the ability to seek help effectively (Roll et al. 2007), using examples during analogical problem solving (Muldner and Conati 2007, 2010), applying specific problem solving strategies (Chi and VanLehn 2007), and successfully engaging in self-regulated learning (SRL). The latter is a comprehensive meta-cognitive process which leverages many of the other aforementioned meta-cognitive skills. With SRL, students set their own learning goals, plan actions to achieve them and then self-monitor their progress towards these goals as they engage in the target activities (Azevedo et al. 2009; Tan et al. 2006).
Some of this research has already generated evidence confirming our preliminary findings that scaffolding the suitable meta-cognitive skills can impact short-term student performance (e.g., Muldner and Conati 2010; Roll et al. 2007; Azevedo et al. 2009; Rau et al. 2009; Crippen and Boyd 2007; McLaren et al. 2008; Aleven and Koedinger 2002). On the other hand, there is still limited evidence that computer-based support to meta-cognition can impact student meta-cognitive abilities in the long term. Leelawong and Biswas (2008) did not find significant differences in transfer and retention of SRL skills between students who received explicit scaffolding for these skills and students who did not. Similarly, Roll et al. (2007) did not find any significant difference in retention of help-seeking strategies between students who were scaffolded for them and those who were not. Chi and VanLehn (2007) have shown in a laboratory experiment that students scaffolded by a computer tutor to apply specific problem solving strategies in one educational domain were able to apply the same strategies in a different domain, without scaffolding. Although this work shows the potential of meta-cognitive scaffolding for teaching transferable domain-independent skills, all problem solving happened during the same experimental session, thus the study does not provide evidence of long-term skill retention.
The lack of findings on the long-term effects of providing computer-based support to meta-cognitive skills may be in part due to the fact that few studies have tried to provide such instruction and guidance over the longer periods of time that may be required for students to truly learn and understand how to apply these skills. Furthermore, it is very possible that a specific ILE focused on teaching metacognitive skills is only a part of the answer. Effective acquisition of meta-cognitve skills might require exposure to a range of situations in which the learner can experience and practice these skills, and might depend of the target meta-cognitive skills and on pre-existing student attitudes and abilities. Understanding how to tackle the complexity of these issues with suitable technology and adequately crafted studies is a key challenge and an important direction of future research in this field.
Understanding the Value of Student-Adaptive Scaffolding
As illustrated by our SE-Coach work, there are different, non-mutually exclusive, approaches that can be used to provide computer-based support to meta-cognition. One approach is to design interface affordances that can facilitate the desired meta-cognitive processes that students can leverage at their discretion (e.g., Atkinson et al. 2000; van Joolingen 2000; Luckin and Hammerton 2002; Azevedo et al. 2009). A second approach consists of providing more proactive support during interaction to complement existing interface affordances. Within this approach, the choice can be made on how much the real-time support should be tailored to the student’s specific needs.
Most researchers have investigated the value of non-tailored support, namely all students are prompted to practice the target meta-cognitive skills at every relevant opportunity (e.g., Chi and Vanlehn 2007; Aleven and Koedinger 2002; Crippen and Boyd 2007; McLaren et al. 2008; Rau et al. 2009). Mitrovic (2003) went a step further and devised support for self-explanation during problem solving that was tailored to simple indicators such as whether a problem solving action was new for the student or was incorrect. Some researchers have looked at providing more tailored interventions in line with those generated by the SE-Coach. For instance, Tan et al. (2006) Betty’s Brain is an environment that uses teachable agents to help students learn both domain and self-regulatory skills. To support the latter, the environment can recognize a number of interaction patters indicating poor SRL (e.g., teaching the agent new concepts but no relationships between them, inability to identify correct and incorrect links from graded quiz questions), and provides feedback accordingly. Muldner and Conati (2007, 2010) devised an ILE (EA-Coach) that supports users during analogical problem solving (APS) by selecting examples that foster the appropriate meta-cognitive skills (self-explanation and min-analogy, namely transferring knowledge from the example to the problem when needed as opposed to brute force copying of steps). The EA-Coach includes a probabilistic student model that encodes the student’s current domain knowledge and proficiency in the target APS skills. Example selection is driven by a decision-theoretic mechanism that computes, given the current assessment in the student model, the utility of each example in the EA-Coach’s pool to help the student both solve the current problem as well as learn from the process.
While there is some evidence for the value of each of the approaches above, more systematic ablation studies should be conducted to understand their comparative effectiveness, not only in terms of learning outcomes, but also considering student acceptance and satisfaction. In particular, researchers should strive to identify precise guidelines for when different levels of scaffolding can be suitable, given the meta-cognitive skills(s) to be taught. For instance, the Conati and VanLehn (2000) study only compared the fully adaptive SE-Coach with multiple levels of scaffolding against providing simple untailored prompts at every example step read by the student. More combinations of the two dimensions level of interface scaffolding (e.g., simple prompts vs. interface tools to scaffold explanations) and how to scaffold (e.g., scaffold at every step vs. scaffold in a student-adaptive manner) should be compared to identify under which conditions they might be effective.
Improving Student Modeling for Meta-Cognitive Support
Student modelling is recognized to be a very challenging computational problem because of the gap that often exists between the student’s behaviors that an ILE can observe and the student’s states and traits that the ILE needs to assess. Arguably, the more complex the user’s states and abilities to be captured, the more difficult they are to assess unobtrusively from basic interaction events. For instance, assessing student self-explanation unobtrusively requires recognizing when students selfexplain spontaneously in their head. This requirement is hard to fulfil for a software system that has no direct access to students’ thoughts or verbalizations.
The difficulty of capturing the relevant cognitive and meta-cognitive abilities is a bottleneck to providing highly personalised support to meta-cognition. However, advances in sensing technology have made it possible to go beyond interaction events in student modelling for meta-cognitive support, with the potential of reducing the gap between student observable behaviors and mental states to be assessed. For instance, Conati and Merten (2007) relied on eye-tracking technology to improve recognition of student spontaneous self-explanation in the context of learning by exploration of an interactive simulation. They showed that information on user attention patterns collected via an eye-tracker significantly increase a student model’s ability to capture self-explanation compared to a lower-level time-based predictor. This, in turn, increases the model’s accuracy in predicting student learning outcomes. It should be noted that some eye-trackers also provide data on pupil dilation (which has been reliably correlated with cognitive load) as well as on distance of the head from the screen (an approximation of posture, which is a good predictor of engagement/disengagement). Thus, future research on using eye-trackers in student models of meta-cognition should look at this additional information which comes “for free” with gaze data. Other data sources that are promising, although so far they have only been investigated for the detection of a student’s affective states, include: (i) electroencephalography (EEG), which can provide direct information on the brain areas that are activated during learning; (ii) software for recognition of facial expressions and sensors for electro dermal activity (EDA), which provide different ways to detect states of concentration, flow and engagement, namely affective states that may indicate meta-cognitive activity as opposed to superficial studying.
Researchers have also leveraged advances in Data Mining and AI to move toward richer student models for meta-cognition. For instance, Kinnebrew et al. (2013) and Sabourin et al. (2013) used data mining approaches to identify the SRL behaviors that distinguish effective and ineffective learners with two different ILE designed to scaffold SRL skills. They had the long-term goal of building student models that can help the ILE foster the appropriate behaviors and discourage the detrimental ones, as needed by each student. Aleven et al. (2004) discuss the Geometry-Tutor, an ILE that supports self-explanation during problem solving by allowing students to type their self-explanations in free form text and using natural language processing to provide detailed feedback on their quality. Currently the Geometry Tutor still requires students to self-explain every problem solving step they encounter, however, the assessment of self-explanation quality generated by the natural language module could potentially inform a student model of the student’s overall domain knowledge and explanation abilities. This student model could then be used to scaffold self-explanations in a more selective manner, i.e., if when the student is assessed to have limited knowledge and/or self-explanation abilities. Interestingly, a study that compared the effectiveness of the Geometry Tutor against a version that gave students menu-based tools to generate their self-explanation showed no significant differences in the learning outcomes of the two group. Aleven et al. (2004) indicate that a possible reason for this outcome is that even though there is an advantage of generating self-explanations in natural language, it did not materialize because these explanations took too long to generate. Asking students to self-explain only when a student model indicates that it is advantageous may increase the effectiveness of the NL approach proposed in this work.
In general, all the advances that are being made in applying student modeling technologies to gain a better understanding of students’ self explanation and other meta-cognitive abilities, combined with the assessmen of student affect and engagement, are a very promising direction of future research, which may eventually lead to a more comprehensive theory of Self Regulated Learning, and of how to measure and scaffold SRL in computer-based learning environments.
In this commentary, we provided an overview and discussed the SE-Coach, one of the first examples of Intelligent Learning Environments (ILE) that target activities beyond problem solving, doing so by providing adaptive support to apply relevant domain-independent meta-cognitive skills, as opposed to targeting domain-dependent knowledge. There is already strong evidence that ILE can have a substantial impact on education when they target problem solving skills. With the advancement of Massive Open Online Courses (MOOC), where a student educational experience is mostly based on on-line tools, providing ILEs that can support a variety of student-adaptive learning activities and foster long-term, domain independent learning skills has the potential to shape the future of education. We have discussed some of the advancements that have been made on how to design these ILEs, as well as some of the remaining challenges. We expect this line of research to be one of the dominant threads in AIED in the years to come, an exciting interdisciplinary effort bridging advances in Cognitive Science, Education, Artificial Intelligence and Human Computer Interactions.
At the standard level of statistical significance, p < 0.05
- Aleven, V., A. Ogan, et al. (2004). Evaluating the effectiveness of a tutorial dialogue system for self-explanation. Intelligent Tutoring Systems: 7th International Conference, Maceió, Alagoas, Brazil, 443–454.Google Scholar
- Chi, M. T. H. (2000). Self-explaining expository texts: the dual processes of generating inferences and repairing mental models. In R. Glaser (Ed.), Advances in instructional psychology (pp. 161–238). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
- Chi, M., & VanLehn, K. (2007). The impact of explicit strategy instruction on problem solving behaviors across intelligent tutoring systems. In Proceedings of the 29th annual conference of the cognitive science society (pp. 167–172). Mahwah, NJ: Erlbaum.Google Scholar
- Conati C. and Merten C. (2007). Eye-tracking for user modeling in exploratory learning environments: an empirical evaluation. Knowledge based systems, vol. 20, issue 6. Amsterdam, The Netherlands: Elsevier Science Publishers B. V.Google Scholar
- Conati, C., & VanLehn, K. (2000). Toward computer-based support of meta-cognitive skills: a computational framework to coach self-explanation. International Journal of Artificial Intelligence in Education, 11, 398–415.Google Scholar
- Kinnebrew, S., Mack, D., and Biswas, G. (2013). Mining temporally-interesting learning behavior patterns. In Proceedings of EDM 6th International Congerence of Educational Data Mining, 252–255.Google Scholar
- Leelawong, K., & Biswas, G. (2008). Designing learning by teaching agents: the Betty’s brain system. International Journal of Artificial Intelligence in Education, 18(3), 181–208.Google Scholar
- Luckin, R., & Hammerton, L. (2002). Getting to know me: helping learners understand their own learning needs through metacognitive scaffolding. In Intelligent Tutoring Systems (Ed.), Proceedings of the 6th international conference ITS 2002 (pp. 759–771). Berlin/Heidelberg: Springer.Google Scholar
- McLaren, B., Lim, S., & Koedinger, K. (2008). When and how often should worked examples be given to students? New results and a summary of the current state of research. 30th annual conference of the cognitive science society (pp. 2176–2181). Austin, TX: Cognitive Science Society.Google Scholar
- Mitrovic, T. (2003). Supporting self-explanation in a data normalization tutor supplementary proceedings of AIED 2003, pp. 565–577.Google Scholar
- Muldner, K. & Conati, C. (2007). Evaluating a decision-theoretic approach to tailored example selection. In: Proc. of the 20th International Joint Conference on Artificial Intelligence (IJCAI’07), 483–489.Google Scholar
- Muldner, K., & Conati, C. (2010). Scaffolding meta-cognitive skills for effective analogical problem solving via tailored example selection. International Journal of Artificial Intelligence in Education, 20(2), 99–136.Google Scholar
- Rau, M., Aleven, V., & Rummel, N. (2009). Intelligent tutoring systems with multiple representations and self-explanation prompts support learning of fractions. 14th International Conference on Artificial Intelligence in Education, AIED 2009 (pp. 441–449). Amsterdam: IOS Press.Google Scholar
- Roll, I., Aleven, V., McLaren, B., & Koedinger, K. (2007). Can help seeking be tutored? Searching for the secret sauce of metacognitive tutoring. In R. Luckin, K. Koedinger, & J. Greer (Eds.), Proceedings of the 13th international conference on artificial intelligence in education (pp. 203–210). Amsterdam, the Netherlands: Ios Press.Google Scholar
- Tan, J., Biswas, G., & Schwartz, D. (2006). Feedback for metacognitive support in learning by teaching environments. Meeting of the cognitive science society, (pp. 828–833). Vancouver.Google Scholar
- van Joolingen, W. (2000). Designing for collaborative discovery learning. In: Proceedings of the fifth international conference on intelligent tutoring systems (ILE 2000). Montreal, Canada, pp. 202–211.Google Scholar