Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Concept maps [1] are the product of mapping one or more categorical propositions. These propositions are composed of two classes, known as the referent and the relatum, and a term, representing a binary or dyadic relation. Graphically, these elements take the form of nodes and labeled directed arcs, respectively. The nodes represent concepts or ideas within a subject area or domain, and the labeled directed arcs are binary relations which explain how two concepts are related. They have been applied to enhance both individual and collaborative learning, and there is strong evidence that their use is associated with increased knowledge transfer and retention across several instructional conditions, settings and methodological features [2, 3].

However, despite their graphical simplicity the construction of concept maps is complex and difficult for students, especially for newbies. Consequently, learner support or feedback is recommended. For instance, some researchers such as Cimolino et al. [4], have found that when students start out badly, with incorrect propositions, they tend to continue with further incorrect propositions until the map is grossly incorrect. In particular, feedback is a core aspect of all the known psychological perspectives about cognition and learning (see [5] for a thorough discussion of these perspectives) and it has been an important element in machine-mediated education since the days of Pressey’s teaching machines [6, 7]. Moreover, from a review of 12 meta-analyses that have included specific information on feedback in classrooms (based on 196 studies and 6972 effect-sizes), Hattie [8] found that the average effect size was d = .79 which places feedback among the top 10 influences on educational achievement.

From a human-computer interaction (HCI) perspective, there are three facts that underline the importance of feedback, at least when it comes to educational software. First, an information technology savvy generation, defined by the terms digital natives, homo zappiens, Net generation, iGeneration, Google generation, etc. (see [9] for definitions and references), which really understands what they are doing with information technology and, use it effectively and efficiently does not exist. In a review of the literature, Kirschner and van Merriënboer [9] found that learners do not really have deep knowledge of technology, and what knowledge they do have is often limited to basic office suite skills, e-mailing, text messaging, Facebook, Wikipedia and surfing the Internet. Social media, such as Blogs and Wikis, is used as a passive source of information and not as a tool for actively creating content, interacting with others, and sharing resources.

Second, the assumption that providing learners with control over the learning tasks they work on fosters their self-regulated learning skills and results in personalized learning trajectories [10, 11] is false. Most students do not reflect spontaneously on their learning processes [12] and consequently have difficulty in controlling and regulating their own learning. In particular, there is solid evidence, especially for computer-based learning environments, that students, particularly novices who lack prior knowledge of the learning tasks, do not apply and acquire self-regulation skills merely by engaging in self-regulated learning, but rather need additional support such as prompts or tutoring that stimulate them to reflect on their learning processes [9, 13, 14].

Third, the constructivist hypothesis that people learn best in an unguided or minimally guided environment is false. Following Kirschner et al. [15] this minimally guided approach has been called by various names including: discovery learning, problem-based learning (PBL), inquiry learning and constructivist learning. However, there is not a clear body of research using controlled experiments indicating that unguided or minimally guided instruction was more effective than guided or direct instruction. In fact, controlled experiments almost uniformly support direct, strong instructional guidance rather than constructivist-based minimal guidance during the instruction of novice to intermediate learners [15–17].

Several conclusions can be reached from these facts:

  1. 1.

    Students are not highly effective at managing their own interactions with the technology and, should not be trusted to be in control of these interactions.

  2. 2.

    The ubiquitous presence of technology in the lives of the learners has not resulted in improved information retrieval, information seeking or evaluation skills.

  3. 3.

    When it comes to reflecting and regulating their learning, students need additional training or instructional support.

  4. 4.

    Learners should be explicitly shown what to do and how to do it, especially when dealing with novel information.

In this sense, this paper analyzes four computer-based concept mapping tools that claim to provide some form of feedback and guidance for the learners [4, 18–20] addressing the following research questions: (a) what type of feedback does the software use?; (b) does the feedback provided adheres to a specific model found in the literature and if so which one?; (c) are there any controlled experiments or in-class studies that give account of the efficiency of the feedback provided by the software?

The rest of this paper is structured as follows: In Sect. 2, some of the definitions, purposes, typologies and models of feedback are addressed. In Sect. 3, the four concept-mapping tools are analyzed. Finally, in Sect. 4, some conclusions and future work are stated.

2 Feedback: Definitions, Purposes, Typologies and Models

Although feedback is highly cited in the learning and performance literature, there is a plethora or definitions, typologies and models. For instance, Mason and Bruning [21] define feedback as any message generated in response to a learner’s action, while Mory [22] states that it is information presented to the learner after any input with the purpose of shaping the perceptions of the learner or any message or display that the computer presents to the learner after a response. More recent definitions include the following: (a) information provided by an agent (e.g., teacher, peer, book, parent, self, experience, computer) regarding aspects of one’s performance or understanding [23]; (b) information communicated to the learner that is intended to modify his or her thinking or behavior for the purpose of improving learning [24]; (c) post-response information which informs learners about their actual state of learning or performance in order to regulate the further process of learning in the direction of the learning standards strived for [25].

When it comes to establishing the aim, goal or purpose of feedback, the literature presents a similar scenario. Mason and Bruning [21] state that feedback should help and guide learners to identify errors, become aware of misconceptions and regulate their learning. Mory [22] asserts that feedback should help learners on the correction and analysis of errors, with a predominant focus on all the metacognitive variables (e.g., reflection) involved in this process and should also keep students motivated. Hattie and Timperley [23] claim that the main purpose of feedback is to reduce discrepancies between what is understood, what is aimed to be understood (i.e., the learning goal(s)) and performance. Shute [25] maintains that the main purpose of feedback is to direct learners in order to increase their knowledge, skills, and understanding in some content area or general skill (e.g., problem solving). Narciss [24] declares that the goal of feedback is to contribute to the regulation of a learning process in such a way that learners acquire the knowledge and competencies needed to master learning tasks.

There is not, either, a unified typology of feedback. Mason and Bruning [21] provide a typology as well as Vasilyeva et al. [26], Shute [25] and Thurlings et al. [27]. Because of the lack of space, only the typology of Shute is presented as it has several similarities with the one’s of Mason & Bruning and Vasilyeva et al. In particular, Shute classifies feedback, based on its complexity, in the following way:

  1. 1.

    No Feedback. It refers to conditions where the learner is presented a question and is required to respond, but there is no indication as to the correctness of the learner’s response.

  2. 2.

    Verification. It is also called “knowledge of results” or “knowledge of outcome.” It informs the learners about the correctness of their responses (e.g., right–wrong, or overall percentage correct).

  3. 3.

    Correct Response. It is also known as “knowledge of correct response.” Informs the learner of the correct answer to a specific problem, with no additional information.

  4. 4.

    Try Again. Also known as “repeat-until-correct” feedback. It informs the learner about an incorrect response and allows the learner one or more attempts to answer it.

  5. 5.

    Error Flagging. Also known as “location of mistakes.” Error flagging highlights errors in a solution, without giving correct answer.

  6. 6.

    Elaborated. General term relating to the provision of an explanation about why a specific response was correct or not and may allow the learner to review part of the instruction. It may or may not present the correct answer.

  7. 7.

    Attribute Isolation. Elaborated feedback that presents information addressing central attributes of the target concept or skill being studied.

  8. 8.

    Topic Contingent. Elaborated feedback providing the learner with information relating to the target topic currently being studied. May entail simply reteaching the material.

  9. 9.

    Response Contingent. Elaborated feedback that focuses on the learner’s specific response. It may describe why the incorrect answer is wrong and why the correct answer is correct. This does not use formal error analysis.

  10. 10.

    Hints/Cues/Prompts. Elaborated feedback guiding the learner in the right direction, e.g., strategic hint on what to do next or a worked example or demonstration. Avoids explicitly presenting the correct answer.

  11. 11.

    Bugs/Misconceptions. Elaborated feedback requiring error analysis and diagnosis. It provides information about the learner’s specific errors or misconceptions (e.g., what is wrong and why).

  12. 12.

    Informative Tutoring. The most elaborated feedback, this tutoring presents verification feedback, error flagging, and strategic hints on how to proceed. The correct answer is not usually provided.

There are several models of feedback in the literature [21, 25, 27]. The most simple is the one proposed by behaviorism, where feedback acts to provide a reinforcing message that would automatically connect responses to prior stimuli—the focus being on correct responses [21, 27] and where the cognitive architecture of learners is not taken into count.

The rest of the models in the literature, as Mory [22] points out, propose a more elaborated examination of feedback that takes into account how feedback affects cognitive engagement with tasks and how engagement relates to achievement. One of the most cited models is the one proposed by Butler and Winne [28], which tries to understand the process of self-regulation as it relates to feedback. This model is shown in Fig. 1, and considers self-regulation a recursive process of interpreting information (i.e., feedback) based on beliefs and knowledge, goal setting, and strategy applications to generate both mental and behavioral products [22].

Fig. 1
figure 1figure 1

Butler and Winne’s model of self-regulated learning

Full size image

Another proposed model in the literatures is the one of Hattie and Timperley [23] depicted in Fig. 2. This model. According to these researchers, feedback must answer three major questions asked by a teacher and/or by a student: Where am I going? (What are the goals?), How am I going? (What progress is being made toward the goal?), and Where to next? (What activities need to be undertaken to make better progress?). How effectively answers to these questions serve to reduce the gap is partly dependent on the level at which the feedback operates. These include the level of task performance, the level of process of understanding how to do a task, the regulatory or metacognitive process level, and/or the self or personal level (unrelated to the specifics of the task).

Fig. 2
figure 2figure 2

Hattie & Timperley’s model of feedback

Full size image

Perhaps, the most recent model is the one by Narciss [24]: the Interactive Tutoring Feedback model (ITF). Narciss’s model views feedback as one of several basic components of a generic feedback loop. However, when regulatory paradigms from systems theory are applied to an instructional context, in which learners are provided with feedback by an external feedback source (e.g., teacher, peer-student or digital instructional medium), two interacting feedback loops must be considered (see Fig. 3): the learner’s feedback loop and the feedback loop of the external feedback source. Additionally, the model takes into account that the effects that an instructional activity can have are determined by: (a) the quality of the instructional activity (e.g. scope, nature, and structure of the information provided, and the form of presentation); (b) individual learning conditions (e.g. prior knowledge or level of competencies, meta-cognitive strategies, motivational dispositions and strategies) and (c) situational conditions of the instructional setting (e.g., instructional goals, learning content and tasks).

Fig. 3
figure 3figure 3

Narciss’ interactive tutoring feedback model

Full size image

With the exception of the behaviorist model, the rest of the models in the literature, as Mory [22] points out, belong to the Information Processing perspective [5] on cognition and learning. Consequently, most studies have been carried out from this perspective. Although it is outside the scope of this paper to search and discuss these models, given that constructivist approaches tend to avoid feedback and guidance as much as possible [18, 19], the existence of such models must be scarce. In the next section, the feedback provided by the four computer-based concept mapping tools mentioned in the introductions is analyzed.

3 Computer-Based Concept Mapping Tools

Nowadays there are a lot of tools supporting different activities with concept maps. However, as Anohina and Grundspenkis underline [29], most of them only provide functions such as concept map construction, navigation and sharing, but do not analyze the learners’ concept maps and do not provide appropriate learner’s support in terms of feedback and help. In the following, four concept mapping tools, that claim to provide feedback for learners, are presented and analyzed in a chronological order.

3.1 Chang, Sung, and Chen’s Concept Mapping Tool

The tool developed by Chang et al. [18] supports two kinds of learning strategies for students to construct concept maps. One of them is the ‘construct-on-scaffold’ version, which provides an incomplete expert concept map with some blank nodes and links/relations. The students then select concepts or relations from the concept or relation list and fill in the appropriate blanks in the scaffold with these selections. In particular, the tool has a hint button, which can be used on demand, and that gives hints to students according to the comparisons between student and expert concept maps. The hints are presented in a partial proposition type, such as [Meiosis result in ???]. Additionally, there is an ‘expert concept map’ button which is enabled when the students have worked on constructing their maps for over 30 min.

To test their concept mapping system, the researchers carried out and experiment with forty-eight seven-grade students (N = 48), 23 females and 25 males, selected from three classes of one junior high school in Taipei. All the students were studying their second semester course of General Biology. Each class was randomly assigned to one of the following concept map construction groups: ‘construct-by-self’ using the tool, ‘construct-on-scaffold’ using the tool, and ‘construct by paper-and-pencil’ without the tool.

The experiment employed a pre-test post-test control group design. A one way ANCOVA was conducted on the post-test scores of the three groups. Pre-test scores were used as the covariate to control the potential differences in the students’ biology knowledge. In particular, the ANCOVA revealed significant differences between the 3 groups: F(2,44) = 3.79, p < .05. A Post hoc analysis using Fisher’s Least Significant Difference (LSD) test indicated that the ‘construct-on-scaffold’ method had a better learning impact on students than the ‘construct-by-self’ and ‘paper-and-pencil’ ones.

3.2 The COncept MaP ASSessment Tool (COMPASS)

COMPASS [4] is a web-based concept mapping that supports the elaboration of assessment activities employing various mapping tasks such as the construction of a concept map from scratch (“free construction” task) and the completion and evaluation of a concept map using an available list of concepts/relationships (“concept-relationship list completion/evaluation” task). In particular, after the learner has completed the assessment activity, COMPASS activates the diagnosis process for (i) the identification of errors on the learner’s map according to a predefined set of errors (see [4]), based on the similarity of the learner’s map to the teacher’s one, and the qualitative analysis of the errors, (ii) the qualitative diagnosis of learner’s knowledge, which is based on a predefined classification of errors (see [4]) which concerns the identification of unknown concepts, incomplete understanding and false beliefs, and (iii) the quantitative estimation of learner’s knowledge level on the central concept of the map and subsequently on the assessment activity, which is assigned to one of several characterizations: Insufficient (Ins), Rather Insufficient (RIns), etc.

Furthermore, the tool provides a “Visual Feedback” option and an “Interactive Feedback” option. If the learner selects the “Visual Feedback” option, COMPASS graphically annotates the errors on the map, if any, following the proposed error categorization. In the student selects the “Interactive Feedback” option, COMPASS activates a process denominated “Knowledge Reconstruction + Refinement (KR + R)” which aims to provide feedback, tailored to each individual learner in order to support the reflection process, to tutor and guide the learners and subsequently to enable them enrich/reconstruct their knowledge structure.

In particular, the “KR + R” process incorporates informative and tutoring feedback components (ITRFC) and combines a stepwise presentation (see [33]) of these components with a multiple try strategy. The ITFC include (i) an initiating question (IQ) consisting of the learner’s belief, and a prompt to think of the concepts included in the proposition and to write any keywords describing the concepts, (ii) specific error-task related questions (E-TRQ), (iii) tutoring feedback units (TFU) relevant to concepts/relationship included in the concept map, and (iv) the knowledge of correct response (KCR).

To evaluate the efficiency of the tool, 2 studies (N = 6) where carried out. The first study investigated whether the design of the E-TRQ, as the only source of feedback, helped learners to identify, reconsider and correct their error appropriately. The second study researched whether the E-TR and the TFU helped learners to identify, reconsider and correct their error appropriately. No inferential statistics and no control group where employed in these studies. The results showed that E-TRQ alone helps students, especially those with knowledge level above average, in revising their beliefs and refining their knowledge. In cases of students with low knowledge level, these improved their performance after the TFU + E-TRQ were provided and they identified and corrected a considerable number of errors.

3.3 The Verified Concept Mapping System (VCM)

VCM [19] is intended for explicit mapping tasks that have been carefully defined by an instructor or teacher. For example, the teacher might provide students with learning resources to study and then ask them to construct concept maps that capture their understanding of that material. In particular, VCM allows students to focus on the concept-mapping task as long as they need to complete it. Then, when the learner is ready for feedback, he or she moves to the analysis phase and the system displays both a learner model and some suggested elements for checking.

Feedback is provided by checking for expected propositions and, for any missing proposition, VCM produces a message intended to help the student check her or his map. The messages are previously encoded by the instructor or teacher. For instance, if an expected proposition “Concept1 link1 Concept2” is missing, the teacher might code a message asking the student to consider ways to connect “Concept1”. If the teacher anticipated a misconception in the form “ConceptA linkA ConceptB”, the message might ask the student to check this proposition. Examples of messages are: where should concept x be in the hierarchy? what is the definition of x? can you change the link between concept x and concept y?

A qualitative evaluation of the tool was carried out using a think aloud approach with four university level students (N = 4) coursing their first year of Computer Science. An experienced tutor was asked to perform the experiment so that input could be gained from one person at expert level, but independent of the design team. The mapping task involved scalability, a topic that is quite conceptual and hence suited to concept mapping.

The students spent between 1 and 2 h on the task, while the expert only spent 30 min. One student failed to complete the task and found it a frustrating experience. None of the students (or the expert) appeared to spend much time reading the supplied reading material. Nor did they make reference to it as they attempted to construct the map. Moreover, students used the analysis phase somewhat differently from what it was originally intended. Rather than wait till they had completed the map and then do the analysis, they used this facility at regular points through the mapping activity. They would do a part of the map, then stop and run the analysis to get feedback on the partially completed map.

3.4 The Intelligent Knowledge Assessment System (IKAS)

IKAS [20] is a system developed with the following goals in mind: (a) the promotion of process-oriented learning by supporting assessment focused on the process of knowledge acquisition by students; (b) to promote students’ knowledge self-assessment; (c) to support teachers in improvement of study courses through systematic assessment and analysis of students’ knowledge. Following Anohina-Naumeca et al. [20], the usage scenario of IKAS assumes that a teacher divides a course into several assessment stages. A stage can be any logically completed part of the course, for example, a chapter. For each stage, a map is created by specifying relevant concepts and relationships among them in such a way that a map of particular stage is nothing else than an extension of the previous one. During knowledge assessment, a student solves a task corresponding to the assessment stage and after the submission of his/her solution the system compares the student’s and teacher’s maps and generates feedback.

According to Lukasenko et al. [30], only one type of feedback is provided to students during the solving of a task: checking of a proposition. The idea is that a student points out his/her created proposition and the system checks its correctness. In case of incorrectness the system presents explanations of both concepts involved in the proposition. After the submission of a task a student’s map and a window with quantitative and qualitative data is provided. Quantitative data is a set of numerical indicators aimed to inform a student about his or her performance and degree of achievement in a given task. They are interpreted by the student and no explanation or pedagogical remarks are provided. A qualitative description is a text summary which explains a student how well he or she has mastered concepts in a given task. A text summary points out concepts which require revision. In the student’s map relationships are colored in different tones according to their correctness. The student can acquire detailed information about each relationship by clicking on it. In this case contribution of all parts of a relationship (linking phrase, type, direction and placement of concepts) to the correctness of a relationship. Lukasenko et al. [30] provide screenshots of this functionality.

Starting from 2005 all IKAS prototypes were evaluated in different courses by asking students to fill-in a questionnaire after solving a set of tasks. No other type of evaluations was carried out. In particular, these questionnaires allowed gathering student opinion about concept maps as knowledge assessment tool and the functionality of IKAS. For instance, during evaluation of the first three prototypes, students always found that it would be helpful to provide more informative feedback and to improve the system’s response to user actions.

In the next section the analysis of the tools w.r.t. the research questions stated in the introduction is discussed.

4 Conclusions and Future Work

For the first research question, Chang, Sung and Chen do not declare the type of feedback they use in their tool. Nonetheless, it is clear that they use hints/cues/prompts at a basic level and do not provide the elaboration complexity considered in Shute’s typology. COMPASS is in the same situation but it can be inferred that it provides: basic hints/cues/prompts (i.e., IQ and E-TRQ), correct response, topic contingent (i.e. TFU) and Bugs/misconceptions (i.e. E-TRQ). VCM only uses basic hints/cues/prompts while IKAS has correct response as well as topic and response contingent feedback.

Most of the reviewed tools do not adhere, explicitly, to a specific model of feedback. Only COMPASS proposes a feedback framework of its own: the Adaptive Feedback Framework [31]. Only Chang, Sun and Chen, and COMPASS provide evidence of the effectiveness of the feedback provided. Nevertheless, although Chang, Sung and Chen report that feedback makes a difference w.r.t to achievement, they fail to assert how important the difference is in terms of effect. The evidence provided by COMPASS is severely limited, as no control group was included in the study and only descriptive statistics were used.

In sum, with the exception of COMPASS, the feedback strategies of the rest of the analyzed tools seem to have been designed by intuition and, without taking into account the large body of literature about feedback, in the field of Educational Psychology. Nevertheless, the most important finding of this article is the lack of methodologically sound studies that prove the efficiency of the tools. There is a big software engineering effort, but without appropriate studies the effort amounts to nothing and the field does not advance. In a time where educational interventions with an effect size below .40, are deemed as not worth the effort [8], carrying methodologically sound studies, as well as including what is currently known [32] about psychological constructs such as feedback and motivation, is a necessity. More so if we consider that recent meta-analyses have shown that the impact of technology in learning has an average effect size of 0.33 [33].