An Evaluation Methodology for Concept Maps Mined from Lecture Notes: An Educational Perspective

Abstract

Concept maps are effective tools that assist learners in organising and representing knowledge. Recent efforts in the area of concept mapping work toward semi- or fully automated approaches to extract concept maps from various text sources such as text books. The motivation for this research is twofold: novice learners require substantial assistance from experts in constructing their own maps, introducing additional hurdles, and alternatively, the workload required by academics in manually constructing expert maps is substantial and repetitive. A key limitation of an automated concept map generation is the lack of an evaluation framework to measure the quality of concept maps. The most common evaluation mechanism is measuring the overlap between machine-generated elements (e.g. concepts) with expert maps using relevancy measures such as precision and recall. However, in the educational context, the majority of knowledge presented is relevant to the learner, resulting in a large amount of information being retrieved for knowledge organisation. Therefore, this paper introduces a machine-based approach to evaluate the relative importance of knowledge by comparing with human judgment. We introduce three ranking models and conclude that the structural features are positively correlated with human experts (r_s ~ 1) for courses with rich content and good structure (well-fitted).