Keywords

1 Introduction

Designing effective adaptive systems, from a usability perspective, is seen as a challenging task [1]. Adaptive e-learning systems tailor instructional material to the learner’s needs by, for instance, providing personalized learning paths, changing the interface layout or hiding some material links [2–4]. Meeting the learner’s requirements, providing relevant instructional material and supporting the learner-system interaction goals are increasingly important concerns in e-learning systems [2, 3].

Adaptive systems may, however, violate standard usability principles such as consistency, privacy and learner controllability [1, 5]. Eliminating the negative effects on usability is an essential part of the iterative design process of adaptive systems [6]. It is argued that if an e-learning system is not sufficiently usable, learners become frustrated and focus on the system rather than on the learning content [7]. Furthermore, an e-learning system may be usable in terms of its usage but not in terms of the pedagogical perspective. This issue, therefore, may lead to less effective and less efficient learning with these systems. Usability represents a challenge that should be taken into account when designing and evaluating adaptive e-learning systems [5, 8]. There is a requirement for a better understanding of where adaptation in e-learning systems is useful, and where it is harmful [7].

Although there are many adaptive e-learning systems that have been designed and implemented, they suffer from a lack of experimental evaluation in general [4]. More particularly, usability evaluation is not usually considered as one of the main criteria in the iterative design process of these systems or in determining their ease of use. According to Zaharias “very little has been done to critically examine the usability of e-learning applications” [9]. It is not always clear how easy and pleasant an adaptive e-learning system is to use.

This paper aims to fill a research gap in understanding the usability of adaptive e-learning systems. An adaptive e-learning system, based on knowledge level and learning style, was designed and implemented. It provides personalised learning paths by organising material links according to their relevance to a particular learner; it also provides adaptive guidance and feedback to support learner-system interaction goals. Using a standard usability instrument an experimental evaluation, concerning learners’ perception of usability, was conducted to compare the adaptive e-learning system with a non-adaptive version. The main aim is to determine whether adaptivity influences perceived usability given the fact that both systems have the same interface layout. It is expected that a highly usable e-learning system will increase the satisfaction, engagement and motivation of learners, and therefore, enhance their learning [7, 9]. This study is focused on usability. The evaluation of the learning outcome and learner satisfaction in using the system has been reported elsewhere [10].

The paper is organised as follows. Section 2 presents related work. Section 3 describes the implemented adaptive e-learning system. Section 4 provides the usability evaluation method. Section 5 discusses the usability results, the main usability issues, identifies its limitations and points to future work. Section 6 concludes the paper.

2 Related Work

Adaptivity is an approach designed to meet the different needs of different learners when it is incorporated in e-learning systems [2, 3, 11]. Many adaptive e-learning systems have been designed and built [2, 4]. For example, the SQL-Tutor is an intelligent e-learning system that customizes the sequence of SQL lessons based on the knowledge level of learners [12]. An approach that takes into account the learning style of learners to provide instructional recommendations to learners is represented by the eTeacher system [13]. The Protus system combines knowledge level and learning style to personalize learning material for teaching Java programming [14].

Despite some successful systems, the literature also reports some failures [4, 15]. Adaptive e-learning systems provide different designs and presentations to meet the diverse requirements of learners; therefore, usability evaluation plays an important role in the early stages of design, in order to help produce better systems. According to Bangor, Kortum, and Miller “it has become clear that a generalized assessment of the usability and customer satisfaction for different delivery types of interfaces is valuable information to have when trying to determine which technology would be best suited for different deployments” [16]. This highlights the importance of usability evaluations of adaptive e-learning systems in particular, since learners encounter different designs and presentations of content adapted to their characteristics (e.g., learning style). In addition, usability investigations should be taken into account to achieve a harmony between the learner, the learning task, the learning context and the e-learning system [8]. An adaptive e-learning system may be usable in terms of its usage but not in terms of the pedagogical perspective [7].

Usability has a clear connection with learning and also with adaptivity. However, usability evaluation is not usually taken into account; neither when forming the iterative design process of these systems nor in determining how they are easy and pleasant to use [9]. This situation warrants a study of the design and development of an adaptive e-learning systems and of its usability evaluation.

3 AdaptLearn: An Adaptive e-learning System

AdaptLearnFootnote 1 is an acronym for ‘Adaptive Learning’ which is an adaptive e-learning system. It takes into account learning style and knowledge level as important learner characteristics in order to construct personalized learning paths and adaptive feedback and guidance. A screenshot of AdaptLearn is presented in Fig. 1. The adaptive e-learning framework used as a basis for AdaptLearn, and relevant technical details are presented in our previous work that can be found in [10]. The main components of AdaptLearn are the domain model, the learner model and the adaptation model. They are briefly described below.

Fig. 1.
figure 1

A screenshot of the AdaptLearn interface

Full size image

In AdaptLearn, the domain model is structured as a hierarchal network (i.e., a tree-like structure) storing knowledge elements related to the application domain. This structure is widely used as a method to representing domain models in related work [17]. The learner model incorporates knowledge level and learning style as learner characteristics in order to provide adaptivity. The knowledge level is an important characteristic that should be taken into account in online learning [2]; it is initialized by using a pre-test, and maintained based on learner-system interaction, mainly test items (i.e., associated with each knowledge element) as a main source of interaction data. The learning style is also assumed to enhance learning when it is integrated in the learner model [18]. It is initialized by using a learning style questionnaire following the Felder-Silverman model which is considered as a valid and reliable learning style identification tool [19].

The adaptation model aims at recommending relevant instructional material to learners to support their interaction goals. It uses the information stored in both the learner model and the domain model in order to provide adaptivity. The adaptation model provides two main adaptive methods including: personalized learning paths and adaptive guidance. The output of the adaptation model is transferred to the interaction model (i.e., the interface).

Personalized learning paths are constructed and continually updated for individual learners based upon their knowledge level and learning style. These paths prioritize links to knowledge elements, hide or remove links to the elements which are not yet ready to be studied (according to the current learning state of the learner), and/or generate links to more relevant knowledge elements, as needed. Examples of personalized learning paths (as provided by AdaptLearn) are presented in Fig. 2. The provision, removal and ordering of elements in the recommended learning paths are expected to meet the needs of learners by taking into account their learning style and their knowledge level. This helps eliminate the effect of cognitive overload, support their interaction goals and optimize the learning process [3].

Fig. 2.
figure 2

Examples of personalized learning paths for different learners.

Full size image

Adaptive guidance is another form of adaptivity. It is integrated to guide learners as they progress through the learning process to successfully accomplish their learning tasks. It is mainly activated when the learner answers some test items that are related to a specific knowledge element. The system may help the learner review and understand a particularly difficult knowledge element by fetching related supplementary material from the domain model and recommending it to the learner. The system may also highlight some important points for the learner to consider. It may advice the learner to revise a specific knowledge element, and provide general feedback about the learning progress.

4 Usability Evaluation

This work contributes to the need to compare different types of interface or system technologies in critical application domains such as e-learning [16]. The learners’ perception of usability, using a standard usability instrument, is measured by conducting a controlled experiment in a computer laboratory. A number of experimental sessions were conducted, and each session lasted for up to 2 h. Two experimental conditions were proposed: adaptive condition and non-adaptive condition. In the adaptive condition, a group of subjects interact with an adaptive version of the system. In the non-adaptive condition, another group of subjects interact with the same system but without adaptivity, and a fixed learning path is provided. The main hypothesis that is put forward for this study is that: an adaptive e-learning system based on learner knowledge and learning style has a higher level of perceived usability than a non-adaptive e-learning system.

In both conditions, the same interface layout and learning material were used, and the experiment was completed within an equivalent timeframe. Learning material is related to computer security (i.e., the application domain) covering the topics of private-key encryption, public-key encryption and key exchange protocols. Each topic has a number of interrelated knowledge elements. Participants were asked to complete all the lessons provided by the system and to precisely follow its recommendations. The main distinction between the two conditions is the provision of adaptivity.

With regards to the experimental procedure, the participants introduced to the main objectives of the experiment, the system features and informed of the procedure. Second, the participants were asked to access the system via an Internet browser. Then, the participants registered with the system, completed some personal information (e.g., username, age and gender), the Index of Leaning Style (ILS)Footnote 2 questionnaire based on the Felder-Silverman model [19] and completed a pre-test containing 22 multiple-choice questions to initialize the learner model. Once completed, the system randomly assigned the participant to a specific study condition: the adaptive condition or the non-adaptive condition. The system then directed the learner to the main page to study the learning material. At the end of the learning process, the participant completed the System Usability Scale (SUS) questionnaire [20]. This tool is a quick, reliable and widely used test of system usability [21]. SUS is a 10 item questionnaire with 5-point Likert scale with anchors ranging from “strongly disagree” to “strongly agree”.

5 Results and Discussion

The experiment was successfully completed by 75 (57 % male, 42 % female) participants. They were undergraduate students in a computer science degree program. The mean age was 22.21 (SD = 3.13). The experimental sample involved 39 participants in the adaptive condition and 36 participants in the non-adaptive condition.

The usability of the adaptive e-learning system and the non-adaptive system was measured and compared. The two systems (i.e., the adaptive and non-adaptive systems) in the experiment are based on the same interface layout and learning material with an important distinction of the provision of adaptivity. The usability scores for the adaptive system (M = 79.46) and the non-adaptive system (M = 71) were good and acceptable in general. This implies that both systems are useful and valuable in learning, and the learners found them easy to use. However, the two systems were compared to get deeper insight into their usability, and whether the provision of adaptivity has any impact. Usability may lead to satisfaction, engagement and motivation of learners, and therefore, highly usable systems are expected to improve learning [7, 9].

As there was homogeneity of variance between the study groups as assessed by Levene’s Test for Equality of Variances, F = .07, P = .79 and data were normally distributed, an independent sample t-test was conducted in order to compare the two systems using an alpha level (α) of 0.01. It was found that there is a statistically significant difference between the general usability score of the adaptive system (M = 79.46, SD = 13.14) and the non-adaptive system (M = 71, SD = 13.67), t(73) = 2.73, P = 0.008. The effect size, which provides an objective measure of how important the effect is, was between medium and large (d = 0.63). The hypothesis is therefore confirmed, and it can be inferred that the adaptive-learning system based on learner knowledge and learning style has a higher level of perceived usability than the non-adaptive e-learning system.

The experiment, however, was based on two extreme conditions, adaptive and non-adaptive systems; different levels of adaptivity including controllability should be compared with more experimental conditions to get deeper understanding of their effect. However, designing such experiments to evaluate their usability and learning effectiveness represents a significant challenge [15].

In an investigation of the SUS tool items, the most significant difference is that participants would use the adaptive e-learning system more frequently than the non-adaptive system in response to the item “I think I would like to use this website frequently”. This is a very important point in usability of systems, which may provide a justification for the adaptivity and recommendation mechanisms provided by the adaptive system. Adaptivity may influence the learners to perceive that the system would help them when needed, provide them with dynamic support according to their characteristics. The recommendations of the system may enhance their intellectual curiosity, satisfaction, and engagement. In contrast, learners may find the non-adaptive system rigid and unresponsive to their needs. They may be less encouraged to use the non-adaptive system as a tool for learning.

Another significant point in relation to the item “I thought this website was easy to use” is that, participants find the adaptive system, in the long run, easier to use than the non-adaptive system. This is a surprising finding because of the inherited complexity of adaptivity. It was expected that the usage of adaptive systems would not be easier than the usage of traditional or non-adaptive systems. Learners may find it more helpful and easier to use a system that provides personalized feedback and recommendations based on their system interaction. For example, the adaptive system suggests to learners what to do if things go wrong, what to do next or what the current situation of the learning process. It may be the case that once learners gain an appreciation of the adaptive system, they may find it easier to use and more useful.

However, the non-adaptive system may seem to be better than the adaptive system in relation to system learnability. This finding was based on the response to the item “I think I would need the support of a technical person to be able to use this system”. It may imply that the non-adaptive system has less problematic issues when it comes to learning how to use the system very quickly. Another point is that, learners who use the adaptive system are more likely to need technical support to be able to initially use the system. Although this finding is not significant, it should be taken into account when designing adaptive e-learning systems.

Although the adaptive and the non-adaptive e-learning systems had the same interface layout, significant results related to the usability of the adaptive system were generated. It is observed that high level of adaptation in e-learning systems enhances usability. The high level of perceived usability may lead learners to be more satisfied, engaged and motivated to use the adaptive e-learning system [7, 9]. It is expected that highly usable adaptive e-learning systems improve learning.

The findings were based on a short-term study with a relatively small number of participants and with few learning resources. With regards to the SUS tool, it was used to test general usability; there is a need for a more specific tool that examines the usability of adaptive e-learning systems in particular. Such a tool would highlight specific usability issues related to adaptivity which will help in designing better systems. In order to improve the adaptive e-learning system, other important factors such as cognitive load, metacognitive skills and affective state may be integrated. Controllability and transparency between the learner and the system can also be investigated.

6 Conclusion

This paper has presented an adaptive e-learning system that was designed and built based on knowledge and learning style to support learner-system interaction goals. An initial experimental evaluation of the system usability was conducted with 75 participants in a learning environment; it produced significant results. They indicate that the adaptive-learning system based on learner knowledge and learning style has a higher level of perceived usability than a non-adaptive e-learning system. As usability has an influence on the learner’s satisfaction, engagement and motivation when using e-learning systems, learning enhancement is expected when the system is highly usable.

The experiment was based on a short-term study with relatively small number of participants and with few learning resources incorporated in the system. A long-term evaluation with more participants is desirable in future experiments. An experimental evaluation is also being undertaken that builds empirically on the finding of this experiment to investigate learning effectiveness, efficiency and usability when controllability and different levels of adaptivity are provided.