Keywords

1 Introduction

As for all interactive systems, usability plays an important role in the success of Interactive Adaptive Systems (IAS)Footnote 1 [1]. The usability evaluation of IAS represents an essential part of their development process. It may be conducted through the use of suitable Usability Evaluation Methods (UEMs). In the IAS literature, several UEMs are available [3, 4]. These methods aim mainly to detect the usability issues. Given this variety, non-specialists and even specialists can encounter difficulties in selecting the most appropriate UEM(s) in particular settings [4, 5]. Choosing appropriate usability evaluation method(s) is a crucial task of the IAS evaluation process [6]. This task depends usually on different factors, such as number of stakeholders, available time, etc. [4]. Applying a Multi-Criteria Decision Analysis (MCDA) method for the suitable UEMs choice is one strategy to deal with multiple and conflicting factors. One of the main steps of the decision analysis process is to define the factors that impact the choice of alternatives. In this research, we provide a general overview of the factors affecting the choice of alternative UEMs. We also classify three common usability evaluation methods for IAS according to the considered factors. These include heuristic evaluation, cognitive walkthrough, and usability test. Lastly, we use the finding of this analysis to guide the choice of suitable methods for the usability evaluation of a target adaptive system as a whole using ELECTRE I (Elimination and Choice Translating Reality) method.

It is common to identify in the IAS literature two kinds of evaluation. The first one is named layered evaluation, which aims to separate the adaptation process into its layers and to assess each one individually where necessary and feasible [7]. The second one is evaluation as a whole (or traditional evaluation), which considers the adaptation as one block. Various UEMs can be applied in conjunction with the layered evaluation as well as the evaluation as a whole. In the IAS field, a limited number of works have been focused on the guidance of the choice of appropriate UEMs. For instance, Paramythis et al. [7] proposed a layered evaluation framework that breaks the adaptation process into five separate layers. They provided a comprehensive overview of the appropriate evaluation methods and attributes to be applied in individual layers and evaluation as a whole. Regarding the use of MCDA, it has been observed that very few studies exist, apart from the ones in our previous works [8, 9]. These studies focus on the choice of suitable evaluation methods for the layered evaluation given particular evaluation settings. While assessing adaptation layers individually allows one to answer questions which are not possible to approach in a “monolithic” entity, there are some assessment questions that require treating the adaptation process as a whole. One example of the questions that can be examined when conducting the evaluation as a whole is “does the adaptive system achieve its goals?” [7]. To the best of our knowledge, there are no attempts to date that provide a review of the common factors affecting the selection of UEMs and that explore the most potential benefits of MCDA to identify appropriate methods for the usability evaluation as a whole of IAS. For instance, the use of MCDA allows the consideration of a variety of criteria that are important for the decision analysis by considering both quantitative and qualitative aspects.

The present paper is organized as follows. First, we briefly outline the usability evaluation methods for IAS, focusing on three common ones (Sect. 2). Second, we present the proposed decision process for choosing the appropriate usability evaluation methods for IAS (Sect. 3). Then, the study investigates the main factors that can affect the choice of usability evaluation methods for IAS and analyzes the considered UEMs in relation to these factors (Sect. 4). An application of an MCDA method is provided in Sect. 5. The aim is to guide the selection of the most suitable methods for the usability evaluation of a given adaptive system as a whole. Lastly, we conclude the paper with a summary and some future directions (Sect. 6).

2 Focus on Three Usability Evaluation Methods for Interactive Adaptive Systems

2.1 Heuristic Evaluation

It describes a method in which expert evaluators examine a user interface in order to discover the usability problems [10]. In the IAS field, expert evaluators need to have expertise in heuristic evaluation and they are required to understand the meaning of the particular heuristics applied for adaptive systems [7].

2.2 Usability Test

The main purpose of this method is to give a group of real users well-defined tasks to perform and to ask them to record what happens [11]. Certain observational methods can be applied in conjunction with this method such as co-discovery, retrospective testing, etc. In the IAS field, certain modifications to observational methods are required. For instance, in contrast to interactive (non-adaptive) systems, an IAS necessitates interrupting the users in order to ask them about the adaptations that occur explicitly.

2.3 Cognitive Walkthrough

During this evaluation method, a group of expert evaluators construct typical user tasks in order to detect the difficulties encountered by novice users [12]. When evaluating adaptive systems, some modifications to this method are needed. For instance, multiple-action sequences per task have to be given to expert evaluators. For each action, the expert evaluators have to examine four main questions: “Will the user expect to be asked to do this?”, “Will they notice the control (e.g., button)?”, “Will they realize that the control is appropriate for this step?”, and “Will progress be apparent once it has been used?” [7].

3 Usability Evaluation Methods’ Choice Process

As shown in Fig. 1, the decision analysis process starts with a preparatory step, where the goal of the decision problem is defined. The aim consists in selecting the appropriate UEMs for interactive adaptive systems. This step also defines the actors involved in the decision process, including a Decision Maker (DM) and an analyst. In this research, a decision maker can be a novice evaluator; s/he can also be an expert evaluator who needs to be assured in the suitable UEMs. Finally, s/he can be a project manager who needs to be aware of the UEMs to be applied given particular evaluation settings. The next step consists in determining the set of alternative UEMs that define the aspects relevant to the decision problem. Then, the factors (or criteria) that can affect the choice of these UEMs are identified. Different criteria should be considered when selecting appropriate UEMs for interactive adaptive systems. Once relevant criteria are retained, the performance table should be established. Each alternative UEM is classified with respect to the considered criteria after a detailed analysis of studies such as [3, 4, 7]. Next, an appropriate MCDA method needs to be selected in order to solve the considered decision problem. The decision maker has then to give information about the evaluation constraints of the target IAS. Usually, some parameters need to be set up in an MCDA method, such as the weight associated to every criterion. The weight refers to the relative importance of each criterion. It can be determined by the DM or estimated using a specific weighting methodFootnote 2. The next step consists to establish the outranking relations for the different alternative UEMs. Before proposing the final list of appropriate UEMs, it is essential first to test the robustness of the results by varying the MCDA method’s parameters and observing the effect on the results. On the basis of such an analysis, it is possible to study the validity of the results. The results are said to be robust only in the case where they are not modified to any significant extent by varying the parameters [13]. After an analysis of the results, the DM has to express the satisfaction level s/he obtained from the proposed UEMs.

Fig. 1.
figure 1

A flowchart illustrating the decision analysis process for the choice of Usability Evaluation Methods (UEMs).

Full size image

A score is attributed; it may adopt either (1) useful, when the list of UEM(s) satisfies the evaluation constraints, or (2) not useful, when the list of UEM(s) is not suitable due either to the non-satisfaction of the evaluation constraints (e.g., exceed the available number of users, etc.) or to the lack of proposals. In this case, a relaxation of constraints has to be performed by proposing a list of actions to the DM.

4 Factors Affecting the Choice of Usability Evaluation Methods

One of the main parts of the UEMs’ choice process is to define the factors that reflect the impact of each alternative on the decision problem. This step has a great influence on the decision process’s success. As already stated, this research seeks to present the common factors that influence the choice of UEMs in the IAS field. As shown in Table 1, three groups of criteria are considered (e.g., situational factors, characteristics of stakeholders, and adaptivity aspects). It is essential to highlight here that this list is not exhaustive and may be completed with other factors.

Table 1. List of criteria affecting the choice of usability evaluation methods for IAS
Full size table

4.1 Situational Factors

The choice of UEMs for interactive adaptive systems depends on different factors related to the evaluation situation. In Table 2, we classify the characteristics of the considered UEMs (cf. Sect. 2) according to these factors.

Table 2. Classification of UEMs according to the situational factors
Full size table

Stage of Development Life-Cycle.

The evaluation of IAS can occur at three stages of the development life-cycle of adaptive systems [7]. These include (1) specification, which refers to the phase taking place before any system implementation, (2) design, which occurs during the IAS’ development, and (3) implementation, which occurs after the implementation of a prototype of the system functionality.

Temporal Resources.

The application time of a usability evaluation method represents an important factor which affects its choice [4]. The duration can be represented by an ordinal scale (i.e., low, medium, or high).

Financial Resources.

Another factor distinguishing UEMs is the required budget [4]. This criterion can be assessed using a three-level scale (i.e., low, medium, or high).

Style of Evaluation.

Usability evaluation methods may be performed under laboratory conditions, as well as in the work environment [15]. In order to evaluate this criterion, yes is used to model the evaluation conducted in laboratory conditions, and no otherwise.

Type of Data.

Usability evaluation methods can be distinguished according to the type of data they deal with, specifically, whether these data are qualitative or quantitative [16]. In order to evaluate this criterion, yes is used to model that an evaluation method provides qualitative data, and 0 otherwise.

4.2 Characteristics of Stakeholders

A crucial consideration when selecting UEMs is the characteristics of the stakeholders involved in the usability evaluation process. In Table 3, we present a classification of the considered UEMs according to the characteristics of stakeholders.

Table 3. Classification of UEMs with respect to the characteristics of stakeholders.
Full size table

Number of Users.

This concerns the total number of users to be involved to use a specific UEM [17].

Number of Evaluators.

This refers to the total number of evaluators to be involved to apply a given UEM [17].

Availability of Direct Access to Users.

The evaluation of interactive adaptive systems can be carried out in the presence of real or representative users [18].

Level of Expertise of Evaluators.

This refers to the evaluator’s expertise and knowledge of UEMs [19]. The level of expertise can be: low, medium, or high.

4.3 Adaptivity Aspects

One other important consideration when choosing UEMs is the adaptivity aspects (e.g., reusability adaptation rules, and intrusiveness of adaptivity). Table 4 presents a classification of the considered UEMs based on these aspects of adaptivity.

Table 4. Classification of UEMs according to the adaptivity aspects.
Full size table

Reusability Adaptation Rules.

The reusability adaptation rules can influence the choice of UEMs [17]. It may adopt two values either simple or complex. In order to evaluate this criterion, yes is used to model simple reusability adaptation rules, and no otherwise.

Intrusiveness of Adaptivity.

This underlines the frequently-given suggestions of the interactive adaptive systems [17].

5 Illustrative Example of Using an MCDA Method for Choosing Appropriate UEMs

Let us assume that a DM, who is in this case a novice evaluator, is interested in identifying the appropriate UEMs for the usability evaluation of a given adaptive tourist guide system. The adaptive system adapts the user interface and the content according to the needs and requirements of the tourists. It helps them to easily find the most appropriate itineraries, schedules, etc., according to their requirements and preferences. Furthermore, the system can be adapted based on the device on which it is displayed (i.e., desktop, Smartphone). Many adaptive systems of this type have been studied and proposed in the literature over the last decades [20, 21].

As already presented, the interest of this study is related essentially to choosing the appropriate methods for the usability evaluation as a whole. To support the decision process of the choice of UEMs, the DM has to consider the evaluation constraints of the whole system to be evaluated without separation between its adaptation layers. This situation deals with choosing problem (P.α)Footnote 3, where the goal is to select one or a combination of UEM(s). For this study, one MCDA method is adopted, namely the ELECTRE I method. In the sub-sections that follow, we give a brief overview of the ELECTRE I method and details to justify the choice of this MCDA method as well as an application of the ELECTRE I method.

5.1 Brief Description of the ELECTRE I Method

ELECTRE I method is an MCDA method based on an outranking relation (aSb)Footnote 4 that aims to increase a set of alternatives in a reduced subset called kernel set [22]. This latter contains the best alternative(s). The ELECTRE I method is based essentially on the following steps:

Calculating the Concordance Index (C(a,b)).

The aim is to test the strength of the criteria coalition in favor of the agreement to the outranking relation. In this step, the discordance index is computed as shown in (Eq. 1), where Wk represents the weight for each criterion and \( f_{k} \left( a \right) \) is the score for alternative a under criterion j.

$$ \begin{aligned} & C(a,b) = \frac{1}{W}\sum\nolimits_{{j:f_{{k(a) \le f_{k(b)} }} }} {W_{K} } \\ & Where\;W = \sum\nolimits_{k = 1}^{m} {w_{k} ,\;w_{k} > 0} \\ \end{aligned} $$
(1)

Calculating the Discordance Index (D(a,b)).

The aim is to measure the rejection against the assertion aSb. In this step, the discordance index is computed as shown in (Eq. 2). More details about the ELECTRE I method can be found in [22].

$$ \begin{aligned} D(a,b) = & \left\{ {\begin{array}{*{20}l} 0 \hfill & {{\text{if}}\;{\text{f}}_{k} (a) > f_{k} (b),\,\forall k} \hfill \\ {\frac{1}{\partial }\hbox{max} [f_{k} (a) - f_{k} (b)],} \hfill & {Otherwise} \hfill \\ \end{array} } \right. \\ & Where\;\partial \, = \,{}_{a,b,k}^{max} [f_{k} (a) - f_{k} (b)] \\ \end{aligned} $$
(2)

5.2 Why Adopt ELECTRE I Method?

Numerous MCDA methods exist in the literature. Each one has some advantages depending on where it is applied. According to [24], the choice of a suitable MCDA method depends mostly on the type of information available and the nature of the decision problem to be solved. As stated earlier, the decision problem to be treated in this study corresponds to (P.α). A number of MCDA methods are appropriate to decision problems involving choice, such as the ELECTRE I method and its variant ELECTRE IS. Some differences between these methods exist. The main novelty of the ELECTRE IS method, for example, is the use of indifference and preference thresholds [22]. These discrimination thresholds aim essentially, in this case, to take into account the imperfect knowledge character of the DM with respect to the evaluation of alternatives. Such imperfect knowledge may arise when two alternatives are susceptible to be characterized with the same performance, which disables the DM to clearly express a preference relation for any pair of alternatives. For the problem analyzed in this case, it is presumed that neither indifference nor preference thresholds are necessary to model the preferences of our DM. Then, ELECTRE I method is retained since it seems to be appropriate for this decision problem. The main advantage of this MCDA method consists in using pair-wise comparisons between alternative UEMs, so that one can select the appropriate one or a combination of UEMs according to different criteria [25].

5.3 Application of the ELECTRE I Method

The use of ELECTRE I method requires the identification of a set of input data about the given decision problem. Firstly, the DM has to identify the UEMs applicable to the candidate usability attributes to be assessed. Three representative UEMs are considered in this study, namely cognitive walkthrough, heuristic evaluation, and usability test (cf. Sect. 2). After determining the set of criteria, a performance table should be established which consists of the evaluation of alternative UEMs through the retained criteria (Cf. Tables 2, 3 and 4). Then, the decision maker is asked to answer a questionnaire in order to explore the constraints about the usability evaluation of the given adaptive system. An example of these questions is: “When will the usability evaluation of the given adaptive system be done?”. In this study, the evaluation as a whole occurs in laboratory conditions during the implementation stage. As already stated, ELECTRE I is retained. A set of input data has to be determined for this MCDA method such as the relative importance of criteria. In this study, the DM considers that the decision criteria at the same level have equal weights. The outranking relation of ELECTRE I method involves two calculations, including the concordance and the discordance indices (Cf. Sect. 5.1). The outranking relation is based on the concordance and discordance thresholds. Table 5 presents the concordance matrix, which aims to measure the strength of the criteria coalition in favor of the agreement to the outranking relation.

Table 5. Concordance matrix.
Full size table

Then, the discordance matrix, which aims to measure the rejection against the assertion (aSb) is calculated (Table 6).

Table 6. Discordance matrix.
Full size table

In order to interpret the information shown in the concordance and discordance matrices (Tables 5 and 6), two thresholds (p and q) should be defined. These thresholds aim to establish the outranking relations between alternatives and to define the desired concordance and tolerated discordance. The concordance threshold p refers to the minimum concordance index needed for outranking whereas the discordance threshold q reflects the maximum discordance index required for outranking. The values for both threshold parameters are fixed as follows: p = 0.67 (represents the average of the concordance matrix); q = 0.48 (represents the average of the discordance matrix). Table 7 illustrates the outranking relations between alternative UEMs.

Table 7. Outranking relations between alternative UEMs
Full size table

The results obtained by the ELECTRE I method may be expressed in the form of a graph, as shown in Fig. 2. The arrows emerging from the nodes represent the outranking relations between alternatives. Each node corresponds to an alternative UEM. Once the analysis of the robustness of results is carried out, the decision maker has to express his/her satisfaction level obtained from the proposed UEMs. In this study, the Usability Test (UT) method has no incoming arrows. It outranks Heuristic evaluation (HE) and Cognitive Walkthrough (CW) method. The latter outranks heuristic evaluation method. Hence, the usability test can be stated to be the most appropriate method for the evaluation as a whole of the given adaptive system. It is essential to highlight that these results depend essentially on the given evaluation context and can change from a situation to another.

Fig. 2.
figure 2

Outranking graph

Full size image

6 Conclusion and Future Work

During the usability evaluation of IAS, a careful selection of UEMs should be performed in order to fit better the situation of the evaluated system. As a contribution to the IAS field, this paper summarizes firstly the most common factors that affect the selection of UEMs for a particular situation. Three of the most commonly used methods for evaluation as a whole are classified then according to the identified factors. Indeed, the knowledge obtained in this research is used in order to support the choice of appropriate methods for the usability evaluation as a whole. Towards this end, the ELECTRE I method is retained. An application of this MCDA method is presented to support the UEMs choice process for a target adaptive system in the field of tourism.

Future work will investigate to apply our proposal in other areas of adaptive systems (e.g., adaptive learning system, adaptive e-commerce system, etc.). We also intend to propose a Multi-Criteria Decision Support System (MCDSS) that integrates ELECTRE I into its model base subsystem. This MCDSS will guide our DM by providing powerful capabilities in the exploration and the comparison of alternative UEMs.