Keywords

1 Introduction

Nowadays, user-centered quality criteria become more and more important, even in areas of application which have focused on technical and organizational quality criteria before. Therefore, the value of usability engineering methods rises as well, even though the stakeholders often do not know the set of different usability engineering methods. In this context, the mobility and transportation sector represents a typical area of application, where a change to a stronger user orientation can be observed. Mainly driven by the introduction of new communication technologies and mobility offerings, the user has become the center of attention.

In addition to this circumstance, the mobility and transportation sector possesses comprehensive knowledge, e.g. ticket and travel statistics, platform capabilities and restrictions, as well as common problems of travelers. This knowledge can be integrated into user-centered development and evaluation procedures. Nevertheless, this process is complicated by the extent of usability engineering methods and the low accessibility for experts from other areas of application. This paper describes an approach of how the selection of usability engineering methods can be based on an analysis of the existing knowledge of a defined area of application. The mobility and transportation sector serves as an example for this approach, as it provides a heterogeneous and complex context of use [1] and the described change towards the user. In addition, different case studies have shown the value of combining existing transportation knowledge and usability engineering activities [2, 3].

2 State of the Art

There are different approaches for the classification of usability engineering methods, e.g. by their position in the usability engineering process, the type of data collected, their form and the approach used. This way of classification can be shown exemplarily for usability evaluation methods:

  • Nielsen [4] describes a differentiation between formative and summative evaluation. Formative evaluation aims to improve an interface’s usability in an iterative evaluation procedure, while summative evaluation is mainly used for the assessment of existing, that is, finished interfaces, and for quality assurance. Summative evaluation can also be used for the decision “between two alternatives or as a part of competitive analysis to learn how good the competitor really is” [4]. Formative evaluations are used during the design process; summative evaluations take place at the end of product development and, hence, assess the (almost) final product [5].

  • A distinction can also be made between the types of data collected. Summative evaluations provide quantitative data [6], i.e. data that are collected through the measurement of certain relevant things that represent facts and can be analyzed through statistical inferences and comparisons of numbers. The counterpart of quantitative data are qualitative data, which deal with understanding certain behavior from the perspective of people observed or interviewed. These data cannot be measured; however, they provide an insight into the informant’s point of view [7].

  • Another distinguishing characteristic is the form of the evaluation. Usability evaluation methods can be either formal or informal. Formal testing is also called traditional laboratory-based testing, while informal testing can also be named naturalistic usability, or, in an extreme form, field usability [8]. This differentiation applies to both the method itself and the evaluation environment. Formal evaluations take place in an artificial environment, embedded in a usability lab. They are expensive, time-consuming and characterized by a limited degree of realism with regard to their ecological validity, that is, the authenticity of the scenarios resorted to during the evaluation. Informal tests, on the other hand, are reduced versions of formal tests, hence, they necessitate less effort and cost, but also provide a more realistic evaluation environment. They are also called quick and dirty tests and may require compromising with regard to efficiency measures or the recruitment of test participants. The advantage of informal tests is that they allow unique results, concerning user satisfaction. In addition, full usability testing may not be affordable in very early development stages. Informal testing should be combined with expert assessments in order to increase the quality of results collected [9].

  • One of the most substantial distinguishing marks is the one between the analytical and the empirical approach. Analytical usability methods, also called usability inspection methods, encompass drawing on experts that empathize with real users when assessing the system evaluated, whereas empirical methods, also called usability testing methods, are based on observing and surveying real users as they use the system [10]. Several authors categorize in that sense [5, 11], specifying analytical methods as expert-oriented and empirical methods as user-based [5]. This distinction goes back to Schriver who developed a model that classified different forms of evaluation by their degree of user target group involvement [12]. In addition to usability inspection and usability testing methods, there are also survey methods. These are characterized by the use of standardized questionnaires for evaluating a system [13].

Criticism of the various different terms and forms used for categorizing usability methods, among others, comes from Dumas and Fox, stating that what the aforementioned notions “denote is not always clear. They add to the ambiguity about what a usability test is” [14]. They recommend classifying methods making use of a combination of the purpose of the test, the scope of the system evaluated, the location of the test sessions, the presence or absence of a moderator and the functionality level the product has [14].

Despite the disparity and diversity of present classifications for usability evaluation methods, the necessity of a differentiation by itself is undoubted. For example, if it is inevitable to have real users evaluate the system, the choice is, hence, limited to usability testing methods, that is, methods that involve real users as part of the evaluation. It is also possible to combine different categorizations in order to make an informed decision regarding which method would be best suited for the evaluation planned [5]. In addition, e.g. for usability evaluation, nowadays, using a combination of different corresponding methods is recommended in order to combine strengths and marginalize weaknesses of specific methods [15].

The aforementioned suggests that all of the classifications outlined matter in the decision-making process stated above. However, there is no clear and generally valid approach as to how to choose and combine the right methods for the specific context of use and area of application. Such an approach has to include not only the characteristics of different usability engineering methods, but also the special characteristics of the area of application, especially the existing knowledge. This leads to the approach introduced as follows.

3 Approach

3.1 Framework

For reaching the aim of providing a framework that allows for a context-based selection and combination of methods suited for predetermined circumstances, a new approach is used, which is currently being developed within the scope of a PhD thesis. This approach combines (1) considering the individual preset conditions of each usability method used with (2) factoring in company-based and product-specific measures and (3) regarding a knowledge-oriented analysis of the existing knowledge and knowledge gaps in the specific area of application.

  • With regard to (1), the approach makes use of exhaustively researched, detailed descriptions and critical analyses of each usability method. Based on the critical evaluation of each method, criteria for a categorization of methods in the form of a method catalog will be developed and used for defining conditions specific to each method.

  • Referring to (2), users of the framework will be requested to provide basic information relevant for assessing the situation of the corresponding enterprise.

  • With reference to (3), there will be an analysis of knowledge which is assumed to be existent. As a consequence, gaps needing to be filled will be identified. It will then be evaluated which and how specific usability methods may allow for filling these gaps.

The combination of these three aspects will enable a comprehensive and easy-to-use methodology for choosing and combining usability methods.

3.2 Analysis of the Expertise in the Area of Application

This paper focuses on the third aspect mentioned in the framework, the analysis of the knowledge in the specific area of application in which the selected usability method(s) are supposed to be applied. Thus, a new application-oriented access to usability engineering methods, based on the existing knowledge, is provided to ease the dissemination of the methods in different areas of application. The approach is derived from the results of different projects in the mobility and transportation sector and combines analytical and empirical case studies. Figure 1 shows the basic procedure.

Fig. 1.
figure 1

Schematic description of the procedure for defining suitable methods

Full size image

First of all, the existing knowledge and knowledge gaps have to be identified using either an internal or an external analysis of the area of application and related documents. To reduce the effort of using the final framework, a question-based procedure will be developed to extract the relevant information for the selection process. Subsequently, the relevant information will be categorized and analyzed with regard to the selection criteria. The selection will cover a detailed analysis of the following criteria:

  • Content of the existing knowledge, e.g. user, task, system and design information

  • Form of the existing knowledge, e.g. implicit or explicit knowledge

  • Type of approaches used so far, e.g. quantitative or qualitative approaches

The combination of the selection criteria provides a deeper understanding of the existing knowledge as well as knowledge gaps and allows for the final selection of suitable methods. The result of this step will include one or more methods, which can be used to close the identified knowledge gaps, and enables the user of the framework to select methods, based on short descriptions.

Additionally, steps (1) and (2) of the framework support a detailed assessment and selection process, if special restrictions, e.g. budget or technical conditions, have to be considered.

4 Utilization of the Approach: Case Study Mobility Scenarios

In the following, a simplified version of the approach for this analysis and first results will be applied exemplarily to the mobility and transportation sector as a case study. As previously described, the mobility and transportation sector provides high complexity in regard to the context of use and the existing knowledge. Therefore, the mobility and transportation sector allows for a detailed evaluation of the approach and its applicability in complex areas of application.

4.1 Characteristics of the Mobility and Transportation Sector

The mobility and transportation sector is characterized by heterogeneous users, a combination of physical and cognitive tasks, different mobility offerings and information systems and a dynamic physical and social context [1, 2]. Even though it appears evident, the application of usability engineering methods as well as the user-oriented development of information and mobility systems is nevertheless rare. Furthermore, the base for usability engineering methods, from requirements analysis to evaluation, is insufficient, although different activities and results from requirements engineering activities can be used in the usability engineering process.

Content of the Existing Knowledge.

The mobility and transportation sector has extensive knowledge in different areas. The knowledge covers e.g.:

  • Travel behavior, e.g. distances, usage times, means of transport [16, 17],

  • Ticket sales, e.g. percentage of different tickets sold [17],

  • Technical regulations, e.g. communication interface standards.

However, knowledge gaps include e.g. descriptions of mobility users, including expectations and experiences, as well as tasks and concrete information needs along the journey.

Form of the Existing Knowledge.

Some of this knowledge, especially travel behavior, ticket sales and technical regulations, is documented extensively. Regular publications of statistical reports include this knowledge. Knowledge about mobility users, tasks and user interface design is present among some experts from mobility companies but a consistent documentation of this knowledge is missing, and therefore, this knowledge cannot be accessed or easily used.

Type of Approaches Used So Far.

The set of methods used in the mobility sector comprises mainly quantitative methods. Analysis of travel behavior, derived from large questionnaires and statistical analysis of system data, are typical representatives within this set.

Based on this first analysis of the mobility and transportation sector, Tables 1 and 2 exemplarily show the results for the requirements analysis and evaluation phase, which are the base for the described selection process.

Table 1. Exemplary characteristics for requirements analysis
Full size table
Table 2. Exemplary characteristics for evaluation
Full size table

4.2 Requirements Analysis

Requirements analysis, with respect to Mayhew’s usability engineering lifecycle, is an integral aspect of usability engineering. It consists of several different activities to be carried out, e.g. generating user profiles, conducting task analysis and applying general design principles. Finally, usability goals are achieved, which are then transferred into a style guide, forming the basis of the actions in the following steps [18].

Given the considerations displayed in Table 1, the framework supports the identification of suitable methods. A method that meets the identified knowledge gaps is the personas design tool. Personas were developed by Cooper in the late 1990ies, quickly becoming a popular method used in the software industry [19]. Personas consist of a variety of user-oriented data, e.g. user goals, professions, tasks and responsibilities, user knowledge and expertise, behavioral patterns and strategies as well as user expectations regarding a new design solution [20]. A detailed description of the method’s functionality can be found in [21].

As personas represent user needs [20], they are perfectly suited for requirements analysis. But there are more methods that may well be applied in this phase of the usability engineering lifecycle. For example, the user needs analysis, defining goals as well as limitations of the target audience and their use of the design, could be useful in this regard [22]. Another option would be resorting to scenarios that can be used to exemplarily describe the future interaction of users with the system developed. In this way, scenarios can fill the gap between gathering requirements and drafting respective solutions [20].

In order to decide which method(s) would be best to use in the actual context, the first and second phase of the framework introduced earlier should be consulted. Factoring in both the specific conditions of each of the suitable methods and the company-related and product-based measures will help to refine the first selection of methods.

Different studies in German public transport, e.g. [1, 2], have shown that the persona method provides the necessary insights to reduce qualitative knowledge gaps in the mobility sector. In combination with e.g. focus groups and expert interviews, the implicit knowledge in the mobility sector can be derived and documented for different stakeholders.

4.3 Evaluation Phase

The evaluation phase described in Mayhew’s usability engineering lifecycle is part of what follows the requirements analysis and, among others, is characterized by iterative testing [18]. The primary benefit of iterative testing is that the design can be constantly refined and usability problems, arising during the iterations, can be removed in between iterations [4]. As the evaluation procedure per se is one of the crucial aspects of usability optimization [23], the evaluation phase deserves a particular emphasis as follows.

Based on Table 2, usability field tests appear to be recommendable, considering the lack of actual user testing in the mobility and transportation sector. Field tests are rather informal variations of usability testing [8], in which representative users conduct predefined tasks within the real context of use [24]. Hence, there is no artificial imitation of the system’s actual context of use [25]. On the other hand, field usability testing is considered expensive, time-consuming and hardly controllable, compared to laboratory testing [26].

It seems natural that, as an alternative to usability testing in the field, usability testing in a lab-based environment could also be applied in the evaluation phase. Testing in laboratories eases recording the sessions and provides a more structured, controllable environment. Yet, because of the artificially created environment, results tend to not be as relevant for the actual context of use [26]. In addition to lab-based testing, evaluations based on heuristics would also be conceivable. Heuristic evaluations are based on a set of well-established usability principles and involve a few expert evaluators that examine the system with regard to their compliance with these principles, which are called the heuristics [27]. As has been highlighted in Table 2, user data are especially rare. Hence, it may be preferable to resort to usability testing methods involving real users. Also, further analyzing the situation on hand, making use of the first and second aspect of the framework explained earlier, could potentially lead to further insight as to which method(s) may be used best for the actual context.

Field usability tests in public transport have already proven their value in the mobility sector [3, 28]. Knowledge gaps can be closed and user feedback can be incorporated into the evaluation of technical systems. A combination of field and lab-based tests seems suitable for the different tasks and the existing expertise, throughout the course of the iterative testing process.

5 Discussion

The described approach combines the selection of usability engineering methods along the development and evaluation process with an analysis of the existing expertise and knowledge in a specified area of application. While this analysis, prior to the selection of methods, is challenging, it provides new access to user-centered development and usability engineering methods for different stakeholders in different areas of application.

An issue that arises is the necessity of transferring the analysis, described exemplarily with regard to the mobility and transportation sector above, to a generally applicable process that can be used to identify knowledge gaps in a particular area of application. It is not yet decided how to design this process. There are two options to approach this issue: The first one would require consulting usability engineering experts every time knowledge gaps are to be identified and ask them to analyze the intended area of application prior to a usability engineering activity. The second option would be a generic procedure that, based on information provided by the user of the framework, is able to give competent support in selecting methods.

At this point, the second option, i.e. the generic approach, seems to be more suited when it comes to achieving the overall aim of providing a comprehensive, easy-to-use framework for selecting usability engineering methods. Deciding in favor of the first approach would mean introducing an additional barrier created by involving another service provider. It would lead to more effort and costs and, hence, might be a hindrance for using the framework altogether.

6 Conclusion

First results from the mobility sector show that the expertise-based approach provides new insights and a more systematic approach for the selection of methods. The analysis of existing knowledge and knowledge gaps is challenging, but the approach holds great potential to overcome the limitations of other approaches, that focus on method characteristics and economical factors only.

This paper does not provide a ready-made solution but rather an approach to be discussed, in order to obtain more stimuli and possibly derive new solutions. The approach to be refined was applied to one area of application only and should be broadened, in order to be able to factor in more fields of use. With this aim, focus groups with experts in the field of usability engineering are being conducted, in order to derive insights into which significant criteria could form the basis of the framework needed.