Keywords

1 Introduction

Nowadays users are surrounded by a broad range of networked interaction devices (e.g. smartphones, smartwatches, tablets, terminals etc.) for carrying out their everyday activities. The number of such interaction devices is growing, new possible interaction techniques are emerging and modern UIs are expected to be available across a wide range of diverse platforms.

In the context of our research and development project with our industrial partner Diebold Nixdorf AGFootnote 1, we have focused on the development of cross-device user interfaces that can span across various platforms and support a cross-channel banking experience.

Figure 1 shows the underlying example scenario where cross-device UIs are used in the context of a cross-channel banking web application. The idea is that banking services are accessible through different channels and depending on the situation, customers can access the service through any channel they wish to use. For example, if the customers pursue a cross-channel interaction for a payment cashout process, they can begin an interaction using one channel (Prepare Cashout at Desktop-PC at home), modify the transaction on their way on a mobile channel (Edit Cashout via Smartphone), and finalize it at the automatic teller machine (ATM). As the described scenario covers different devices with varying platform properties and interaction techniques, each target platform has specific needs regarding its UI. Especially the target platform ATM has different hardware capabilities. In the example scenario, the coupling between Smartphone and ATM can be established through Authorization via NFC technology or via the classical way using a banking card (Authorization via Card). For authentification, an eye-tracker is integrated into the ATM where the customer can enter a password by gazing at predefined password symbols (Authentification via VisualPin).

Fig. 1.
figure 1

Example scenario

Full size image

As the motivational example scenario indicates, UI development is a challenging task due to the trade-off between efficiency and usability: creating different UIs available across various devices while assuring high usability for heterogeneous users.

In the past, model-driven user interface development (MDUID) approaches were proposed to support efficient development of UIs. Widely studied approaches are UsiXML [8], MARIA [15], and IFML [3] that support the abstract modeling of user interfaces and their transformation to final user interfaces. Although those MDUID approaches increase the efficiency and consistency in the generation of multiple UI variants, existing usability studies show that completely automated approaches are not optimal in terms of the quality regarding the usability of the UI [2].

As usability is an important criterion for user acceptance and user experience of interactive systems, usability issues have to be taken into account when developing multiple variants of a UI, especially for a cross-device usage scenario as in our case. Therefore, a proper investigation of applying MDUID for cross-device UIs and its impact on the usability of the generated UI is essential to identify usability problems and determine whether acceptance by the user is given.

While previous work in the context of usability evaluation of model-driven UIs covered single- [1] and multi-device [2] UIs, our goal is to investigate the usability of generated cross-device UIs. In this regard, our analysis focuses on the usability evaluation of model-driven cross-device UIs and the user’s perception of the cross-device interaction based on our MDUID approach. By conducting a usability evaluation experiment with different users, we aim to understand current limitations of our MDUID approach for cross-device UIs and identify improvement potential.

The remaining sections of this paper are organized as follows: Sect. 2 presents our model-driven cross-device UI development approach. In Sect. 3, we describe the setup of our usability evaluation study and present the main results. Related work is presented in Sect. 4 and finally Sect. 5 concludes the paper and gives an outlook on future work.

2 Model-Driven Cross-Device UI Development Approach

In this section, we recapture our approach for model-driven cross-device UI development from previous work [17] and show its application for the motivated cross-channel banking scenario.

2.1 MDUID Approach for Cross-Device UIs

An overview of our model-driven cross-device UI development approach is depicted in Fig. 2. The model-driven development process for cross-device UIs is divided into three phases. In the first phase, Modeling, a Domain Model described as a UML class diagram and an Abstract UI Model based on the Interaction Flow Modeling language (IFML)Footnote 2, serve as specification of the data entities as well as structure, content and navigation needed to characterize the UI in an abstract manner.

Fig. 2.
figure 2

Model-driven cross-device UI development approach

Full size image

The second phase, Transformation, deals with the automated transformation of the Abstract UI Model to different final UI (FUI) representations for the varying target platforms PC, ATM and Smartphone. For this purpose, several model-to-text transformation (M2T) templates were defined that transfer the Abstract UI Model to the final UIs.

The last phase, Execution, is dedicated to executing the heterogeneous UI views on different target platforms. For this purpose, a common UI framework can be used were data is exchanged between different platforms/channels. In order to support a seamless handover between channels and allowing task-continuity for the user, our approach includes an Application and Synchronization Server, which is responsible for storing and sharing of data (e.g. UI state or user preferences). The UI state, including entered input data by the users, is stored and restored, allowing users to move across channels while seamlessly continuing their tasks.

2.2 Application of the Approach

In the following, we describe how the above described model-driven approach was applied to the motivated example scenario (see Fig. 1) for generating cross-device UIs for the cross-channel web application.

Based on the existing IFML Editor Eclipse pluginFootnote 3, developers are able to specify the domain and abstract UI model.

In Fig. 3 an excerpt of the specified domain model is shown in form of a UML class diagram. The depicted domain model covers main concepts and relations to represent a banking application. In our example case, each User of the cross-channel banking web application has at least one account. To each account any number of a BankCard and Transaction are assigned. A transaction includes the respective attributes like date, amount of money, balance etc. Each transaction can in turn be detailed in form of a Denomination describing the specific banknotes that are wished to be withdrawn. The enumeration on the right side of the domain model show possible expressions for certain data types, such as bank card type or user profile.

Fig. 3.
figure 3

Excerpt of the domain model for the cross-channel banking application

Full size image

After specifying the domain model, we specify the abstract UI model based on IFML. Figure 4 shows a small excerpt from the IFML model that is characterizing the UI for our cross-channel banking web application. Essentially, one can see the abstract UI modeling of the masks for the registration, login, and main windows. The three main windows themselves contain more elements to enable a detailed specification of the interaction objects nested therein. For example in the login window, there is a form “LoginForm”, which in turn has two input fields (SimpleField) for entering username and password. The LoginWindow contains also events - (shown in a circle) - such as “Register” or “Log in” for triggering actions from the business logic (see for example Action “Register”). If you follow the event “Register” in the login window, the corresponding navigation edge follows to the registration window. In a similar way, the other masks “RegistrationWindow” and “MainWindow” are modeled to specify the abstract UI model.

Fig. 4.
figure 4

Excerpt of the IFML model for cross-channel banking application

Full size image

The specified domain model and the IFML model serve as input for our code generator to transform them to the specific final UIs for the varying platform. For transforming these models into final web UI views, we implemented an XtendFootnote 4 plugin that maps the IFML model elements to specific HTML5 elements. The Xtend plugin includes different Xtend templates to transfer the IFML source model into web UIs supporting manifold devices. During the transformation process, the application’s view is built upon basic components with a custom look & feel, like buttons, text input fields, dropdown lists, tables, etc. As a basis for these components, we implemented components based on the HTML5 Web ComponentsFootnote 5 specification promoted by Google as W3C standard. Our custom components are sensitive to the application environment they are being used in (desktop, mobile, ATM) and adapt themselves accordingly. On mobile devices, for example, buttons are larger and more suitable for touch operation than on desktop devices. During the execution phase, the generated web views build up a HTML5/JavaScript single-page application running in a web browser. It exchanges JSON messages with the back-end server through HTTP/REST. The back-end server is implemented in JavaScript and uses Node.jsFootnote 6 as its runtime environment. It is built upon Google’s V8 JavaScript engine also used by Google Chrome and provides a high-performance runtime environment for non-blocking and event-driven programming. In order to support task continuity and transfer application state between devices, the current state name and its associated context are saved to the Application & Synchronization Server. Inside a view controller and prior to saving a state, all context information necessary for recovery is added to a state-context object. This includes the UI’s view-model, as well as any other necessary information associated with the current state. To invoke a previously saved state, the application just needs to retrieve the current state name and invoke it.

Figure 5 shows the different generated final UIs for our cross-channel banking web application. To be more specific, the MainWindow for the cash withdrawal process is shown for each involved device Desktop, Smartphone and ATM.

More technical details regarding the transformation and execution phases can be found in our previous work [17] where we present used technologies, languages and frameworks for the generation and execution phases in more detail.

Fig. 5.
figure 5

Generated UIs for different device types

Full size image

3 Usability Evaluation

While the previous section recapitulated our existing solution for model-driven cross-device UI development and showed its application in the context of a cross-channel banking application, the main goal of the paper is to analyze the usability of such a generated cross-device user interface.

Therefore, in the course of our industrial research project we conducted a usability test to assess the quality of model-driven cross-device web UIs. The goal was to evaluate how well or poorly the generated cross-channel banking web application called “KoMoS” and its user interface performed, and how satisfied the users were in completing two withdrawal tasks, one based on the classical way using a single access point on the ATM and one based on a cross-device usage scenario. In this section, we first describe the setting and procedure of our usability study. After that, we present the results of our usability study measured in terms of effectiveness, efficiency and satisfaction as suggested in ISO 9241-11 [7].

3.1 Setup Usability Study

We set up a usability laboratory at Diebold Nixdorf AG to conduct the usability test. Figure 6 shows the general setting of our usability lab. The usability test area contains the involved devices for the cross-device interaction: Desktop-PC, Smartphone, and ATM. Additionally, there are two cameras that serve to record the user interaction with the overall system. Camera A focuses the user, to track user behavior (facial expressions, gestures, etc.) and Camera B serves to observe the input of the user. Beside the test area, we have a question answering area, where participants answer an online questionnaire using a tablet after completing their task.

Fig. 6.
figure 6

Usability setting

Full size image

The test involved 15 persons where we had five participants from Paderborn Universities (mainly CS students), five employees from Diebold Nixdorf AG (from different departments like marketing, R&D, etc.) and five elderly persons where two of them lived in a nursing home. The interaction between the user and the generated KoMoS banking application was video-recorded and logged. An instructor and one independent observer participated in the study. The instructor interacted with the user while the observer took a record of observations and assisted during the usability test. Users were informed that the usability of the generated KoMoS banking application was going to be tested to check whether the system met their needs. No pretask training was scheduled and all of the participants were going to see the generated application for the first time. The design of the usability test followed a logical sequence of events for each user and across tests. The general usability testing procedure was as follows:

  • Users were given information about the goals of the test and the setup of the usability laboratory. They were also informed about the video recording for analysis purposes.

  • Users were given a series of clear instructions that were specific for the test. They were advised to try to accomplish the tasks without any assistance, and that they should only ask for help if they felt unable to complete the task on their own. Users should also use the think-aloud technique while accomplishing the task, so that possible usability problems could be analyzed.

  • Users were asked to complete two different types of tasks. The first task was to withdraw a predetermined amount of money from the ATM - a classical cashout at the ATM using a bank card and PIN. For this task, the participants received a bank card with the corresponding PIN. The second task was to withdraw money based on the cross-device interaction scenario, so the users had to start preparing a cashout at the Desktop-PC, edit the cashout at the smartphone and finalize it at the ATM. For avoiding a possible ceiling effect, there was no time limit to complete the tasks.

  • Users were then asked to fill in an online questionnaire after completing the last task. This questionnaire is arranged in a hierarchical format and contains: (1) a demographic questionnaire which is used to confirm their age, job description and gender. They also scored their attitude towards technology usage and especially their experience with the involved device types. (2) several measures of specific interface factors based on the IsoMetrics questionnaire [4] for the evaluation of graphical user interfaces (ISO 9241/10). Each item of this questionnaire is rated on a scale from one to seven with positive statements on the right side and negative statements on the left side. Additional space that allows the users to make comments is also included in the questionnaire. (3) Additional specialized questions regarding cross-device interaction and usage of eye-tracking were incorporated in the questionnaire to assess the specific details of the KoMoS banking scenario.

  • After finishing the test, users and the instructor had the possibility to talk about their experience and exchange ideas about lessons learned. Users were given a small reward for their participation.

3.2 Results of the Usability Study

In this section, we discuss the results and findings of the usability study regarding the criteria effectiveness, efficiency and satisfaction.

Effectiveness. Effectiveness relates the goals of using the product to the accuracy and completeness with which these goals are achieved [7]. It was measured using the following measures proposed in the Common Industry Format (CIF) [16] for usability tests: the completion rate and the frequency of assists. The completion rate is the percentage of participants who completed each task correctly. The frequency of assists is the number of times that the instructor assisted the participant. Regarding effectiveness our usability study showed that each participant except one older woman (with the age of 86 and big vision problems) was able to complete both tasks. In some few cases the instructor had to give some smaller hints to complete the task: for example that the cashout preparation step was successfully accomplished and that the participant could move to the next step at the ATM or if the participant was unsure about the input possibilities a small hint was provided that input via eye-tracking or pin pad was possible. Nevertheless the assisted completion rate for both tasks was 14 out of 15 participants.

Some observed usability problems regarding effectiveness:

  • Layout/Design: The denomination dialogue which supports the selection of specific banknotes during the withdrawal process was not intuitive for all users. Some of the elderly participants for example clicked directly on the banknote symbols (which were inactive by construction) instead of the “+” Symbol for selecting specific banknotes. Furthermore the structure of the menu was not optimal for some users. Although the most important functions were directly accessible, some participants needed more time to get used to the menu and the different button types which were used in the KoMoS banking application.

  • System Feedback: The participants missed at some points in the system dialog feedback from the system which helps them to know about the next steps. For example, some of the users were not sure after preparing the cashout on the smartphone if they have finished this task. Furthermore, the users wished feedback on possible input techniques. As we have different input methods (Pin-Pad, Touchscreen and Eye-Tracker on the ATM) they were not sure when to use which input method.

  • Difficulty with operating the application via eye-tracking: The users were not familiar with controlling an application via eye-tracking. So different questions arose: Where can one look without causing a wrong entry? How long I have to gaze at a specific UI element (symbol) to select it?

Efficiency. Efficiency relates to the resources expended in relation to the accuracy and completeness with which users achieve goals [7]. For the evaluation of the efficiency, the required time for the different two tasks was measured. Table 1 shows a comparison between the needed time for accomplishing both tasks. For accomplishing the classical cashout process using the generated KoMoS banking application only on the ATM, the participants needed approximately 64 s in average. Compared to this reference scenario the completion of the task in the cross-device scenario takes approximately 135 s in average. In this regard, we should notice that the classical cahout process is a common known activity for all participants (although the used app is new). In contrast, most of the users were not very familiar with the cross-device scenario where different devices and interaction techniques were involved. Nevertheless, in the cross-device scenario we can observe that the needed time for finalizing the withdrawal process at the ATM takes only approximately 46 s in average since the transaction has been already prepared on the smartphone. The majority of the overall time with 135 s in average was needed for preparation at the smartphone (89 s in average). Thus, the stay at the ATM is reduced by about 25% by a cross-device interaction compared to the classical ATM cashout process. If we imagine that the preparation of the transaction on the smartphone is done in an idle time, this can be seen as a time saving.

Table 1. Efficiency: Comparison between ATM reference scenario and Cross-device scenario
Full size table

Satisfaction. Satisfaction is defined as freedom from discomfort, and positive attitudes towards the use of the product [7]. For the evaluation of the satisfaction criteria, we measured and analyzed the questionnaire answers of the participants. As depicted in Fig. 7 the evaluation of the satisfaction criteria shows a positive feedback of the participants regarding the generated UI of the KoMoS banking application. In total average, the interaction with the UI received the score 5,2 out of 7 which is quite acceptable. While, participants especially honor the criteria like ease-of task handling, learnability or liquid usage across different device types, they also point out that some aspects like system feedback, personalization and adaptability are not optimally solved through the generated UI of the KoMoS banking application.

Fig. 7.
figure 7

General usability results regarding satisfaction criteria

Full size image
Fig. 8.
figure 8

Usability results regarding specific aspects

Full size image

The second part of the questionnaire covers specific aspects about the cross-device baking scenario. In this regard, user’s satisfaction of the generated UI regarding authorization via card vs. smartphone (NFC), authentification via PIN vs. Eye-Tracking and the device switches were assessed based on questions. Figure 8 shows that there are not big differences in the authorization via card or smartphone regarding comfortability and quickness. However, the participants honor authorization via smartphone (NFC) as more hygienic. In contrast to that, classical authorization via bank card is perceived as more secure by the users. Regarding the both authentification types, the classical solution based on PIN entry is seen as more quick but rather unhygienic, while the Eye-tracking authentification takes longer but increases the satisfaction regarding hygiene as the users do not have to touch the Pin-Pad.

In summary, the usability evaluation results show that cross-device UIs automatically generated based on our model-driven approach are accepted by the majority of the users. The high rate of effectiveness for both tasks indicates that it is possible to automatically generate the UI for a cross-channel application scenario with the needed functionality. However, detailed feedback regarding satisfaction and user comments also show that there is improvement potential. Especially the aspects layout design, system feedback, personalization and adaptability show potential for further improvements. In this connection, existing model-driven UI development approaches have to be incorporated with explicit modeling and transformation techniques for better supporting user assistance and adaptation.

4 Related Work

In recent years, a number of approaches have addressed the problem of UI development for multi-device and cross-device user interfaces. While there are several existing approaches in this direction, the number of usability evaluation studies focusing on UIs generated based on a model-driven approach is limited to a few examples. In this section, we especially review prior work that explores the development of multi-/cross-device user interfaces and usability evaluation of model-driven user interfaces.

4.1 Multi-/Cross-device UI Development Approaches

The development of multi-device UIs has been subject of extensive research [13] where different approaches were proposed to support efficient development of UIs for different target platforms. Model-based and model-driven UI development approaches were proposed to create multi-device UIs based on the transformation of abstract user interface models to final user interfaces. Widely studied approaches are UsiXML [8], MARIA [15], and IFML [3] that support the abstract modeling of user interfaces and their transformation to final user interfaces.

Previous work by the research community has also covered concepts and techniques for supporting the development of cross-device user interfaces. One of the concepts here is called UI migration, which follows the idea of transferring a UI or parts of it from a source to a target device while enabling task-continuity through carrying the UI’s state across devices. In [14] for example, the authors present an agent-based solution to support migration of interactive applications among various devices, including digital TVs and mobile devices, allowing users to freely move around at home and outdoor. The aim is to provide users with a seamless and supportive environment for ubiquitous access in multi-device contexts of use. A more recent model-based approach which allows designers and developers to specify how to distribute interfaces at various granularity levels, ranging from entire user interfaces to parts of single interactive elements is presented in [10]. This solution includes run-time support for keeping the resulting user interfaces synchronized and customization tools that allow end users to dynamically change how the user interface elements are distributed across multiple interactive devices in order to address unforeseen situations.

In the case of web applications, most solutions rely on HTML proxy-based techniques to dynamically push and pull UIs [5]. An extension of this concept is presented in [12], where the authors propose XDStudio to support interactive development of cross-device UIs. In addition, there is also existing work on the specification support for cross-device applications. In [20] for example, the authors present their framework Panelrama which is a web-based framework for the construction of applications using distributed UIs. In a similar work [6], the authors present Conductor, which is a prototype framework serving as an example for the construction of cross-device applications.

4.2 Usability Evaluation Approaches

Due to the low number of usability evaluation studies existing for model-driven UIs and different scope of analysis criteria, it is very difficult to make comparisons among the existing usability evaluation approaches. In earlier work, Chesta et al. evaluated a multi-platform user interface generated by TERESA [11] according to several criteria: tool interface (intuitiveness, learnability), tool functionalities (completeness, developer satisfaction), final product obtained by employing the tool (user satisfaction, maintainability and portability), and approach cost/effectiveness (development efficiency, integrability). Their results suggest that the usage of the MDE approach improved some of these metrics compared to a manual approach where the user interface is manually produced.

Abrahao et al. [1] conducted an experimental study testing the usability of user interfaces that were automatically produced by MDE techniques. Based on their usability evaluation study the authors were able to identify main usability problems for the generated user interface. They argue that valuable feedback based on such experimental studies can be used to improve the used models and their transformation to final UIs. In this context, they also underline the concept of usability proven by construction as the insights from the usability study can be used for defining design guidelines or anti-patterns for developing highly usable UIs.

A further usability study for multi-device UIs generated by MDE techniques was presented by Aquino et al. [2]. The authors describe an MDE approach that generates multi-platform graphical user interfaces (e.g., desktop, web) that are subject to an exploratory controlled experiment. The usability of user interfaces generated for the two mentioned platforms and used on multiple display devices (i.e., standard size, large, and small screens) were examined in terms of satisfaction, effectiveness and efficiency. The results of the paper suggest that the tested MDE approach should incorporate enhancements in its multi-device/platform user interface generation process in order to improve its generated usability.

While above mentioned approaches focus on the usability evaluation of generated UIs for single or multi-device platforms, our goal is to analyze usability of generated UIs across different devices. Therefore we especially take into account the perception of the users regarding the cross-device interaction. A similar usability evaluation approach to our study is presented in [9] where the authors introduce a cross-platform usability analysis model. While this approach provides a conceptual framework for analyzing usability of cross-device UIs, it does not focus on generated UIs using MDE techniques and also do not provide an experimental study in this regard.

5 Conclusion and Outlook

In this paper, we presented a usability study for cross-device user interfaces that were automatically generated based on a model-driven approach. We conducted a usability test with 15 different users based on a cross-channel banking web application to analyze strength and weaknesses in the usability of the generated cross-device UIs regarding effectiveness, efficiency and satisfaction. The usability evaluation results serve as an indicator to identify usability problems and can be used to further improve the existing model-driven UI development approach that was applied.

In ongoing work we are focusing on the usability evaluation of adaptive UIs that have been promoted as a solution for context variability due to their ability to automatically adapt to the context-of-use at runtime. Therefore, we have established a model-driven engineering approach [18] for adaptive UIs including an authoring environment [19] and design a suitable usability evaluation method for adaptive UIs.