Keywords

1 Introduction

Wearable devices such as smartwatch, activity tracker, and head-mounted display refer to the electronic devices that transmit information in real time to the body. Since wearable devices can communicate at the closest distance to the user than any other device, they provide a new user experience in some aspects. For example, wearable devices, along with functional aspects such as traditional smartphones, have expressive aspects; accessories or clothing, for example. In addition, since user has to “wear” the product, factors such as comfort became important which were not considered before.

According to Gartner (2016), the number of wearable devices will be over to 322.7 million by 2017, with more than 20% of them being covered by smartwatch. The growth of such wearable devices is closely related to the development of the Internet of Things environment, and devices such as activity trackers are good examples. Activity Tracker tracks user behavior, sends it to the computing environment, and provides feedback to the user in real time. This process is similar to that of the desktop environment, but there is a fundamental difference in that the desktop user uses the device stationary (Lumsden and Brewster 2003). Thus, a wearable device is required to have a different evaluation framework than a traditional computing environment because it is more influenced by the context, or the user’s value, as well as the function of the device.

However, until now, researches related to evaluation of wearable devices have been evaluated in the field of usability such as the accuracy of control or efficient input method. Of course, the usability-oriented viewpoint should be performed during the whole product design process of the high-end product, but it should be accompanied with the consideration as to what user value the wearable device can provide. Therefore, designers and manufacturers should design a product with a sufficient understanding of the user experience (UX) in wearable devices, and a development of a user experience evaluation framework for wearable devices is required to reduce the time and cost burden in the evaluation process.

The research objective of this study is to provide a systematic framework that assist in the evaluation and design of wearable devices. For this, we first defined features and functions of wearable devices. Based on this, we constructed three core areas of UX evaluation: design space, evaluation factors, and context of use. The relationship between design space and evaluation factors was examined through a case study by applying the developed evaluation framework.

2 Related Works

2.1 Framework for Understanding Product UX

Although the definition of UX is different for each researcher, according to ISO DIS (2008), it is defined as ‘a person’s perceptions and responses that result from the use and/or anticipated use of a product, system or service’. It is said that the UX is inherently variable because it can change according to the characteristics and context of use of the product and the external environment. Therefore, UX should evaluate not only values of interaction with the subject, but also values of the interaction before and after interaction with the subject (Vermeeren et al. 2010). In order to evaluate these long-term and changing experiences, we thought that we needed to include user values and contextual factors in the evaluation framework.

Prior to the evaluation framework of UX, we highlighted the differences between usability and UX for this study. Some studies suggested that usability can be interchanged with UX, but in general, UX is seen as a larger concept involving usability (Saffer 2010). Usability begins with a consideration of how easy it is to use design elements of interacting products (Heo et al. 2009). Therefore, usability differs from UX in that the issue is related to task performance regardless of user’s emotional state or attitude (Kaye 2007).

In order to measure or evaluate the UX of wearable devices, it is necessary to identify the factors that directly and indirectly affect it (Schulze and Krömker 2010). In Wechsung (2014)’s framework for evaluating multimodal interactions, for example, they classified factors that affect UX as user, context, and system. User referred to the user-level variables that interact with the product, including demographic information, personality, needs, abilities, and emotional states. Context included environmental factors and service factors while system was divided into functional factors and agent factors. In order to create the evaluation framework, the design principles are collected and reconstructed according to the subject.

Design principles have been developed for good product design, but they can also be effective guidelines for evaluating objects. Also, when evaluating new products that have not existed before, it is necessary to present new evaluation factors in accordance with the product. In addition, it is important to determine which design spaces should be evaluated and how to evaluate them to build a user experience framework (Heo et al. 2009).

2.2 Features of Wearable Devices

It is important to understand the specific characteristics of wearable devices to evaluate UX. Wearable devices basically share properties with mobile devices as they are mostly used mobile. Mobile devices enable to provide information by themselves presenting the following three main characteristics: (1) they usually work in the hands of users, (2) they mostly operate without a physical connection like a cable, and (3) they provide additional features such as new applications and internet connectivity (Weiss 2003). On the other hand, the characteristics of wear for wearable devices can be summarized in two ways: the user must always be with them, and their appearance is exposed.

Characteristics of mobile devices changes in the environment of wearable devices. That is, they are no longer used by holding it in one hand, as they do not need to be hold body of the device since it is already worn by the user. On the other hand, the rest two mobile devices characteristics can be considered as more emphasized in wearable devices, where the non-linear characteristics are directly related to the battery and communication speed of the device. As devices become more compact, designers intensively deploy multiple technologies within smaller devices. Likewise, in order to provide additional functions, the wearable device usually operates by providing the application itself or by operating the application in cooperation with the smartphone.

In recent mobile computers, such as smartphones or tablet PCs, weight reduction of the product was an essential factor. However, since the wearable device is in direct contact with the user’s skin, weight factors as well as comfort factors are important. For example, if you are playing a game or watching an image through a Head-Mounted Display, this can provide a greater immersion experience than interacting with an existing desktop monitor. However, over-heavy display weight and eye fatigue due to fast screen switching can interfere with user engagement and provide a negative user experience. The characteristic of being exposed is a feature that restricts the form and wearing manner of the wearable device because the device should not cause the device to have a sense of heterogeneity. For example, smartwatch, which is the most widespread wearable device at present, suggests a way to replace the watch as its name suggests, and its shape is also a form that does not deviate greatly from the existing form of the watch.

Further, since the form of the product can be determined from its function, it is necessary to search for the function of the wearable device. The development of various wearable devices has also diversified its functions. However, the wearable device must basically provide the role that the existing product performs, such as time confirmation and vision adjustment. In addition, wearable devices require display because they provide common mobile product functions such as dialing, sending messages, scheduling, and running applications.

Although wearable devices evaluated in our study are smartwatches and HMDs, these interface features and functions are similarly defined in other types of wearable devices. Of course, there will be a few differences between them because there will be so many different wearable devices and they will be released. Products that do not provide their own output, such as a fitness tracker, differ in appearance from functional differences, such as small or no display, when compared to smartwatch. Despite these small differences, however, we can provide a consistent design space for constructing our evaluation framework in the context of a product’s wear and movement.

3 Evaluation Framework for Wearable Devices

Figure 1 shows the overall evaluation framework for wearable devices which is based on the product design space and evaluation factors. The framework enables to explore user value from the product and evaluate them. We aimed to propose an evaluation framework taking into consideration aspect of user values as evaluation factors, design space of wearable devices as well as context of use.

Fig. 1.
figure 1

Evaluation framework for wearable devices

Full size image

Context of use is the area to be considered in product development apart from the above two areas, and includes external factors related to the user experience. From the proposed framework, it is able to create an evaluation table that shows the relationship between UX evaluation factors and design space. In this way, it enables designers to quickly identify UX problems and modify correctly in the according design spaces that need improvement.

3.1 Design Space

Wearable devices share design spaces with the mobile device. In previous research, the design space of mobile device has been divided into Linguistic User Interface (LUI)/Physical User Interface (PUI)/Graphic User Interface (GUI) dimension or Hardware and Software dimension (Heo et al. 2009; Kim et al. 2008). LUI is a concept that includes menus and navigational structures as interfaces related to information content and structure for task execution. A GUI is an interface that displays task-related information graphically or visually, such as icons or fonts. A PUI is an interface that can be physically touched, such as a keypad and a microphone, and the user actually performs the task through the PUI.

As shown in Table 1, in this study, the design spaces of wearable devices are divided into physical and functional aspects by reconstructing the sub-categories of existing studies. The physical design space refers to an external aspect of a product that a user can physically touch or perceive. Physical design spaces are divided into three sub categories: input, output, and body and structure. The input refers to the part used by the user to operate the product, such as a controller. In the traditional computing environment, the mouse and the keyboard were used for input manipulation. However, with the birth of the touch screen, introduction of the gyro sensor and the improvement of the speech recognition rate have made it possible to manipulate the device with finger, voice and gesture. The output means the part where the user receives the sensory feedback from the product like the display. In recent wearable devices, auditory feedback and tactile feedback are often provided together. Finally, the body and structure refers to the rest of the product’s exterior, excluding the input and output parts, such as the shape of the device or strap.

Table 1. Design space of the evaluation framework
Full size table

The functional design space refers to an area that actually functions based on a software system. It consists of three categories: GUI, application, and specification. The application of a wearable device is defined as a set of functions that a wearable device provides by itself or in cooperation with another device. In addition, the GUI means an icon on the screen, audio feedback, etc. in which functions are output. Finally, specification means the parts inside the product and how they work together.

3.2 Evaluation Factors

Since we have classified the UX into the dimension of user value along with the concept of usability, we reviewed the usability evaluation factors of the existing mobile and wearable products, and studies on the user value that can be obtained through the products. Table 2 shows the evaluation factors of the evaluation framework defined in this study. Usability dimension is a very important evaluation factor from the existing computer interaction environment and we classify it as the ease of use, learnability, and wearability. In addition, user value is a factor that increases the merchandise in terms of marketing. User value dimension of wearable device is divided into utilitarian value, hedonic value, and aesthetic value.

Table 2. Evaluation factors of the evaluation framework
Full size table

In ISO 9421-11 (1998), usability is defined as: the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. In the evaluation area of this study, usability factors were derived by focusing on the meaning of effectiveness. The reason for this is that efficiency shares a lot of scope with the Utilitarian value in the user value domain, and satisfaction is regarded as the overall result of the user experience (Joo et al. 2011).

The ease of use of the wearable device as usability factor is the degree to which the user believes that using the device from free to effort (Davis 1989). This factor may be considered similar to the effectiveness of usability, and it measures how easily and accurately the user can perform a given task. According to Nielsen (1994) usability model, learnability can be said to be the degree to which novice users can easily master the system. Learnability includes sub-principles such as familiarity, consistency, and predictability, and is considered as an important usability evaluation factor in many studies. Likewise, learnability should be considered to be one of the most important indicators of wearable device usability, because most users have not or have not used a wearable device. Based on the research of Gemperle et al. (1998), we defined wearability of a system as the degree to which the user could wear it without disturbing. In their research, guidelines were provided to improve the wearability of wearable products. Principles such as weight and human movement were used in this study to measure the wearability of the device.

Understanding the user value is needed to define the relationship between user experience and user value domain. The user value, second evaluation area of our framework, refers to the experience and benefits that users gain by consuming products (Holbrook 1999). In particular, since the value will affect the experience of interacting with products and services, this relationship should be considered from the outset of the product design process (Kujala and Väänänen-Vainio-Mattila 2008). In this study, we divide the user value dimension into utilitarian and hedonic dimensions according to existing studies (Babin et al. 1994). We also added aesthetics as a new dimension of user value.

The utilitarian value is a value that the user receives from functional and task-related benefits (Babin et al. 1994). A utility system has its purpose in a design that enhances user’s performance and productivity (Van der Heijden 2004), For example, the design of smartwatch for utility value has value in that smartwatch can quickly handle the tasks that smartphones need to handle, such as receiving notifications. According to Kivetz and Simonson (2002), users tend to think utilitarian value is more important than hedonic value when satisfactory function is not satisfied. In other words, utilitarian value is a value expressed when the product meets the minimum functions that must be met to have value.

On the other hand, hedonic value is defined as that value based on the personal experience of fun and playfulness (Babin et al. 1994). The purpose of the hedonic system is to make the product enjoyable and sustainable for the user (Van der Heijden 2004). For example, if a user continues to use a product through challenging content, such as competing through the number of steps measured by a smartwatch, it can be said that the product has a hedonic value. The user feels hedonic value is more important than practical value when the functional minimum level of product is met. Positive experience on hedonic value causes feelings like cheerfulness and excitement (Chitturi et al. 2008)

Finally, the aesthetic value describes the pleasure that emanates from looking at a product without evaluating utility (Holbrook 1980). In some studies, aesthetic value is regarded as a sub-factor of hedonic value (Chitturi et al. 2008). In this study, two concepts were separated to evaluate the role of wearable devices as clothing. Especially, according to Hekkert et al. (2003), it can be seen that the appropriate combination of prototypicality and novelty increases the aesthetic appreciation, which shows that wearable devices can benefit from maintaining the appearance of existing clothes.

3.3 Context of Use

As shown in Fig. 1, the context of use can be defined as a combination of user, task, device, and environment while the users are using the product and achieving their intended goals (Baber 2009). The combination of these contextual information helps to construct multiple evaluation tables. Defining the context of use in the evaluation process has the advantage of enabling a focused approach and providing the basis for developing a replicated evaluation plan (Maguire 2001).

The user type is a factor indicating the familiarity level of the user with respect to the wearable device, and can be classified into the novice and expert level. The type of task means the activity performed by the user during the UX evaluation, and the type of task may vary depending on the purpose of the device or the purpose of the evaluation. The type of environment refers to the ambient conditions of the environment in which the actual product is used and can be divided into field/laboratory. Finally, the types of wearable devices can be divided into smartwatches, HMD devices, smart glasses, and fitness trackers.

4 Case Study

We carried out a case study to investigate the applicability of the developed evaluation framework. The case study was conducted in two parts, the expert evaluation and user evaluation.

4.1 Expert Evaluation

The purpose of the expert evaluation is to discover the relation of the evaluation factor and design space in a relatively short time using heuristic method, thus finding the design space needing improvement. For expert evaluation, we need to create the association table. The rows and columns of the association table contain the evaluation factors and design space defined in the evaluation framework. Strong (S), moderate (M), and low (L) are used to present the correlation between the evaluation factors and design space. The degree of relevance between each design space and the evaluation factor was scored using a five-point scale, and based on the results of the score, those having arithmetic average exceeding 3 was presented as moderated, and more than 4.5 as strong. In this study, two HCI experts with sufficient knowledge about smartwatch and HMD scored the evaluation table.

4.2 User Evaluation

User studies were carried out to investigate the factors affecting actual user wearable device satisfaction and to identify usability problems that were not found yet. Since the user does not have a clear concept of each evaluation factor, the relationship between the evaluation factor and the design space should be identified through an indirect method. In this study, we tried to clarify the relationship between the two areas through self-reported metrics.

Participants.

For the user study, 12 university students with high knowledge in Information Technology (IT) were recruited. Data were obtained from eleven subjects except one subject who complained of dizziness during HMD device evaluation. The data of the abandoned subjects were collected only for four devices except PS VR. The subjects consisted of ten males and two females. Age ranged from 17 to 28 years with an average of 23.6 and a standard deviation of 2.52.

Apparatus.

In order to evaluate the user experience of wearable devices, we conducted experiments with two smartwatches and three HMDs (Table 3). The biggest difference between the Apple Watch 2 and Huawei Watch was the difference in the operating system, and there were other differences, such as the shape of the watch and the display. Likewise, three Virtual Reality (VR) systems also showed differences in PUI, drive system, and wearing style.

Table 3. Apparatus used for the case study
Full size table

Design of the Experiments.

The experiment was conducted in a laboratory environment with a within subject design. HMD devices were evaluated after assessing smartwatches data, because the HMD may cause motion sickness which could influence on the results of smartwatches. The order in the device type was randomly assigned to each subject and after each device a short questionnaire was conducted asking participant to score with a 7-point likert scale to gather data on satisfaction as well as design space and evaluation factor.

The tasks selected for the user study were primarily based on functions that are common for both devices which are considered the most frequently used functions when the product is first introduced or by results of user surveys. The major functions selected for smartwatch were largely classified into application setting and management, communication, and health management. Application setting and management task asked participants to set up and manage the app such as changing of watch face and app install. Communication function were tasks such as confirming and replying to messages or notifications through the actual smartwatch, and confirming/correcting the schedule. Finally, health management were functions like setting the target momentum, tracking the exercise information/physical information, and designing such tasks as the use during the move to be able to evaluate the factors such as the normal wearing comfort at the same time. The task selected for the HMD device was set based on the basic function of the device which were navigation and option setting, video viewing, game activity. The navigation and option tasks asked participants to set up setting such as avatar or profile directly on the main screen. In video viewing, participants watched a 360degree VR video provided without actual operation. The game activity was evaluated through a racing game in which the user directly controlled and accomplished the goal.

Questionnaires were designed to evaluate UX factors and device satisfaction which were generated based on previous papers on usability and user values (Lund 2001; Knight and Baber 2005; and Vosset al. 2003). All users’ evaluation factors were categorized into two questions. Satisfaction scores were evaluated with satisfaction of overall device and satisfaction of each design space. We also interviewed at the end of the experiment and collected qualitative data on specifically satisfied or unsatisfied items. Correlation analysis was performed to analyze the relationship between device satisfaction and evaluation factors.

5 Results

5.1 Results from Expert Evaluation

The results of the expert evaluation for smartwatches and head-mounted display are shown in Tables 4 and 5. For smartwatches, there was a strong correlation between input and ease of use, and body and structure and wearability for usability evaluation factor and physical design space. Moreover, only one strong correlation between physic design space and user value was encounter: body and structure and aesthetic. In the case of smartwatches, there were more strong correlation between functional design space with evaluation factor of usability and user value. For instance, GUI and Application showed strong correlation with ease of use (Table 4).

Table 4. Results of expert evaluation for smartwatches
Full size table
Table 5. Results of expert evaluation for head-mounted display
Full size table

For the results of the expert evaluation for head-mounted display, it is possible to see that there is difference with respect to the results of the smartwatches. For instance, there are more strong correlation between physical design space and evaluation factor (Table 5). Input was rated to have strong correlation with ease of use and learnability, while body and structure showed strong correlation with respect to wearability. For the user value evaluation factors, aesthetic aspects were considered to have strong correlation with input and body and structure physical space (Table 5). No strong correlation was found between functional space and usability evaluation factor, however, functional space showed strong correlation with user value evaluation factor.

5.2 Results from User Evaluation

Results of the user evaluation can be divided into two parts. First, we show the score of satisfaction and evaluation factors for usability and user value for the difference devices (Table 6).

Table 6. Results of User questionnaire on smartwatches and head-mounted display
Full size table

Results from the correlation between satisfaction of design spaces and evaluation factor for smartwatches are shown in the following table (Table 7). Satisfaction on physical design space for input is strongly correlated with ease of use, learnability, and aesthetic. For output, there is a strong correlation for ease of use, learnability, utilitarian value, and aesthetic value. Lastly, for body and structure, there was only significant correlation with respect to wearability. For the functional space, GUI had significantly strong correlation with ease of use and learnability. Application was the one having the most number of evaluation factors significantly correlating, which were ease of use, learnability, utilitarian value, hedonic value, and aesthetic value. Aesthetic value was shown to be significantly correlated for all of the evaluation factors (Table 7).

Table 7. Correlation matrix of user evaluation for smartwatches
Full size table

For the overall satisfaction, usability evaluation factors like ease of use and learnability were found to be significantly correlated. Evaluation factors of user were all found to be significantly correlated with satisfaction of users.

Table 8 shows the results from the correlation analysis between user evaluation factors and satisfaction of design space for head-mounted display. The results of the overall satisfaction show that satisfaction is significantly correlated to all of the evaluation factors selected: ease of use, learnability, wearability, utilitarian value, hedonic value, and aesthetic value. The strongest correlation was between satisfaction and hedonic value (r = 0.691), followed by aesthetic value (r = 0.557) (Table 8). For the physical design space, input and output, all the evaluation factors of usability and user value were shown to have significant correlation between both. However, in the case of body and structure physical design space, there were significant correlation with respect to learnability (p < 0.05), wearability (p < 0.01), and aesthetic value (p < 0.05) (Table 8). In the case of the functional design space satisfaction, there were less significant correlation with evaluation factors. That is, GUI satisfaction was significantly correlated with user value evaluation factors (utilitarian, hedonic, and aesthetic values). For the application satisfaction, all of the evaluation factors were found to have significant correlation except for wearability (r = 0.227). Lastly, the specification functional design space was found to be significantly correlated with utilitarian, hedonic, and aesthetic value.

Table 8. Correlation matrix of user evaluation for head-mounted display
Full size table

6 Discussion and Conclusion

The research aimed to present a framework for user experience evaluation of wearable devices based on evaluation factors and design space. Based on previous researches, we presented a framework that explains the importance of separating the design space of the product in physical and functional space since they refer to different aspect of the product. Also, we divided the evaluation factors of the product in usability evaluation factors, which are commonly used by previous studies, and added the user value evaluation factors that are more related to the user experience throughout the usage stage of the product.

Two evaluation process were presented in this study which consisted on an expert evaluation and a user study. The proposed expert evaluation enables to understand that expert on the field are also enabled to score and evaluate the relation between evaluation factors and design space. In this way, the presented method might provide a fast way of encountering design issues and importance of design space aspect for the development and design of a product. The second method used in this research consisted on a user evaluation. User evaluation results also showed that there exists different relation between design space and evaluation factors for different product even in the same product group such as wearable devices. These methods allow to get a more detailed analysis on the results of the correlation and know how users rate importance and satisfaction of each of the area.

With the results obtained from the expert evaluation and user evaluation case study, we can conclude that there is a necessity to divide and categorize the design space of the product and evaluation factor; for better understanding the needs of the user and help designs to improve the product. The framework presented in this research can be used as basis for future research to develop a more systematic evaluation tool of user experience. Moreover, this research enhance researcher to be aware and continue researching and investigating the need to develop new form of evaluation for user experience in product design and development.

This research has limitation in three aspects. First, the research was conducted with small number of participants which might have bought biases on the research. However, it is important to note that different products are categorized in the same product family as wearable device; which needs to consider on different design space as well as evaluation factor. Secondly, the research only takes into consideration three evaluation factors for usability evaluation and three for user value. This might be reinforced in future researches, adding more evaluation factors relevant for the product design and perception of user experience. Finally, in this research, there was lack of contextual consideration as well as user factors consideration. Therefore, future research might be conducted by adding contextual factors for the development of the evaluation framework as well as user factors such as age and gender.