Keywords

1 Introduction

Human-Computer Interaction (HCI) is a multidisciplinary area that is concerned with providing design guidelines to developers who create applications to users needs and expectations. In this process, the HCI includes the project, the implementation, and evaluation of the interaction between users and the computer systems [40].

The evaluation is specifically performed for validation of the application from the user’s point of view and, depending on the type of interface, an analysis methodology is chosen. The literature presented some approaches of evaluations that focus on Usability and User Experience [44].

Usability aims to evaluate how the communication between the user and the system is. How easy and quick it is for the user to understand the application, interact with it, evaluate the effectiveness of the User Interface (UI) in execution tasks, and how the system and the user react to an error [48]. Then User Experience is an approach that evaluates not only the usability of the system but the user’s feelings and perceptions.

UI evaluation approaches for everyday devices, such as smartphones and computers, are based on Graphical User Interface (GUI), where interaction occurs through the screen, with graphic elements manipulated by touch, or auxiliary devices such as the mouse and the keyboard [44].

The difference for the applications based on Tangible User Interfaces (TUI) is the presence of physical objects as elements of interaction. In a scenario of tangible interactions, there is the object and a set of movements or actions that the user can perform with this physical element that recognizes this interaction and reacts visually or about the object itself or the environment [30].

Hence, if in a TUI the physical element is the input and output device of the interface, it can be assumed that the interaction process is more intuitive and natural for the user with a real-world analogy [29].

Interacting in a TUI application is different from a GUI, it is suggested that the evaluation methods currently used regularly for common graphical interfaces may not fit fully into the evaluation of a tangible application.

The objective of this paper its presents the potential of using UX [57] as the main evaluation approach in TUI, starting from results of a Systematic Review of Literature (SRL), in which it found other studies published in the last five years that evaluate TUI applications, analyzing the methods and tools used to perform them, relating these with an experiment of evaluation at AR Sandbox application, discuss in order to contribute of the proposal of new methodologies that aim to evaluate the applications of tangible interfaces, considering their due particularity.

Next section presents a theoretical background about Tangible Interaction, Evaluation Methods and Tools; Sect. 3 presents and discuss the results of the Systematic Review of Literature (SRL) about Evaluation Methods and Tools used in TUI applications; Sect. 4 describes a practical experiment of TUI application evaluation using AR Sandbox; and Sect. 5 discusses the results and main contribution of this article which is to show the potential of the UX approach to evaluation in TUI.

2 Tangible Interaction, Evaluation Methods and Tools

The development of TUI applications is a new process and recent research is emerging that discusses a way to evaluate this type of interface. Usually, the methods that are being applied for the development of TUIs are the same methods for UI already used in daily life. Therefore, it is probably that there are specific evaluation criteria for tangible interfaces, since this is an unconventional approach to human-computer communication.

Tangible Interaction is a term suggested by Hornecker and Buur to present a comprehensive field than TUI, considering social interaction through tangible applications, thus including the issue of interaction with the environment and body gesticulation [26].

[31] brought the term Reality-Based Interaction to conceptualize new user interaction styles for user skills. This context suggests that interaction with digital information is closer to interaction with the real world.

Reality-Based Interaction has four concepts:

  • Intuitive Physics: the user’s perception of the real world;

  • Body consciousness: the user’s notion of his body and the ability to coordinate his gestures;

  • Environmental awareness: the user’s perception of the environment around him and his ability to interact with it;

  • Social understanding: the perception that the user has with other users in the same environment, the communication between them and the ability to perform tasks together to achieve the same goal.

In the literature, there are already different approaches and evaluation tools that could measure these concepts of Reality-Based interaction in TUI applications. Several instruments are used, which can be quantitative and/or qualitative. The obtained results are grouped, with an analysis in order to discuss some conclusions about what was intended to be evaluated and/or validated at the application.

Some evaluation instruments are:

  • Interview: the evaluator asks a user a series of questions in order to understand how his experience was when using the application; during the course of the interview other issues may also be addressed;

  • Questionnaire: After the user makes use of the application, a questionnaire is applied with a set of questions that can be for descriptive or scaled answers between two opposing adjectives (I liked-dislike/agree-disagree);

  • Observation: the user is monitored while using the application and can be recorded for later analysis, it is possible to capture the user’s behavior and abilities;

  • Think Aloud: the evaluator takes notes that the user expressed orally while using the application.

For each type of evaluation methodology, it is possible to use one or more evaluation instruments. The choice of evaluation instrument is an important issue to consider in planning phases of evaluation and, specifically with TUI applications, determines how the result is useful.

About the evaluation tools and methodologies for TUI applications (especially User Experience), the next section search to verify the existence, adaptations or suggestions of use that contemplate the concepts of Reality-Based Interaction.

3 Systematic Review of Literature (SRL) of TUI Applications Evaluations Methods

The SRL method was used for the delimitation of this study of the evaluation methods/tools used in TUIs. SRL is an exploratory analysis methodology, through search engines of scientific articles. Following specific protocols that allow a better understanding of state of art on what was published/researched in a certain area of knowledge.

3.1 Methodology

The systematization of this SRL used the software StArt (State of the Art through Systematic Review) as a toolFootnote 1 [19], which allows the creation, execution, selection, and extraction of data, within an information management software that can be shared by a group of researchers.

Two research questions were answered in the articles raised:

  • (a) What approaches are used in TUIs evaluation?

  • (b) What tools/instruments are used to measure the proposed goal in these TUIs evaluation?

The SRL protocol also demands to specify the search string generated based on a set of keywords defined from the most recurring ones found in the articles preliminary listed in the search:

(“TUI” or “tangible user interface” or “tangible interface”) AND (“evaluate” OR “evaluating”) AND (“UX” OR “usability” OR “communicability”)

This search string was applied to scientific indexers who returned the collection of articles. In this mapping the following Academic Search Engines (ASEs) were adopted:

These ASEs were selected because they aggregate a considerable amount of work within the research area considered.

In order to restrict the amount of work retrieved in this stage of selection, for subsequent extraction of the data, some criteria were used for the exclusion/inclusion of articles.

Criteria for inclusion of articles:

  • Full articles;

  • Published as of 2013;

  • Presents some TUI application with the evaluation process;

Exclusion Criteria for Articles:

  • Complete book, abstract, poster or short article;

  • Be focused on another research area other than HCI discussion;

  • This article presents TUI application but does not present the evaluation process.

3.2 Results and Discuss

The SRL was performed on the ASEs and as a result, 703 references were returned, retrieved and stored in the StArt tool. The total set of articles resulting from this initial phase, classified according to the search engine used, is presented in Table 1.

Table 1. Distribution of articles found in each ASE
Full size table

The first filter the group of researchers carried out a screening by analyzing: title, keywords and abstract. In order to make this selection, we used the exclusion and inclusion criteria of articles, established on SRL protocol, resulting in a subset of 86 articles. Then, the final filtering cycle involved the three researchers with a complete reading of the articles to identify the answers to the research questions. Thus the final set listed in this SRL comprises the total of 50 articles.

About the research questions listed above, the answers give some information to discuss:

(a) What approaches are used in TUIs evaluation?

As shown in Fig. 1, most articles use the approach focused on usability for evaluations. This validates the operation of the technology, as designed, evaluating its efficiency/effectiveness.

Fig. 1.
figure 1

Evaluation approaches used in selected articles

Full size image

The listed articles that work with usability focused approach are: [2, 6,7,8,9,10,11,12, 15, 20, 21, 25, 27, 28, 32, 33, 37,38,39, 41, 42, 45, 46, 49,50,52] e [61].

User Experience was the second most commonly used approach, considering the user’s perceptions and feelings regarding their relationship with a TUI application.

These are the listed articles that work with a User Experience approach: [8, 13, 17, 18, 22, 24, 34, 37, 49, 54,55,56, 58,59,60] e [3].

Seven articles presented tests that did not allow a categorization about the evaluation approach used, applied only functional tests of the developed technologies: [4, 5, 35, 36, 43, 47, 53].

Of the articles listed, three of them [8, 37, 49] combined the Usability and User Experience approach, using a questionnaire as the main evaluation tool.

[37] is an example of this joint approach, presents a scenario of evaluations about perceived spatial affordances of hover interaction above Tabletop Surfaces.

At the same time that efficiency-effectiveness tests are presented with quantitative performance and error rates, there is concern about sensations in the use of the application. Was applied attractiveness measurement tools and the care given with characteristics that go beyond the standard questionnaire used, such as the height and size of participants’ arms, for example.

(b) What tools/instruments are used to measure the proposed goal in these TUIs evaluation?

In general, there are very different types of instruments that have been used to answer the evaluation objective within a given proposed approach.

Table 2 shows, the use of interviews (structured or semi-structured), questionnaires and the observation of the specialists were the instruments most used for TUI application evaluations returned at SRL articles.

Table 2. Evaluation instruments used in selected articles
Full size table

Figure 2 relates the evaluation approaches used with instruments that used.

Fig. 2.
figure 2

Evaluation approach vs. evaluation instrument in selected articles

Full size image

Interviews were usually used in a semi-structured way, with some key questions and others free, in order to evaluate how a certain group of users felt (UX) about the execution of certain tasks (Usability) in TUIs application.

[8] presents and his paper an evaluation of TUI application which goal was to improve learning conditions of children with dyslexia and attention disorder. In this scenario, a simplified questionnaire with children “liked or disliked” type, combined with interview applied to therapists, allowed a more accurate evaluation.

Focus groups were used in [28, 34] and [21] as a way of generating a collaborative observational quality in association with interviews.

Although relevant, the interviews had a much lower occurrence of use in the selected articles than the use of questionnaires and observation of the process by specialists.

Several patterns of questionnaires established have emerged from articles, such as the Scale Usability System (SUS) [14], used in [15] and also in [21] combined with Focus Group.

Another recurrent questionnaire model was the AttrackDiff [23], which an instrument to measure the attractiveness of interactive products and UX. This instrument was used in [37, 49] and [16].

The observation was the most referenced instrument in the final articles selected. Figure 2 gives indications that there is a relation to the use of this in a complementary way to the use of questionnaires. This is justified because questionnaires focus on user-generated perceptions, while observations are organized by specialists and mediators of the evaluation process. For example, [55] used the concepts of “gamification” for history teaching to primary school students using an augmented reality application in a TUI. In this work [55], a questionnaire was used to evaluate the interaction process with the students, and direct observation of the use was made by the specialists who recorded all the sections of the evaluation.

Based on SRL results, some observations about evaluation processes applied to TUIs:

  • Most articles did not use an only single evaluation instrument or approach to evaluate TUI applications. This use of more than one approach evidences the concern of not only the efficiency of the application but also UX. At this point, the use of instruments that allow users to collaborate during the evaluation process is growing.

  • There was no exact correlation between a particular TUI application evaluation approach and a specific package of instruments. According to the proposed objectives, the scenario or user profile (for example, children) defines which instruments will be used. Evaluation with children, for example, the reports of the articles indicate the difficulty of use questionnaires or more specific tools, requiring the involvement of other professionals in the evaluation process.

  • None of the selected articles proposes any specific evaluation tool that considers the more general characteristics of interaction with TUIs, nor adaptations of existing tools.

  • The concepts of Reality-Based Interaction, considering: intuitive physics, body consciousness, environmental awareness, and social understanding was not reflected in evaluation tools or instruments used. Observations on some points such as the height of the elements by the hands of users, arrangement and lighting of the environment, the design of the elements and use of materials more suitable for that type of interaction, although, none evaluation considered these particularities or how they can impact the interaction process.

  • Another important issue is that none of the articles (even the 703 from the initial phase) proposes to discuss a specific evaluation process with tangible interactions or to review the application of several existing methods in TUI.

The next section presents a practical of UX evaluation at TUI application named AR Sandbox using some instruments listed at SRL.

4 AR Sandbox: UX Evaluation for a TUI Application

AR Sandbox is a project developed by UC DavisFootnote 6, consists of a sandbox where a topographic map is projected on top of the surface (Fig. 3). The user shapes the sand to represent reliefs as if shaping the topographic map and, raising his hand over the sandbox, the system recognizes it as a cloud and simulates the flow of water.

Fig. 3.
figure 3

Users testing AR Sandbox

Full size image

The project consists of a didactic and educational TUI application that offers the user a dynamic practice of representing the topographic map. Users can apply their topography knowledge to compose scenarios, simulate precipitation and observe flow, as well as can shape real-world scenarios, rather than creating a model, the user can play it in the AR Sandbox by turning the relief into a moldable and interactive material.

In the work [16], this TUI application was implemented and the usability and UX evaluation were done. This section discusses the procedures and results of this evaluation, relating them to SRL conclusions.

4.1 Procedures

To evaluate the AR Sandbox application, the AttrakDiff tool was used, offering questionnaires for users, as well as generate graphs with the result obtained from the answers.

The evaluations of UX in AttrackDiff are separated by dimensions, which are: PQ, HQ-I, HQ-S, and ATT, where:

  • PQ (Pragmatic Quality): describes the functional quality of an application and indicates the degree of success through the user’s objectives achieved using it;

  • HQ-I (Hedonic Quality - Identity): indicates the level of immediate identification of the user with the application;

  • HQ-S (Hedonic Quality - Stimulus): indicates if the application supports the user’s needs concerning originality, interest and, especially, stimulus;

  • ATT (Attractiveness): it is the most comprehensive measure that quantifies the overall attractiveness of the application, based on the perception of quality by the user.

For evaluation, users freely experimented the application, executing some tasks such as shaping the topographic map, reproducing mountain, plateau, plain, depression, beaches, simulating rainfall and observing water flow.

After the experiment, each user evaluated the TUI application through 2 questionnaires: one form containing the Usability evaluation, and another form containing the UX. Questions and graphs of both forms were based on AttrackDiff instrument.

80 volunteers participated in the evaluation, 74 undergraduate students, two post-graduation students, three teachers and one “other”. 43 considered themselves as “Non-specialist” (computer science students or teachers), and 37 were considered “Specialist” (geology or hydric engineering students or teachers).

4.2 Results and Discuss

Figure 4 presents the UX evaluation of the AR Sandbox. The questionnaire users indicated their opinions on a semantic scale that varies from one adjective to another.

Fig. 4.
figure 4

Semantic scale results of UX in AttrackDiff for AR Sandbox TUI application.

Full size image

The first pair “Technical - Human” received 0.7 scores, most of the users were in doubt of the meaning of Technical and Humanized adjectives for the application, therefore, they did not know to answer and they marked neutral. Other users indicated “Human” because the interaction is “more natural”.

“Predictable - Unpredictable” pair received a 0.4 scores, the response range for this question was \(-3\) to 3, but the evaluation shows that the mean of the responses tended to be “Unpredictable”, indicating that users believe the AR Sandbox it is surprising.

“Expensive - Cheap” pair received 0.2 scores, users believed that the cost to implement the AR Sandbox is not cheap. They had difficulty evaluating this issue because they did not know the value of the equipment or the complexity of the computational system.

Furthermore, the “Challenger - Easy/Not Demanding” pair received a rating of 0.3 scores, which indicates that although application manipulation is simple (shaping the topographic map and simulating rain), other users were motivated to use their imagination to simulate different scenarios.

In the Fig. 5 the results of the UX evaluation fields of AttrackDiff are displayed. In general, the evaluation of the fields was uniform, between 1.7 and 1.8, and all fields were above average (0).

These numbers allow us to conclude that the UX evaluation of AR Sandbox had a good result, users were able to perform tasks accurately, interested in the application, felt stimulated, attracted, and believes that application works.

Fig. 5.
figure 5

The evaluation result of UX in AttrackDiff dimensions for AR Sandbox TUI application

Full size image

Introducing the discussion of the UX evaluation results of the AttrackDiff questionnaire for the AR Sandbox application, many considerations are related to observations in the SRL about evaluation methods in TUI:

  • Questionnaires quoted in the SRL, and the AttrackDiff that used in this evaluation with the AR Sandbox, do not consider a full observation for a TUI application. Environmental factors, physical body limits for use of this type of interface are ignored.

  • Specialists comments on the use of the AR Sandbox allowed the discussion of questions that could help in the evaluation and design of the TUI application.

  • Accessibility is not included in the AttrackDiff questionnaire nor in the others cited in the SRL. When applying evaluation with the AR Sandbox a wheelchair user had difficulty using the TUI application because of the height of the table. There are not topics about how accessible the application is for a person with any kind of disabilities or about the environmental factor that could interfere in the UX.

  • A TUI application aims to extract the user’s physical abilities for interaction in a natural way with their interface. During the tests of AR Sandbox, users proved that this characteristic is present, but do not explain or evaluate this at questionnaire. The tasks performed were simple and intuitive, requiring no further explanation for use of the interface.

  • Facial expressions of satisfaction and how users interact with each other encourages collaboration in the use of the application. As for example, some groups of students simulated the dam breaking to observe the flow of water and other groups shaped the highest mountains. However, the AttrackDiff questionnaires and several other instruments cited in the SRL are answered individually and do not present questions about social engagement and collaboration about TUI application use.

  • Some questions in AttrackDiff instrument were unusual, repetitive and others were not relevant for the evaluator. For example, the “Technical - Humanized” and “Cheap - Expensive” adjective pairs were questions that the users had different interpretations, causing some confusion of evaluation of the application.

5 The Potential of UX for TUI Evaluation

Academic research and commercial applications with TUI have been growing at the HCI area. Although the projects in TUI are more specific application contexts, it was noticed at SRL and in the experiment of evaluation with AR Sandbox, these evaluation methods currently used privilege interactions through GUI.

The concept of usability has become more comprehensive, but still, evaluations with this approach have as their main objective to assess the quality of software, efficiency, and effectiveness with a look to technology. Because TUIs are a relatively recent technology, most of the benchmarked evaluations focus on testing whether the “application works”.

On the other hand, according to ISO 9241-110: 2010, UX is defined as: “user perceptions, feelings, and sensations that result in the use and/or use of a product, system or service” [57].

In this way, UX is a more comprehensive approach of evaluation that allows analyzing TUI applications not only by the functional characteristics of the system. UX takes into account user interaction with the entire application, evaluating user’s thoughts and feelings [1].

To evaluate a product by UX, qualitative methods such as interviews, questionnaires, and written documents can be used. The data collected by observations can be verbal quotations from the user expressing their opinions, feelings, and knowledge.

This was verified with this work since the closed questionnaires do not usually provide space for dialogue between the evaluators with the users and between them.

The results from the observations are detailed descriptions of the behavior and actions of the user and the data collected in documents are summaries, citations or reports.

Quantitative methods allow the grouping of the answers of users in order to analyze what is common between the answers, and with the generalization, can extract information for comparisons, ideas and meanings of the events occurred in the evaluation.

The evaluation of UX with the application AR Sandbox used the AttrackDiff, which is one of the instruments referenced to evaluate with this approach and allows to relate that the ergonomic quality takes into consideration the usability because the user feels more comfortable when it has the control of the situation.

But, even AttrackDiff does not present basic questions that contemplate the full scope of the interaction experience with a TUI. If evaluations are made with the inadequate instruments there may be a misinterpretation of the results and design of new applications in TUI may have impacts.

Thus, the first step in proposing a specific methodology for TUIs is to understand that existing instruments are flawed. For most applications today use the GUI the evaluation tools do not extract the maximum potential when used in TUI.

Hence, it is suggested to adopt some basic considerations to enhance the use of the UX approach to TUI evaluation:

  • The main element of TUI applications are physical objects as elements of interaction, the choice of how this object will be and how to evaluate its use is an important issue when developing the application, where designers should be concerned with the size, object, as this should interfere directly with the UX.

  • TUIs demand the user’s body gestures to shape and move objects in the interaction process and such aspects should be present in the evaluation of the application. Consider that people have different biotypes and may not have all the preserved capabilities (accessibility).

  • Do not use only a single instrument to evaluate UX in TUI, but consider using them together depending on the context of a specific use of that application, user profile. For example, with children, a particular instrument will not bring results while with adults it can works.

  • Social collaboration is a basic and fundamental characteristic during the use of a TUI application. It has a major impact on UX and creating ways to evaluate this during the interaction process is critical. Focus groups, thinking aloud, expert observations or methods that enable users to talk to each other and with evaluator should be incorporated.

  • Consider the concepts of Reality-Based Interaction (intuitive physics, body consciousness, environmental awareness, and social understanding) during the evaluation process. If the instruments used to allow this type of information to be interpreted or if it is necessary to adapt or build a specific tool for it.

Also, from the general analysis of the articles referenced in the SRL, it is possible to generate categories of evaluation types based on the proposed objectives:

  • Comparison of user interfaces: aims to compare the use of traditional interfaces, such as GUI, in the relation of TUI, evaluating the gains/benefits of tangible interaction. Usually, they use the same traditional evaluation tools, with criteria developed for GUI, applied in a new use with TUI, having questionnaires and usability as a focus;

  • Collaborative interaction: TUI has as the main characteristic to privilege the process of collaborative interaction, in a certain physical space. This group of articles tries precisely to evaluate if this characteristic of the TUIs potentiates a certain UX in an application that needs more of this process of collaboration. In these cases, the use of evaluation approaches and instruments such as focus groups, observations, and UX questionnaires is preferred;

  • Education instrument: aims to propose solutions based on tools focused on the teaching/learning process for children and young people. They are of specific contexts and presented a variation in the instruments of evaluation, precisely due to the complexity inherent in the application of structured questionnaires with children.

These categorization groups help in the understanding that the evaluations have several solution proposals and that realizing this potentiates the results. How objective is pursued in the evaluation process impacts the prescription of the UX evaluation method itself.

6 Conclusion and Future Works

Applications that make use of Tangible User Interfaces are something new and incorporate various forms of interaction, and just as common interfaces are evaluated the TUIs should also be.

The forms of interaction of the TUIs are diverse, the possibility of the user interacting with their body to manipulate the interaction element and the responding application in the same object stimulates the feeling of immersion.

This work evidenced the need to develop an appropriate methodology for TUIs. According to the results of the SRL developed in this work, inadequate methods are being applied in a generic way for TUIs, complicating the possibility to improve the applications.

Between them, even though it is not the most used, the User Experience is the closest approach to evaluating TUIs. UX shows its potential to be applied in the evaluations process for this type of interface, required customization according to TUI application to adapt to the many particularities that the interaction has to offer.

From this study, as future works, it is intended to elaborate a set of guidelines on how best to apply a UX evaluation approach in TUI applications, incorporating the issues discussed in this paper, to be validated during the test stage with users for evaluation of interface, consolidating in a methodology for TUI evaluation.