Keywords

1 Introduction

Can you treat Virtual Reality (VR) as a coworker? What does it mean to trust VR?

Trust is a fundamental concept in everyday life, being a factor influencing several types of interaction, from interpersonal to group behavior, from economic to politic [1]. It can be assumed that for every situation where an interaction is provided, trust is indispensable. Thus, many researchers in the past decades studied the concept of trust applied in several domains such as inter-organizational trust [2], or trust in leadership [3].

Nowadays, the nature of interaction has changed with the introduction of technologies. Thus, other than with people, many interactions occur between people and technologies and, also for this type of relationship, trust is an essential part. It is clear that trusting people and trusting technologies is different [4]. As suggested by [5], person-person trust is “a psychological state comprising the intention to accept vulnerability based upon positive expectations of the intention of the behavior of another” (p. 395). This definition cannot be applied to trust in technology, since the technology does not have any “intention”. This assumption led some authors to state that only people can be trusted [6]. Conversely, several authors (including the authors of this paper) suggest that some characteristics of person-person trust are extendable to the trust in the use of technology [7, 8]. Reference [4] specified that when referring to the relationship with technologies, “trust situations arise when one has to make oneself vulnerable by relying on another person or object, regardless of the trust object’s will or volition” (p. 3). In literature, trust in technology is considered a more problematic form of relationship to the one between person-person [4], mostly because technologies may not guarantee to people the same level of assurance and support to reach their goals (e.g. pay for a service, fill a form) that another human being can guarantee [6]. However, in each interactive situation, trust in technology, intended as a set of beliefs in the use of a product (i.e., belief in the functioning, reliability, safety, etc.) is a key factor for a satisfactory relationship between user and product [4]. A clear example of the importance of trust in technologies is the use of personal data to buy goods on websites [9]. Moreover, research has shown that trust is a key factor for the success of technologies in several domains such as decision to purchase a product online [10], in the business-to-consumer field trust has a direct effect on the vendor opinion and an indirect effect on consumer intention to purchase [11]. Concurrently, in social networks, [12] found that technological factors of trust are as important as interpersonal trust.

Regarding the definition of trust in technology, [4, 8] claim that trust in technology reflects at least three main beliefs about the attributes of a technology: (i) belief about the functionality of the product, which refers to the capability of a technology to perform specific tasks; (ii) belief that the technology is helpful. Thus to the fact that the technology will help the user to perform a specific task; (iii) Belief that the technology is reliable, hence, the perception that a technology works properly.

Even though the studies on trust in technology are growing, currently there is not a clear framework on the factors affecting users’ trust before and during the interaction with different technologies. However, some theoretical and empirical studies have suggested that trust in technology can be supported or prevented by the perceived usability of the system [13]. Concurrently a relationship between trust in technology and usability was underlined in several domains, such as web retail [13], information systems [7] and e-health [14].

Usability is defined as the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use [15]. A low level of usability could compromise the users’ interaction with a product, thus affecting the individual’s trust in the technology functionality, reliability and helpfulness.

Although the relationship between trust in technology and usability is known in HCI, currently there are no empirical studies that analyzed the relationship between trust and usability during the interaction with virtual reality (VR) environments. VR is defined as an experience where an individual is placed in a 3D environment generated by a technology, and is able to interact inside this environment [16]. VR systems are today commonly applied in several situations – such as phobia treatment [17–19], pain reduction [20–23], training [24] etc. Nevertheless, as [24] suggested these systems are, still today, quite expensive, and often researchers and practitioners are so focused on testing the functionalities of their systems that they missed to explore, with standardized tools, aspects such as the perceived usability and trust.

Present work aims to discuss the initial results of ongoing research on trust in VR.

The objective of the research is to identify the main factors of trust, and validate a model to assess trust in the use of VR. The initial framework hypothesizes that trust in VR is influenced by at least three main factors: (i) Usability, (ii) Presence [25], and (iii) Acceptance [26] (see Fig. 1).

Fig. 1.
figure 1

The framework of trust in virtual reality

Full size image

An ongoing set of data on the relationship between usability and trustFootnote 1 of different VR environments will be presented and discussed in this paper. To gather data, three VR systems - a Desktop 3D tool, a Cave Automatic Virtual Environment (CAVE), and a flight simulator - were tested using twenty five participants – ten participants for the 3D Desktop, ten for CAVE, and five for the simulator. Whilst three different virtual environments were experienced for different experimental aims, the same methodology was used after each interaction to assess both trust and usability. This methodology is described below.

2 Experimental Design

2.1 Materials

Three experiments were organized from October 2014 to December 2014. Each study used a common set of standardized evaluation tools (i.e., questionnaires) to gather data about perceived usability and trust.

  • System Usability Scale (SUS). The SUS has been built by [27]. SUS is composed by two factors: Usability and Learnability [28–30].

  • Trust in Technology Measures (TTM) questionnaire, developed by [4] has been built to investigate trust in technology.

2.2 Hypotheses

The hypotheses of the study were:

  1. 1.

    The trust and SUS questionnaires are reliable for subjective measures of VR products.

  2. 2.

    There is a correlation between trust and usability.

  3. 3.

    This correlation exists in different types of VR systems.

2.3 Study 1 – Desktop Virtual System

The first study was conducted at The University of Nottingham for the project LARTE (Live Augmented Reality Training Environments, in collaboration with HoloVis Ltd and Jaguar Land Rover (JLR). LARTE project aims to investigate the effectiveness of a virtual training for automotive services. Data from ten participants (5 male, 5 female; Mean age = 26.5, SD = 4.77), recruited among students of The University of Nottingham in UK, were used to observe the relationship between usability and trust. In study 1, participants, through a pc desktop, were trained on how to perform a service procedure on a 3D car model experienced on the LEGO ® Digital Designer (LDD) virtual environment. Each participant was initially trained in the use of the LDD for 15 min by an expert. After that, participants received a video training on the service procedure and were asked to perform the procedure on the LDD and on a real model of a car. After the performance, participants were asked to assess the usability (SUS) and the trust in the use (TTM) of LDD application as a virtual training tool.

2.4 Study 2 – CAVE Environment

The second study, conducted at JLR Virtual Innovation Centre (VIC), was designed with a twofold objective, as follows: (i) to investigate the applicability of the trust questionnaire to VR environments experienced in a CAVE; (ii) to have a first set of data on the relationship between trust and usability in CAVE. The participants were 10 all were JLR employees (7 Male, 3 Female; Mean age = 33.21, SD = 15.01). Participants of this study received an initial training (15 min) to learn how to use a CAVE system and become familiar with the manipulation of the objects in the 3D immersive environment. After that, participants were asked to interact with the technology to perform a set of assembly and reassembly tasks with a limited amount of time. After the performance participants were asked to assess the usability and the trust in the use of a CAVE application to perform object manipulation and assembly and reassembly tasks.

2.5 Study 3 – Flight Simulator

Study 3 aimed to investigate the reaction of pilots of Airbus Group to a new interface for a jet commercial aircraft. A part-task simulator was used to allow pilots to interact with the new interface. Moreover, the simulation experience also used flight gear for a short haul flight.

Study 3, as well as the above studies, is ongoing research. For the aim of this work, we discuss here only the outcomes collected by five male pilots (Mean age = 50.2, SD = 8.23). Pilots have both military and commercial licenses and are based at Airbus Group. Each participant experienced the interface of fly simulator to achieve three emergency activity tasks given to the participants. The emergencies given to the participant include; engine fire, fuel leak and a combination of the two. Each task starts during cruise phase; consequently the participants do not have to take-off or land.

The procedure for experiment three requires two experimenters in the assessment. The first experimenter is the lead experimenter, who looks after the participant, explains the tasks and delivers the questionnaires. The second experimenter’s role is to handle the simulator, explain the cockpit philosophy to the pilot and act as Air Traffic Control (ACT).

Firstly, the lead experimenter briefs the participants regarding the flight including the route, what they would be required to do i.e. to manually fly the aircraft, the weather, etc. This can be considered similar to what would occur during a briefing session before a flight was to take place. The participants are then introduced to the system and asked to have a trial run using the interfaces. Pilots were instructed to freely interact with the system until they were comfortable and ready to start the task. The order of the tasks is controlled to reduce the learning effect. During the task they need to maintain aircraft control, attend to the emergency using the relevant checklists and comply with the ATC instructions (delivered by the second experimenter). After the performance of the three tasks, pilots are asked to complete the SUS and Trust questionnaire about the use of the fly simulator application to learn how to react in emergency situations.

2.6 Data Analysis

Pearson’s correlation coefficient analysis was used to assess the correlation between SUS and Trust questionnaires. Moreover, Cronbach alpha index is used to check the scales reliability. Finally a linear regression with stepwise method was performed to observe the effect of usability on trust. The results of the TTM are converted in percentage.

All the data were processed by IBM SPSS ® 22.

3 Results

The three VR applications tested in the studies were perceived by participants as usable – average SUS overall score equaled 78.9 % - and trustworthy – average TTM score equaled 77.9 %. Nevertheless the three applications have different levels of usability and trust, in particular the flight simulator resulted in the most usable and trustable VR environment among the studies (see Fig. 2).

Fig. 2.
figure 2

Overall scores (in percentage) of SUS and TTM scales for each study. Study 1 LDD application experienced through a Desktop pc. Study 2 a VR environment experienced through a CAVE. Study 3 a VR application experienced in a fly simulator.

Full size image

Independently from the hardware experienced by participants (Desktop, CAVE and simulator), SUS and TTM questionnaires resulted reliable tools when applied to measure usability and trust in the use of VR environments. As Table 1 shows SUS reliability ranges from .65 to .92, while TTM ranges from .792 to .884.

Table 1. Cronbach’s alpha (α) of SUS and TTM scale estimated in each study
Full size table

The overall scale of TTM and SUS questionnaires correlated in study 1 (r = .665, p = .036) and 2(r = .672, p = .033). Concurrently, as Table 2 shows, TTM overall scale correlate with the usability factor of SUS in all the three studies.

Table 2. Correlation between the overall scale of TTM and the usability factor of SUS. Correlation was measure through the Pearson’s correlation coefficient (r).
Full size table

Linear regression, as showed in Table 3, revealed that overall scores of SUS predict TMM overall scores only for participants of study 1 and 2. Nevertheless, the usability factor of SUS resulted as a good predictor of TTM scores for the participants of all the three studies. Figure 3 shows the relationship between the usability factor and TTM in the three studies.

Table 3. Outcomes of linear regression to observe the effect (F, β and p) of the usability factor of SUS on the overall level of trust, and to analyze the percentage of explained variance (AdjR2).
Full size table
Fig. 3.
figure 3

Relationship between usability factor of SUS and TTM for each study

Full size image

4 Discussion

The relationship between usability and trust has usually been studied in web and software domains. These three studied aimed to investigate if this relationship is applied also to VR systems. This section will discuss the results of the three studies in light of the hypotheses explained above in the paper.

Hypothesis 1: TTM and SUS are Reliable for Subjective Measures of VR Products. The results of the Cronbach’s alphas indicate that the two questionnaires are reliable for VR domain, mainly in study 2 and 3. Since there is lack of literature regarding the applicability of these questionnaires in VR, these results can have a high impact on the future work in the VR fields and in the way of measure usability and trust.

Hypothesis 2: There Is a Correlation Between Trust and Usability. The results of the correlation analyses demonstrate that there is a significant correlation between the overall SUS and TTM in study 1 and 2. Whist these results show that there is a relationship between SUS overall scale and trust for study 1 and 2, this is not demonstrated in study 3. However, significant outputs were found in all the three studies when investigating the influence of the usability factor of SUS on trust, shown in Table 2.

As for the correlation, also the linear regression shows significant results for the relationship between overall SUS and TTM for study 1 and 2 and a significant relationship between the usability factor of SUS and TTM for all the three studies.

The lack of significant result for study 3 could be due to two different reasons. First of all, the small sample. It is possible to hypothesize that with a bigger sample size, the results will align with study 1 and 2. The second reason could rely on the nature of the participants’ experience. Indeed the study 3 is the only one where the users were actually experts. Further analyses on the influence of expertise on the relationship between SUS and TTM should be done.

Hypothesis 3: This Correlation Exists in Different VR Systems. As the three studies display similar results, this indicates that the relationship between usability and trust is demonstrated in different VR systems with different characteristics. This is mainly demonstrated by the results of study 1 and 2, which use two different VR systems. For example, the CAVE used in study 1 had a 3D environment in comparison to the 2D desktop set up, different interaction modalities, and a different level of immersion.

5 Conclusion

As stated in the introduction, VR is a growing technology and it is applied in several domains. Investigating the factors influencing the trust that users have in the technology could change the way people interact with VR and the way VR is developed. This study aimed to have a first set of data on the relationship between one of the most known factor of HCI, usability, and trust. It is important to address that this is one of the first studies where trust is applied to VR systems.

Generally, it can be said that the results of the three experiments fulfill the hypotheses of the study, demonstrating (i) that standardized tools largely applied in HCI studies may be reliably used to test the user experience of different VR systems (ii) that a strong relationship between usability and trust in VR systems exists, thus usability could be a predictor of users’ trust in VR systems and (iii) that the relationship between usability and trust is present in different VR technologies.

Further data will be collected in the next months, to add more users to our evaluation cohorts. Nevertheless, current results support the idea that the designers who want to deliver a successful VR application have to carefully define the functioning of the system by proving usable tools. As said above, trust is a wide and multi factorial concept, therefore understanding the usability role is just one step toward the full comprehension of the trust in technology concept applied to VR application. In tune with that, future works are needed to investigate the role of usability in the development of people beliefs and attitudes (such as trust) toward a technology, but also to understand the effect of other factors (such as Presence and Technology Acceptance) on trust in VR.