Keywords

1 Introduction

Ambient Intelligence environments (AmIE) are responsive systems designed and implemented to support everyday living and working activities [1]. As with any other technology, trust plays a vital role in the wide adoption of Ambient Intelligence environments. Its role has been acknowledged for almost all application domains [3, 4, 15, 18–20]. Therefore, to ensure the acceptance of Ambient Intelligence and support its wide adoption, it is critical to take into account the trust factor.

The major scientific problem is the lack of instruments for a runtime trust measurement. While some questionnaire-based instruments for empirical measurement of trust exist (cf. [5, 10, 11]), to the best of our knowledge, there is no computerized system that can directly measure users’ trust in a system at runtime.

As we discussed in our previous publication [13], although we cannot directly measure trust, some of its determinants can be measured at runtime, which allows us to estimate the levels of trust with sufficient certainty. Examples of such determinants are usability, context information quality, system decision-making uncertainty, and interaction conflicts.

Theoretically, such data could be produced within large-scale real-life experiments. However, conducting real world experiments requires the full scale development and maintenance of sensors and actuators networks. This makes data collection from real deployments scarce, expensive and time consuming [8]. A solution would be conducting online experiments to collect the necessary data to produce trust estimation models.

However, a problem with conducting online experiments for this purpose is that AmI environments are relatively new for end users. As a result, most of the users do not have any experience with such systems, thus lack the necessary mental models to correctly assess the systems under study. To overcome this problem, interactive virtual environments of AmIE can be created to help develop the appropriate mental model for users, which is expected to improve their realization of AmI environments [12]. In our previous work, we have analyzed to which extent short-term experience with virtual reality based simulation of intelligent environments supports the development of novice users mental models [16]. Our results indicated a positive correlation between virtually experiencing intelligent environments and enhancements in mental models.

We have conducted mentioned virtual experiment and reported in [13]. In this paper, we describe more in detail the testbeds technical aspects. To make this paper more self-contained, we summarize literature in Sect. 2. As our major contribution, we describe the design and development of a virtual testbed for studying trust in AmIE. To overcome the above-explained problem of incomplete or missing mental models, we used 3D Virtual Simulator (3DVSim) of AmI. This is because 3DVSim of AmI will provide users with initial experience supporting the development of mental models. Improving subjects’ mental models will decrease subject bias [7], i.e., through the extended interaction with the 3DVSim, users will better understand the system under assessment, which allows for low biased answers.

The remainder of this paper is organized as follows. In Sect. 2 we discuss research on measuring trust in automation systems and web-based surveys as an instrument in psychometric measurements. Section 3 presents our proposed method for virtual testbed development and an example experiment we conducted using the proposed 3DVSim.

2 Literature Study

To date, trust in automated systems has been under explored. However, some significant research exist.

In [17] the effect of automation error on system trust in the domain of route planning was studied. Participants experienced both manual and automatic mode of route planning. The number of errors was the independent variable of this study. Participants were recruited and assigned to 4 experimental condition. The experiment was done on the computers at laboratory. After completion, participants were asked to rate their trust in automatic system by a score between 1 to 7.

In another study [2] the effect of displaying system confidence on the user trust in context-aware mobile phone was investigated. The experiment showed that displaying confidence information increased users trust.

In another study [4] an experiment conducted to investigate the effect of dynamic adjustment of the level of automation on users trust in ambient-aware environments. The experiment was conducted on an experimental smart home environment.

All the above experiments were conducted in physical laboratories with 14–90 students and at most with 4 experimental conditions. One limitation of such experiments is the number of participants. Besides physical experiments are very time consuming and expensive. These challenges led us to seek for other experimental testbeds.

Online surveys help researchers to recruit large number of participants at a relatively low cost, with fewer physical resources. Factors such as costs of acquiring observers, participants, equipments and time consumption led to collect data from users by online survey tools (e.g., survey monkey.com) [6].

Behrend et al. (2011) collected both traditional university pool samples and Mechanical Turk samples to conduct a survey. The online sample were more diverse in demographic characteristics and showed higher internal consistency. They concluded that the reliability of data from online sample is as good as or even better than the university sample.

However, a challenge of online surveys when applied to AmI environments is that subjects may not have developed adequate mental models that are necessary to understand the system under study and deliver proper answers to the post experiment questions. Simulators or virtual environments were used to conduct human-computer interaction studies before. An instance is [14] that proposes an approach for user-user conflict resolution in smart environments.

Several challenges such as expensive appliances, collecting information from sensors, reasoning from knowledge base and the time and monetary cost make it difficult for researchers to study in a real smart home. To overcome mentioned challenges, we propose a virtual testbed for studying trust in AmIE. To overcome the problem of incomplete or missing mental models, we used 3DVSim of AmI. This is because 3DVSim of AmI will provide users with initial experience supporting the development of mental models.

3 The Proposed Virtual Testbed

There are nine main steps that are required for the development of the virtual testbed that we propose in this paper. These steps are shown in Fig. 1. It describes both the technical preparations and necessary steps of the proposed research method. Based on an example, we describe how to use the virtual testbed for studying trust in intelligent environments. Suppose that we aim to measure the relation between trust and interaction conflicts. Our main hypothesis is that higher conflict density results in higher loss of trust. In the next sections, we describe how to go through the mentioned steps to prepare the testbed that will allow to study the above hypothesis.

Fig. 1.
figure 1

The proposed method

Full size image

3.1 Developing Instruments

In this step, we designed experimental conditions in which conflicts of various intensity and frequency were expected to occur at different stages (beginning, middle, or end) of the experiment. We name each of the designed experimental condition a test scenario. Each test scenario is clearly distinguishable within the prepared interactive 3D virtual environment.

In each scenario, subjects are expected to conduct five typical tasks of the analyzed study domain (e.g., smart home). Examples were turning on lights, opening a door or window, etc. Based on the interaction conflict components identified in the previous step and our hypothesis, we designed 3 types of scenarios: scenarios with low conflict densities, which contain no conflict or one conflict; scenarios with medium conflict densities, which contain 2 or 3 conflicts; and scenarios with high conflict density, which contain more than 3 conflicts. Figure 2 describes the formulation of the scenarios.

Fig. 2.
figure 2

Scenario formulation

Full size image

3.2 Developing 3DVSim

This step concerns with the development of the 3D simulation. Based on the scenarios sketched in the previous step, we have developed a 3DVSim, a virtual environment that supports interaction in the virtual/simulated smart home. It simulates the experimental scenarios. It provides users simulated smart devices such as smart TVs, connected lights, air conditioners, and so on. In addition, a virtual user interface is embedded into the 3DVSim that allows users to control the virtual smart devices involved in the virtual test. This component empowers users by allowing explicit command and control of their virtual intelligent environment. However, in order to induce conflicting scenarios, the 3D simulation also undertakes automated actions. Firstly, this simulates the intelligent behavior of the smart home system under study. Secondly, it triggers interaction conflicts. Through this nearly realistic experience, users are expected to (further) develop their mental models of the concepts under study (e.g., conflicts in Ambient Intelligence environments). As explained in section one, this will help improve the validity of users responses to trust measurement questionnaires (trust scales).

3DVSim allows to validate and test the dynamic behavior of strategy components, reactive agents, device composition platforms and adaptive user interfaces. It becomes possible by manipulating ambient settings, environmental conditions or device states, as these are the most relevant input parameters determining the behavior of mentioned components.

Fig. 3.
figure 3

Dynamic 3DVSim generation process

Full size image

The process shown in Fig. 3 generates the 3DVSim dynamically. At the beginning of the process, the Environment Manager retrieves a complete environment overview in order to acquire the required data of the physical room, such as list of discovered devices, the 3D representation of each device, as well as the device profile, which is needed to retrieve the corresponding Device Control Interface (DCI). These DCI are usual controls, e.g. radio buttons or context menus. The overview defines by researchers in an editing tool which creates an abstract description of the experiment.

The Environment Monitoring component is responsible for device discovery and monitoring. This part of AmIE observes the whole environment and provides information about important changes to the other components such as the Interaction Appliance and automation agents (implicit interaction), e.g., changing in a device property like the on/off state of a lamp, or a change in room properties like the lighting. This component is necessary for both physical and virtual environment, because we need to detect and analyze changes in AmIE components.

After defining overview, we must select devices and behavior of each device in order to model virtual environment. Each device represented within the 3D-view has a unique ID. This ID allows the user interface to link the 3D object with a physical device which is usually controlled by our system. Then we need the position and orientation information for each object, which is possible with XML description files. Each file can contain name, device ID, position of the object, orientation, affordances and behaviors of the object. An XML-file describing an example room is shown below:

figure a

At the end of the process, the Environment Monitor writes all the information needed to create the user interface into an XML file which is then sent to the Interaction Appliance.

Fig. 4.
figure 4

Screenshot of 3DVSim developed by Unity game engine (Color figure online)

Full size image
figure b

Using the output XML file and a database containing 3D objects, the 3DVSim generated using the Unity game engine. The output is WebGL. The Unity game engine (version 5) supports WebGL and can transfer the content directly to a web server. WebGL is a powerful API that is incorporated into Firefox, Chrome, etc. and can render 3D content within the browser [9] without requiring application installation or additional components. Figure 4 shows screenshots of the developed testbed. Controllable devices are signaled with red arrows.

3.3 Choosing Appropriate Trust Scale

Along with previous steps, appropriate trust scale must be chosen. Users evaluate the trust in the system with a proper scale. The questionnaire presented by Jian et al. (2000) presents a scale for human-automation trust measurement. Their proposed scale can be utilized in all automatic systems, such as AmI environments. It proposes several components for trust. It contains 12 items, each one evaluating a proposed component of trust such as familiarity, honesty and so on. All the items aggregately represent the user trust. This scale is the only empirically developed human-automation trust scale. We decided to use this scale because of its generality and usage in similar research studies [4]. For more detailed information on this scale, readers are referred to [5].

Fig. 5.
figure 5

Website IT architecture

Full size image
Fig. 6.
figure 6

Screenshot of the webform containing the questionnaire

Full size image
Fig. 7.
figure 7

Screenshot of the experiment website - users are redirected to the webform through “Go to Questionnaire” button

Full size image

3.4 Developing Website and Recruiting

In this step the prepared virtual experiment needs to be extended to run within a web-page. This will allow a large-scale and wide area participation. Additionally, website contains the questionnaire. Users will be requested to read and complete the questionnaire (trust scale) After they have interacted with and experienced the simulated environment. Please note that through this short term experience, users are expected to have developed the necessary mental models. The experiment is conducted over an Internet connection, and it is hosted at shexperiment.hcilab.ir/en. The website uses a PHP server-side scripting language and a MySQL database to store 3D contents and participant responses. The online survey form is written in PHP/HTML. The experimental data are processed by a PHP script and stored in MYSQL database.

The experiment can be accessed from any system with an Internet connection and Firefox version 5 or newer, Chrome version 12 or newer, Safari version 5.1 or newer or Opera version 11 or newer. At least 256 MB of graphic memory is required. Figure 5 shows the IT architecture of the experiment website. The experiment link was shared on our social networks in Facebook, LinkedIn, and ResearchGate. Participants should be randomly assigned to an experimental 3D scenario.

Again, after experiencing the smart home and the related scenario through the 3DVSim, participants should be redirected to the questionnaire, where they are asked to answer the questions. At the end, the responses will be saved in a csv file for statistical analysis. Figure 6 shows the questionnaire. Figure 7 shows how users are redirected to the webform after having completed the 3D-based virtual experiment.

4 Conclusion

In this paper, we proposed a virtual testbed and its development process for measuring trust in Ambient Intelligence environments using online surveys. We described a web-based experiment to gather data from a large number of online participants. Within the presented testbed, subjects experience an intelligent environment through an 3D Virtual Simulator before attempting to complete a well-known trust questionnaire. Using our proposed method, we are able to conduct an experiment to analyze the effect of interaction conflicts on users trust in intelligent environments. However, our proposed approach can be utilized to measure trust in the presence of any other variable that is hypothesized to have an effect on user trust in such environments.