Keywords

1 Introduction

In today’s world, Virtual Reality (VR) technologies have become more and more common due to increases in computing performance as well as advances in input and output technologies. Probably the most commonly known use of VR is in computer games, which have brought interaction with VR environments in to the reach of large audiences. While the VR industry is heavily driven by entertainment based use cases, especially in the gaming industry, the use of VR for professional purposes is also increasing. When building cities, factories and other large industrial sites, the role of simulation technologies can be essential in enabling cost-efficient exploration of different solutions in the design and planning phases. When exploring 3D models of industrial site architectural plans, it is important to gain a realistic perception of the environment in order to assess different construction designs. On the other hand, for sales purposes, the experiental factors may play a role when communicating with potential customers. For these reasons, it is interesting to explore different interaction technologies and immersion in this context.

Whereas 3D virtual worlds have traditionally been accessed via a PC with a monitor display screen, other interaction technologies are becoming more common, not only for researchers but also for other audiences. Immersive environments such as CAVEs have become less expensive to set up, and HMDs such as the Oculus Rift and Google Glass have caught the attention of developers as well as consumers. In this paper, we describe our work on utilizing HMD and CAVE presentations, rather than a conventional PC set-up to explore an industrial site simulation utilizing 3D models of the physical world, see Fig. 1.

Fig. 1.
figure 1

Test participant exploring the virtual factory environment with Oculus Rift head mounted display Paths taken by users are shown as blue trails.

Full size image

The contribution of this paper lies in:

  • Providing user study based information on the relative benefits of CAVE and HMD visualizations in identifying human perception issues in the factory site design process.

  • Demonstrating that heart rate monitoring can be used as a useful source of data on user perception when exploring a virtual industrial site environment.

2 Related Work

2.1 User Experience with 3D Virtual Worlds

User experience (UX) research covers both utilitarian and hedonic aspects related to the user a product, service or user interface [1]. Whereas a unified definition for UX has not yet emerged [2], it is generally agreed that it goes beyond the instrumental means and measures relevant for usability research. User experience research expands the traditional perspective of usability oriented user research by including hedonic values associated with the interaction, and investigating the system use in a more holistic manner. Whereas UX research on 3D virtual worlds has so far been altogether rather scarce, prior art has addressed selected topics UI design and users’ perceptions on such systems.

Research on the UI design of virtual worlds (VWs) has typically focused on quite narrow topics, such as text readability in 3D games [3], avatars [4], or collaborative and shared space aspects [5]. However, it has been identified that the ability to create realistic visualizations is one of the advantages of using 3D virtual worlds, especially when compared to conventional textual or 2D graphical user interface (GUI) presentations. User perceptions of the use of 3D virtual worlds has been charted e.g. in the context of home health care professionals. One identified strength in the approach was the patient activity monitoring, which could be visualized in a realistic yet in anonymous manner by using activity tracking combined with a 3D virtual world presentation and avatars [6]. When investigating different visualization techniques, Vatjus-Anttila et al. conclude that when 3D models of the physical world are considered, a better user experience is achieved when the VW design matches people’s mental expectations of the physical space [7]. Contrary to our research, in prior art the research focus has been on the GUI design elements, and not in comparing user perceptions when different technology platforms are used to interact with the virtual worlds.

Considering the perception of height in both real and virtual representations, Emmelkamp et al. [8] have presented work focused on the treatment of those with acrophobia, concluding that exposure to the virtual condition was as effective as real exposure in treating the condition. Here, the participants’ heart rate was used as one measure of their response to real and virtual height. In other studies on the treatment of phobias using virtual reality [9] identified galvanic skin resistance as a more useful measurement technique than heart rate. The use of heart rate monitors in exploring user experience in general has been reported on by [10]. Changes in heart rate caused by playing conventional video games has also been studied e.g. Segal and Dietz [11], who identified a significant increase in heart rates caused by playing video games.

Prior art has compared different presentation formats of VR from the point of view of the symptoms induced during its use, e.g. Sharples et al. [12] compare the use of a Head Mounted Display, projection screen and desktop display. Here, the main conclusion being the large variation between subjects. In our study we compare a similar set of display formats, but rather focusing on differences in the utility of each to provide inputs to the architectural design process.

2.2 Immersive Environments and Human Perception

CAVEs. Immersive environments have received much attention in research. One of the most common technologies used being CAVEs [13], which typically surround the user with three large projected walls, and even ceiling and floor, creating an immersive display environment that can be used either in a 2D or stereoscopic 3D mode. As application domains for CAVE type immersive environments, game, architecture and cultural heritage related topic are often considered, see e.g. [14]. CAVE environments have been employed in professional use when investigating and assessing architectural plans both in research [15] and in commercial design studios [16]. Frost et al. [17] present a building design process that incorporated the use of a VR CAVE environment. In their work, future users of the space being designed virtually visited the environment during the design phase and provided feedback to the architects.

Related to our research, earlier work related to human perception is of particular interest. Prior art has investigated what kind of effect the use of immersive environment has on the perception of walking distance [18]. Here, it was found that test participants’ distance estimates were almost identical between real world and 3D virtual world representations. When comparing a projected CAVE with a fish tank type display on a monitor, Demiralp et al. [19] found the different displays had an effect on how suitable they were for VR applications. Also, the effect of the field of view and peripheral vision on the level of immersion experienced by users has been reported in several studies [20, 21]. With desktop virtual reality, the use of Peripheral Lenses has been proposed as a way to increase the level of immersion [21]. Here, these kinds of additional screens next to the main display can be seen as a step towards a CAVE type setup.

HMDs. Head Mounted Displays (HMDs) are proposed as one method to provide the user with a deeper level of immersion in a virtual environment [22]. Since the early work of e.g. Chung in late 1980’s, we have witnessed the evolution from bulky and heavy HMD equipment towards lightweight and truly mobile systems. In the scope of architectural design, Thomas’ early work on using HMDs as an augmented reality tool allowing a building design to be viewed in its physical surroundings [23], providing an interesting base to build upon. Earlier work [24] has discussed the increased immersion provided by HMDs compared to monitors, but has however not included detailed evaluation of the differences.

Together with other development trends, such as display miniaturization and increases in computing power, we now have equipment that can be easily adopted to different kinds of use cases and customized for industrial applications. Recent research using the Oculus Rift HMD has demonstrated e.g. a simulation that enables the user to experience flying like a bird [25]. Related to the domain used in our study, i.e. construction sites, previous research has examined the role of HMDs in building renovation and construction, e.g. [26]. However, their use has been considered rather for Augmented Reality applications instead of Virtual Reality. Comparison of different interactive technologies and charting user perceptions of using them, remains so far unexplored.

3 Study Design

3.1 Industrial Site Virtual World

To evaluate the suitability of HMD and CAVE solutions for our defined use case, we implemented a prototype application that enabled a test user to move around a virtual 3D industrial site (Fig. 2) with HMD, CAVE and a normal PC monitor screen as the output device. In each case a Microsoft Xbox controller was used as the interaction method. The HMD used was an Oculus Rift, Developer Kit 1 version [27]. The CAVE environment used consisted of a 180° stereoscopic 3D projection screen that did not include floor or ceiling projected surfaces. In the CAVE environment users were required to wear polarized glasses to see the stereoscopic effect.

Fig. 2.
figure 2

3D model used in the virtual world

Full size image

Our virtual industrial site used a 3D model that was supplied by R-Taso ltd. [28], a company that specializes in creating stair and walkway solutions for large industrial complexes. The model represented a typical installation for their products and included stairs, walkways and vertical ladders, see Fig. 2. Thus, the evaluation was done with the models that are actually used for professional purposes, neglecting some of the details of high fidelity architectural models. Additionally, we added scenery and other elements to the virtual world to increase the level of realism.

Our application was implemented using the Unity 3D environment, with a first-person camera controller to enable the user to move around the virtual industrial site. An example view of the 3D virtual world provided through the HMD is illustrated in Fig. 3. Physics objects were added to the model to enable the user to climb stairs and walk on the elevated walkways. Ladder climbing was enabled by use of a script such that forward motion when in the vicinity of the ladder would create upward motion, and vice versa. Walking and running speed in the virtual world was 3.8 km/h and 7.8 km/h respectively. The highest walkway in the model was 12.4 m above ground. To further increase the level of immersion we added sounds, such as different footstep sounds when walking on different surfaces.

Fig. 3.
figure 3

View seen by the user through the Head Mounted Display

Full size image

To enable analysis of differences in users navigation routes in the 3 different presentation formats of the application, we included a logging functionality, so that the path taken around the model and the time taken to reach certain waypoints were logged. Additionally, some specific events, such as falling down or starting and finishing a mini task were logged. As prior art had highlighted the effect of virtual world conditions on heart rates, we included heart rate monitoring in our study. Participants put on a heart rate monitoring chest belt at the beginning of the study before completing the initial questionnaire, and wore it for the full duration of the study. The heart rate data was logged by an Android based application for later analysis.

3.2 User Study Set-up

Study Set-up. The three visualization modes compared in the study are shown in Figs. 4, 5 and 6. In each case the virtual world model and the navigation mechanism were identical.

Fig. 4.
figure 4

The three visualization modes compared in the study. Left to right: PC screen Oculus Rift HMD, Stereoscopic CAVE

Full size image

The study was set up in one room; in one corner of the room there was the fixed CAVE system, additional equipment consisted of a laptop to run the simulation, a large flat screen display, an Oculus Rift and an Xbox gamepad. A camcorder was used to film the test sessions. In addition to video recording, users’ heart rates were monitored both during tests and whilst completing the questionnaires between each test mode. During tests and while filling questionnaires users’ comments and reactions were observed and notes made by a test moderator.

Each user was welcomed to the test and was asked to put on the heart rate monitor. Then the user completed a background information form. The user was told about the general phases of the test. The order in which users completed the three test modes was counterbalanced to avoid learning effects.

The user was guided to each test environment in turn and instructed on the use of the gamepad to navigate the virtual environment. The user was then briefly instructed on the tasks to be completed in the virtual environment. The user had to climb either the ladders or the stairs to the top of the silo structure. On the top of the silos there were two task checkpoints that were identified with a red glow, which the user had to find and move to. The location of the checkpoints was randomized. When arriving to a checkpoint, the simulation entered ‘targeting mode’ in which the user was required to find a target object on the ground below the silo towers and mark it by pointing it with a cursor and pressing a button on the controller. After finishing the two tasks, the user had to return to the starting point, either by ladder or stairs (Figs. 2 and 7).

Fig. 5.
figure 5

User using the HMD version of the application. Note the Xbox controller.

Full size image
Fig. 6.
figure 6

User navigating the CAVE version of the application using the Xbox controller.

Full size image

When the user had finished the test for one presentation format, s/he was given a questionnaire related to the experienced realism during the test. After filling the questionnaire the user carried out the tests in the other visualization environments. After the final test the user removed the heart rate monitor and completed a final questionnaire where s/he chose the favorite presentation format and gave reasons for their choice. Users then completed a word selection task, choosing five words from a grid of 52 adjectives to represent each experience. This method was a customized version of the Microsoft Research Product Reaction cards method [29]. Finally, the user was thanked for their participation and a cinema ticket was given as a reward.

Participants. Altogether, 30 participants took part in the user study. The participants were between the ages of 20 and 51 (M = 26.4, SD = 4.8), and 8 of them were women and 22 men. As a background, 71 % of the participants had not used any kind of HMD earlier, but 56 % had used 3D glasses once and 41 % more than once. In total 38 % of the participants played console games about once a year, 25 % every month.

4 Results

4.1 Paths Taken by Test Participants

Inspired by the work on visual presentations of real world user flows presented by Williamson and Williamson [30], we aimed to present similar visualizations for a virtual environment. Figure 7 presents a top-down view of the virtual environment indicating the paths taken by users.

Fig. 7.
figure 7

Top down view of our virtual factory environment, showing the paths taken by all users when exploring the space.

Full size image

Considering the overall paths taken by participants (see Fig. 7) it appeared that with the PC participants took a straightforward, goal orientated path, compared to a more exploratory approach with the Oculus and CAVE. This is also reflected in the mean total time taken for users to complete the task with the PC (M = 3 m 45 s, SD = 76 s) being notably faster that either the CAVE (M = 5 m 8 s, SD = 123 s) or Oculus (M = 5 m 6 s, SD = 91 s). By visual inspection of the detailed navigation paths (e.g. Figure 8), the paths taken when using the CAVE were observed to be generally more spread and less consistent between users than either PC or Oculus conditions.

Fig. 8.
figure 8

Paths taken by users when climbing the stairs, at the level without a safety barrier. Top: Oculus, Middle: PC, Bottom: CAVE.

Full size image

4.2 Subjective Ratings

The users’ subjective ratings comparing each condition are presented in Fig. 9. To examine the significance of differences in the median subjective measures for each display condition, Friedman’s tests were conducted on the responses to each of the Likert questions. Post hoc analysis with Wilcoxon signed-rank tests was then conducted with a Bonferroni correction applied to remove problems of multiplicity, resulting in a significance level set at p < 0.016 (.05/3 = .016). Table 1 details the results of the statistical analysis. For all of the criteria, statistically significant differences were found between the Oculus and both other conditions. Hence the reported perception of space and the feelings of height, motion, control and realism the feeling of space provided by the Oculus were better than either the CAVE or the PC screen case. No significant differences were found between the CAVE and PC conditions.

Fig. 9.
figure 9

Subjective rating for each visualisation condition.

Full size image
Table 1. Analysis of significance of differences in median subjective ratings for each display condition. For the Wilcoxon signed rank tests a Bonferonni correction results in a significance level criteria of p < .016.
Full size table

Related to the feeling of space, half (15/30) of the users commented on the wide field of view provided by the CAVE. However, there seemed to be some disagreement as to whether it was a positive or negative issue, user #7 summing up the general responses “The field of view was too big. But on the other hand it was better than with a small screen, however orientation was hard to achieve” (User #7). In the case of the Oculus, 8/30 users mentioned the effect of moving their head as contributing to the feeling of space. Almost half of the users (13/30) felt that the feeling of height when climbing the stairs or ladders was realistic, with some comments about feeling dizzy, or keeping away from the edge, e.g. “Especially when I was at the top of the ladders and glanced down, I felt like being somewhere high. It felt horrible and frightening” (User #11) and “…I was more aware of edges in the staircase.” (User #12).

Issues related to the poor image quality, lag or low frame rate were mentioned by 8/30 users in relation to the feeling of motion when using the Oculus, e.g. “It was somewhat choppy when you turned your head.” (User #3). One user (User #21) suggested that adding head swaying when walking or running would have improved the feeling of motion.

Users commented on difficulties controlling the movement in the CAVE visualization, with 6/30 users giving comments in this direction e.g. “Not so easy to control as the Oculus and normal screen.” (User #6). In the Oculus case, the control of the camera rotation by both rotating the head and by using the left stick on the joystick was commented negatively by 6/30 users, however 2/30 users gave positive comments related to this functionality, e.g. “Mildly confusing because the view turned with both controller and head.” (User #13).

Considering the overall realism of exploring the virtual world, 15/30 users commented on the low resolution of the Oculus image as on factor limiting the reality of the experience. When asked for their favorite of the three visualization conditions 27 participants selected the Oculus, 2 the Cave and 1 the PC. The reasons given for selecting the Oculus mostly related to the experience being fun or immersive in nature e.g. “The Oculus offered the best immersion, and naturally controlling your view was effortless.” (User #20).

4.3 Product Reaction Cards

The adjectives chosen most frequently by test participants to reflect their experiences with each visualization condition are presented in Table 2. Generally the adjectives selected to describe the Cave were somewhat negative about both its visualization and ease of interaction with it. This corresponded with the descriptive comments given by the participants that described some discomfort with the wide field of view and relatively inaccurate control of the CAVE.

Table 2. Adjectives most frequently selected by users to describe the experience of each visualization mode. Bracketed numbers indicate the number of users selecting each adjective.
Full size table

The majority of the words selected to describe the Oculus were hedonistic in nature, with entertaining, empowering, inspiring and fun being the most chosen adjectives. The PC interface was considered somewhat dull and old fashioned. However, in its favour, the users considered the PC interface controllable, clear and easy to use.

4.4 Heart Rate Analysis

The data logged by the Android application was transferred to Microsoft Excel where analysis was made. When reviewing the logged heart rate data there were many cases where the data was considered invalid. This was for example where the chest belt sensor had poor contact with the participant’s skin during the test. Thus, test data that included heart rates above 180 bpm or heart rate jumps of more than 20 bpm were deemed erroneous. To enable within subjects analysis, participants whose test data included erroneous data for one or more test conditions were excluded from further analysis. Hence, the data from 17 participants was included in the heart rate based analysis (Fig. 10).

Fig. 10.
figure 10

Mean heart rates for each condition. Values for participants with invalid readings have been removed from the analysis.

Full size image

To identify the significance of differences between the participant’s mean heart rates in each condition A one-way within subjects ANOVA was conducted. There was a significant effect of the output modality, Wilks’ Lambda = 0.606, F (2,15) = 4.88, p = .023. Three paired samples t-tests were used to make post hoc comparisons between the conditions with a target criteria of p < .05. However, due to the multiple comparisons a Bonferroni adjusted significance criteria of p < .016 was applied.

A first paired samples t-test indicated that there was no significant difference in the values for PC (M = 85.5, SD = 12.5) and Cave (M = 86.4, SD = 12.8) conditions; t(16) = −0.715, p = .485. A second paired samples t-test indicated that there was a significant difference in the scores for PC (M = 85.5, SD = 12.5) and Oculus (M = 91.7, SD = 12.7) conditions; t (16) = −3.81, p = .006. A third paired samples t-test indicated that there was no significant difference in the scores for Cave (M = 86.4, SD = 12.8) and Oculus (M = 91.7, SD = 12.7) conditions; t(16) = −2.326, p = .033. It may be noted that the latter comparison of Cave against Oculus conditions would have been considered significant if we had not applied the conservative Bonferroni correction. Hence, we consider that there is a borderline significant difference between the heart rates whilst using the Cave and Oculus.

When reviewing the heart rate of individual participants there is some evidence that changes in heart rate correspond to experiences in the virtual environment (Fig. 11). Although taken in isolation no conclusions can be drawn from variations in heart rate, as it is affected by many other factors beyond the experience of the virtual word, it may prove valuable when combined with other data sources. For example, peaks in heart rate at a certain point in the virtual environment could provide the trigger for qualitative interview questions related to that point.

Fig. 11.
figure 11

Example heart rate of participants whilst exploring the virtual world (Top: user #10, Oculus. Bottom: User #1, Oculus).

Full size image

5 Discussion

5.1 Experiental Aspects

Overall the Oculus was the clearly preferred visualization/interaction approach. The playful aspects of the Oculus based interaction identified by the users in both the product reaction cards and free form comments, highlights that playfulness in a key factor in user engagement.

In general our study had strong internal validity, the data obtained from Likert scale questions, and free form comments pointed to similar conclusions. Additionally, the heart rate data supported the findings, with the Oculus based exploration resulting in significantly higher mean heart rates than other methods, suggesting a higher level of immersion.

5.2 Control and Immersion

Although the three visualization approaches we compared were based on exactly the same model and used the identical control method of the Xbox controller, the test participants judged the feeling of control to be different. Specifically, the level of control with the Oculus case was considered to give a significantly stronger feeling of control than either the PC or cave cases. The Oculus’ interaction based on the natural head movement was clearly appreciated, and our duplication of camera direction control with the handheld controller joystick was not positively received.

There was also indication that the PC based solution was perceived as being more controllable than the CAVE solution. Although the Likert responses did not alone indicate a significant difference, the direction was suggested and was supported by the large variance in the navigation paths taken by different users (Fig. 8) and the many free form comments related to the lack of controllability of the CAVE. This is perhaps not surprising, as although all users stood at exactly the same position when using the CAVE, we speculate that their perception of their own position in the 3D space was rather personal, depending on a variety of factors. This suggests that the CAVE type environments may not be suitable for tasks investigating precise navigation in a virtual world with a first-person camera viewpoint, as the user’s exact location in the scene is not apparent. Rather the CAVE environment is more suited to presenting a relatively static view of a virtual world, and has other benefits compared to the head mounted display, for example being viewable by a number of people simultaneously without the need for additional hardware.

One interesting aspect relates to differences in perception caused by the visibility of the user’s own body. While wearing the HMD, users cannot see their own hands and body, whilst in the CAVE presentation these are visible which may act to improve the level of reality, for example by providing a size benchmark.

5.3 Use of Heart Rate

We acknowledge that our use of heart rate is not rigorous in the medical sense, and the large number of results we had to discard due to operational issues was not ideal. However, we believe we have demonstrated that a simple heart rate monitoring set-up can provide useful information on the design of spaces that are being evaluated in a virtual reality environment. In this respect we have shown that, as well as for users with acrophobia as reported by [8], heart rate can be used as an indication of emotional state also for normal users in a virtual world.

5.4 Interaction Technologies as Tools

Our target was the creation of tools and methods to enable industrial site designs to be evaluated, potential problem issues identified and designs corrected before any site construction commences. Of particular interest are safety and psychological issues, for example the perception of height and feeling of personal security when working at height. Based on our user study results we have identified that in particular feelings of discomfort due to height can be well identified by this approach, particularly when experienced via a HMD such as the Oculus used in our study.

5.5 Limitations of the Study

By making a comparison between the three different presentation formats we hope to identify particular pros and cons of each solution. Here, we acknowledge the novelty effect may have influenced the users’ perceptions. However, as few participants had experience of either both the Oculus or CAVE based solutions we feel our comparison of the two is valid. As noted in the users’ free form comments each solution had some level of technical limitations e.g. the limited resolution of the Oculus and the field of view of the CAVE, however we feel that these limitations represent the state of the mainstream of the currently available technology in each case.

As we only used one virtual world model in our study that places a potential limitation on the applicability of our case. In addition, it should be noted that we explored a one-time use, and user’s perceptions may change in the long term when the novelty effect wears off. However, in many situations, such as when presenting the construction site to costumers, management or marketing department, the interaction and the exploration within the 3D virtual world models is of a short duration. Thus, we believe that despite of the limitations, our study offers interesting insights on using different interaction technologies.

6 Conclusion

In our user research (n = 30) we have compared three different access methods for exploring a 3D model of an industrial site design, CAVE, Oculus HMD, and PC monitor. The main target of our work was to identify approaches that could provide user feedback to the architects and designers of the site installation. Our results show that the Oculus HMD was the clearly preferred method, especially for the experiental aspects it provided, but also because of the feeling of control in usage.

We have concluded that CAVE type environments are not ideal for first-person presentation of a virtual environment in cases where precise navigation is needed, e.g. exploring a factory building. We have demonstrated that the use of a commercial heart rate monitor can be used to give useful information on user perceptions when exploring a virtual environment.