The Comparison of Performance, Efficiency, and Task Solution Strategies in Real, Virtual and Dual Reality Environments

Abstract

Using virtual models of a real environment to improve performance and design effective and efficient user interfaces has always been a matter of choice to provide control of complex environments. The concept of Dual Reality has gone one step further in synchronizing a real environment with its virtualization. So far, little is known about the design of effective Dual Reality interfaces. With this paper we want to shed light on this topic by comparing the strategies, performance and efficiency in a real, virtualized and a DR setting given a complex task. We propose a cost and efficiency measure for complex tasks, and have conducted an experiment based on a complex shelf planning task. Our results show that for certain tasks interacting with the virtual world yields better results, whereas the best effectivity can be observed in a Dual Reality setup. We discuss these results and present design guidelines for future Dual Reality interfaces.

1 Introduction

More and more tasks that were previously successfully conducted in a real-world environment are now being virtualized. For example, car design, which was previously done using miniature models, is now done on a PC using a CAD tool. Cars are designed and tested virtually, and architectural design and structural engineering calculation is done completely on virtual models, replacing their real counterparts. Both virtualization and a real environment have their advantages: On the one hand, virtual tools are fast and easy to use, as the physical demand is reduced. The virtual interface can be enriched by additional visual clues, using coloring of parts of the scene, lighting or textual information. For example, in [16], the work piece is visualized so it cannot be occluded, and additional textural information in the form of process data is displayed. On the other hand, a real-world environment with real objects has better haptic feedback. Size, height and distance estimation is easier in the real world, as we will discuss in the next section. The concept of Mirror Worlds, first mentioned by David Gelernter [9] and later by Lifton and Paradiso under the term Dual Reality [12], describes a setting where both worlds, a real and a virtual world, are connected together and influence each other. Lifton and Paradiso define it as follows:

Dual Reality is an environment resulting from the interplay between the real world and the virtual world, as mediated by networks of sensors and actuators. While both worlds are complete unto themselves, they are also enriched by their ability to mutually reflect, influence, and merge into one another [12].

In Dual Reality, an event in one world can, but does not have to, cause a corresponding action in the other world. The users can act either with the real environment or with its virtual counterpart. Every action can be reflected in the corresponding counterpart. Although not directly stated in their paper, a mechanical entity is implicitly needed, which synchronizes both worlds. Consider the model of a virtual apartment in Dual Reality: The user can turn on a lamp in the virtual environment. In the real counterpart, the light is also turned on remotely by the software. In turn, if the user turns on the light in the real world, this is recognized by the software, which then turns on the light in the visualization. Our goal is to understand how people interact with a Dual Reality setup, whether they take advantage of the possibilities of this concept, which possible problems arise and how this interaction can be improved. We are interested in questions such as “Are the users acting the same way in a virtual replica and in the physical environment?”, “How efficient is a Dual Reality setup compared to the real and virtual world?”, and “Which interface type should I use to meet my requirements?”

In order to investigate these questions, we have created an experimental setting, consisting of a real and a virtual environment, representing the same setup and influencing each other. Subjects were asked to perform a complex task requiring both strategic planning and physical actions. We propose a cost function, which allows us to judge and compare the efficiency of each task. As an initial step we were interested to learn more about the differences in terms of performance and behavior between real-world, virtual-world, and Dual Reality interaction. Dan Montello argues in his work [14] that the definition of space is a perceptual problem, and gives the definitions of four different sizes of space. The space that can be seen by the user without locomotion is called the vista space. A bigger space that can only be apprehended with a significant amount of locomotion is called the environmental space. Most of the related work that compares the task solution strategies between a virtual and real environment does not take Dual Reality settings into account, and uses only a small space (such as the size of a table), meaning it is clearly limited to a vista space. The Dual Reality systems that are known from the literature, but which are not evaluated in terms of performance and efficiency, are situated in the environmental space. Our experimental setting, which evaluates a real, virtual, and a Dual Reality interface, resides somewhere between the vista and environmental space: The scene can be viewed as a whole from a single point, but the detailed information and actions needed for the task can only be performed with significant locomotion. The remainder of this paper is organized as follows: The next section will first discuss related work and provide some background information on the topic of Dual Reality. Then we will present the experimental setup and how we designed the task to be performed in a real, a virtual, and a Dual Reality world. We then discuss the experiment, including a pilot study, and its results. Finally, a discussion and conclusion complete the paper.

2 Related Work

For our work, three different aspects are of importance: first, the differences in perception of a virtual world or object compared to a real environment; second, the research field of Dual Reality; and third, comparisons between interaction in a real and in a virtual environment. Research in the field of visual perception has shown that users perceive a virtual world significantly differently than the real world. A first problem is the perception of rotation, orientation and shape of a three-dimensional object. Dobbins and Grossmann [7] showed that orientation can be perceived differently, depending on the position of the object in relation to the beholder. Mistakes in the perception of the metric structure of 3D objects from multiple cues were previously discovered by Todd and Norman [17]. We expect that differences in visual perception will influence performance and behavior in real and virtual worlds.

There exist several examples of such a setup of complex worlds in the Dual Reality paradigm: Back et al. [4] present a Dual Reality chocolate factory, consisting of a real factory and a virtual model. The applications of this project are, first, to allow a virtual trip through the factory, as well as the remote control of the machines inside the factory for authorized persons. In addition, the states of the machines in the real world, as well as interactions with them, are reflected inside the visualization. Davies and Callaghan [6] examined how human behavior can be captured and learned by sensors. Their goal is to create an autonomous virtual avatar, whose behavior seems natural. They instrumented an apartment with motion sensors, and created a corresponding virtual world in 3D. The behavior of the user in the real world is perceived and also visualized in the virtual component. Conversely, interactions with the virtual world (e.g. turning on the light), will also affect the illumination of the real apartment. Khan et al. [11] created a virtual supermarket, displayed on a CAVE (Cave Automated Virtual Environment). The interaction is realized by a “human joystick” principle: The camera is moved with respect to the user’s position in relation to the center of the CAVE. Their objective was to evaluate the user’s experience of pervasive applications within a virtual environment, and to show the potential of evaluating location-based services, especially location-based advertisements in a virtual supermarket. They claim that a virtual world has the advantage of being fully controllable and adaptable to the researchers’ needs, but they did not investigate the differences between their simulated and a real environment. In contrast to the chocolate factory and the virtual avatar mentioned before in this subsection, this visualization has no real counterpart, so it does not form a Dual Reality system. Still, it is the virtualization of an environment as it can exist in the real world. Therefore, the interaction and visualization techniques are related to our experiment.

The design of the two Dual Reality worlds (or the virtualized world), and especially the interaction possibilities, are of interest for our work. We designed our real and virtual environment similarly to these systems in terms of visualization as well as interaction possibilities and techniques, to discover the differences in interaction and behavior between the two environments, as well as a DR setup containing both environments at the same time. In the following we will discuss several studies which compare the performance and behavior of a digitalized 2D representation (e.g. controlled via mouse or touch) with a physical or tangible version for the same task and domain. Kozak et al. [1] observed in the past that training in the virtual world does not necessarily lead to an improved performance in the real world. In their study, subjects had to place a set of cans on a table according to given positions. The first test group was able to train in advance using the real cans and table. A second group used a virtual reproduction of the setup, using a data glove and a head-mounted display. A third group had no training at all. Only the first test group was able to perform significantly better; the group using the virtual version did not perform better than the group without training. A simple task where a ruler or several simple geometric objects have to be aligned to fit a template is presented in [18]. The results show that subjects needed significantly more time on the touch-based system. In contrast to our experiment, the tasks here are tasks which are easy to perform, like aligning a ruler to a given shape. Lucchi et al. [13] compared actions like selection, scaling, rotation and positioning using a touch-based version or tangible objects. Here as well, the tangible version outperformed the touch version in terms of time, precision and number of translations needed to reach the desired goal. Still, a comparison of complex tasks is absent from this paper; only easy rotation and scaling tasks are performed there. More complex tasks are presented in [2, 3]. The participants were asked to solve a puzzle by using either an interactive surface (based on touch input), or real puzzle pieces lying on top of the surface. The results show that the virtual version was outperformed by its tangible counterpart. The behavior and the percentage of time devoted to each sub-task were significantly different. While this is similar to our setup, the task itself can be considered rather simplistic in terms of strategy and behavior, if for example compared to a more complex control task in a factory or when finding a solution to a spatial configuration task.

In summary, several studies have been carried out comparing the performance between a virtual and a real condition for simple tasks, involving simple actions. We will extend this work for a more complex task, which requires a strategy, as well as more complex actions including locomotion. We will introduce a new test condition, namely the Dual Reality condition, which is also compared against the virtual and real conditions. An efficiency measure for all three conditions will be proposed. We will observe and discuss differences in terms of performance, efficiency, as well as the number and duration of actions conducted in order to complete the task, in order to give guidelines on the optimal interface for a given environment and task.

3 Experimental Setup

We decided to take a pick-and-place task, which had to be extended to form a more complex task, requiring strategic planning as well as a higher amount of locomotion as stated in the introduction. Instead of giving the user a specific target location, we gave him a complex formula which scores the placement of the object, depending on its position and which other objects are situated in its surroundings. To form a meaningful task, we decided to replicate a realistic task from the retail domain, namely that of “shelf planning”: Retailers have to plan their shelf layouts (i.e. the order and position of product placements in a shelf) to optimize their profit. We have designed a real and a virtual environment where real and virtual products could be placed at arbitrary positions on the respective shelves in a shelf unit. In the Dual Reality condition of our experiment, both environments can influence each other, and are always “synchronized”. Each product placement or movement which is done in the virtual environment will also be applied automatically to the real shelf, and vice versa. Normally, in a Dual Reality system, the “synchronization” of the real world is done by machines, such as a robot. Details on how this robot was emulated for our experiment will be given later. For our efficiency calculation, we will assume that this task is done by a robot or an automated process.

Each product is assigned to a price category. Depending on the price and the placement within the shelf unit, the overall profit is calculated. Profit is influenced by the placement (some positions in the shelf unit are more profitable than others) and on which other products have been placed nearby (products of the same price category reduce each other’s profit). Details on the profit calculation will be given below. The main task consists of maximizing the profit of the shelf unit as a whole.

3.1 Efficiency Calculation

To be able to compare the efficiency of the tasks, an efficiency function is essential. In this subsection, we give a general formula which has to be refined for a specific task. We will create a specific formula for our shelf planning task in the next section. We define efficiency as the fraction of the performance P (in terms of score reached for the task) and the cost C needed to achieve this result:

$$ Efficiency = \frac{P}{C} $$

The shelf planning task can be divided into several types of sub-tasks. We will denote this type-set of sub-tasks as ST, their elements as ST _i and the number of times this sub-task type was executed during the experiment task as |ST _i |. The cost depends on the number of times each sub-task ST_i in ST is conducted, as well as the cost for each of them (C(ST_i)):

$$ C = \sum\nolimits_{{ST_{i} \in ST}} {(\left| {ST_{i} } \right|*C(ST_{i} ))} $$

Each subtask is conducted by an entity, which can either be a real person, or a machine such as a robot. Each entity has a different cost for operating over a certain period of time. For a human this would be his salary; for a machine, the operating cost. The cost of a given subtask is therefore calculated by multiplying the cost per hour C _hour (E) of the entity E that conducts the sub-task

by the time needed to complete it (Time(ST_i)):

$$ C(ST_{i} ) = C_{hour} (E)*Time(ST_{i} ) $$

The efficiency formula then is calculated as follows:

$$ Efficiency = \frac{P}{{\sum\nolimits_{{ST_{i} \in ST}} {(\left| {ST_{i} } \right|*(C_{hour} (E)*Time(ST_{i} )))} }} $$

This formula needs to be refined in the next step according to the given experiment task that should be measured. There are three domain-specific variables: Time on average needed to complete each individual Subtask ST (Time(ST_i)), Cost per hour of the entities that fulfill these subtasks (C_hour(E)) and Measure of the performance P. It is not our primary goal to show which of the interfaces is most efficient, but to see how changes in different parameters, such as cost per hour/salary of a worker or time needed for a subtask, can influence the efficiency and make another interface concept become the most efficient one. This should give the reader an idea of whether he should prefer a real, virtual or DR interface for his specific task. Calculating the measures is rather straightforward: for example for a pick-and-place task, the performance P could be the number of placed objects divided by the sum of offsets of each object compared to its target position. There is only one sub-task, namely the pick-and-place subtask, whose average time can be computed in a short trial. The cost per hour is the hourly wage of the human performing the task. In the experiment section, we will determine these variables according to our shelf planning task. The next section will give an overview of the schematic setup of the real environment, followed by the design of the virtual environment, and the Dual Reality system which is a linkage of these two environments.

3.2 Schematic Setup

The schematic setup is shown in Fig. 2 and is the same for all three conditions. The worlds are designed in exactly the same way, illustrated in Fig. 1. For the virtual version, we used a 3D scene displayed on a 2D touchscreen for representing a virtual version of the real world.

Fig. 1.

Environment in the real, virtual and Dual Reality conditions

Fig. 2.

Schematic setup of the experiment (all measures in cm)

The products are initially placed in the right shelf unit (“start shelf”), and have to be placed inside the left one (“target shelf”). The touch display running the virtual environment is placed on a table in between the two shelf units. The user stands in front of these two shelf units (“interaction zone”), so he has access to both the real shelves as well as the virtual environment on the display. Behind the shelves is an assistant zone, where the experimenter as well as an assistant can observe the experiment.

3.3 Interaction Design

Real Environment.

The subject uses real products and shelves to accomplish the task. We implemented several assistance systems, which should help the subject in achieving a good score, without reducing the cognitive effort too much. We displayed the actual score as well as the change with respect to the last step on the screen between the shelves, giving the subject an idea of whether his actions were going in the right direction. Additionally, we installed LED lights in different colors on each shelf of the left shelf unit (target shelf). As soon as the subject touches a product or holds it in his hands, the LEDs are turned on, indicating if there is not enough space to place the product (red light); there is enough space, but the score would decrease (yellow light); or there is enough space and the score would increase (green light) after placing the product (see Fig. 3).

Fig. 3.

Lights indicate available space and score

Virtual Environment.

We designed the virtual world to be as similar as possible to the real world in order to ensure comparability. Therefore we decided against a 2D representation and used a 3D view instead. A screenshot of the virtual model is shown in Fig. 4. For additional functions such as switching views between shelves or zooming, several command buttons are displayed in the lower left corner of the interface.

Fig. 4.

Setup in the virtual world condition

Several interaction techniques are provided, based on single-touch input:

• Selection of a product: For selecting a product, a single tap on the product is sufficient. Tapping the product again, or selecting another product, deselects it again. The currently selected object is highlighted by a red overlay on the product.

• Placement of a product: A selected product can be placed by tapping on the exact position on the shelf where the product should be placed. The program automatically moves the adjacent products aside if the selected space is not sufficient, as long as there is enough space on the shelf. If no space is left on the shelf, a corresponding message appears in the upper right corner of the screen as feedback to the user. The product remains selected and can be placed at some other location.

• Moving a product: A product inside the shelf unit can be moved simply by dragging and dropping it. A move is possible only on the same shelf; moving it to another shelf is not allowed using this technique. In that case, a placement action has to be performed.

• Shelf views: When clicking on one of the two rightmost buttons, the user gets an overview perspective of the contents inside the left or right shelf unit. This facilitates a better identification and selection of products. In the new visualization, an additional button appears to get back to the shelf view.

• Zoom: We implemented a zoom functionality, allowing the user to zoom in or out in the current perspective, to have a more detailed view of a shelf unit or product desk. This corresponds to the action of walking towards the shelf unit.

Users were not forced nor encouraged to use zooming or view switching, if they did not want to. We implemented the assistance functions from the real environment (see last section) in the virtual environment as well: Whenever a product is selected, the shelves inside the target shelf are colored according to available space and possible score after the placement of the selected product. The score is displayed in the upper left corner of the screen.

Dual Reality Environment.

In this environment, both the real products and visual cues of the real environment, as well as the virtual system using the touchscreen, can be used. The subject should always have the possibility to switch between the two environments at will. Therefore, both environments always have to be synchronous: Whenever an action is conducted in one of the two worlds, it will be mimicked in the other world. If the subject places a real product on a shelf, the product is also automatically placed in the virtual environment. When the subject uses the touchscreen to place a product in the virtual environment, the real product also has to be transported to its new location.

3.4 Pilot Study

Prior to our experiment, we conducted a pilot study with a slightly different setup in order to find first differences in people’s behavior and performance in a real and in a virtual environment, and to decide which measures might be interesting to observe in our main experiment. Twenty-seven students volunteered to take part in the pilot study. We tested only a real and a virtual condition without any assistance functions. The products were initially placed on two product tables, and had to be placed inside two target shelves, which were standing opposite each other. All subjects were asked to optimize the product placement in the two shelf units in order to maximize the profit of products from three different price ranges (high, medium, low). For more detail on how the profit was calculated, see the later section on the main experiment.

The results reveal that the profit was significantly higher for the real condition (M_real = 0.943, M_virtual = 0.909, p = 0.02). We observed a higher number of interactions, especially product movements (M_real = 16.62, M_virtual = 24.35, p = 0.04), in the virtual condition. Most participants used only one hand to interact with the real world and never picked up two products at once. Two-handed interactions were conducted by only 50 % of all participants. Those subjects performed on average four two-handed interactions per session. The time needed for placing a product was significantly higher for the real environment (M_real = 4371, M_virtual = 3362, p = 0.02).

4 Main Experiment

4.1 Hypotheses

Although there are small differences in the experimental setup, we expect a similar outcome in terms of performance in the main experiment, as all conditions have the same assistance functions guiding the user to get an optimal profit. Therefore we expect the real condition to perform better than the virtual one (H 1). We did not calculate the efficiency for our pilot experiment, but observed a significantly higher number of interactions and a lower performance in the virtual condition. As the Dual Reality interface should combine the advantages of both worlds, we expect the best efficiency to be in this condition, and the second best for the real setup (H 2), as the latter already outperformed the virtual version in our pilot study. Relatedly, we expect the most interaction with the virtual interface and the least within the real condition. The number of interactions in the Dual Reality setup should lie in between (H 3). As interacting with a virtual interface should be easier in terms of physical demand, we assume more interaction and less workload in that condition (H 4). Regarding the Dual Reality condition, we assume that most people switch between the virtual and real environment either in their physical interactions (H 5) or by switching the view focus (H 6).

4.2 Apparatus

Within each shelf unit, one shelf is at eye level and one below. For calculating the “eye level”, we first retrieved the current average body height from the German Federal Bureau of Statistics (DESTATIS),¹ which is 171 cm. From this value, we subtracted 15 cm (a value derived from a pre-test) to arrive at an eye level of about 155 cm. We then designed the shelf to be at eye level, so the placed products can be seen best from that height. We arrived at a shelf height of 142 cm. The second shelf is placed below this, at 105 cm. Shelves in the starting shelf unit had a width of 82 cm, whereas the ones in the target unit were only 56 cm wide. We designed the latter to be smaller so not all the products inside the start shelf unit can be placed in the target; the subjects had to make a selection. The table between the shelves has a squared shape with a width of 90 cm, where only half of the table is visible to the subject. The monitor is placed on the table at a height of 90 cm. On the other side of the shelf is an “assistance zone” for the experimenter and an assistant; this is completely covered by cloth, so it is hidden from the experiment participant. The shelves of both shelf units are accessible from the back as in a puppet theater, so that the assistant can grab and place products undetected by the participants, simulating the “robot” which synchronizes the real world in a Dual Reality environment. The experimenter has a second screen inside the assistance zone, which shows a duplicate screen of the participant’s touch display. It allows the experimenter to “synchronize” the virtual world in a Wizard of Oz style, whenever an action is done in the real environment by the participant.

The experiment was designed in a within-subject design: Each participant performed the task in the real, virtual and Dual Reality conditions, one after another. Additionally, we had three sets of products, A, B and C. Each set contained different products, although some of the products appeared in several sets. Each participant had the possibility to become familiar with each interface in advance, so we did not expect any influence on the experiment results, neither from the order of conditions, nor the order of product sets. Nevertheless, we balanced the order in which conditions and product sets were used. The products of a set are placed in the same style and ordering for each experiment. In order to capture eye movements, subjects use a mobile eye tracker in the Dual Reality condition.

4.3 Participants

We recruited 17 participants over several university mailing lists. Their ages ranged from 21 to 44 years (average 27.5). We had 10 male and 7 female subjects. Each was paid 10 Euro for the experiment.

4.4 Task

The subjects were given a set of products and one empty shelf unit. Every product placement contributed to an overall profit, as we will describe in the next section. The main task was to place the products in the shelves so that the overall profit was maximized. In the DR condition, subjects were free to make use of only the real, only the virtual or both worlds. We set no time limit, and the subjects could change the placement of the products as often as they wanted. The test run was marked as finished once the subject reported being satisfied with the current shelf layout.

The formula to derive the profit for the placements was based on related work from economics [5, 10, 15]. To enable subjects to better estimate the profit of their shelves, we based the formula on a few simplistic assumptions. The first aspect taken into account was the effect of the inventory level on the product demand, as proposed by Drèze et al. [8] and implemented by Hwang et al. [10] and Murray et al. [15]. It reflects the fact that products at eye level receive a significantly higher demand than products located below or above. In our formula, we modeled this effect by a deduction of 10 % for products below eye level. The second aspect is the space and cross-space elasticity as described by Corstjens and Doyle [5]. This earlier theory says, on an abstract level, that if two products of the same price and product category are next to each other, or on the same level in neighboring shelves, the profit of both is reduced.

In detail, the total profit was calculated as follows: Each product has a different price, which is printed on the real product or the 3D model of the product in the virtual interface. We divided the products into three price categories: low-priced products (price < 0.50€), medium-priced products (price between 0.50€ and 1€) and high-priced products (price > 1€). The profit of the target shelf unit is the sum of the profits of its contained products. If a product is placed on a shelf below eye level, its profit is the product price reduced by 10 %. If more than one product from the same price category is placed at the same eye level, the profit is again reduced. We call such a group of products (on the same level and in the same price category) “colliding products”. For every collision, the profit of each colliding product is reduced by an additional 20 %. The reduction can be at most 100 %.

More formally, let C(p) = collisions for a given product p (e.g. the products in the same price level and placed at the same shelf height as p), Pr(p) = price of a product, as printed on the product, and H(b) = handicap of a shelf, respecting its height level. It is 0 for the shelf at eye level and 0.1 for the shelf below. Then the overall profit P is inductively defined as follows:

Profit of a product p:

$$ P_{p} = Pr(p)*(Max(0;1 - (\left| {C(p)} \right|*0.2))) $$

Profit of a single shelf b:

$$ p_{b} = (1 - H(b))*\sum\nolimits_{product\;p\;on\;b} {(P_{p} )} $$

The profit of the target shelf unit is then the sum of the profit of all of its shelves. The setup included more products than could be placed into the shelves. The subjects had to decide which products they wanted to use, and which would remain in the starting shelf. Participants were informed in advance of this effect by the experimenter. Apart from the assistant systems described earlier, we gave no further advice or strategy on how they could best complete the task.

4.5 Procedure

First, the participants were given written instructions. After possible questions were answered, a training phase started where first a short introduction to the interface was given. The introduction was done in a live demo by the experimenter. Following this, the subjects were given time to familiarize themselves with the interface. Once they stated they were comfortable with it, the main experiment phase started, where the users executed the given task in one of the three conditions. The starting condition was selected in a balanced way, as described in section Apparatus. This step was repeated with the second and third conditions. Each condition ended with a short two-page questionnaire, consisting of a NASA-TLX, and an additional page where users could explain their feelings towards efficiency, learnability and enjoyability of the respective condition, and what differences between conditions they perceived.

4.6 Efficiency Formula Refinement for the Experiment Task

The interface allows two sub-tasks: The placement task, e.g. picking up the product, transporting it to the desired location and dropping it off, and the movement task, which is moving the product inside the shelf without picking it up. In our example, let us assume that we have three different entities that can conduct this task: The expert has the highest qualification and salary. His part in the task is to design the layout for the shelf. The assistant has a lower qualification. He can place the products according to a given layout, but cannot design a shelf layout himself. The robot is the cheapest entity, but can only reproduce an interaction in the real environment if it was done in the virtual environment, as described in our definition of Dual Reality. In our three conditions, we then have the following sub-tasks that have to be conducted, in order to get a shelf layout using real products:

• Real Environment: The placement of the real products is done directly by the expert; we have only the cost of his placement and movement tasks. Our set of sub-tasks ST therefore consists solely of sub-tasks done by the expert: ST = {ST_{plc_r_exp};ST_{mv_r_exp}}

• Virtual Environment: All actions by the expert are done virtually. Therefore, an assistant needs to fill in the real products after the design is finished (sub-task ST_{plc_ass}): ST = {ST_{plc_v_exp};ST_{mv_v_exp};ST_{plc_r_ass}}

• Dual Reality Environment: Some of the expert’s actions are done using the real shelf (ST_{plc_r_exp};ST_{mv_r_exp}), some virtually (ST_{plc_v_exp};ST_{mv_v_exp}). The expert’s virtual manipulations are executed in the real world by the robot entity (ST_{plc_rob};ST_{mv_rob}):

  ST = {ST_{plc_r_exp};ST_{mv_r_exp};ST_{plc_v_exp};ST_{mv_v_exp};ST_{plc_rob};ST_{mv_rob}}

The interaction times needed for each of the subtasks are calculated based on the results of our experiment, which will be given in the next sections. We found the following average interaction times: human placement in the real environment: 4130 ms; human placement in the virtual environment: 1698 ms.

Our experimental setup did not allow us to measure the time needed for a movement, as this action is too fast to be annotated correctly by the experimenter. We estimated the times in a short trial, using a stopwatch: human movement in the real environment: 1933 ms; human movement in the virtual environment: 1633 ms.

To allow a fair comparison, we designed both conditions, the virtual and the real one, as it is typically done by (software) engineers, using a screen for the virtual version, and a physical environment for the real one. Because the real environment typically involves a higher amount of locomotion, it requires more interaction time than the virtual environment. For the Dual Reality condition, we did not have a robot for our experiment, but there are industrial robots like the KUKA KR-16 which can conduct the placement/movement tasks. We estimated the time needed for those tasks using the datasheet² as follows:

• Grabbing/releasing the product: max. 90° rotation by axis 5, 45° by axis 6 → 316 ms

• Pull back/forward to shelf: max. 45° rotation by axes 2/3 → 288 ms

• Swivel between start/target shelf: max. 90° by axis 1 → 577 ms

• Movement inside shelf: max 45° rotation by axes 1/2/3 → 288 ms

  • Time(plcmt.) = grab + pull back + swivel + pull forward + release = 1785 ms

  • Time(movemt.) = grab + movement inside shelf + release = 920 ms

4.7 Measures—Dependent Variables

We divide our measures into two categories: task and sub-task measures. A task measure is taken during the whole test-run, and scores the overall result of the complex task. A sub-task measure is taken in relation to a specific sub-task that is conducted during the overall task. For the task, we recorded for all three conditions: Profit of the target shelf unit with the user’s solution, compared to the optimum solution; Efficiency of the task for each condition, according to our efficiency measure; and Workload according to the NASA-TLX. In the Dual Reality condition, we additionally documented: Number of context switches the user performs within his physical actions and Number of visual context switches in terms of visual attention switches between the two environments.

The idea of the profit measure is to compare how successful the participants were in accomplishing the given task. The profit is a percentual value, comparing the overall profit reached by the subject against an optimum shelf arrangement with the highest possible profit. We calculated this optimum on the given products and space using a brute-force algorithm. Measuring the profit helps us to investigate hypothesis 1. For us, visual context switches are only switches where the user looked at the main part of the screen containing the 3D model. Only looking at the score is not a context switch for us. We used the results from our eye tracker to discriminate between those two cases.

For each sub-task, such as placement or movement of a product, we recorded the number of times the sub-task was conducted during the task as well as the time on average that the user needed to complete the sub-task. The analysis of the measures was done automatically for actions in the virtual environment by the software. For activities in the real environment, the subject’s actions were duplicated in this software by the experimenter on his display inside the assistance zone.

5 Results

The total time for the whole experiment varied from 15 to 30 min, as we set no time limit. On average, each participant needed about 20 min. All measures were pairwise compared using a paired t-test. The answers from the questionnaire were evaluated using chi-square.

5.1 Task Measures

In contrast to our initial hypothesis H 1, the overall profit in the real-world condition was significantly lower than for the virtual condition (M_real = 0.93, M_virtual = 0.96, SD = 0.04, t = 3,51, p < 0.005), as shown in Table 1. The Dual Reality score lay between the two, but without any significant difference from each of the other conditions (M_dr = 0.945). Although this profit was best for the virtual condition, the efficiency, according to our efficiency measure, was highest for the Dual Reality setup. The real setup was significantly less efficient (M_real = 0.67, M_dr = 1.09, SD = 0.64, t = 2.72, p < 0.05), as was the virtual condition (M_virtual = 0.63, SD = 0.59, t = 3.2, p < 0.05). We could not find any significant difference between the efficiency of the real and virtual interfaces.

Table 1. Results of the experimental task: real, virtual and dual reality (DR)

We measured a significantly higher workload for the real compared to the virtual (M_real = 4.95, M_virtual = 2.94, SD = 1.01, t = 8.2, p < 0.005) and Dual Reality (M_dr = 3.02, SD = 1.12, t = 7.12, p < 0.005) conditions. Regarding the DR condition, we observed 7 of 17 participants switching between the real and virtual environments within their physical actions, for example placing a first product on the real shelf and then another product inside the virtual environment on the screen, or vice versa. Those who did such a context switch did it twice on average during the experiment. 15 of the subjects switched their eye focus between the two environments (26 times on average). We recorded the time users spent in both conditions of the DR setup, but could not find any significance. Some users acted mainly in the real environment, whereas others used the virtual environment all the time. Differences in terms of performance or efficiency could not be shown between those two groups. We evaluated the questionnaire using a chi-square test. Only one of the questions was of high significance in favor of the real environment, all others being insignificant: Users stated that the real interface made it easier to solve the task (Χ² = 24.93, p < 0.005).

Table 2 shows the results for the sub-tasks, regarding the number of times the task was conducted during an experiment for both conditions, as well as the average time needed for the task. As the logging of the experiment was done by an annotation tool used by the experimenter, we could not record the time needed for a movement: a movement task is completed too quickly to be annotated correctly.

Table 2. Sub-tasks measure results

In the virtual condition, products were moved significantly more often than in the Dual Reality setup (M_virtual = 4.35, M_DR = 1.18, t = 2.26, p < 0.05). The DR condition again had significantly more movements than the real condition (M_real = 0.18, t = 2.2, p < 0.05). We observed a similar result for the number of placements for each condition, where the participants positioned products at a new location more frequently in the virtual condition than in the DR setup (M_virtual = 21.65, M_DR = 14.88, t = 3.12, p < 0.05) and the real condition (M_real = 13.59, t = 3.64, p < 0.005). The number of placements between the real and Dual Reality conditions remains insignificant. For the placement times, the real condition was slowest, followed by the DR system (M_real = 4130, M_DR = 2816, t = 3.29, p < 0.005), which itself was significantly slower than the virtual condition (M_virtual = 1898, t = 2.85, p < 0.05).

6 Discussion

Hypothesis H 1 is rejected, as it implies that the performance is lower for the virtual than the real condition. Contrary to this hypothesis, and although the assistance systems are the same in both conditions, the overall profit in the virtual world condition was significantly higher (p < 0.005). It seems like an “instrumented” system, e.g. a system that is enriched with assistance functions, works better in a virtual environment. There might be two possible causes of that effect: First, people are used to getting additional information in a virtual system, and are therefore more accustomed to using it. Real environments usually have no assistance functions. Second, we observed a significantly higher amount of interaction within the virtual system, and a smaller workload, as predicted by H 4 and confirmed by our results. The virtual system, which requires less body movement in order to complete tasks, seems to enable users to explore the solution space more than the real environment, which could be a cause of the increased performance. Although the workload is highest for the real condition, subjects stated that this condition made it easiest to solve the task.

Our efficiency measure could only partly confirm H 2, given for the example cost setting of our main experiment: We assigned the expert a reference cost of C_hour(expert) = 100€. For the other entities we set cheaper costs of C_hour(assistant) = 50€ and C_hour(robot) = 1€. Within this example setting, the real condition was not as efficient as the virtual one. Dual Reality outperformed them both. If we change these entity costs, the results can also change significantly: If we reduce the expert cost by a half, so it is equal to the assistant cost, the real interface becomes significantly more efficient than the virtual condition (M_real = 1.33, M_virtual = 0.93, SD = 0.47, t = 3,56, p < 0.005). Dual Reality again outperforms them both. Every interface has its own drawbacks: In the real environment, the user needs extra time for the task, as he has to carry and handle the products, whereas the design on a desktop screen requires an additional person to fill in the shelf afterwards. The Dual Reality setup allows a virtual or a real-world design, but always has to synchronize with the real world using a machine entity. Our formula makes it possible to find an equilibrium, based on entity cost, at which the efficiency of two interfaces is equal. A designer can then investigate which changes in the cost of a specific entity give one interface an advantage over the other. To do so, two simple steps are required: First we need to equate the efficiency formulas of both conditions. In the second step, we resolve the equation to the specific entity cost that we want to modify to achieve our equilibrium. This gives us the entity cost at which both conditions have an equal efficiency. As an example, if we want to find the equilibrium for the virtual and DR conditions, using the cost of the robot entity as a modifier, we set Efficiency_V = Efficiency_DR and resolve the equation by C_hour(robot). We then get C_hour(robot) = 111.114€ as an equilibrium value. In this example, reducing the robot cost gives the DR condition an advantage, whereas increasing the cost raises the efficiency of the virtual condition.

H 3 has been confirmed: the number of interactions was least for the real and greatest for the virtual condition. This effect can also be based on the workload of the different conditions, which is highest for the real condition, lower for the Dual Reality condition and lowest for the virtual condition. Most of the subjects switched between the virtual and real environment in our Dual Reality condition: About half of the subjects (7 of 17) did such a context switch within their physical actions (“hard” context switch), while nearly every subject (15 of 17) changed at least their eye focus between the two environments. This leads to the assumption that the concept of Dual Reality and the design of our DR condition works, as people took advantage of the possibilities such a system offers. H 5 and H 6, claiming that people switch between the environments within their actions as well as within their visual attention, have also been confirmed. Interestingly, the participants always used the same problem-solving strategy in all three conditions: First they placed high-priced products, then medium- and low-priced products, until the score did not increase anymore, although this mostly does not yield optimal results.

7 Design Principles

Based on the results of the user study and the cost/efficiency function that we propose, we give guidelines on which interface type fits best for given preconditions. The guidelines presented here mainly apply to our experimental setting. Whether or not the rules can hold for other complex Dual Reality setups has to be confirmed in future research. We designed our generic efficiency formula so that it can be translated to an arbitrary composed task.

Regarding the performance of the task, as we observed in the pilot study, a real interface works best when the user has to interact without additional information or support functionality. The average user is most accustomed to the real world and can therefore achieve the best results under those circumstances. That changes if an additional instrumentation is done, like highlighting parts of the scene, or adding a capability to view the current score. Virtual environments will outperform their real counterparts in that case, although the number of interactions is likely to be higher within that environment.

If the focus is on the efficiency of the interface, e.g. the cost that is incurred in relation to the result, it depends on the actual cost for each entity. With the settings from our experiment, DR mostly outperformed the other two conditions. But as we have shown in the discussion, a general answer is not possible. The efficiency formula has to be modeled first according to the task, in order to compute the most efficient solution. Using our efficiency formula, it is possible to calculate the point from which one or the other interface type should be preferred, in relation to the cost of the entities involved.

Apart from these analyses, another important aspect is the setup effort of a Dual Reality environment. Whereas the effort is rather low for a real or virtual setup, a Dual Reality setup involves a significantly higher effort, especially for the realization of the synchronization of the real and virtual environment by machines. Our experiment has provided evidence that most people actually use the advantages of the DR environment and can achieve better results with it. Therefore, a Dual Reality setup should always be considered, in addition to a conventional real or virtual setting.

8 Conclusion

More and more tasks are being virtualized, such that users can perform them in the virtual world instead of the real world. Several studies have researched differences between real and virtual environments using rather simple tasks. Our experiment investigated a setup with a complex task, involving complex actions with a significant amount of locomotion, in real, virtual, and Dual Reality settings. We proposed an efficiency function, and compared the performance, efficiency, and number and timing of sub-tasks within the three conditions. In contrast to the hypothesis and our pilot study, the performance in maximizing the profit was significantly better in the virtual version, although the efficiency is best for the Dual Reality setting. We explored the impact of entity cost on the efficiency, and identified guidelines which guide the developer in selecting the right interface for his purpose, depending on the task that has to be fulfilled as well as the subtasks and entities involved.