Explorations in AR: Finding Its Value

Abstract

This paper presents the main findings regarding the overall effectiveness of an AR-based wearable device for different types of users performing basic, common tasks throughout a few demo scenarios, thus identifying opportunities for improvement and further investment. A series of user tests were carried out among participants using this head-mounted prototype while different aspects of their interaction and overall experience were evaluated. It was given special attention to desirability factors, as well as pain points and obstacles to completing the different scenarios.

This paper presents main insights regarding several features for an ideal head-mounted AR experience to the user, as well as their view regarding the future of digital media consumption. It also discusses some limitations and lessons learned for future work. Our results allow us to shed light onto the scope of expectations of potential end users, in order to help in the design and engineering of these devices. We argue that the value of exploratory studies of such as these lies in its potential to delve into innovative paths and their potential implications, otherwise difficult to trace.

1 Introduction

Augmented Reality (AR), the technology that enables computer-generated digital information onto the user’s view and interaction with the real world [13, 9], has gained extensive attention in recent years, both from the industry and the field of research. Although the adoption of AR among consumers is not yet widespread, expectations regarding its potential in terms of their transformative benefits and future market revenue are currently high [5].

Despite this promissory scenario, there is still a large range of design options that needs to be considered for these solutions to achieve sufficient maturity within a consumer market. However, in general, the research capacity within the industry is relatively small. Not only are there high levels of competitiveness and regulatory issues to surmount, but also many times the resources to carry out sounded exploratory research, both in terms of competences and time, are usually missing. And yet these kinds of studies are particularly important when there is uncertainty about the future feasibility and user acceptance of a certain product.

Certainly, in the last years there has been a significant increase in the number of AR papers including some sort of user-centered research. The work so far has mainly focused on user interface design challenges and technology issues, whereas much less has been published regarding users’ characteristics, attitude and expectations about AR solutions thus, the whole potential of user-centered theory has not been fully embraced [10].

In this regard, our research was conceived as an exploratory space where feasible future paths for AR solutions could be traced and validated. To do so, we carried out user tests using interactive proof-of-concept (POC) software on a head-mounted device to learn about users’ expectations in terms of usability and utility, as well as to set out grounded steps for future explorations.

2 Theoretical Background

This section presents previous relevant studies in this field and discusses existing user-centered research on AR.

2.1 Augmented Reality (AR) and Head-Mounted Devices (HMD)

The earliest examples of AR technologies date back to 1960, when head-mounted devices were first used in military aviation, to improve pilot’s ability to react to the environment in enemy fighter aircraft (REF Radical Technologies). However, it only found a niche for consumers with outspread adoption of smartphones and tablets, which bundled the kind of technology that would enable a new generation of AR-embedded applications [12]. Phone-based AR treated the handset as the display surface on top of which information and AR objects were laid.

Soon after, a new generation of AR wearable mediators emerged, with Google Glass being the first of this class to reach final consumers. Currently, the two most well-known AR HMDs are Microsoft HoloLens and Magic Leap One. Both of them integrate digital and physical environments through the use of a head-mounted device that incorporates orientation and position sensors to display digital objects into the material environment whilst allowing interaction with them. Although both of them are intended for the consumer market in the long term, their reach has been exclusively within the industry realm so far, and mainstream adoption of AR headsets is still distant.

There is a considerable number of publications related to prototypes that have been developed for research purposes in different application areas, such as industrial AR applications [1], entertainment areas [2] or commercial ends [4]. More recently, many of these proofs-of-concept have turned into full-fledged applications available for the final consumer. However, despite having an end-user perspective in mind, the amount of AR studies geared towards understanding users’ needs and expectations have not increased at the same pace, which indicates a gap to be bridged in future research.

2.2 User-Centered AR Studies

The first comprehensive review of AR user studies was published in 2005 by Swan and Gabbard [6] and, among a total of 1,104 AR papers, they found only 21 related to user-centered research. All of them revolved around user task performance and usability aspects regarding perceptual and ergonomic issues. After this one, two other surveys were carried out in 2008 (Dünser et al.) and 2012 (Bai and Blackwell) respectively, both following the classification of Swan and Gabbard [11], with a few new categories being added.

The latest review of user centered research in AR was carried out by Dey et al. [3] covering AR papers published between 2005 and 2014 which contain a user study. Based on the type of display examined, their field of application and the methodology followed, the authors classified the literature to provide an overview of the state-of-the-art of the field. Different from previous surveys, they consider a wider set of sources, sifted the papers by their citational impact and broaden the classification categories to include issues experienced by the users [3].

Interestingly, they noted that despite the significant growth of AR papers published between 2005 and 2014, the relative percentage of those specifically regarding user studies had remained equally low throughout the same period [3]. The authors also suggest opportunities to explore for future user studies, such as diversifying evaluation methods and including a broader range of participant populations [3].

Another example of user-centered research involving is Olsson and Salo’s work [10] which evaluates user experience with commercial applications. They measured the degree of AR technology acceptance among consumers and established a series of guidelines for developing successful applications, being curiosity and novelty among the main motivators for using mobile AR applications.

Moreover recently, Kim et al. published an extensive review of all the papers presented at ISMAR, an academic conference which is a reference in the field, from 2008 until 2017 in order to provide an overview of the AR-related research taking place worldwide and uncover emerging trends in the field [8]. One important finding of their work is that evaluation research had substantially increased as compared to a previous survey done by Zhou et al. [14] – which Kim et al. work follows on – that summarized AR work presented at the same conference in the previous ten years. According to Kim et al. [8] the increase of evaluation research in the field of AR is a signal that AR technology has achieved a maturity that brings it closer to consumers, and so there is a need to evaluate different solutions with end-users.

Although this past work provides a consistent theoretical background for AR user studies there is still room for more exploratory research in order to establish priorities, develop operational definitions and provide significant insight into identifying feasible directions for AR solutions from a user centered perspective. In this regard, this research was designed to serve as an independent project aimed at acquiring knowledge about AR user experience through validated explorations.

3 Methodology

In the present study, we developed a series of interactive proof-of-concepts on an AR head-mounted device and tested them with users to set design guidelines based on data collected during the tests. An AR wearable prototype based on Android was built for this study and five demo scenarios were designed in which participants had to perform basic, common tasks during the test. The aim was to assess the overall effectiveness, efficiency and satisfaction of the AR demo scenarios and identify obstacles to completing them. An exaggerated hand gesture was used as primary input method and several UI components were explored. We validated our hypotheses through different methods of data collection: user tests, the verbal protocol – the running commentary that participants make as they think aloud during the tests –, and short interviews with each participant at the beginning and at the end of each test, to learn about their subjective experience when using the device through the different scenarios.

3.1 Participants Sample

Recruitment for this study was conducted in an indoor environment located at the SIDIA R&D Center in Manaus, Brazil. The only requirements for participation were intermediate-level English proficiency – the software was built in this language – and smartphone ownership. A broad screener was important to have a more representative cross section of the participants sample. A total of 14 people took part in the tests, distributed as seen on Fig. 1. Among each of these three characteristics we sought an equal distribution for gender and a spectrum as wide as possible for their age distributions and area of expertise.

Fig. 1.

Distribution of participants’ gender, age, and professional background.

Among all participants, 1 had previously owned a Google Glass, 2 owned a Samsung Gear VR device, and 3 had relevant experience with mobile AR – augmented reality experiences embedded within smartphone applications, such as Pokémon Go and Snapchat lenses. Every participant had some knowledge about Virtual Reality (VR) and 5 had no prior knowledge about Augmented Reality (Fig. 2).

Fig. 2.

Distribution of participants’ previous experience with AR and VR technology.

3.2 Support Material

We used an Android-based prototype wearable device built out of a Samsung Gear VR body, with a centered world-facing camera and two round openings following the eyelines with see-through acrylic where stereoscopic images from two tiny projectors were printed after bouncing from concave mirrors to multiply their size in the short distance.

The software running on the prototype device was built using Unity and Vuforia, providing ad-hoc marker-based experiences where a printed image was responsible for the digital objects’ placement.

3.3 Moderator’s Role

In these tests, the moderator should never give directions or suggestions to guide the participant through the experience. The moderator should ask what the participant believes should be done and encourage he/she to follow their assumptions and think aloud throughout the entire experience – i.e. express in words all doubts and observations.

The only guidance provided by the moderator is to point to the next scenario’s printed marker, so the participant can then initiate the next demo scenario.

3.4 Data Collection

The main goal for these experiments was to quickly and superficially understand people’s impressions on AR, mobile computing devices – especially head-mounted ones –, and hand gestures as UI input type. Therefore, a good user experience was never intended to be achieved, instead all demo scenarios were meant to provoke reactions thus providing valuable insight to a plausible direction to which to steer augmented reality solutions in following endeavors.

Demo Scenarios: General Interactions.

Once a scenario’s printed marker is recognized by the device’s camera, a hand animation with an upwards-pointing index finger appears, in attempt to induce the participant to repeat the gesture thus activating the beginning of that demo scenario. Once each scenario is finished, an animation with a “End of Demo… Restarting.” text message appears. After the end of a scenario, the participant is directed to the starting marker for the next scenario. 2 different sequences were used split evenly among participants (Fig. 3).

Fig. 3.

Close-up shot of the device showing a hand animation prompting action to initiate demo scenario.

Demo Scenario 1: Points of Interest (POI) and Navigation.

The experience starts with minimized cards placed horizontally all around the participant – 360º on a horizontal plane –, each representing a different location, such as restaurants, museums or grocery stores. The card positions do not change, the participant must look around to view other cards. As a card approaches the center of the participant’s Field of View (FOV), it becomes maximized. When a location nears the edge of the FOV, it minimizes into a circle through a center-distance-responsive scaling motion.

After interacting with the card through the upwards-index finger gesture, a full detail card fills the screen and, after a short timer, the participant is presented with a mini-map and directions to the location (Fig. 4).

Fig. 4.

Close-up shot of the device showing a centered maximized card over the monument, and a minimized “circle card” to the right of the monument.

Main issues evaluated:

• How well do users understand there are POIs all around?

• How useful to the users the presented POI information is?

• What are the user’s expectations to POI information?

• How useful to the users is the full map view?

• What are the user’s expectations to navigation information?

• How do users respond to having both gesture-based and time-based actions in the same scenario?

Demo Scenario 2: Lego Shopping.

The experience starts with an animation of the built Lego set with each toy interacting with each other, before presenting a “Hot Deal” badge with a discounted price tag and a shopping button with the gesture animation. Interacting with the shopping button triggers a “Successfully purchased” message prior to ending the scenario (Fig. 5).

Fig. 5.

Close-up shot of the device showing the animated Lego set with price and purchase information.

Main issues evaluated:

• Does the animation increase interest in the product?

• What are users’ expectations to the presented information?

• How do users respond to having a shopping button similar to the one that triggers the start of each scenario?

• How do users feel about how long it takes for them to see the price?

• How clear it is to purchase the Lego toy set?

Demo Scenario 3: Baking.

The experience consists of an animation sequence with timed steps presenting the several preparation stages of a cupcake recipe (Fig. 6).

Fig. 6.

Close-up shot of the device showing an animation presenting one the cupcake recipe preparation steps.

Main issues evaluated:

• Does the animation increase interest in the product?

• How useful is the animation to understanding the product?

Demo Scenario 4: Wine Shopping.

In this experience, the complementary information to the wine has been shrunk significantly to the point of minimal readability just beyond arm’s length. The only way the participant can read the information is by walking towards it less than a foot away (Fig. 7).

Fig. 7.

Close-up shots of the device showing the downsized information (on the left) and how it appears when up close (on the right).

Main issues evaluated:

• How useful is the presented information?

• Is the information layout helpful for understanding the product?

• How clear it is to purchase the wine?

• Do participants understand they can walk towards the marker to zoom in?

Demo Scenario 5: Daily Information

This experience’s printed marker is placed far from the participant in a way that it is difficult for it to be recognized. Regardless of the scenario sequence, this is the final demo scenario for all participants. Once initiated, 3 different dioramas are presented with a short timer triggering their replacement: weather, calendar, and music, respectively. The latter option presents the user with a “Play music” button. For the first time in the entire usability test session, the moderator uses a hidden controller to discretely activate the option and initiates music playback on a mobile phone. A message “Look at the speaker” is presented and when the participant looks down at the speaker on the table, the moderator once again discretely transfers audio to the speaker via Bluetooth, thus giving the participant the impression that the music began playing on the speaker just by looking at it (Fig. 8).

Fig. 8.

Close-up shots of the device showing the music diorama (on the left), looking at the speaker (on the middle), and the moderator’s hidden controller.

Main issues evaluated:

• How useful the presented information is to the user?

• What are user expectations to this scenario’s information?

• How do users feel about the timeout for each step in the scenario?

3.5 Additional Procedures

The sessions were conducted in a controlled environment, provided with a video camera to capture the participant’s face and gestures. A digital voice recorder was also present at the sessions to create a set of audio recordings for backup.

During each of the evaluation sessions a standard protocol was followed. First, the participant was introduced to the user test methodology and acquainted with the equipment and recording systems in the room. Moreover, an explanation of the importance in thinking aloud when performing the usability tests so that an indicator of their confusions’ moments and reasons is clearer. A particular caution to avoid entering into any details about the scenarios the participants would see was imperative as well as about any possible interactions with them, thus ensuring an experience as unbiased as possible.

Accompanying interviews with each participant happened before and after the usability tests. The former to discussing any previous experiences with augmented and virtual reality (AR/VR) and reasons for trying a new technology. The latter is preceded by a form where the participant scores each scenario. Unscripted follow-up questions were then done by the moderator to clarify participants’ behavior and expectations. Finally, both moderator and observer did a debriefing together to contribute their observations about surprises and issues identified and tallied those throughout the sessions.

4 Findings and Recommendations

This section describes the main findings obtained from the study. Both quantitative and qualitative data were analyzed at the end of all the user tests and results were organized according to different categories that were evaluated during the sessions. These are: impressions about the hardware, input controls and user experience. We also look at questions regarding the usability tests in themselves, considering main takeaways from our experience, and draw some insights that might be useful for further explorations.

4.1 Hardware

This category describes some specifics of user’s acceptance and expectations regarding the head-mounted hardware. For each of them, we also give recommendations to improve future iterations of the prototype.

Focal Length.

Different focal lengths between the digital projection and physical surfaces led to misconnection between the two thus resulting in little spatial interaction – i.e. little to no moving around the room to view information from other angles or distances. This suggests that in order to make use of spatial interfaces, the projection must have a focal length that matches with the real world. Virtual projections that are mismatched with the physical world’s focus length don’t seem to provide credible spatial behavior.

Ergonomic Device.

Almost every participant complained about the device being big and uncomfortable. Even when they had enjoyed the scenarios and found the experience to be useful, the ergonomic aspect of the device in general prevailed over other considerations. This aspect of the product probed to be essential to users. Even an extremely useful software experience is quickly overshadowed by any hardware-caused discomfort. Indeed, bulky hardware was considered a main hindrance in user’s acceptance of AR wearable devices. Therefore, despite its usefulness, a large and heavy hardware is not well regarded with the sole exception for where helmets or big goggles are already in use.

Small and Simple.

The small form factor was also put forward as a crucial element when considering future adoption. Several participants regarded as an acceptable size, something they could carry in their pockets. Indeed, most participants found more desirable a small device that offers a simple and convenient experience, than a big, robust and powerful solution.

Modular Approach.

Participants who wore prescription glasses encountered several issues when wearing the device, due to incompatibility between the available space left to fit the user’s face and several larger shapes for glasses’ frames. Moreover, having a touchpad on the right side of the device is an issue for left-handed individuals. This suggests that considering a modular approach might be an interesting path to explore when aiming at a consumer market. This way, the same base hardware could fit a plethora of glasses, attend to right- and left-handed individuals if using touchpads on the side, and also allow temporary upgrade to more powerful and bulky devices.

4.2 Interface

Here we present main issues that occurred during participants’ interaction with the interfaces. Participants were not specified how to interact in order to learn about user’s expectations of how interaction was most likely to happen by letting them spontaneously decide the appropriate response.

Useful Animations.

Some participants stated that they would appreciate a more guided experience. In this regard, the use of animations to draw attention to a specific object proved to be extremely helpful. As an example, animations were successfully used to draw users ‘attention to a place where the camera needs to better analyze for image recognition purposes.

Clear Design Elements.

There should be a clear understanding of what is interactable and what is purely informational. Participants were confused whenever the visual representation of an interactable content differed from repeating interactive elements seen previously. Keeping the visual unity guides the user on how to interact and what to expect as a result, as well as provide context awareness through that element. Exploiting repetition and consistency of design elements, users can rapidly define mental models for greater interaction with the UI.

Feedback for Interactions.

Another recurrent issue was a perceived lack of feedback whenever an interaction was initiated or ongoing. In some cases, the experience was initiated accidentally and in other situations participants tried out different gestures, stating that they felt uncertain of whether they were interacting or not. This indicates that letting users know they are being listened to or that their input is being recognized in action, matters for them. A possible solution to this difficulty is to always provide feedback for interactions – be it visual or audio –, not just for confirmation of performed action, but also for ongoing ones.

“Do not Disturb” Option.

During the tests, a few participants felt overwhelmed with a continuous flow of information before their eyes. Some of them suggested having a gesture to dismiss the information at any moment during the experiences. Questions of invasiveness were also put forward during the debriefing interviews. In this regard, being mindful of an intrusive interface (e.g. providing an easily accessible ‘Do not disturb’ option or ‘Silent mode’ that could be triggered whenever the user deems necessary) can minimize this potential hindrance for user’s acceptance of this kind of solutions.

Ground’s Visibility.

An important issue that was brought forward during the interviews with several participants, was a security concern regarding the possible risks of using an AR wearable device in public places that could potentially isolate them or even blind them against the environment. Indeed, the fact of blocking the ground’s view might disguise physical obstacles, thus possibly causing accidents. Devices that have their projections positioned straight on users’ eyesight, are likely to take up too much space of user’s Field of View (FOV), thus getting in the way of critical information from the surroundings. This must be considered when designing wearable devices for outdoor use by consumers.

4.3 Input Controls

This category refers to questions related to how participants used the input controls during the test and presents the main insights about desires and expectations regarding those.

Hands-Free Experience.

According to participants the fact of having to interact with the interface felt rather natural at times. However, it was highlighted that there would be moments where they might not be able to use their hands to interact with the UI. Another point that came to the fore, was that most participants felt shy about performing the raised finger gesture even during the user tests and most of them explicitly stated that they felt awkward while performing gestures in the air and that they would not be willing to do it in a public place. This shows a gap that can be considered in future work, by exploring real hands-free experiences for moments when interacting through gestures is not possible or in situations where social awkwardness refrains users from performing certain gestures.

Physical Surfaces.

The previously mentioned social awkwardness and natural feel also speak for the possibility of using physical surfaces for more natural haptic feedback. Not only can physical surfaces make interaction feel more natural, it can also bring discretion to the experience.

Non-invasive Confirmations.

During the interaction with the different scenarios, sometimes participants accidentally activated an action they were not, in fact, willing to take. The problem of false positives was a common occurrence that should be avoided. At the same time, it is important to do so, while also avoiding excessive steps for interaction (for example, a 2-step gaze interaction or temporary gesture/voice activation could be used).

Multiple Input Options.

During the test, we observed a great variety of intended gestures when interacting with the interface. Indeed, natural controls and gestures depend to a great extent on existing mental models and cultural traits. One way to embrace idiosyncrasies is to allow personalization of such controls. The possibility of doing so by providing multiple options depending on the user’s context and preferences, instead of only one, might be an interesting path to explore.

4.4 Experience

This section presents main findings in relation to the overall user experience, as it was described and reflected upon by participants during the interviews that followed the usability tests.

Convenience Beats Options.

Throughout the accompanying in-depth debriefing interviews with participants, most of them agreed that they valued convenience in a solution, rather than a wide range of options. Therefore, it is significantly more valuable to a user an interface that, despite its simplicity and lack of options, is easy and pleasant to interact with.

Embrace User Guidance.

In general, participants valued very positively user guidance during the experiences. In the study, it was realized that an introductory experience presenting the interactions works well, however other resources such as simple interaction hint animations for users to mimic proved to be highly effective. Also, using redundancy in information provides helpful guidance through the user experience (e.g. use animation, text and sound to explain an action).

IoT Controls.

In several cases, participants manifested a desire to have control over objects in a seamless fashion, for instance, being able to control smart locks, lights, speakers, microwave ovens or washing machines.

5 Discussion

Our research investigated the overall user experience with an AR-focused wearable head-mounted device, thus identifying opportunities for improvement and further investment. The study was designed as an ad hoc exploration aimed at validating expectations and mental models of users towards a head-mounted display. As with any other ad-hoc reporting, the work involved a customized, small-target approach, which entails inherent limitations, especially regarding the generalizability of the findings. On the other hand, this kind of ad-hoc reporting allows to get actionable insight on specific questions that might be difficult to get from more generic reports.

We organized the recommendations in different categories that span the different aspects of the experience that we wanted to evaluate. These are: hardware, interface, input controls, and overall experience. For each of them, we identified several issues perceived during the user tests that somehow hindered the device usability and, thus, we offer some suggestions for improvement. In the case of hardware, the main takeaway is that any hardware-caused discomfort severely compromises the entire experience, regardless of how useful the software was considered. Thus, it is important to look at the AR experience as a whole, in which advances in software and hardware should walk hand in hand to get out the full potential of the technology, since the ergonomic factor seems to be a crucial element for its usage. Regarding input controls, it was observed that the range of gestures which users spontaneously came up with varied widely among participants. This suggests that a possible path that so far has remained rather underexplored is to consider input control personalization. Given that natural gestures are culturally shaped to a large extent, allowing for a degree of personalization of input controls can be one way to turn the experience with AR devices easily amenable to such diversity. A main point that the user tests revealed is that AR wearable devices should not deter user’s visibility and interaction with the physical surroundings. Avoiding filling the user’s field of view with a digital projection is a way out to this hindrance, as well as being mindful of an intrusive interface that might stand on the way of user’s interactions with the real situations in which they find themselves.

On another level, this study discusses other questions that were encountered while carrying the usability tests that go beyond the specific purposes aimed at evaluating. In particular, some highlighted aspects perceived during the user tests are believed to certainly become opportunities for improvement in forthcoming research. These questions relate to the method used to gain insights into the usability and utility of the prototype and reveal both hits and opportunities that are worth considering regarding the user tests per se. First of all, using superficial simulations proved to work well for testing. It was observed that it allows more explorations with less effort, while facilitating any direction shift on a certain design strategy, without losing too much work. Moreover, using simulations is less prone to usage error or crashes, and bypass controls are highly effective to avoid user frustration with recurring failed attempts at interaction. Simulating the success of interactions by bypassing them with remote controls are highly effective in keeping participants engaged, while allowing for greater freedom in interactivity.

Another takeaway of this experience is that the use of repetitive elements was very helpful for participants during their navigation through the scenarios and served to conduct the sessions more easily. For example, the use of information such as “End of Scenario” upon completion of each of the demos was highly effective to keep a good pace throughout the user tests. Besides, at the moment of wearing the device, an image was immediately displayed to serve as a reference when adjusting the headset when worn for the first time. This ensured that participants could effectively see the interface as soon as it appeared and helped to create a smoother experience.

As a final note to this section, we argue that the primary value of this study lies in its exploratory character, that is, as a key step to establish priorities, develop operational insights and help improve subsequent research. Allowing for this kind of studies can provide industries with leverage to define a consistent roadmap and bring priorities into focus. In this regard, the findings reported here, although lacking statistical generalizability, still provide grounds for analytical inquiry valuable for the design of ensuing studies. In this regard, the validation presented can serve as a first step for other explorations considering a more user-centered approach to AR devices, in addition to drawing recommendations relevant to future investigations on this field.

6 Conclusion

This paper has presented main insights regarding several features for an ideal head-mounted augmented reality experience to the user, as well as their view regarding the future of digital media consumption. Among the findings, one major takeaway of our study is that hardware issues regarding the size and ergonomics of the device were given fundamental importance by participants, no matter how much utility or power they might gain through the use of a device. This means that a primary concern when designing an AR experience is first and foremost to ensure the hardware is comfortable to wear, to use, and to purchase. Through the accompanying in-depth debriefing interviews, another key finding is that convenience is much more important than options. Therefore, it is significantly more valuable to a user an interface that, despite its simplicity and lack of options, it is easy and pleasant to interact with. We have also discussed some lessons learned for future work, specifically regarding how user tests, as a method widely employed in user-centered studies, can be improved.

Our results allow us to shed light onto the scope of expectations for potential end users, in order to help in the design and engineering of AR wearable devices. We argue that the value of exploratory studies such as these lies in its potential to delve into innovative paths and their implications, otherwise difficult to trace within the industry.