Keywords

1 Introduction

The Microsoft HoloLens is one of the latest members that joined the growing number augmented reality (AR)-capable displays. Although Microsoft did not label this unit as a sole AR display, they prefer holographic computing, it definitely facilitates typical AR applications. A huge benefit is the embedded tracking capability of the HoloLens, which allows the device to compute its position and orientation relative to the environment. In an AR scenario tracking facilitates to augment physical objects with virtual information, thus, to enrich the environment and to convey instructions. One caveat, the HoloLens tracking enables self-tracking and does not allow one to recognize and track physical objects with its onboard technology, which is a necessity for assembly assistance.

One goal of our research is to improve AR-assisted assembly and assembly training. In an assembly scenario, the display is used to convey assembly instructions, shown as registered 3D models. An assembly task is usually a procedural task. The operator follows a given sequence of assembly steps. Thus, the 3D models demonstrate step-by-step what an operator is supposed to do (Fig. 1). Text, voice, and videos can provide additional help, if a user requests them.

Fig. 1.
figure 1

(a–c) The images show the view through a HoloLens (recorded via the HoloLens’s RGB camera). The 3D models are used to convey assembly instructions step-by-step. (A video is available at https://youtu.be/VApDEJMMc0M)

Full size image

The HoloLens is a good base platform for an AR-assisted assembly scenario. It has features and advantages that make it an ideal choice for this scenario, however, it disadvantages may also disqualify it. The advantages: It comes along with a high-quality display, which allows for a high-quality rendering, it is self-embedded, thus, untethered, and its tracking latency is not noticeable. Its display renders clear, bright colors, still slightly transparent, but in our experience, users have no problems to see virtual instructions. It is also untethered, predecessors with a similar display quality were always tethered, which was always an obstacle for companies and interested user groups. A highlight is clearly the low tracking latency. Low since technically latency is still present, however, it is not noticeable. In our opinion, these three features are a benefit for users. The disadvantage is the limited field of view, especially when working in an assembly scenario. Everything that happens during the assembly of a product happens in arm-lengths of a user. Given this distance, almost every 3D model appears cropped. As a consequence, the instruction that a particular 3D model delivers is incomplete or can be misunderstood.

The objective of this study is to obtain the user’s attitude towards the HoloLens for an AR-assisted assembly scenario and to determine whether the advantages outperform the disadvantages. In general, when working with the HoloLens, it is possible to catch the notion of the delivered instructions, although with some effort. The user needs to step back or turn his/her head often around. Previous research, which employed previous-generation head-mounted displays (HMD) for their studies regularly reported that a small field of view is one obstacle [1, 2]. Cables, low-quality displays, low comfort in general is an additional obstacle [3,4,5]. However, studies also report that users prefer AR despite bulky and inconvenient displays [4, 5]. Our assumption is that the attitude towards this technology changed due to the popularity the technology gained in the last month, thus, users might be more open to try and work with devices such as the HoloLens. Along with features such as good display, comfortable to wear, and not noticeable latency, users might see devices such as the HoloLens as an asset. To determine this attitude is our objective.

Therefore, we asked a group of students to participate in a study, to test the HoloLens using an AR assembly assistance application, and to report their opinion using a questionnaire. In the following, we introduce our test setup and the application (Sect. 2). Section 3 describes the study and reports the results. We close with a conclusion and an outlook in Sect. 4.

2 Test Setup and Application

2.1 Test Setup

Our test setup is shown in Fig. 2. It is a typical tabletop setup which mainly incorporates a Kinect camera, along with the subsequent computer, and a HoloLens. The Kinect camera is connected to the computer. We use the Kinect to recognize and track single objects on the breadboard in order to augment the physical parts with virtual instructions. Our tracking software processes the range data from the Kinect to compute a pose for all physical parts of interest on the workbench with respect to a camera coordinate system [6, 7]. The HoloLens is used to render virtual objects, such as 3D models to convey assembly instructions (Fig. 1).

Fig. 2.
figure 2

The system setup.

Full size image

An overview of the software architecture is shown in Fig. 3. We use two software tools, one for tracking and one to render virtual content with the HoloLens. Our tracking tool TrackingExpert [2] facilitates to track objects using the range camera data, which is transformed into a point set. It employs feature descriptors matching and the Iterative Closest Point (ICP, [8]) algorithm. A feature descriptor is a mathematical model to describe the curvature characteristics of the surface around a point. It is used to represent the surface characteristics of a part as a compressed signature, reduced to a set of parameters. We employ axis-angle feature descriptors and histogram matching, inspired by [9, 10]. Feature descriptor matching is used to find the object in the input point cloud and to estimate a pose. Subsequently, ICP is used to refine the pose and to align a reference point set with the point cloud fetched from the Kinect camera. ICP increases the registration fidelity. The outcome of this tracking procedure is a matrix [R|t] representing the pose of an object in camera coordinate system. The pose is forwarded to a HoloLens application via TCP/IP.

Fig. 3.
figure 3

Software architecture of the test system

Full size image

Our HoloLens application, called HoloHelper, is a general rendering application written with Unity support. Its basic functionality is to receive tracking pose data for different objects i and to render virtual content assigned to these objects using the given pose. The application was prepared as a general renderer, not tied to any specific purpose. The software also includes the application logic to switch between assembly steps. The entire configuration for a particular application is read from an xml file.

The HoloLens needs to be calibrated and aligned with the outside Kinect tracking system to display virtual objects correctly. Correctly means in most of our application cases that a 3D model needs to appear aligned with a physical object. We use marker based tracking for calibration. An ARToolkit [11] marker is attached underneath the Kinect camera (Fig. 2). Before the application starts, the HoloLens asks the user to align the HoloLens’s front camera with the marker to estimate the spatial relation between HoloLens and camera. This procedure initializes an initial vector along with an orientation delta. When the user now starts to move, the HoloLens can track its own position relative to this initial points, which further facilitates to calculate the pose of each object with respect to the HoloLens’s camera coordinate system, hence, to render virtual objects accordingly. If the position of the Kinect does not change, further recalibration is not always necessary because the HoloLens stores the location as anchor point.

2.2 Assembly Application Scenario

We prepared an assembly application scenario in which users were asked to assemble a piston motor as shown in Fig. 4. The piston motor is a hydraulically controlled, variable (L/K series) motor. The entire unit measures roughly 7” tall, 7” wide, and 9” long including the output shaft, it weighs 34 lb. It contains 28 major parts, which need to be assembled in 14 assembly steps. For our test, we asked user to assemble eight parts in four assembly steps. We reduced the number of steps since the assembly of the complete motor takes approximately 45 min. Since we intended to work with a group, we had to prevent that the other members of the group have to wait for two hours until everybody assembled the motor once.

Fig. 4.
figure 4

(a) The axial piston motor was object for this study. (b) We asked uses to assemble a shaft, a bearing, a gasket, an endcap, and five screws.

Full size image

The AR application we prepared indicates the assembly steps using 3D models; three steps are shown in Fig. 1. Each 3D model is animated. The animation shows the assembly direction. Animations are not always necessary, since the assembly direction is obvious for many parts. However, for parts such as the shaft, a swash plate, a piston, the animation provides necessary information. All animations run in an infinite loop. We use voice commands to switch between assembly steps, the user needs to say “next” to step to the next assembly step. The voice commands are processed using the built-in Cortana voice control.

3 Focus Group Study and Results

The goal of this study was to obtain the users’ attitude towards the HoloLens, in particular, whether the HoloLens is an adequate device for an application such as assembly assistance and whether the advantages of the HoloLens outperform the disadvantages. We recruited volunteers, students from a mechanical engineering class. The group was limited to students from a machine design class for three reasons. First, the volunteers should have a connection to the application in order to notice advantages. The students of this class learn about shafts, gaskets, pistons, bearing, know their functionalities beyond layman knowledge and also learn how to assemble components into a working product. Thus, they are familiar with the topic. People from different disciplines may have a harder time to assess whether the application can contribute to assembly. Second, the students are most likely not pre-coined to other assembly assisting technologies, due to their lack of experience, and at the same time are literate in using modern technology. Third, we intend to integrate devices such as the HoloLens into classroom education where appropriate. Thus, we intended to level the students’ attitude to using and learning with such a device. In this context, this study is part of a larger ongoing study which investigates opportunities for this technology in mechanical engineering education. Therefore, the study was a part of the students’ homework. The study is sanctioned by the Iowa State University Institutional Review Board.

3.1 Procedure and Data Collection

We assembled groups of three to five volunteers for the study, 32 students in total (30 men, 2 women, 9 with glasses, nobody used the HoloLens before). Each group was guided as a whole into the study area. First, the group was introduced to the lab area and the HoloLens including a short introduction about the HoloLens’ functionality and capabilities. Each student could test the HoloLens, the standard holograms for a few minutes. The experimenter also introduced the piston motor and explained its functionality to the students.

After the introduction, the students were asked to assemble the piston motor following the introduction as shown by the HoloLens. The students were separated for this task and the remaining students were asked to wait until everybody assembled the motor once. Each trial took about 6–8 min. The students were asked to not discuss the HoloLens application at this point and to not share their experience.

When all students completed the task, we assembled the group and first asked the students to complete a questionnaire. The questionnaire presented eight statements. We used a 5-point Likert scale to obtain the students’ attitude where 1 means strongly agree and 5 strongly disagree:

  1. 1.

    I could easily understand and follow the instructions shown as 3D models.

  2. 2.

    The instructions shown though the display were helpful.

  3. 3.

    The HoloLens is convenient to wear.

  4. 4.

    I feel confident that I performed well in completing this task.

  5. 5.

    I would recommend the HoloLens to other people as visual instruction device.

  6. 6.

    The HoloLens is fun to work with.

The first questions aimed to learn whether the volunteers understood the instructions. The 3D models are cropped by the HoloLens display and it might be challenging to understand the instructions without looking from the left to the right. Question two measured whether the instructions where considered as necessary.

The remaining two statements were

  1. 7.

    I looked past the HoloLens’s display to assemble the piston motor.

  2. 8.

    I had to move my head often to completely see and understand the instructions.

The students could answer with never, randomly, sometimes, often, very often. We asked question 7 because we noticed in previous tests that users sometimes look underneath the holographic display to obtain a free look towards the physical object. The last questions measures whether the students noticed that they had to move their head often when working with the HoloLens.

After the students completed the questionnaire, we asked them to discuss and agree on further advantages and disadvantages of the HoloLens in context of machine design, assembly, and instructions in general. To encourage the discussion, we gave them keywords such as, wearing devices, voice control, field-of-view, and safety. We asked the group to discuss each point for a few minutes and to write down their opinion. We further encouraged them to discuss other topics, if they have additional ones. A facilitator moderated the discussion, however, did not contributed or influenced opinions.

3.2 Results

The results of the questionnaire are shown in Fig. 5. and the quantitative values are listed in Table 1. Each box diagram shows the results for one question. The overall mean for all questions is 1.66 with a standard deviation of 0.42. In general, the students agree that the instructions were helpful and that they used them to assemble the piston motor. The students disagree about the convenience of the HoloLens. We notice almost a 50% split into people who had no problems wearing the HoloLens and people who are neutral. The questions 4 to 6 yield a positive result from our point of view. With a few exceptions, the students like to work with a HoloLens.

Fig. 5.
figure 5

Results of the questionnaire

Full size image
Table 1. Quantitative values for each question, mean (standard deviation)
Full size table

The result to Question 7 and Question 8 is inconclusive. The majority of students said they walked too much when assembling the piston motor to obtain an overall view and that they also looked pass the HoloLens’s displays to obtain a better view of the situation.

The results of the students’ discussion are mostly positive. The majority of the students told us that they liked to work with the HoloLens. Some reported that they like the device’s functionality even if they do not like to wear it. All students agree that it is fun to see graphics in the environment.

On the negative side, the most common statement we notice is regarding the small field of view of, which is an obstacle when working with the device in close range to objects. A larger group of students also mentioned that the gesture is tiring after a few minutes of use (we did not allow the students to use the clicker). They wish for a more convenient way to manipulate objects if they have to do it for a longer period of time. The same comments were made regarding the scrolling gesture. A few students reported that they would prefer “to work on a computer display” if they have to walk around when using the HoloLens. Nobody commented on voice control or on safety concerns.

3.3 Discussion

The results indicate a clear attitude of the students in favor for the HoloLens for a close range application such as assembly instruction. The questionnaire results are mostly positive. The additional feedback that we obtain also does not add many negative points. As expected, the students want to have a larger field of view and more convenient hand gesture. On the positive end, although the field-of-view is reported to be an obstacle, the majority of students can arrange to work with it; it would prevent them from using the device. We notice a similar result considering the convenience. Students reported that they dislike wearing the device, although they would work with it, if they would be asked to. One caveat that needs to be considered is the limited time the volunteers worked with the HoloLens. Thus, it is unclear whether a longer period would change their opinion.

Second, several students agreed that they can imagine that an application such as assembly assistance is a perfect use case for the HoloLens. They can further image to use the application in class to demonstrate the functionality of machines. Shafts, pistons, and other machine components are never demonstrated in operation in class. Thus, students mentioned that an application that would virtually demonstrate how parts interact can help.

On the flipside, some students reported that they just do not like to wear the device. Mostly because they disliked that the device is tied around their head. No further reasons were reported such as safety or privacy concerns. We also noticed a dislike among glass wearers. They did not feel comfortable using the HoloLens while wearing their glasses. Removing the glasses for the time of the experiment increased unease because the HoloLens did not support lens correction and the volunteers were not used to not wear their glasses.

4 Conclusion and Outlook

In this effort, we conducted a focus group study with three groups a five volunteers to obtain the students’ attitude towards the HoloLens and an instruction application. The HoloLens can convey instructions to users using virtual content and we were interested to study whether students can consider to work with devices such as the HoloLens. The volunteers were asked to use the HoloLens for an assembly task and to complete a questionnaire. The results of the experiment and the questionnaire support the assembly application use case. The volunteers reported that the HoloLens was helpful assembling a physical part. Although we found that some people just do not like to wear the HoloLens, which appear to be a personal trait against wearing devices.

In future, we intend to focus on an improved application design that leverages the advantages of the HoloLens such as context-related information. Application features that can further demonstrate advantages of the HoloLens in comparison to other means such as textbooks, etc. may change the students’ attitude.