1 Introduction

With the propagation of digitalization in engineering education, visualization methods like Virtual and Augmented Reality (VR, AR) are increasingly focused by educators. The process of manually creating content for Mixed Reality (MR) applications is however still time consuming and requires special knowledge in areas of 3D modeling, programming, choice of device and software platform.

The authors developed an Augmented Reality Content Management System (ARCMS) to assist educators with creating AR manuals in the context of laboratory experiments, which allows for an easy-to-use and simple-to-learn creation of AR based step-by-step instructions.

Educators use a web-interface to create content for AR manuals. They define individual steps to be performed by the students. Each step can contain text and multimedia elements as well as a hint that is displayed in AR.

The system was implemented and is used in different departments in the authors’ institution under experimental conditions to further validate the findings presented in the paper at hand. Educators pointed out the ease-of-use in creating content for their laboratory experiments. However, no formal study from the students’ point of view has been conducted before the paper at hand.

1.1 Augmented Reality

Azuma defines AR as a context-aware overlay of virtual information on a fixed position of the real world. Augmented Reality is defined by three key characteristics: a combination of the real and virtual, real-time interaction and 3D registered, spatial context, so that virtual content appears at a fixed position independent of the user’s position [3]. This stands in contrast to VR where the user immerses himself in a fully virtual environment. However, a clear difference between those technologies is not always obvious. Milgram et al. [14] place both technologies on their Mixed Reality (MR) spectrum where the real world dominates the user experience in AR applications and the virtual world is dominant in VR.

To allow for a spatially fixed visualization of virtual content the system needs to track the position of the user. The most common way to access AR content today is via smart devices like smartphones and tablets [1]. Those devices offer high-resolution cameras and displays as well as a variety of sensors, which assist in calculating the exact orientation and position of the device in the surrounding space. Furthermore, they provide high processing power and come in compact form factors. While other AR systems like Head-Mounted-Displays and Smart Glasses exist, the hardware is still in its infancy and not as reliable and proven as in smart devices [6].

1.2 Augmented Reality in Engineering Education

Common industrial use-cases for AR include MRO (maintenance, repair and operation) processes and marketing. AR in education however becomes an increasingly focused topic [1, 4]. Research on AR in education has been undertaken in various educational levels, including higher education at universities as well as primary and secondary education, and over a variety of fields like engineering, language teaching, mathematics [4], medicine [9] or arts [8].

Systematic reviews of literature confirm the various benefits AR brings to education. On the one hand it supports the learning process by enhancing motivation [1] and satisfaction [4]. The immersion provided by AR as well as the integration of interactive multimedia content and the personalized learning possibilities are further benefits of AR in education [4]. On the other hand, it should be noted that while most reviews of literature show positive effects on students learning, negative effects of AR are rarely mentioned and mostly focus on cognitive overload [15] or technical problems [1].

AR is used for a wide range of use cases in engineering education. Lin et al. [11] show that using AR to teach basic physical concepts like elastic collisions is notably improved over 2D animations. Akçayir et al. [2] use AR in laboratory environments to improve laboratory skills of first-year university students. They show that AR makes it easier and more comfortable to complete the experiments, which in turn allows students to complete the experiment in less time. Martín-Gutiérrez et al. [13] present different use-cases for AR in electrical engineering education that present animations and 3D models on predefined markers to provide step-by-step instructions, help with understanding electrical plan reading or teach fundamental physical concepts like magnetic fields. However, they developed an individual, specialized AR application for each use-case.

This underlines that the creation of content for AR is still a major challenge before AR can become more widespread in education [17]. Kerawalla et al. [10] identify the educator’s flexibility in creating custom AR experience as a primary requirement for using AR in education. To this day, it is still time consuming and requires a very special skillset. Content is often created using 3D modelling tools, which offer a lot of advanced functionality, but also have a steep learning curve [5]. Instead of directly modelling content for AR, 3D models can be obtained by scanning real-world objects using 3D scanners or video cameras [5, 13]. While also time-consuming, this process requires additional hardware as well.

To provide an easier solution for content management in AR, Cubillo et al. [7] propose a client-server-architecture that allows educators to create multimedia content in a web application. The data is then associated with a marker that students can scan with their mobile device to view the created content. The markers are then incorporated into existing educational material.

However, this solution is neither suitable for step-by-step instructions nor does it take advantage of new, advanced AR functionality, like spatial awareness and simultaneous localization and mapping (SLAM). SLAM allows free movement even when no image marker is visible to the camera. The authors therefore propose a different approach for an ARCMS.

2 Aims and Requirements

To make the creation of content for AR manuals easier, the authors propose an intuitively usable ARCMS. In a typical use case three actors can be identified. An educator creates and prepares the step-wise instructions. He is interested in using AR but does not necessarily have any prior knowledge or experience in using AR or creating content for AR. Students are performing laboratory experiments. They might not have any prior knowledge with the experiments and are getting their knowledge from paper-based manuals or from an oral introduction by the educator. AR based manuals can help with providing structured information overlaid on the real world that improves orientation as well as the motivation of the students. To achieve this goal, a manual is divided into discrete steps that have to be completed in sequence. Each step describes a singular task and accompanying descriptions, hints and multimedia content that help completing it.

An ARCMS consists of three parts. All information needs to be stored in a central datastore. A web-application is used by the educator to create the manual and save them in the central datastore. Finally, a tablet-based AR application is used by the educator to spatially register the steps’ hints at the real-world equipment. After that, the students can view the prepared manuals.

Educators must be able to select from a given set of hints, like different types of arrows, that they can position freely in AR. Instructions should contain texts, as well as multimedia elements like images. Furthermore, it should be possible to add safety or warning symbols to each individual step. Because a reliable wireless network connection cannot be always guaranteed, the system needs to provide a mechanism for storing and synchronizing all data necessary to the tablet application. Students must be able to move freely around the machinery with the AR application to perform the various tasks. While an image-based marker can be used as a reference point for initialization, it should not be necessary to have this marker visible at all times, as this would severely hinder the students’ mobility.

3 Concept

To create and use an instruction, three steps are necessary. First, the content is created in a web-application by an educator. A step can contain an AR hint, like an arrow, that then needs to get a position assigned at a real-world machinery in the second step. After each step has an assigned position, the gathered data is saved in the data-store and can be viewed by students on other smart devices. This three-step process is shown in Fig. 1.

Fig. 1.
figure 1

Creating and using an instruction takes three steps. Creating the step-by-step instruction, assigning a position for each step and then using the instruction.

Full size image

The three subsystems of the ARCMS are implemented in two components. A webserver provides the central datastore as well as the interface for the web application. It also provides an Application Programing Interface (API) built based on the Representational State Transfer (REST) paradigm. The second part of the system, the tablet-based AR application, uses the API to query and update data in the datastore. This is shown in Fig. 2.

Fig. 2.
figure 2

Implementation of the system in its two parts, the webserver and the tablet application.

Full size image

A manual contains general metadata, like a unique identifier, a revision, a title and a description, and as the central element of the approach list of tasks to perform. The actual content is structured in a list of steps and offers additional descriptions on each step. Those descriptions can not only contain short texts but also show additional images or documents. They can furthermore attract special attention by showing hazard symbols and accompanying safety instructions. For each step within an instruction, a visual hint is presented to the user in the AR overlay to assist with the orientation around the laboratory equipment. Educators can choose hints from a given set of icons, including arrows and hazard-symbols to reduce the complexity when creating instructions.

Steps can be grouped together to give structure to the manual. Like steps, groups provide a textual description, hazard symbols and attachments. In contrast to septs groups do not store a position or AR based hint, but show all hints of the child elements.

Usually, tasks in an instruction are supposed to be performed in a designated order. Therefore, steps can be set to be explicitly prohibitive and need to be marked as completed by a user, so that subsequent steps are only available after all previous steps have been completed. Because sometimes steps should always be available, e.g. because the step gives some general information, this behavior can be turned off for each individual step.

After the content has been defined in the web application, each step needs to get a position defined at which the AR hint can be shown. This is done in the tablet application. An image marker is printed out and attached close to the machinery to provide a frame of reference for spatial initialization. The application detects the marker and stores all positions and orientations relative to its detected position. After the marker has been detected, the educator can then select steps and assign a position to them by tapping on the desired location in the tablets’ camera image. The spatial position in three-dimensional space is position calculated relative to the marker and the hint is placed at the appropriate position at the real-world machinery. The educator can relocate the position for each step’s hint later without changing any other data of the manual. Once placed, each AR hint can be scaled and rotated with the usual smart device gestures. The manual can be accessed by students only after each step has a position assigned.

The student must detect/scan the marker to start using a manual. Afterwards, a list of all steps in the manual is presented to him. When selecting a step, it’s description, attached safety and hazard symbols and images are shown in the lower-right corner of the tablet’s display. The primary focus on the screen is the camera image with the relevant overlaid AR hints. When a single step is selected by the student, the chosen hint is presented at the position associated with the step. For a group, the application shows all hints from all children of the group. Then, the user can tap an AR hint to see the information the educators have provided for the step. In combination with steps that are available even if previous steps have not been marked as completed, this allows for a flexible system that can be used to provide informational content rather than just providing actionable steps.

4 Prototypical Implementation

All subsystems have been implemented prototypically. A server is responsible for both the datastore with a database built on the CouchDB NoSQL database management system and a filesystem for storing attachment files. It also runs an Express webserver on top of NodeJS serving both the web application as well as the REST-API.

CouchDB was chosen as a document-based database because of its native ability to store JSON-Data as well as it’s REST-API, which makes it easy to integrate into NodeJS applications. Furthermore, CouchDB provides an identification and revisioning system that can be utilized for the synchronization of data between the sever and tablet clients. Both manuals as well as user data are stored in a single database. The webserver was implemented using the Express framework for NodeJS. The web application allows users to register and log into the application where they have the option to create a new manual or edit an existing one. When editing a manual, the user can add, remove and reorder steps using a drag-and-drop interface. The user can enter text for titles and descriptions, select AR hints and safety symbols from prepopulated dropdown-menus and upload files. When the user saves the manual from the web application, CouchDB automatically increases the revision number to avoid synchronization conflicts.

Because each step needs a defined position, adding new steps requires the educator to assign a position using the tablet application after the manual was edited. However, the position is saved if the educator only edits or reorders existing steps. Small changes and improvement can therefore be made without much redundant work. The web-interface of the prototype is shown in Fig. 3.

Fig. 3.
figure 3

The web-interface used to create and edit instructions

Full size image

The tablet application is used by educators to assign positions to each step as well as by students to view manuals. It is built for iOS on top of Apples ARKit AR framework and runs on both iPad and iPhone, although the use of tablets is recommended because of the increased screen size. To use the application, the user first needs to register using the web application. Then, the same credentials can be used to log into the tablet application as well. Educators can select from a list of manuals they have created. After selecting a manual, the application requires them to point the devices camera to the marker. The application is initialized once the marker is recognized. The educator then selects each step from a sidebar and taps on the screen to assign a position to the step. After each position is defined the manual is saved into the database. The educator can then choose to publish the manual to make it available students.

Students only get to see a list of published manuals, which have positions assigned to each step and are therefore marked as published. When a marker was detected students can select each step that is available to them. In addition to the AR hint that is shown at the saved position relative to the marker, the additional information is shown to them. This includes all texts, safety symbols as well as attached images associated with the step. When a step can be marked as completed, a button is also shown that creates a checkmark next to the step and advances the manual to the next step. The tablet application is shown in Fig. 4 as seen by students.

Fig. 4.
figure 4

The tablet application as seen by students.

Full size image

5 Usability Study

An interactive AR based instruction was created using the stated ARCMS to conduct a usability study. This instruction describes a real-world experiment on actual equipment from the field of material science, namely a conductivity measurement of a thin film materials library which was sputter-deposited on a silicon wafer [12, 16]. In the experiment the participants have to perform a variety of tasks. This includes turning on and setting up measurement equipment, aligning a silicon wafer under the probe head and maneuvering the head to a defined starting position using a joystick control. Furthermore, the participants have to set up the measurement software using a PC. Participants get a short, around 1-minute-long introduction in the purpose of the experiment as well as the general usage of the AR tablet application. However, no additional information is given on what steps the participants are asked to perform. Participants are asked not to request additional help or information from supervisors during the experiment. Instead, they should perform the steps to the best of their ability using only the information provided to them by the AR application.

The instruction incorporates different features of the system and shows detailed descriptions for each step, images and hazard symbols as well as visual hints. The participants of the study are students from different fields that have not previously conducted this exact experiment. Supervisors measure the time needed and the number of handling errors whilst participants perform the experiment. After finishing the experiment, students evaluate their experience while using the system and the instruction with a provided questionnaire. Students will judge different aspects of the system, whether the system helped them performing the task and how well they felt supported with the orientation at the complex instruments, as well as rating several motivational aspects and their own success rate, as they experienced it, of solving an unknown lab experiment while using an augmented reality application.

The questionnaire asks for general metrics about the students and their usage of smart devices and social media to assess their technical knowledge and overall digital competence concerning tablets prior to the task. Furthermore, students evaluate their perceived workload while performing the experiment using a Task Load Index designed and used by the NASA and adapted into the German language and used by leading research facilities such as the German Federal Institute for Occupational Safety and Health (BAuA, Bundesanstalt für Arbeitsschutz und Arbeitsmedizin). Finally, the participants judge the ease of use and functionality of the ARCMS itself. The participants also were given the opportunity to write additional textual comments on the instruction and the used application based on their learning experiences.

The results will be analyzed with regard of the student’s technical prior knowledge, their actual and their perceived success rate while working on the experiment. The goal of this study is to evaluate the motivational aspects of using AR applications in laboratory settings as well as to investigate the differences between actual and perceived learning experiences while working with AR. Another goal is to further understand the usability of the application to make it even more intuitive and usable for novices and experts alike.

5.1 Outcomes

16 participants, all students at the Ruhr-University Bochum and between 21 and 31 years old, (62.5% male and 37.5% female) took part in the study. 60.0% were students in engineering, 33.3% studied in humanities, 6.7% said they studied something else, and one participant did not give his or her subject of study. Since it was suspected that engineers would be more used to the overall experimental laboratory setup and the use of smart devices for learning, students of other disciplines were asked to take part in the experimental study for comparison. After finishing the experiment, they were asked to assess their use of smart devices using a Likert scale asking for the frequencies of certain uses. Then they had to assess their experiences while working on the experiment using a Likert scale consisting of five characteristics: Low, rather low, medium, rather high, high.

Table 1 shows the accumulated frequency of positive answers (high or very high) to the questions asked concerning the prior technical knowledge and overall use of tablets and smart devices.

Table 1. Prior knowledge and use of smart devices (n = 16).
Full size table

The numbers show how often and for what purpose smart devices are used in the test group. Overall, 93.8% of all participants use smart devices often or very often. Smart devices are most used for communication (93.8%) or social media purposes (60.0%). 55.3% of the participants also use their smart devices for learning, but only 8.3% have used AR applications in the past.

The students were asked to validate the physical and psychological work load they experienced while using the AR application to solve the laboratory experiment. They were given a scale reaching from low to high (Table 2).

Table 2. NASA instrument for assessing mental and physical load (n = 16).
Full size table

Next, the students were asked to validate the physical and psychological work load they experienced while using the AR application to solve the laboratory experiment. They were given a scale reaching from low to high (Table 2).

Concerning mental activity and cognitive workload most students (86.6%) said that it was somewhere between rather high and rather low, with the median being right in the middle of the scale. This means that most students assessed their cognitive load as neither too much nor too low. 75.0% said that their physical activity while using the AR application was rather low or medium. The remaining participants assessed their physical activity as low or rather high. None of the participants said that they felt their physical activity was too high, none of them found it straining to hold a tablet and navigate through the application. When asked to assess the perceived time pressure while working on the experiment 75.0% said it was low or rather low. While the students did not know about any time limit, they did know that the time needed for the experiment was measured. 18.8% found the time pressure to be medium. One participant found it rather high. Asked for their overall strain while fulfilling the given task 56.3% said it was low or rather low, 25.0% said it was medium. Two students found it rather high and one said the experienced strain was high. Assessing their frustration while working on the given task 87.5% said it was low or rather low. One student ticked medium and one student ticked rather high. Finally, they were asked to assess their own performance during the experiment. Despite the fact that some found it straining or difficult to finish the experiment, the majority felt they completed the task well (25.0%) or rather well (43.8%). The rest (31.3%) thought they did neither too good nor too bad.

Table 3 shows the accumulated frequency of positive answers (high or very high) to the questions asked concerning the ease of use and learning experience using the AR application. While most participants found the information given in the application helpful and useful, 68.8% of them said that it helped with their understanding of the lab experiment itself. It should be noted, however, that the particular instruction focused heavily on the actual instructions and did not offer any additional background information about the experiment.

Table 3. Ease of use and learning experience (n = 16).
Full size table

This underlines that the quality of the actual content heavily influences the perceived usefulness. The difference between the positive answers to this question and the significantly higher numbers in other questions might also be explained by the fact that not all participants had a background in engineering and may have lacked certain basic knowledge that helped the other students to understand the experiment at hand better.

5.2 Conclusion

Most students needed 10 to 13 min to solve the hitherto completely unknown laboratory experiment using the AR-application. One student was slightly faster and finished in under 8 min, one was slightly slower and finished it in 15 min. Most students (68.8%) made only two, three or four mistakes that could be easily corrected. Three participants made even fewer mistakes, two made five or six mistakes. All students, independent of their experience and study subject, were able to finish the experiment quickly and successfully.

The numbers show that smart devices, although not frequently used for AR applications, are already important for students when it comes to learning and are overall very widely and frequently used. Therefore, tablets work as an intuitive instrument and medium for teaching university students.

Interestingly, three questions in the questionnaire showed a wide variance of answers:

  • How do you asses your mental load during the use of the application?

  • How do you asses the pressure of time during the use of the application?

  • How do you overall asses your strain while fulfilling the given task?

The rather high deviation in the participants’ answers can be explained because some participants had already done lab experiments in the past and some didn’t. Secondly, some participants had no engineering background and had not used AR applications, meaning they were not familiar with working in a lab, using an application and were, overall, not familiar with the science behind the experiment. Therefore, these students assessed their mental load higher, the time pressure bigger and the strain equally higher than the rest of the group. These outcomes might indicate that this particular instruction might not be enough for novice learners, since those still required additional help and support to finish the task.

On the other hand, even those learners who had to work harder to navigate through the experiment were able to finish it successfully in the end and were overall pleased with their own performance, meaning that the developed application fosters all students’ motivation, independent of their academic background and learning experience.

Overall, most participants felt that the given information was enough to help them with the experiment, that the AR application was helpful while solving the given task, and that it was both easy to understand and easy to operate. 93.8% said that they would like to see more such learning applications in education.

To sum it up, this first study showed that students can work very well with the application, that it supports their learning and even enables them to do lab experiments independently from scientific staff, even when they have never done a lab experiment in the past. The outcomes also show that the developed AR instruction has a positive effect on student’s motivation.

6 Summary

All in all, this first study regarding the usability of the designed system shows promising results. The authors developed the instruction used for the study in about one hour, which underlines how easy and fast content creation becomes with the ARCMS tool. The student participants were satisfied with the usability of the application. However, the mixed results in the satisfaction regarding the actual content shows the importance of the actual instruction independently from the used framework. However, the authors observed a boost in motivation not only for students working with the instructions but also for educators who prepare such experiments. The novelty of AR combined with the easy-to-use ARCMS can help educators in creating better and more structured instructions for their experiments.

The authors plan further development of the system to make it available to a broader range of users. This includes other use-cases for education in engineering, education in medicine as well as in sheltered workshops. Furthermore, the authors plan to test industrial applications for AR based step-by-step instructions in MRO (Maintenance, Repair and Operations) processes. Future studies should further investigate the correlation between the use of AR applications and students’ motivation. Also, when other scientific fields of study start adopting AR based instructions, that different groups of participants and different kinds of students can be investigated and compared to one another.