Augmented Reality for Maintenance
Augmented reality for maintenance is a technologic solution encompassing both software and hardware resources with the aim of providing additional virtual information as a support for assisting technicians in performing complex tasks.
Augmented reality (AR) is considered a computer vision solution aimed at “augmenting” the real world with additional information in the reality-virtuality continuum by Milgram and Kishino (1994). Differently from virtual reality (VR), where entire virtual reproductions of the real world are generated, in augmented reality the central idea is to add virtual objects (or overlay visual aids) into a real scene. Augmented scenes are conveyed to the user either in see-through head-mounted displays or in images captured by a camera. Computer vision algorithms have the key role of analyzing the images provided by the camera and solve the positioning and orientation problem of virtual objects within the real scene.
The development of augmented reality applications in manufacturing processes represents the evolution of the industry in the digital era. Starting from seventeenth century, during the first industrial revolution, mechanical production systems as a substitute of hand production methods appeared for the first time in manufacturing processes. During the second industrial revolution, large-scale manufacturing of machine tools together with improvements in transports lead the industry to mass production methods. More recently, the introduction of information technology in the industry revolutionized again the manufacturing processes by pervading them with the digital transformation. For instance, programmable controllers and computer-based software applications added another dimension to the control capabilities of machine tools. The last epochal evolution in manufacturing concerns the switch from the introduction of information technology to the integration of cyber-physical systems, which is characterized by a higher degree of design complexity.
In this context, characterized by a trend in developing ever more complex manufacturing systems, all maintenance processes become ever more challenging, also for experienced operators. The variety of adopted industrial equipment and machinery are becoming more functionally rich. Indeed, new industrial technologies are characterized often by a combination of advanced electronics and software, thereby requiring operators to possess an in-depth knowledge which can be built only after years of work in their field. Additionally, it should be also taken into account that the fast pace of change in technology-related areas requires constant updates for the technical personnel. Often, even the more experienced operator has to deal with new situations to solve technical issues in complex cyber-physical systems. As a result, operational and functional failures are more difficult and expensive to detect, troubleshoot, and repair.
The superimposition of virtual objects and hints onto the real world in real time
The possibility to take into account of the user’s viewpoint by means of a tracking system
The possibility to insert virtual objects in a real world scene by automatically calculating all their 6 degrees of freedom (DOFs) which control the objects’ location, orientation, and scale
Benefits of Using Augmented Reality in the Industry
Augmented reality gives the opportunity to make easier for industry workers to deal with specific industrial processes. Typical applications range from maintenance tasks, where specific and punctual information concerning cyber-physical systems are contextualized in real settings, to virtual engineering, where virtual prototypes constitute the basis for engineering tasks (Vilacoba et al. 2016).
The most common application concern the transformation of conventional manuals into digital instructions – showing operations to be performed at the right time in the right place – which may have the characteristics of being always updated thanks to connectivity, thereby providing additional documentation available on demand and representing an effective tool for guided troubleshooting (Ghimire et al. 2016). Based on the situation faced by an operator, a visualization system – either in the form of handheld devices (HHD) or head-mounted displays (HMD) – retrieve all necessary information and provides appropriate instructions back to the user through the augmented reality display and other components (e.g., audio). Instructions can be of arbitrary choice, from simple reproduction of technical manuals, design diagrams, schematics to contextualized 3D static models, virtual animations, short movies, images, and audio instructions. Instructions are superimposed to the field of view of maintenance operators, which can safely work on faulty equipment.
Augmented reality can be seen as a methodology to provide just-in-time training, by delivering only the information needed in a particular context while performing a specific task. Technicians exploit this information to operate on the task and are able to increment their prior personal knowledge in a lifelong learning process. This way, they are also allowed to deal with an increased range of maintenance tasks than before with less training, which translates in evident advantages from the point of view of a more cost effective allocation of the workforce in the industry. In fact, in this scenario, a single operator is able to handle several equipment and customers’ machineries, thus improving responsiveness and customer satisfaction.
It has been proven that augmented reality allows users to perform maintenance tasks with a lower rate of errors and faster than using paper-based instruction manuals (Lamberti et al. 2014, 2017; Henderson and Feiner 2011). This is due to an improved user experience driven by the fact that virtual tools can be directly shown in real working environments and real-time information is always available in the scene for the technician. It is worth recalling that faster intervention times at the factory floor reduces machinery downtime, thereby reducing loss of profits for an industry. Moreover, other key drivers for the adoption of these systems concern the reduction of technicians’ mental workload during operations, the possibility to count on technical assistance during maintenance, and the reduction in training time for new technicians.
Trends and Challenges
To maximize the effectiveness and usability of augmented reality interfaces, their design should seamlessly integrate computer graphics with real environments by avoiding the use of encumbering technologies. Today, the majority of mobile augmented reality systems are based on head-mounted displays (HMDs) and see-through devices. HMDs involve the use of complex equipment to be worn by the user (e.g., the Oculus Rift). See-through systems became popular due to the rapid prototyping possibilities offered by several software libraries (e.g., ARToolkit). However, a current bottleneck is constituted by the use of transparent display technology, which is not mature yet.
Projecting virtual models onto objects in the real world
Integrating virtual models into camera views obtained from video cameras
Projecting virtual components onto the user’s retina
Visualizing the virtual components through a semitransparent display while observing the real world
In industrial scenarios, the last two options should be preferred since they provide unhindered views while remaining safe and secure. In fact, portable and unobtrusive devices like see-through glasses guarantee hands-free operations by repair technicians.
Virtual retinal displays (VRDs) have been also recently introduced to overlay diagnostics and repair instructions directly onto the view of maintenance operators.
Whatever the technology used to visualize the augmented components, one of the key problems in augmented reality is the determination of the position and orientation of a camera in 3D space, known as camera pose estimation problem. It should be considered that the level of accuracy needed in pose estimation in augmented reality applications for maintenance is, in general, more demanding than in other fields. Indeed, small changes in a pose can be the cause of huge deviations in the visualization of the virtual content (Lakshmprabha et al. 2015).
The majority of augmented reality applications are based on the use of markers (like QR codes) to solve the real-time camera pose estimation problem (Antonijević et al. 2016). Marker-based techniques are particularly suitable for stationary objects. They require to modify the aspect of the real world manufacturing scenario by introducing specific markers on the top of machines or within the real environment to correctly locate virtual assets. Therefore, often the use of markers is perceived as a limitation of augmented reality (De Crescenzio et al. 2011). Marker-based techniques require that illumination is sufficient and markers are always visible.
On the other hand, marker-less techniques permit both to track moving parts and avoid the use of external markers to be placed in the environment (Lamberti et al. 2017). Marker-less techniques are based on image processing algorithms aimed to detect image features and perform 3D reconstruction. The process involves different phases, namely feature extraction, features detection, features matching, and image registration. The most promising algorithms used for feature extraction are Scale Invariant Feature Transform (SIFT) and Speed-Up Robust Features (SURF). Features detection algorithms apply classification mechanism like Support Vector Machine (SVM), K-Nearest Neighbor (KNN), and Sparse Representation-based Classification (SRC). Features matching, which consists in finding the correspondence between two set of features, can be achieved by applying the well-known k-d Tree algorithm. Finally, Image Registration concerns the calculation of the geometrical transform that aligns the featured points of two images of the same object. A suitable methodology is provided by random sample consensus (RANSAC) (Dandachi et al. 2015).
Given the overview of the algorithms used in augmented reality, a possible bottleneck concerns the need of sophisticated processing on devices that should be portable. The challenge is to develop solutions capable to take into account the currently achievable speed and accuracy. In fact, one of the issues with today’s augmented reality applications is the lack of accuracy in 3D resolution (Vilacoba et al. 2016), which translates into a problem in precise positioning of virtual components.
Additional challenges are raised by the interaction paradigm needed for using the application effectively and in natural way. Though many augmented reality applications are developed just as an informative means (e.g., furniture placement in interior design), thus letting the interaction through touch-based screen devices be suitable for the specific task, the use cases which can be faced in manufacturing and industrial scenarios have different requirements in terms of interaction. In fact, technicians should carry out hands-free tasks. This requirement makes troublesome the use of input devices based on hand interaction, e.g., touch- and multitouch-based. A possible solution is related to the use of speech recognition technology, today enough mature to ensure an intuitive and natural user experience. Recent proposals in the field encompass also the possibility of automatically generating icon-based interfaces to associate unconstrained voice commands issued by the user to application functionalities based on the semantic similarity with the evoked command (Lamberti et al. 2017). However, it should be considered that speech recognition technology is robust in controlled situations, i.e., non-noisy environments. Thus, the noise at the factory floor should be taken into account in the design of an augmented reality solution. Gaze-based communication represents a possible alternative solution to the problem of hands-free interaction. Gaze recognition provides a thoughtful and natural interaction mode. On the other hand, special purpose hardware (i.e., infrared cameras) is needed to realize a robust gaze-based interface, which can be in competition with the hardware used for augmented reality (e.g., HMD and see-through glasses). Other possible solutions concern the use of gesture/pose recognition technologies, where the user could interact with the application interface both by using hand gestures/static poses (e.g., recognized through consumer devices such as the Leap Motion and Microsoft Kinect) and wearable solutions (e.g., inertial-based trackers) Henderson and Feiner (2010). However, in all these latter cases, the technician should suspend the maintenance procedure to interact with the application.
Conclusion and Discussion
Augmented reality domain can be applied in different application scenarios, encompassing gaming, entertainment, cultural heritage, tourism, construction, maintenance, education, health care, navigation, and military. Among them, industrial maintenance and repair represents a strategic thread for the application of augmented reality-related technologies, due to the economic revenues resulting from the possible reduction in time required for maintenance of complex systems and the automation of training processes. This entry described the benefits of using augmented reality technologies in the maintenance and repair field. Open issues are also discussed, mainly focusing on technological problems that currently limit a further spread of augmented reality solutions.
- Antonijević, M., Sučić, S., Keserica, H.: Augmented reality for substation automation by utilizing IEC 61850 communication. In: 39th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), Opatija, pp. 316–320 (2016)Google Scholar
- Dandachi, G., Assoum, A., Elhassan, B., Dornaika, F.: Machine learning schemes in augmented reality for features detection. In: Fifth International Conference on Digital Information and Communication Technology and its Applications (DICTAP), Beirut, pp. 101–105 (2015)Google Scholar
- Ghimire, R., Pattipati, K.R., Luh, P.B.: Fault diagnosis and augmented reality-based troubleshooting of HVAC systems. In: IEEE AUTOTESTCON, Anaheim, pp. 1–10 (2016)Google Scholar
- Henderson, S., Feiner, S.: Opportunistic tangible user interfaces for augmented reality. IEEE Trans. Vis. Comput. Graph. 16(1), 4–16 (2010)Google Scholar
- Henderson, S., Feiner, S.: Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Trans. Vis. Comput. Graph. 17(10), 1355–1368 (2011).Google Scholar
- Lakshmprabha, N.S., Kasderidis, S., Mousouliotis, P., Petrou, L., Beltramello, O.: Augmented reality for maintenance application on a mobile platform. In: IEEE Virtual Reality (VR), Arles, pp. 355–356 (2015)Google Scholar
- Lamberti, F., Manuri, F., Sanna, A., Paravati, G., Pezzolla, P., Montuschi, P.: Challenges, opportunities, and future trends of emerging techniques for augmented reality-based maintenance. IEEE Trans. Emerg. Topics Comput. 2(4), 411–421 (2014).Google Scholar
- Lamberti, F., Manuri, F., Paravati, G., Piumatti, G., Sanna, A.: Using semantics to automatically generate speech interfaces for wearable virtual and augmented reality applications. IEEE Trans. Hum. Mach. Sys. 47(1), 152–164 (2017).Google Scholar
- Milgram, P., Kishino, F.: A taxonomy of mixed reality visual displays. IEICE Trans. Inf. Syst. E77-D(12), 1321–1329 (1994)Google Scholar
- Navab, N.: Developing killer apps for industrial augmented reality. IEEE Comput. Graph. Appl. 24(3), 16–20 (2004).Google Scholar
- Vilacoba, D., Trujillo, M.Á., Viñuales, A., Weber, P.: Press dedicated machine show-room, a direct application of augmented reality in industry. An industrial Augmented reality experience. In: IEEE International Conference on Industrial Technology (ICIT), Taipei, pp. 1990–1995 (2016)Google Scholar