With the increasing requirements on machine tools such as higher complexity, individuality and digitization, their maintenance becomes also more complex, which increases time consumption and demands special knowledge. The maintenance engineer can be supported in this process using the Augmented Reality (AR) technology, mobile devices and suitable production data. This requires the integration and adaptation of the maintenance documentation as well as further production systems and their data such as the CAD system, product data management (PDM) system, manufacturing execution system (MES) and CNC. For this purpose, a method is developed that allows an efficient integration process of various production data into an AR supported maintenance documentation. This includes concepts for the integration process of different production systems and data into an AR maintenance system as well as the design of the AR maintenance system architecture. Basis is the analysis of the required production and AR systems, data and processes. The aim of the method is to reduce the high expenditure of the overall planning process and the necessary expertise of the maintenance planner of the AR technology. The developed method is verified with different maintenance scenarios, production systems and data as well as machine tools and AR devices.
The high complexity and individuality of machine tools and their maintenance lead to more difficult tasks for the maintenance engineer and require specialist knowledge about the particular machine tool. The result is a higher potential for mistakes of the maintenance engineer, increasing maintenance time, production downtime and consequently increasing production costs .
By digitization in production and deployment of new technologies such as Augmented Reality (AR) and mobile devices, the maintenance engineer and processes can be supported . AR offers new ways for human-machine interactions by superimposing the real environment with virtual information in the right location in the user’s field of view . With suitable AR mobile devices and a digital maintenance documentation, the maintenance engineer can use interactive step-by-step instructions to reduce the complexity of the process and sources of error can be shown directly on the machine tool based on current production data to improve the maintenance process .
Problem of the production data integration process
The basis for the maintenance work is the maintenance documentation , containing a detailed description of the machine tool, user instructions for manual tasks and pursuing documents, photos and graphics. The maintenance documentation is often delivered on paper that can be time-consuming, out-of-date, not action-oriented and unfavorably structured for the maintenance engineer . The created digital form in a document has not the appropriate data structure for using in an interactive AR application. Additionally, data from CAD systems and product data management (PDM) systems are required or can expand the AR documentation but has to be integrated with an efficient process. Furthermore, data from diagnosis, condition monitoring and further production systems, e.g. a manufacturing execution system (MES) as well as CNC data, can also support the maintenance engineer by detecting defects during the maintenance. Therefore, interfaces and visualization elements have to be available to avoid much programming effort.
On the one hand, AR has the potential to improve the maintenance process by supporting the engineer , on the other hand the integration process of various production data and the creation of a new digital AR maintenance documentation has a high expenditure with many different data sources and data formats and needs a high level of AR expertise that cannot be covered by the employees. Therefore, an efficient integration process is required that includes various AR components (e.g. camera image processing, tracking of the environment, visualization, interaction) and production systems to support the maintenance planner.
State of the art of Augmented Reality for maintenance of machine tools
In recent years, several explorations were carried out to support the maintenance engineer with AR based instructions using 3D models for manual working steps, e.g. a gearbox maintenance for milling machines . A product disassembly process for maintenance without expert intervention is developed in . For this purpose, a disassembly sequence table is generated based on the product ontology and linked with virtual information. Moreover, AR systems were created that establish a connection to the CNC to use online data [8, 9]. This allows the visualization of all CNC values for monitoring, error detection, process control, simulations and movement of 3D models.
For the integration of AR required data various approaches were explored [10, 11]. Besides the video recording of real workflows, programming and manual addition of 3D models and instruction texts, the usage of an authoring application is a common concept. This allows the user to append and manage the maintenance data in a special application that creates the AR based data structure. In  an AR system for assembly instructions is described that integrates a special process planning application to create the assembly sequence and include CAD data. Furthermore, the AR assembly system described in  additionally imports PDM data to the authoring process.
As a basis for maintenance AR applications, general AR programming libraries and frameworks provide the opportunity to use several AR components. However, they need high-level expertise in AR, enhancements and programming for the maintenance requirements . Existing AR systems for maintenance, which are using authoring applications, require additional software in the planning process. Furthermore, they cover only special elements, cannot map the required complexity or require a high level of expenditure for the integration process. Neither sufficient automated processing of the data, nor specialized efficient workflows for the production exist. Moreover, it is not sufficient to integrate existing maintenance documentations and comprehensive data sources.
Figure 1 illustrates a typical current integration process of AR based maintenance data. In this case, a special authoring system lies in the center of the process where the maintenance documentation and all additional data is entered and managed. The necessary 3D model from the CAD system has to be prepared separately in an application for digital content creation (DCC). This process is characterized by a high manual expenditure and new software systems, data formats and workflows that have to be used by the maintenance planner.
The authors’ preliminary work describes the development of the AR framework ARViewer, which is specialized for production usage . It includes various AR components and interfaces for different use cases and is designed for including external systems such as CNCs  and PDMs  as well as different devices such as tablet computers and head-mounted displays (HMDs) . It provides the basis for realizing a new method for the efficient integration of production data into the AR based maintenance of machine tools.
Aim and approach
Aim of the work
The aim is the design of a new method for the efficient integration of production data into the AR based maintenance of machine tools by developing a new creation process for the AR based documentation including different production systems as well as a concept for the overall AR maintenance system architecture. With this method the high expenditure of the AR planning process and the necessary AR expertise of the maintenance planner should be reduced.
Figure 2 illustrates the aim for the new integration process of AR based maintenance data. The center of this process comprises software systems that are already in use. This includes CAD, documentation and PDM systems that are used to create and compose the AR based maintenance data. Existing production data and available AR components can be applied for different maintenance use cases. In addition, CNC and MES should be integrated for online data support. The target process is characterized by a high level of automation, a continuous data model and an industry-oriented workflow.
Approach of the work
To achieve the aim of the work, the state of the art of current AR concepts for the maintenance of machine tools was researched. The next step is the analysis of the required AR based maintenance process and data as well as the creation process for the AR based documentation as a basis for the design of the new method. Subsequently, the new method for the efficient integration process of production data into the AR system is developed and a design of the overall AR maintenance system architecture is created. Finally, the developed method is verified with a sample implementation for different maintenance scenarios and AR devices.
Analysis of the AR based maintenance
Definition of the AR based maintenance process
For developing the method of an efficient integration process of production data, the AR based maintenance process as a basis has to be defined and is shown in Fig. 3. First step is the identification of the specific machine tool by tracking an attached label with the AR device. Using the tracked number, the connection to all required external systems is established. Now all maintenance tasks are available for the maintenance engineer. By selecting the desired task, all data is visualized step by step and the maintenance engineer executes the task. The data includes the prepared maintenance documentation for AR with superimposed 3D models as well as live data of connected production systems. This is repeated until the maintenance of the machine tool is completed. Finally, the results are stored in the system for subsequent evaluation. Different AR devices, e.g. tablet computers and HMDs, with different options for visualization and interaction should be supported for the maintenance process.
Required AR based maintenance data
Table 1 summarizes the data required for the AR system and the maintenance engineer for the specified AR based maintenance process shown in Fig. 3. The method should consider various AR devices, maintenance scenarios and production systems. Therefore, different properties of AR systems, interfaces to online data sources from external systems and the maintenance documentation with manual tasks have to be included in the integration concept. The goal is to integrate and use all data in one single AR system to cover various maintenance scenarios and to minimize the integration expenditure for the planning process. The basis for the concept is the increasing level of digitization in production today and the available production systems and data.
Requirements to the creation process of the AR based documentation
The requirements to the AR based maintenance process and data that are defined in the last sections have to be considered to the creation process of the AR based maintenance documentation. That implies that many different existing data types and sources have to be integrated to create an efficient and automated process without much manual effort. That includes the usage of 3D models of the CAD system without a manual converting process, the usage of the existing documentation system without creating the instructions twice for the AR system as well as the usage of interfaces to further external systems such as PDM, MES and CNC to extend the AR based documentation with enhanced information. Additionally, the requirements to the AR technology shown in Table 1 have to be considered without extensive expertise of AR or programming of the maintenance planner.
Method for the efficient integration process of production data
Integration process of production data into the AR system
The industrial process of maintenance planning has to be adapted to the AR technology. The goal is a continuous process of data linkage from the first planning steps up to the real maintenance of the machine tool. This leads to a continuous data model and continuous engineering to minimize the expenditure for development and service. By using the AR technology, the maintenance data is visualized in the right location on the machine tool. For this purpose, a 3D model of the machine tool is necessary and the existing CAD system and CAD model is used as a starting point of the method.
The following fundamental properties of the method are defined to achieve the goal of high efficiency of the overall integration process of production data:
Usage of existing source systems All maintenance data is created with software systems that are already available in the production and the user is familiar with, e.g. CAD, PDM and documentation systems. An additional authoring application is not used. Therefore, the current maintenance planning process is adapted by using these systems in the entire process.
Usage of existing production data All the production data already available in the production systems is used to avoid the creation expenditure of additional AR required data, e.g. CAD models, maintenance documentation and PDM data. Furthermore, MES and CNC are directly connected via defined interfaces for the enhancement with online data.
Creation of a user-oriented workflow A fast and short process is realized by using existing production systems and data as well as templates for new information (e.g. maintenance labels in the CAD system) and automation of particular steps.
Automation of manual processes To reduce the integration expenditure, time-consuming manual processes are automated, e.g. the necessary polygon reduction and format conversion of CAD data and the adaptation of the AR application and user interface to the current maintenance scenario.
Creation of uniform interfaces The application is usable with different AR devices, maintenance scenarios, data sources and machine tools. By providing various interfaces, a specialized customization by the maintenance planner is avoided.
Providing various AR components To cover different use cases, simplify the integration complexity and avoid keen knowledge about AR and programming, various developed AR components are used and configured.
For realizing the presented six fundamental properties and the goals shown in Sect. 3 and Fig. 2, Fig. 4 illustrates the sequence of the new method for the efficient integration process of the production data into the AR based maintenance. Initial point is the CAD model of the machine tool whose components have to be restructured for highlighting or moving special components depending on the maintenance use case. Furthermore, special visual labels have to be appended for highlighting the maintenance tasks and AR tracking purposes. An automated polygon reduction and format conversion is processed after the export of the 3D model to eliminate manual work.
The second step is creating the maintenance documentation with the technical documentation system. With these systems the information can be entered by the maintenance planner with graphical elements in a user-friendly way but are stored in a structured form commonly in an xml  based format. The data structure is divided into the main maintenance tasks that consist of many single steps that have detailed information attached such as descriptions of the manual work, materials and tools. Additionally, the external data e.g. 3D models, PDM or CNC data sets are attached by their names. With these technical documentation systems and the structured data format, the documentation can be used for different output mediums such as digital documents, paper documents and AR data. Additionally, pursuing documents, photos and graphics can be included for detailed information.
All created and required maintenance data is stored and managed in a systematical structure in the PDM system. Additional PDM data as well as configurations for AR components, MES and CNCs are also stored in the PDM system and linked with the maintenance documentation.
For the maintenance process, the PDM system deploys all required data via network to the AR device. This device processes an automatic adaptation of the user interface based on the current maintenance use case and deposited device properties (see Table 1) by calculating graphic elements and activating interaction methods. For verifying the results, it is possible to obtain a preview of the AR application during the planning process.
With the described new method, the high expenditure with many manual steps, different data sources and data formats is reduced by using existing systems, data and automated processes. With the user-oriented workflow, provided interfaces to external systems and AR templates the complexity of the process and the necessary AR expertise is reduced.
Design of the AR maintenance system architecture
For realizing the new method defined in Sect. 5.1 and Fig. 4, a concept for the AR maintenance system architecture is designed (see Fig. 5). It consists of five main AR application components. The data management component integrates the existing maintenance data from the production systems. This includes the created maintenance documentation from the PDM system as well as online data from MES and CNCs. The camera image component is required for tracking the machine tool and for the visualization for the user. Therefore, different camera interfaces are available for different AR devices. With the tracking component the maintenance information is visualized in the right location by calculating a 3D coordinate system based on the 3D model. For different environments and machine tools different tracking methods are available. Through the interaction component, the user selects and enters information. The interaction methods are depending on the specific AR device, e.g. with a touchscreen on a tablet computer or gestures with HMDs. The visualization component shows the appropriate 3D models as well as graphical elements such as texts, photos and graphics of the current maintenance documentation step. Additionally, it adapts these graphical elements to the different requirements on different devices.
In the presented concept, the AR application is directly connected to the source systems of the maintenance documentation and integrated into the production infrastructure - without using an additional authoring system. The PDM system is used as the central system for the prepared maintenance instructions. In the PDM system, all created maintenance data from the CAD system, documentation system and pursuing documents is stored, managed and provided via network. Online data from the machine tool and production is used directly via defined interfaces to the CNC and MES, e.g. for diagnosis and condition monitoring. The interfaces are designed with standard technologies and can connect different types of systems to reduce the expenditure of customization.
The presented process and concept can be implemented with different software systems, so that they are customizable for different requirements and companies. Examples are described in the verification Sect. 6 and details about individual components of the concept are available in . With the presented concept of the AR maintenance system architecture and its interfaces, various data sources and data formats can be integrated without a high expenditure and changes of the AR application.
Verification of the developed method
Aim and approach of the verification
The developed method for an efficient integration process of production data into an AR maintenance system and the design of the AR maintenance system architecture are verified with several maintenance scenarios of real industrial environments to demonstrate the feasibility, viability and advantages of the method. Therefore, the developed AR application, different machine tools and AR devices are used. Basically, the particular components of the method can be implemented with different systems depending on the company’s requirements. For the verification, each component is exemplary implemented and tested for the specific maintenance scenario. Starting with the CAD model the described maintenance planning process is executed and the result is visualized on the example machine tools. Finally, an analysis of the executed planning process and conclusions about the created method are discussed.
Sample implementation of the developed method
The developed method was verified on different maintenance scenarios including a CNC turning and milling center  and a wire drawing machine . Extracts of their paper-based maintenance documentations and CAD models were used as basis for the AR based documentation. An example use case is the maintenance of the slideway wipers of the CNC turning and milling center including the steps disassembly, inspection, cleaning and reassembly. The task requires carefully cleaning of all relevant parts in the machine, without omitting any working steps. Target AR devices, to assist the maintenance engineer, are industrial tablet computers  and HMDs . The implemented components of the AR application shown in Fig. 5 are based on the AR framework ARViewer (see Sect. 2 and ).
The method was applied as shown in Fig. 4, considering the defined six fundamental properties (see Sect. 5.1). The structure in Fig. 6 illustrates the implemented setup with the included systems. The maintenance planning is realized with the defined standard production applications at the office workspace (a) not requiring an additional authoring system. The created maintenance documentation is stored and managed on a server with the PDM system (b). On the shop floor, the maintenance engineer accesses the data from the PDM server and the online production systems (CNC, MES) using the AR maintenance application on a tablet computer (c).
For the exemplary CNC turning and milling center, the 3D model was designed with the CAD application Autodesk Inventor . To move the machine axes and to show 3D model components in correct position in the AR application, assemblies had to be restructured in the model hierarchy. The corresponding assemblies themselves were used as markings in the AR view. Markers for tracking were added to the machine as predefined assemblies with a defined name. The open file format Wavefront obj  was used for exchange with the AR application. It offers various options for the resolution, whereby a small number of polygons is sufficient for the accuracy of the AR visualization.
In the next step, all other required data was arranged, such as photos of real machine components (e.g. jpg file format) and additional documents with detailed descriptions and tables (e.g. pdf file format). In order to gain access to the data of the MES and CNC during the maintenance, a configuration file was created in the xml file format  with entries for the connection data and required data records of the machine (only required once per machine). Next, the individual maintenance steps were created with the provided data. For this purpose, the technical documentation system XMLmind XML Editor  was used as an example, whereby the data is stored in a structured xml based form for the system requirements but is created in a user-friendly form with a clear graphical user interface. The used standard DocBook  specifies, checks and stores specifications for the structure in an xml file .
The created maintenance data and described systems are integrated in a structured form in the PDM system Siemens Teamcenter . With the existing API (application programming interface) of the PDM system and its client-server architecture, the current maintenance data can be accessed from any device and location via network, which is used by the AR maintenance application. The papers [12, 13] and  describe more details about the usage, interfaces and implementation for the source production systems (CAD, PDM, documentation system, CNC, MES).
At the machine tool, the maintenance engineer receives the created step-by-step instructions including 3D models and online data for the maintenance tasks, so that all information is available on a single device without a separate handling of external systems. Thereby, the online systems are used to support the maintenance engineer with the latest machine information and to detect problems on the machine tool. The user interface is adjusted automatically with different visualization elements based on the current working step data. For the different AR devices, e.g. with different display sizes and interaction devices, separate approaches for visualization and user interaction were implemented to create the best benefit.
Evaluation of the sample implementation
The application of the new developed method in the various maintenance scenarios demonstrates their applicability for the maintenance of machine tools. The goal of the efficient process of integrating production data into AR based maintenance applications (see Fig. 2) is achieved by realizing the defined properties of the method described in Sect. 5.1:
By using the existing source systems (CAD, PDM, documentation system) the maintenance planner works with well-known software applications, needs no induction to new systems and uses its knowledge and experience.
By using the existing production data, less data has to be created, has not to be manually transferred to a new system and fast updates are executed since no intermediate system is required.
A user-oriented and short workflow is created by the integration of available production systems and data, the automation of processes and the usage templates for maintenance instructions.
The automation of the process is applied by the polygon reduction and format conversion of the 3D models, the adaption of the AR user interface and by connecting and using available production systems and online data. Thus, for the creation process no programming is required.
By implementing uniform interfaces, the AR application is used with different maintenance cases, data sources and machine tools.
By providing various AR components, the application is used with different AR devices appropriate to the current use case, e.g. with different visualization and interaction components for tablet computers and HMDs.
A summarized comparison of the typical previous and the new developed creation process is shown in Table 2. This shows that the effort in the planning process, the required expertise of the maintenance planner and the overall complexity is significantly reduced.
Especially the use of the existing standard documentation system and the integration of external systems for generating the AR based documentation as well as the automatic update, if changes of the documentation are necessary, can reduce the additionally time for the AR based documentation by half, because all texts, pictures and the data structure have to be created only once. The 3D model components have to be integrated additionally, but the effort is reduced by automated processes. For complex machine tools with text-based documentations over a hundred pages, the savings can be multiple days. In a company with many different machine tools, the expenses can be reduced accordingly.
The presented work describes a new developed method for an efficient creation and integration process of the maintenance documentation and production data into AR based maintenance applications for machine tools, thus reducing the high expenditure of the process and the necessary expertise of the maintenance planner for the AR technology. As basis for the development of the new method, an analysis of the AR based maintenance process and required data as well as user and technology requirements was processed. The result is an improved planning and integration process for required production data and systems that is user-oriented and partly automated. New maintenance instructions are created in a structured form and in an action-oriented manner with existing standard applications. Furthermore, existing production data can be included through uniform interfaces. Updates of the maintenance data are processed fast because no additional software and manual working steps are required. Besides the proposed creation and integration process, a design of an AR based maintenance software architecture with several AR components was created to realize the verification of the method. The implementation and verification of the new method in various maintenance scenarios demonstrates their applicability for different AR devices and machine tools.
Duffuaa SO, Raouf A (2015) Planning and control of maintenance systems - modelling and analysis. Springer, Berlin
Henderson S, Feiner S (2011) Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Trans Vis Comput Graph 17(10):1355–1368
Azuma R, Behringer R, Feiner S, Julier S, Macintyre B (2001) Recent advances in augmented reality. IEEE Comput Graph Appl 21(6):34–47
VDI - The Association of German Engineers (2006) VDI guideline 4500 sheet 1: technical documentation - definitions and legal basics, Beuth, Düsseldorf
Abramovici M, Krebs A, Schindler T (2013) Design for usability by ubiquitous product documentation. Smart product engineering. Springer, Berlin, pp 633–641
Tianren W, Yue L, Yongtian W (2008) Infrared marker based augmented reality system for equipment maintenance. In: Proceedings - international conference on computer science and software engineering, CSSE 2008, vol 5, pp 816–819
Chang MML, Ong SK, Nee AYC (2017) AR-guided product disassembly for maintenance and remanufacturing. Proc CIRP 61:299–304
Zhang J, Ong SK, Nee AYC (2010) Development of an AR system achieving in situ machining simulation on a 3-axis CNC machine. Comput Anim Virtual Worlds 21(2):103–115
Liu C, Cao S, Tse W, Xu X (2017) Augmented reality-assisted intelligent window for cyber-physical machine tools. J Manuf Syst 44:280–286
Wang J, Zeng C, Liu Y, Li S (2013) Integrated content authoring for augmented reality based product manual assembly process instruction. In: 43rd International conference on computers and industrial engineering, pp 16–18
Sääski J, Salonen T, Hakkarainen M, Siltanen S, Woodward C, Lempiäinen J (2008) Integration of design and assembly using augmented reality. Int Fed Inf Process 260:395–404
Klimant P, Kollatsch C, Schumann M (2017) Augmented reality solutions in mechanical engineering. In: Proceedings of the ASME 2017 12th international manufacturing science and engineering conference, MSEC 2017, Los Angeles, vol 3: manufacturing equipment and systems
Kollatsch C, Schumann M, Klimant P, Wittstock V, Putz M (2014) Mobile augmented reality based monitoring of assembly lines. In: 5th CATS 2014–CIRP conference on assembly systems and technologies, Dresden, Procedia - CIRP vol 23, pp 246–251
Kollatsch C, Schumann M, Winkler, S, Klimant, P (2018) Integration of product data management systems in Augmented Reality maintenance applications for machine tools. In: IEEE COMSOC MMTC communications - frontiers, special issue on holographic communications and distributed collaborations 12/2:22–31
Vogel U, Herold R, Richter B, Baumgarten J, Fehse K, Leo K, Sieler M (2011) Interactive see-through augmented-reality smart display system. SID Symp Dig Tech Pap 42(1):143–146
Ray ET (2003) Learning XML: creating self-describing data. O’Reilly Media Inc, Newton
Niles Simmons Industrieanlagen GmbH: Turn & Mill Center N20 MC. https://niles-simmons.de/en/products/cnc-turning-centres-mc-series/n20-mc, 24.09.2020
Kieselstein International GmbH: machines and plants. https://www.kieselstein.com/en/Machines-Plants_1090.html. Accessed 24 Sep 2020
Dell: dell latitude rugged. https://www.dell.com/en-uk/work/shop/laptop-computers-for-businesses/latitude-7220-build-your-own/spd/latitude-12-7220-tablet/xctol7220emea. Accessed 24 Sep 2020
Herold R (2011) Ein Beitrag zur Realisierung von Systemarchitekturen für Head-Mounted Displays auf Basis bidirektionaler OLED-Mikrodisplays. Shaker, Waterford
Autodesk: inventor. https://www.autodesk.com/products/inventor/overview. Accessed 18 Jan 2021
Martinreddy.net: object files. http://www.martinreddy.net/gfx/3d/OBJ.spec. Accessed 18 Jan 2021
Becher M (2009) XML: DTD, XML-Schema, XPath, XQuery, XSLT, XSL-FO, SAX, DOM. W3L GmbH, Herdecke/Witten
XMLmind Software: XMLmind XML Editor. https://www.xmlmind.com/xmleditor. Accessed 18 Jan 2021
Walsh N, Hamilton RL (2010) DocBook 5: the definitive guide. O’Reilly Media Inc, Newton
Siemens: Siemens PLM. https://www.plm.automation.siemens.com/global/de/products/teamcenter/. Accessed 18 Jan 2021
The project “AVARE - ServAR - Augmented Reality-based System for Assistance in Commissioning and Maintenance of Machine Tools” is supported by the Federal Ministry for Economic Affairs and Energy on the basis of a decision by the German Bundestag.
Conflict of interest
The authors declare that they have no conflict of interest statement.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Kollatsch, C., Klimant, P. Efficient integration process of production data into Augmented Reality based maintenance of machine tools. Prod. Eng. Res. Devel. (2021). https://doi.org/10.1007/s11740-021-01026-6
- Design method
- Integration process
- Production data
- Augmented Reality
- Machine tool