Creation of Interactive Virtual Reality Scenarios as a Training and Education Tool
- 640 Downloads
A novel approach to the creation of realistic training scenarios for safety-critical industrial applications is presented. This is based on virtual reality techniques but extended by the incorporation of a range of options for interactivity, which permit the trainee to take actions in realistic ways in the simulated environment. These actions may include potentially dangerous errors, with realistic consequences simulated, but in complete safety. Any real environment may be simulated both visually and functionally in the virtual environment. An innovative feature is the use of virtual buttons displayed on the image of the user’s hand, thus avoiding the need for accessories such as haptic gloves. The system enables trainee specialists to gain realistic operational experience without the anxieties of causing damage in a real environment, but it is also relevant to a wide range of applications where rich interactivity is needed.
KeywordsInteractivity Learning Safety-critical Scenarios Training Virtual reality
It is probable that the majority of uses of modern Virtual Reality (VR) technologies are for entertainment. However, VR technologies also routinely allow people working in safety-critical industries to undergo training in a realistic environment, without the risk of hazards such as injury or equipment damage. Electrical power distribution is an example of a major safety-critical industry and a need arose to create a virtual training simulator for power substation operators, who have to be ready to react rapidly, correctly and safely to a wide range of routine and emergency situations. The work presented here discusses the development of an innovative VR training system for such cases, but the principles established could be used in entertainment or creative applications as well.
The target need arose from an approach by the Russian power industry, for which over 30% of faults and blackouts have historically been caused by errors during switching  and hence the improvement of operators’ training is a priority: it is probable that the situations in other industries and in other industrialized countries will show significant similarities.
19.2 Limitations of Current Two-Dimensional Simulators
Many simulators exist for training of industrial personnel, but they mainly rely on two-dimensional screen displays. There is no doubt that the underlying algorithms are sophisticated; for the power substation operator case they typically include: control of simulated operation order with both routine and non-routine events; estimation and recording of the operators’ decisions made during such simulated events; rapid evaluation of the parameters of the modelled system.
Against this, such simulators have deficiencies, such as: a two-dimensional display of the main control room and substation equipment does not give a realistic scenario to develop skills; routine normal operations tend to be perfunctory and without deep insight for the trainees; navigation around the installation is not a realistic representation of the real situation.
Given the powerful impact of three-dimensional simulations now routinely available for entertainment applications, it was felt that the development of such technology could deliver a major improvement in training simulators. A VR simulator for industrial operatives would be vastly more meaningful, although there would need to be significant improvements in the quality of interaction, in comparison with entertainment-oriented VR devices.
19.3 Three-Dimensional Approaches to Scenario Simulation
19.3.1 3D Technologies
There are two main three-dimensional training technologies: 3D Helmet and 3D CAVE (cave automatic virtual environment). Both of them can be used as forms of VR simulator and both deliver complete immersion in the virtual environment. However, the CAVE demands a relatively large amount of space: it leaves the user unfettered by a headset, but nonetheless has significant deficiencies of realism: in particular, the user only sees a 2D projected image, albeit on a surrounding wall. Thus, if the space is limited or true 3D is essential, the 3D Helmet technology is preferable.
The 3D CAVE consists of a cube-shaped space, normally with a volume of at least 8 m3, with display screens surrounding the user on all sides . The user may work with virtual objects with hands or a joystick, but it is difficult to display these realistically since the surrounding screens are basically two-dimensional; further, the user’s mobility is limited by the cube perimeter. For 3D Helmet VR technology, two small displays are mounted in front of the user’s eyes so that they can view the simulated environment . The user then has the opportunity to work with credible 3D representations of virtual objects, either with their hands or a joystick.
19.3.2 Modelling Requirements for Appropriate Environments
A primary need is a 3D model of the environment: this must have a good degree of verisimilitude in both its visual and spatial similarity to the real thing, but it must also have functional similarity in the operation of relevant switches, indicators, etc. The verisimilitude of a 3D model leads the user (operator) to develop the skills of operation of the selected environment during the training, such that after completion they will not need to adapt to the real environment, due to the realism of the VR interaction.
Spatial analogue representations of objects and the environment are very important: operators need to know the time needed to move from one point to another in the space concerned and this can be critical in emergency scenarios. Further, the need for functional similarity in the 3D model is paramount and the model must react to the user’s actions in the same way as in the real environment.
19.3.3 Scenarios of Training
The scenarios of training have to include both regular and emergency cases: regular operations refine the skill of routine switching and understanding of the basic operation on the station. Emergency scenarios need to start in the same way as regular training but then there will be the insertion of an emergency at a time and of a nature that must be unpredictable to the trainee, who then has to take correct decisions as fast as possible .
The user can choose between training mode and test mode. If they choose training mode, they can select either emergency or regular training and the type of scenario. In test mode, the user does not know what type of training scenario they will encounter: it will be a random choice of the simulator.
19.4 Implementation of Virtual Reality Simulator
A substation of typical Russian design was chosen to be simulated. This contained: six 110 kV overhead line connections; six 35 kV overhead lines and one 35 kV cable line connection; thirty 10 kV cable line connections; outdoor switchgear for 110 kV; outdoor switchgear for 35 kV; indoor switchgear for 10 kV; three three-phase transformers. The area of the prototype was about 18,000 m2. This diversity of equipment enables many scenarios to be implemented.
The digitalization of the prototype has three main stages, as follows : acquisition of a cloud of object points for the prototype hardware; mesh calculation and optimization; texture calculation and optimization. The object point cloud becomes a skeleton model in the virtual space: the surfaces of the prototype objects are then given appropriate textures and the whole 3D model is optimized.
Creation of training scenarios is a specialized issue dependent on the application. For the substation operator case, the regular scenarios included taking out of, and putting into, service the following: transformers; circuit-breakers; disconnectors; overhead lines. For the emergency cases, the following scenarios were chosen. 1. Taking circuit-breakers out of service due to: emergency powering off of transformer; emergency voltage transformer fuse failure; emergency current transformer explosion. 2. Taking an overhead line out of service due to emergency collapse of column insulation of disconnector. 3. Taking a sulfur hexafluoride circuit-breaker out of service due to emergency gas depressurising. 4. Bringing a disconnector into service due to an emergency earth fault on an overhead line. The totality of these scenarios covers about 90% of typical emergency operations in a substation.
19.5 Technical Description of the Simulator
The game engine part of the simulator is based on Unreal Engine 4  (Epic Games, 2014). A software model of the chosen prototype was written in the C language, enabling the latency to be reduced to less than 10 ms, which is essential for a convincing experience. The VR headset used an umbilical cable to link to the host computer because there is no current wireless link technology that can transmit two channels of HD video plus three channels of gyroscopic information and a USB-3 channel (for the forward-looking camera—see below) with acceptable latency.
To create a functioning simulator, it is necessary to combine the technical functions of the modelled system (the simulator logical part) with the capabilities of modern 3-D shells (graphic engines). The integration of these two subsystems is discussed in Sect. 19.6.1.
19.6 Software Implementations
19.6.1 Functional Simulations and Graphics Engine
In principle, the system consists of two subsystems that constantly interact and exchange information. The first subsystem is the logical part of the simulator: this performs the technical algorithm that describes the real behaviour of the simulated system and its response to operator actions, including button pressing, switch knob turning, cable unplugging and, above all, mistakes.
The second subsystem is a graphical 3-D shell (graphics engine) responsible for the visual presentation of the substation model. The connection between the two subsystems is event-based and is performed through the interaction interface. The protocol is thus that the simulator logical part proceeds to a new state calculation for the system only after receiving the appropriate command from the 3-D shell, this resulting from the user having performed some action that triggers the need for a new response. Following this, the simulator logical part evaluates the action taken and, if it causes a linked visual change in the environment display, sends a command to the 3-D shell to display the corresponding change.
Minimum: the user can effect changes in the simulated equipment using a minimum set of peripheral devices—a monitor, keyboard and mouse are needed, connected to a computer.
The optimal set of controllers consists of a virtual reality helmet with gyroscopic position sensing and a forward-looking camera to track the position of the user’s head, arms and body.
Maximum: differs from the optimal by adding an omnidirectional treadmill, which allows the trainee to move around in a virtual environment, ensuring the maximum presence effect, and consequently the effectiveness of training.
19.6.2 Operating Modes
The system can operate in two basic modes: functional preparation mode and training mode. The functional preparation mode consists of environment preparation (see Fig. 19.2): the equipment models are created, connection diagrams loaded and training scenarios are recorded in the corresponding databases on the server.
Basic individual training can be conducted on a dedicated PC: the simulator application is installed at the trainee’s workplace; a model file of the trainee’s or instructor’s choice is then loaded. Depending on the configuration, model files can be located on a workstation or on a server with access to model files using the CIFS protocol.
19.7 Creation of Scenarios
Analogous to a computer game, the graphics engine can create an arbitrary environment into which the trainee can enter when using the VR headset. However, as stated above, to reduce latency to less than 10 ms, as is essential for a realistic experience, the use of high-level language and packages was avoided and instead the model was entirely written in the C language. For the prototype that was developed, the client required a realistic model of a high-voltage substation, both the outdoor high-voltage equipment and the indoor control room and low-voltage equipment. All of this had to have simulated functionality, including moving parts, to cover all of the requested training scenarios.
There are no virtual controls on the right hand, but the system detects the index finger pointing forward as a tool to press buttons, while a grasping action using all of the fingers is seen as a tool intended to turn a control knob or grasp a plug.
19.8 Outcomes and Conclusions
A three-dimensional stereoscopic software representation of an industrial installation was created, including a large range of virtual interactive functions.
The implementation was designed to exploit a virtual reality headset with enhanced functionality, including gyroscopes to determine the movement of the headset and a forward-looking infrared camera which was used to detect the position of the hands and fingers of the user, thus avoiding the need for haptic gloves. Further, virtual buttons were located on the hands of the user in the virtual reality space, thus avoiding the need for hardware control buttons. The implementation also prioritized extremely low latency in the display, to give maximum reality in the simulation. To ensure this, the headset was connected to the controlling computer by an umbilical cable, as no standard commercial wireless link would have sufficient capacity and adequately low latency. In addition, the code representing the simulated environment was written in C to avoid latency issues that could be caused by higher level programming approaches.
The simulated environment in the prototype was an industrial plant (electricity substation), created with great attention to verisimilitude. Excellent realism was achieved and this included functional button switches and electrical connectors, all of which could be operated on by the user’s own hands without the need for haptic gloves or other technical adaptations. The industrial environment simulated was dictated by the primary sponsor, but an arbitrary range of alternative environments can be envisaged. A potential disadvantage is the large amount of specialist labour needed to create the simulation: 1200 person-hours in the present case, not including work on assembling the hardware.
The system functioned completely according to specifications and has been adopted for the training of substation operators by a major electrical power distribution company in Russia. A particular advantage of a virtual reality training simulator in an application of this type is the avoidance of any danger resulting from errors made by the operators during their training. The system has been tested with a number of trainee operators and results have been successful, in the sense of perceived realism and relevance of the training. Work is now proceeding to implement other environments.
Acknowledgements and Note
The authors are grateful to the PJSC Interregional Distribution Grid Company of Central Russia for funding and technical support to develop the VR simulator.
This chapter is an expanded version of a presentation originally given at the 2018 European Conference on Technology-Enhanced Learning .
The copyright of all the images in the figures belongs to R. R. Nasyrov.
- 1.Russkih, A.A., Nasyrov, R.R.: Electrical Safety System, pp. 14–21. PCS, Moscow (2015). ISBN 978-5-905485-81-7Google Scholar
- 3.Thompson, J.I.: A three dimensional helmet mounted primary flight reference for paratroopers. Thesis. Jason I. Thompson, Second Lieutenant, USAF. AFIT/GCS/ENG/05-18. Department of The Air Force, Air University, Air Force Institute of Technology (2005)Google Scholar
- 4.Nasyrov, R.R., Suleimanov, I.R., Churkin, A.I., Pilyugin, A.V., Marchenkov, D.V.: Switching training simulator based on virtual reality. Electrichestvo (Electricity) 3 (2016)Google Scholar
- 5.Epic Games. Unreal Engine 4. Epic Games Inc, Cary NC, USA (2014). https://www.unrealengine.com/en-US/
- 6.Leatham, J.: VR omnidirectional treadmills making gains towards full immersion and cardio. VR Fitness Insider website, Boise, ID, USA (2018). https://www.vrfitnessinsider.com/vr-omnidirectional-treadmills-making-gains-towards-full-immersion-and-cardio/
- 7.Nasyrov, R.R., Excell, P.S.: New approaches to training of power substation operators based on interactive virtual reality. In: Pammer-Schindler, V., Pérez-Sanagustín, M., Drachsler, H., Elferink, R., Scheffel, M. (eds.) Lifelong Technology-Enhanced Learning. EC-TEL 2018. Lecture Notes in Computer Science, vol. 11082. Springer, Cham (2018)Google Scholar
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.