1 Introduction

It is probable that the majority of uses of modern Virtual Reality (VR) technologies are for entertainment. However, VR technologies also routinely allow people working in safety-critical industries to undergo training in a realistic environment, without the risk of hazards such as injury or equipment damage. Electrical power distribution is an example of a major safety-critical industry and a need arose to create a virtual training simulator for power substation operators, who have to be ready to react rapidly, correctly and safely to a wide range of routine and emergency situations. The work presented here discusses the development of an innovative VR training system for such cases, but the principles established could be used in entertainment or creative applications as well.

The target need arose from an approach by the Russian power industry, for which over 30% of faults and blackouts have historically been caused by errors during switching [1] and hence the improvement of operators’ training is a priority: it is probable that the situations in other industries and in other industrialized countries will show significant similarities.

2 Limitations of Current Two-Dimensional Simulators

Many simulators exist for training of industrial personnel, but they mainly rely on two-dimensional screen displays. There is no doubt that the underlying algorithms are sophisticated; for the power substation operator case they typically include: control of simulated operation order with both routine and non-routine events; estimation and recording of the operators’ decisions made during such simulated events; rapid evaluation of the parameters of the modelled system.

Against this, such simulators have deficiencies, such as: a two-dimensional display of the main control room and substation equipment does not give a realistic scenario to develop skills; routine normal operations tend to be perfunctory and without deep insight for the trainees; navigation around the installation is not a realistic representation of the real situation.

Given the powerful impact of three-dimensional simulations now routinely available for entertainment applications, it was felt that the development of such technology could deliver a major improvement in training simulators. A VR simulator for industrial operatives would be vastly more meaningful, although there would need to be significant improvements in the quality of interaction, in comparison with entertainment-oriented VR devices.

3 Three-Dimensional Approaches to Scenario Simulation

3.1 3D Technologies

There are two main three-dimensional training technologies: 3D Helmet and 3D CAVE (cave automatic virtual environment). Both of them can be used as forms of VR simulator and both deliver complete immersion in the virtual environment. However, the CAVE demands a relatively large amount of space: it leaves the user unfettered by a headset, but nonetheless has significant deficiencies of realism: in particular, the user only sees a 2D projected image, albeit on a surrounding wall. Thus, if the space is limited or true 3D is essential, the 3D Helmet technology is preferable.

The 3D CAVE consists of a cube-shaped space, normally with a volume of at least 8 m3, with display screens surrounding the user on all sides [2]. The user may work with virtual objects with hands or a joystick, but it is difficult to display these realistically since the surrounding screens are basically two-dimensional; further, the user’s mobility is limited by the cube perimeter. For 3D Helmet VR technology, two small displays are mounted in front of the user’s eyes so that they can view the simulated environment [3]. The user then has the opportunity to work with credible 3D representations of virtual objects, either with their hands or a joystick.

3.2 Modelling Requirements for Appropriate Environments

A primary need is a 3D model of the environment: this must have a good degree of verisimilitude in both its visual and spatial similarity to the real thing, but it must also have functional similarity in the operation of relevant switches, indicators, etc. The verisimilitude of a 3D model leads the user (operator) to develop the skills of operation of the selected environment during the training, such that after completion they will not need to adapt to the real environment, due to the realism of the VR interaction.

Spatial analogue representations of objects and the environment are very important: operators need to know the time needed to move from one point to another in the space concerned and this can be critical in emergency scenarios. Further, the need for functional similarity in the 3D model is paramount and the model must react to the user’s actions in the same way as in the real environment.

3.3 Scenarios of Training

The scenarios of training have to include both regular and emergency cases: regular operations refine the skill of routine switching and understanding of the basic operation on the station. Emergency scenarios need to start in the same way as regular training but then there will be the insertion of an emergency at a time and of a nature that must be unpredictable to the trainee, who then has to take correct decisions as fast as possible [4].

The user can choose between training mode and test mode. If they choose training mode, they can select either emergency or regular training and the type of scenario. In test mode, the user does not know what type of training scenario they will encounter: it will be a random choice of the simulator.

4 Implementation of Virtual Reality Simulator

A substation of typical Russian design was chosen to be simulated. This contained: six 110 kV overhead line connections; six 35 kV overhead lines and one 35 kV cable line connection; thirty 10 kV cable line connections; outdoor switchgear for 110 kV; outdoor switchgear for 35 kV; indoor switchgear for 10 kV; three three-phase transformers. The area of the prototype was about 18,000 m2. This diversity of equipment enables many scenarios to be implemented.

The digitalization of the prototype has three main stages, as follows [3]: acquisition of a cloud of object points for the prototype hardware; mesh calculation and optimization; texture calculation and optimization. The object point cloud becomes a skeleton model in the virtual space: the surfaces of the prototype objects are then given appropriate textures and the whole 3D model is optimized.

Creation of training scenarios is a specialized issue dependent on the application. For the substation operator case, the regular scenarios included taking out of, and putting into, service the following: transformers; circuit-breakers; disconnectors; overhead lines. For the emergency cases, the following scenarios were chosen. 1. Taking circuit-breakers out of service due to: emergency powering off of transformer; emergency voltage transformer fuse failure; emergency current transformer explosion. 2. Taking an overhead line out of service due to emergency collapse of column insulation of disconnector. 3. Taking a sulfur hexafluoride circuit-breaker out of service due to emergency gas depressurising. 4. Bringing a disconnector into service due to an emergency earth fault on an overhead line. The totality of these scenarios covers about 90% of typical emergency operations in a substation.

5 Technical Description of the Simulator

The game engine part of the simulator is based on Unreal Engine 4 [5] (Epic Games, 2014). A software model of the chosen prototype was written in the C language, enabling the latency to be reduced to less than 10 ms, which is essential for a convincing experience. The VR headset used an umbilical cable to link to the host computer because there is no current wireless link technology that can transmit two channels of HD video plus three channels of gyroscopic information and a USB-3 channel (for the forward-looking camera—see below) with acceptable latency.

There are three main stages of the creation of the training system (Fig. 19.1): preparatory stage, software creation, system operation.

Fig. 19.1
figure 1

Creation stages of VR training system

Full size image

The preparatory stage consists of the selection of the target environment, carrying out 3-D scanning of the real objects in that environment, creation of a library of 3-D models of objects and equipment in the environment, and finally documentation and refinement of the scenarios (Fig. 19.2).

Fig. 19.2
figure 2

Component preparation for the developed system

Full size image

To create a functioning simulator, it is necessary to combine the technical functions of the modelled system (the simulator logical part) with the capabilities of modern 3-D shells (graphic engines). The integration of these two subsystems is discussed in Sect. 19.6.1.

6 Software Implementations

6.1 Functional Simulations and Graphics Engine

In principle, the system consists of two subsystems that constantly interact and exchange information. The first subsystem is the logical part of the simulator: this performs the technical algorithm that describes the real behaviour of the simulated system and its response to operator actions, including button pressing, switch knob turning, cable unplugging and, above all, mistakes.

The second subsystem is a graphical 3-D shell (graphics engine) responsible for the visual presentation of the substation model. The connection between the two subsystems is event-based and is performed through the interaction interface. The protocol is thus that the simulator logical part proceeds to a new state calculation for the system only after receiving the appropriate command from the 3-D shell, this resulting from the user having performed some action that triggers the need for a new response. Following this, the simulator logical part evaluates the action taken and, if it causes a linked visual change in the environment display, sends a command to the 3-D shell to display the corresponding change.

The main training station acts as a server, to perform functions of administration, programming and information storage. The trainee interacts with the 3-D shell using a specific set of controllers connected to their personal computer. There are three options for the sets of controllers, with increasing levels of functionality:

  1. 1.

    Minimum: the user can effect changes in the simulated equipment using a minimum set of peripheral devices—a monitor, keyboard and mouse are needed, connected to a computer.

  2. 2.

    The optimal set of controllers consists of a virtual reality helmet with gyroscopic position sensing and a forward-looking camera to track the position of the user’s head, arms and body.

  3. 3.

    Maximum: differs from the optimal by adding an omnidirectional treadmill, which allows the trainee to move around in a virtual environment, ensuring the maximum presence effect, and consequently the effectiveness of training.

6.2 Operating Modes

The system can operate in two basic modes: functional preparation mode and training mode. The functional preparation mode consists of environment preparation (see Fig. 19.2): the equipment models are created, connection diagrams loaded and training scenarios are recorded in the corresponding databases on the server.

The training mode is configured for two types of system operation: a multiplayer mode and an individual training mode. A schematic diagram of the operation in multiplayer training mode is shown in Fig. 19.3. In this mode, the personal computers of the staff in training are connected to a single local area network which includes the main training station, which is an automated workstation for the instructor. There are servers for the 3-D shell and simulator logical part at the main training station, providing communication between training stations during group training sessions, as well as a complete database covering training scenarios and the library of equipment in the environment. Administration and programming functions are possible from the main training station. Programming involves entering training scenarios into the simulator logical part using the “Form editor” application, as well as further operational adjustments as required by the scenarios created.

Fig. 19.3
figure 3

The training mode system structure, when the main training station is a separate supervisor

Full size image

Basic individual training can be conducted on a dedicated PC: the simulator application is installed at the trainee’s workplace; a model file of the trainee’s or instructor’s choice is then loaded. Depending on the configuration, model files can be located on a workstation or on a server with access to model files using the CIFS protocol.

7 Creation of Scenarios

Analogous to a computer game, the graphics engine can create an arbitrary environment into which the trainee can enter when using the VR headset. However, as stated above, to reduce latency to less than 10 ms, as is essential for a realistic experience, the use of high-level language and packages was avoided and instead the model was entirely written in the C language. For the prototype that was developed, the client required a realistic model of a high-voltage substation, both the outdoor high-voltage equipment and the indoor control room and low-voltage equipment. All of this had to have simulated functionality, including moving parts, to cover all of the requested training scenarios.

To create the environment model simulating a real example of a substation, about 1200 working hours were required, occupying six people, a modeller (mesh creator), two programmers to create mathematical models, one unifying programmer and two electrical power engineers. The three-dimensional information was derived partly from scale drawings and photographs and partly from three-dimensional laser scans. The results show an excellent degree of verisimilitude (Fig. 19.4). Further images showing the realism that was achieved in modelling of the outdoor hardware are given in Figs. 19.5, 19.6 and 19.7.

Fig. 19.4
figure 4

Top: photograph of the real substation that was simulated. Lower: modelled equivalent (differences are mainly due to a slightly different viewing angle)

Full size image
Fig. 19.5
figure 5

3-D model of 110 kV outdoor switchgear yard

Full size image
Fig. 19.6
figure 6

3-D model of a 110 kV sulfur hexafluoride circuit-breaker

Full size image
Fig. 19.7
figure 7

3-D model of a 110 kV disconnector

Full size image

The same techniques were used in modelling of the interior of the control room and low-voltage switch room (Figs. 19.8 and 19.9), although much more functional detail had to be included here, such that switches could be turned, buttons could be pressed, lights could come on and even plugs on flexible cables could be pulled out in the virtual environment. Figure 19.10 shows the detail that was incorporated for this purpose: the plugs shown may be pulled out of their sockets in the virtual world and all of the switches, buttons, and lights function in it as well.

Fig. 19.8
figure 8

3-D model of the main control room

Full size image
Fig. 19.9
figure 9

3-D model of 10 kV indoor switchgear group

Full size image
Fig. 19.10
figure 10

Detailed 3-D functional model of 10 kV circuit-breaker

Full size image

For the full VR experience, the equipment shown in Figs. 19.11 and 19.12 is required. The volume of the region that can be accessed is determined by the placing of the motion sensors attached to the tripods, but a cube with 3 m side length is usually adequate, especially if an omnidirectional VR treadmill is available [6]: in fact, the treadmill would allow a smaller volume to be used, e.g. a cube with side length of 2 m.

Fig. 19.11
figure 11

The appearance of the simulator’s core elements

Full size image
Fig. 19.12
figure 12

Plan layout of the simulator’s elements

Full size image

Figure 19.13 shows the front view of the headset, which has a forward-looking infrared camera attached to its front: this is used to collect the imagery of the user’s hand so that the system can determine what the hand is doing and it can also be used to implement some virtual controls on the hand (see below). Figure 19.14 shows the pair of stereoscopic views that are generated in the two eyepieces of the headset, including a representation of the user’s hand reaching to press a button. Figure 19.15 shows the system in use: the user is wearing the VR headset and reaching out to grasp one of the plugs attached to a flexible cable (Fig. 19.10). The left image shows the view that the user experiences: the virtual image of the hand is shown extracting the plug from its socket in the switchgear.

Fig. 19.13
figure 13

Trainee operator interacting with a 3-D model: the forward-looking camera is noticeable. The marker on the index finger was optional in an early version, but not now needed

Full size image
Fig. 19.14
figure 14

The synthesized stereoscopic view seen by the user shown in Fig. 19.13

Full size image
Fig. 19.15
figure 15

A user interacting with the system. Right: the user is wearing the VR headset and reaching out to grasp one of the plugs in Fig. 19.10. Left: the view that the user experiences: the hand is shown extracting the plug from its socket

Full size image

Figure 19.16 is a diagrammatic recreation of the “virtual buttons” that are made to appear on the left hand of the user in the virtual view. The cube below the wrist is touched when it is desired to change the view significantly, for instance, to move from the control room to the outside switchyard in the combined representation of the substation. The virtual button near the left-hand thumb is used to advance the user’s virtual standing position forward in a way rather similar to Google Streetview. The virtual button in the palm of the left hand has been provided for future functionality but is not currently used.

Fig. 19.16
figure 16

The virtual button controls provided on the image of the left hand, as detected by the forward-looking camera

Full size image

There are no virtual controls on the right hand, but the system detects the index finger pointing forward as a tool to press buttons, while a grasping action using all of the fingers is seen as a tool intended to turn a control knob or grasp a plug.

8 Outcomes and Conclusions

A three-dimensional stereoscopic software representation of an industrial installation was created, including a large range of virtual interactive functions.

The implementation was designed to exploit a virtual reality headset with enhanced functionality, including gyroscopes to determine the movement of the headset and a forward-looking infrared camera which was used to detect the position of the hands and fingers of the user, thus avoiding the need for haptic gloves. Further, virtual buttons were located on the hands of the user in the virtual reality space, thus avoiding the need for hardware control buttons. The implementation also prioritized extremely low latency in the display, to give maximum reality in the simulation. To ensure this, the headset was connected to the controlling computer by an umbilical cable, as no standard commercial wireless link would have sufficient capacity and adequately low latency. In addition, the code representing the simulated environment was written in C to avoid latency issues that could be caused by higher level programming approaches.

The simulated environment in the prototype was an industrial plant (electricity substation), created with great attention to verisimilitude. Excellent realism was achieved and this included functional button switches and electrical connectors, all of which could be operated on by the user’s own hands without the need for haptic gloves or other technical adaptations. The industrial environment simulated was dictated by the primary sponsor, but an arbitrary range of alternative environments can be envisaged. A potential disadvantage is the large amount of specialist labour needed to create the simulation: 1200 person-hours in the present case, not including work on assembling the hardware.

The system functioned completely according to specifications and has been adopted for the training of substation operators by a major electrical power distribution company in Russia. A particular advantage of a virtual reality training simulator in an application of this type is the avoidance of any danger resulting from errors made by the operators during their training. The system has been tested with a number of trainee operators and results have been successful, in the sense of perceived realism and relevance of the training. Work is now proceeding to implement other environments.