1 Introduction

Helicopter offshore operations are conducted in rough environments under adverse conditions, e.g., strong winds or limited visual conditions. Under these conditions helicopter pilots have to operate very close to obstacles, like wind turbine poles and blades, towers or cranes. Even though specific operations ensure safety as much as possible, hoisting near a wind turbine or landing on an offshore platform remain quite challenging operations.

Helicopter operations in the proximity of obstacles are more or less daily business for helicopter pilots, equal to on- or offshore. Nevertheless, obstacles represent a serious issue concerning operational hazards and accidents [1]:

“Operational Safety Issues:

Helicopter Obstacle See and Avoid: Obstacle collisions are the second most common accident outcome in this domain, making obstacle see and avoid one of the key safety issues. This involves the provision of the best equipment and strategies to help flight crew maintain safe clearance from obstacles during take-off and landing.”

When operating in the vicinity of obstacles, pilots are forced to tackle different problematic aspects. Firstly, they have to estimate the distance to an obstacle, in particular between an obstacle and the helicopter main rotor. The situation becomes more difficult when obstacles get or already are in the back of a helicopter, out of pilot’s field of view. This might end up in a tail rotor strike. Accordingly, pilots need a clear view and understanding of the overall obstacle situation, as well. Apart from that, pilots have to manage the helicopter systems, have to conduct specific procedures, and have to be prepared for degraded situations.

1.1 Objective

Some helicopters offer helmet mounted displays (HMD) or more generally head worn see-through displays. HWDs combine and relate additional computed information with the reality. HWDs consider the head orientation of the pilot, which offers various possibilities for the HMI creation. Hence, HWDs allow pilots keep looking out of the cockpit window, paying attention to the surroundings, especially when operating in the vicinity of obstacles. Nevertheless, pilot’s head orientation and field of view is limited.

This contribution introduces a new display concept, which combines the native egocentric perspective with an exocentric orthogonal coplanar 360 degree top view in a head-worn see-through display (HWD). The egocentric perspective comprises the natural view of the pilot to surroundings plus the indication of the primary flight information. The primary flight information comprises all mandatory information, the pilot needs to continue flying any time, e.g., the present attitude, altitude, heading, air speed, ground speed and vertical speed of the helicopter. Accordingly, the exocentric orthogonal coplanar 360 degree top view has been added as inset beside the primary flight symbolics. Thus, pilots should be able to catch the overall obstacle situation as well as being able to determine the distances to each obstacle while looking out and keep flying.

Figure 1 depicts the DLR’s primary flight information symbolics for HWD. Pilots see all the green drawn contents in addition to the reality [2]. The background shows the offshore wind park Alpha Ventus, which is located in the North Sea [3]. The research offshore platform FINO1 is placed in front of the picture. Figure 2 shows a HWD integrated in the DLR’s simulation and evaluation environment, the Generic Experimental Cockpit (GECO). Outside the cockpit the offshore wind park Alpha Ventus is depicted, too.

Fig. 1.
figure 1

DLRs primary flight information symbolics for HWD [2] (Color figure online)

Full size image
Fig. 2.
figure 2

See-through HWD in DLR’s Generic Experimental Cockpit [4]

Full size image

On the one hand, the extension of an exocentric top view to the egocentric perspective seems to be promising in order to combine the pros of perspective with the pros of a coplanar view. On the other hand, the combination of the top view with the egocentric view and furthermore in a see-through HWD may confuse, cause additional workload or attentional tunneling effects. Consequently, the main question related to the developed display concept is, if the display concept is suitable to provide situational and spatial awareness as well as reducing the workload of helicopter pilots operating in the vicinity of obstacles for safer operations.

1.2 Use Cases

The first evaluation of the developed HMI concept regards two target operations: a winching operation at the lower access point of an offshore wind turbine and an offshore platform landing.

Figure 3 depicts the situation of the first target operation, the hoist operation at the lower access point of an offshore wind turbine. Usually, offshore wind turbines have two access points, an upper one at the top, close to the wind turbine’s head and blades plus a second one, at a lower level, a few meters above the sea surface. In the majority of cases, the upper one is used. Only if materials have to be lowered or if injured persons have to be picked up, the lower one is used. The lower access point bears two main challenges. The first results from the fact that pilots can not hover the helicopter straight above the target position. Due to the main rotor size, pilots have to keep a certain distance in order to avoid a collision of the main rotor with the wind turbine pole. The wind turbine rotor is stopped cross to the wind direction at this time. The second issue is that a helicopter pilot is not able to observe the wind turbine pole. Because winches being mounted on the side of a helicopter, pilots have to hover abeam the wind turbine looking to the side with approximately 90 degree offset to the cockpit and helicopter orientation.

Figure 4 illustrates the second target operation, landing on a fixed offshore platform. Similar to the previous case, helicopter pilots operate in the vicinity of high obstacles, like towers, cranes or wind turbine poles, e.g., on construction ships waiting for their installation. When approaching a landing position, the helicopter always faces the wind direction. The landing position and the obstacles have a fixed orientation. Hence, the landing position on an offshore platform may be either before, abeam or behind an obstacle (refer Fig. 4a–c) depending on the wind direction. Among these, case (c) is the most challenging one. When the obstacle comes behind the helicopter, it will be out of pilots field of view. Apart from that, offshore platforms can have more than one obstacle. Thus, in reality you have combinations of the depicted cases (a), (b) and (c). In comparison to that, in the first case the hover position is not fixed. The wind turbine is always abeam to the side of a helicopter. The position can be all around the wind turbine pole due to the fact that the lower access point offers a 360 degree access. Accordingly, the approach to the hoist position of use case one is always the same (equal to Fig. 4b).

Fig. 3.
figure 3

Hoist operation at a wind turbine

Full size image
Fig. 4.
figure 4

Offshore platform landing

Full size image

1.3 Project Context

The work of the contribution represents an outcome of the project “Development of powerful and efficient Avionic-Platforms for Fixed and Rotary Wing Aircraft” (AVATAR). The joint project comprises industrial partners and research institutions. It is funded by the German Federal Ministry of Economics and Energy in the frame of the national Aeronautical Research Program V “LuFo V”.

1.4 Contents

The following Sect. 2 describes related work, concerning helicopter obstacle awareness and warning systems, the pros and cons of orthogonal and perspective representations as well as preliminary work done by DLR. Section 3 introduces the two main aspects of the developed HMI concept, this means the extension of the egocentric perspective view by an exocentric orthogonal view plus the integration of the 360 degree top view in the HWD. Section 4 depicts the developed HMI concept regarding the two target operations. Section 5 comprises the conclusions.

2 Related Work

Present helicopters are equipped with several flight information and warning systems. So far, obstacle awareness and warning systems (OAWS) do not belong to the helicopter equipment. As described in the previous section, it is highly desirable assisting helicopter pilots with a system detecting obstacles nearby. Furthermore, an OAWS should provide a proper HMI in order to support pilot’s situational and spatial awareness.

2.1 Helicopter Obstacle Awareness and Warning Systems

Until now, two promising helicopter OAWS had been prototyped and evaluated, one by Agusta Westland and the other by Airbus Helicopters. In 2014 M. Brunetti from Agusta Westland presented a novel obstacle proximity Lidar system (OLPS) [5]. The system used three Lidar sensors detecting obstacles 360 degrees around the helicopter, with a range of up to 25 m. Each of the three sensors covered approximately 210 degrees with an accuracy of circa 10 cm and 0.25 degrees. The sensors had been mounted below the helicopter main rotor, one in front looking ahead and the other two with 120 degrees to the left and right. One year after Waanders et al. from Airbus Helicopters presented a competitive Rotor Strike Alerting System (RSAS) [6]. This system used commercial of-the-shelf radar sensors from the automotive domain. Four sensors covered 360 degrees. Each sensor covered 100 degrees with an accuracy of 10 cm and less than 10 degrees azimuth up to 80 m. Each sensor was mounted below the main rotor every 90 degrees beginning at 45 degree offset from the helicopter front. Both sensor systems had been evidenced being able to detect obstacles properly in order to realize an OAWS. Hence, from a technological point of view, helicopters might be fitted with sensors enabling an OAWS in future.

2.2 Human Machine Interface of Obstacle Awareness and Warning Systems

The mentioned OAWS offer a display concept for head down displays (HDD), presented on one of the panel mounted cockpit displays or on an additional display beside the cockpit displays. Both display concepts used an orthogonal coplanar 360 degree top view. Agusta Westland developed a 360 degree \(\times \) “25 m” polar grid [5], while Airbus Helicopters implemented three concentric circles representing three distances and alerting levels [6]. The latter display concept offered a 360 degree circle divided by sectors with 9 degrees. A more or less filled sector indicated the distance to the shortest detected obstacle. According to the three mentioned alerting levels, the color green, yellow or red were used, if a distance is below the alerting levels, i.e., 35 m, 15 m and 5 m to the helicopter main or tail rotor. In comparison to Airbus Helicopters, Agusta Westland draw the detected obstacle outline over the grid. The polar grid of Agusta Westland’s display concept used 15 m, 10 m and 5 m distance circles in relation to the helicopter main and tail rotor. As a special feature, Agusta Westland used a beam highlighting the shortest obstacle distance and direction.

Beside the panel mounted head down display concepts of the aforementioned OAWS, Agusta Westland and Airbus Helicopters applied a multimodal HMI using audio. Agusta Westland implemented a variable frequency tone and vocal announcements, i.e., warning and caution, while Airbus Helicopters evaluated discrete tones, indicating the distance to the closest obstacle. Apart from that, research does also investigate other human senses as HMI, e.g., audio or tactile cues. For instance, you can design a multimodal HMI using audio in addition to or instead of any display in order to put the pilots attention to the closest obstacle [7]. You can also imagine tactile cues in addition or instead of visual information presentation, e.g., soft stops, vibrations or directed ticks to the helicopter controls, in order to prevent pilots flying further in the direction of an obstacle [8]. However, so far multimodal HMIs seems to be best practice to emphasize one information, for example, relating to the most critical obstacle.

From the authors’ point view, the best way to gain the overall spatial obstacle situation on the pilots side, seems to be a visual figure. Hence, the authors are looking forward to evaluate the potentials of head worn display concepts, as a basic visual display concept as OAWS HMI. Anyway, further investigation concerning multimodal HMIs, providing an optimal holistic balanced HMI for OAWS, will follow.

2.3 Situational Awareness

Situational awareness means the understanding of what happened, what happens plus what may happen based on the consideration of all corresponding aspects [9]. Situational awareness comprises the perception of information, understanding and processing the perceived information in order to tackle the current and next situation [10]. Further, situational awareness is the product of different sub-items, i.e., system and mode, operational, task, and spatial awareness [11, 12]. The system and mode awareness addresses the understanding of features, functionality and behavior of the technical system in use. It also includes the awareness concerning the level of automation, i.e., knowing the available modes as well as being aware of the present mode of operation of any assistance system. In this context for instance, the flight attitude, the mode of the autopilot, the hover or landing assistance and the obstacle awareness and warning system. The operational and task awareness comprise all procedures as well as corresponding actions that have to be conducted by the pilot, e.g., the hoist procedure or the landing procedure for offshore platforms. The spatial awareness considers close and far surroundings. In the present case, the flight path, the sea surface, wind orientation, humidity and sight, wind turbines, installation ships or offshore platforms.

Even though the presented display concept for HWD primarily addresses the spatial awareness, the other previous mentioned aspects of the situational awareness can not be omitted. Hence, one very important issue of the evaluation is, to evidence a proper and balanced generation of the spatial and situational awareness without attentional tunneling effects or an increased workload due to visual clutter.

2.4 Orthogonal vs. Perspective Representations

Regarding spatial awareness, one has to respect the properties of perspective and orthogonal coplanar representations regarding their effects to the visual perception.

Perspective representations depict the natural stereoscopic view of human beings. They fit very well to our three-dimensional mental model and imagination of surroundings. On the other side the perspective causes the line of sight ambiguity [13, 14] and includes the depth compression [15]. Both effects prevent an accurate estimation of distances, heights and orientations, which represents a serious issue. In addition, human beings underestimate distances under good visual conditions and vice versa. The estimation of distances and heights can be assisted by the application of grids [16,17,18]. Nevertheless, the negative effects of perspective depictions cannot be compensated completely.

In comparison to that, an orthogonal, coplanar representation properly supports the determination of distances and orientations. Unfortunately, coplanar representations do not provide a proper three-dimensional picture. Terrain elevation, i.e., mountains and valleys, may be indicated using color codes. Heights of obstacles are textually annotated. You have to read and to compare numbers in order to know which obstacle has which height. A direct visual comparison per se of heights of different objects causes some efforts. Hence, perceiving heights is more or less difficult and matching coplanar representations with the reality as well. Due to the fact that coplanar representations by itself differ very much from the natural view, users have to become familiar with its usage.

In conclusion, orthogonal coplanar representations do not support a 3D mental model of surrounding objects properly and perspective views hinder the determination of distances and orientations. Consequently, neither a perspective view nor an orthogonal representation for itself will assist the spatial awareness sufficiently. Both views already are utilized in cockpits. Synthetic vision primary flight displays provide a perspective view and synthetic vision navigation displays an exocentric orthogonal coplanar top view (as 120 degree arc or 360 degree circle).

2.5 Previous Work

The very first step related to the presented concept was made by the development of so called “virtual aircraft-fixed cockpit instruments” (AFCI) [19, 20]. The AFCI intend to provide a more flexible and customizable HMI using conventional cockpit instruments as virtual representatives in a HWD besides the primary HWD symbolics (see Fig. 5). The AFCI include a Primary Flight Display (PFD), the first of two insets of the lower half in Fig. 5, a Navigation Display, to the right of the PFD inset. Furthermore, an airport arrival chart is shown below the two aforementioned insets. The position of the insets are related to the helicopter airframe. Hence, if the pilot moves his head then the instruments remain on their airframe related position and may get out of the display area of the HWD. Further, the AFCI concept offered the possibility to change the placement and configuration of the depicted virtual instruments. After entering a changing mode, the pilot is able to use a cursor to grab one of the previously mentioned displays and replace or resize it. The cursor is bound to the center of the HWD and follows the head orientation (see Fig. 6).

Fig. 5.
figure 5

Virtual aircraft-fixed cockpit instruments [19]

Full size image
Fig. 6.
figure 6

AFCI interaction concept [19]

Full size image

3 Method

The developed display concept comprises two key aspects. The first is related to the combination of the egocentric perspective view of a HWD with the exocentric orthogonal coplanar 360 degree top view. This mainly addresses the primary motivation, i.e., the generation of a proper situational and spatial awareness concerning obstacles close to helicopters plus the flight state. The second aspect regards the integration of the inset within the HWD beside the primary HWD symbolics, in other words the placement of the inset. This aspect concerns potential display clutter and the amount of displayed information.

3.1 Extension of the Egocentric Perspective View by an Exocentric Orthogonal View

According to the aforementioned explanations in Sects. 1.1 and 2.4, an orthogonal coplanar 360 degree top view has been implemented as inset beside the primary flight information symbolics in the egocentric view of the HMD (see Fig. 7). The inset shows the aircraft in the center of a compass rose. The compass rose contains a polar coordinate system with 12 sectors. Sector arcs depict the available space to surrounding obstacles within each sector. The space behind the obstacle is indicated by further sector arcs.

Fig. 7.
figure 7

Orthogonal coplanar 360 degree top view in a HWD depicting surrounding obstacles (Color figure online)

Full size image

The display is comparable to the filled sectors of the Airbus Helicopters OAWS display. Alternatively, only one arc may be shown indicating the distance to an obstacle, but this might be insufficient to be perceptible by the pilot. Depending on the resolution and accuracy of the selected sensor system in use, it is also possible to draw the outlines of detected obstacles like the Agusta Westland obstacle display. However, providing too much details might not be helpful. Figure 7 depicts the implemented display concept as black-and-white image.

3.2 Integration of the 360 Degree Top View in an HWD

As shown before in Fig. 7, the obstacle situation display is placed beside the primary flight information symbolics. On the one hand, the display concept should provide a complete situational awareness, i.e., concerning all the different aspects as described in Sect. 2.3. On the other hand, in order to provide a balanced HMI it must be evaluated, how the obstacle situation is used by a pilot. This means, do pilots use it more or less simultaneously together with the primary flight information, i.e., switching frequently by eye movement in between or do pilots want to have more distance to the primary flight symbolics in order to be able to manage the amount of information as well as the focus on the different information moving their head.

Fig. 8.
figure 8

Integration concept for the 360 degree top view inset in the HWD

Full size image

Hence, three different inset orientations had been implemented for the evaluation: an aircraft fixed, an HWD fixed and an aircraft related orientation. The first option implements a virtual aircraft fixed instrument. The inset is fixed to a certain position related to the helicopter airframe, like a real panel mounted cockpit instrument displayed on a LCD or like a head-up display. Depending on the head orientation, the inset can be inside or outside of the field of view and display area of the HWD (see Fig. 8a). In the second layout, the inset has also a fixed position. In comparison to the first layout, the inset is bound to the display area of the HWD. Accordingly, the inset remains visible all the time at same position inside the HWD, even if pilots move their head (refer to Fig. 8b). The third option represents a combination of the first and second option. The inset is located at a certain position related to the helicopter airframe, too. Whenever this position is within the display area of the HWD according to the head orientation, then the inset is displayed and remains at this position. Otherwise the inset is shown at the border of the HWD display area, pointing in the direction of the aircraft related position of the inset. Thus, pilots may focus on it as well as they are guided to it when changing the line of sight from the primary symbolics to the obstacle situation display (Fig. 8c).

4 Results

The concept is implemented using the programming language C and OpenGL and standard PC hardware. After the concept study, the implementation will be ported to an industrial hardware platform for future avionic systems. The display concept is applied to the helmet mounted see-through display JedEye™ (see Fig. 2). The JedEye™ is an industrial high performance prototype, developed by Elbit Systems Ltd. The main features of this monochrome green, binocular optical see-through HMD are its wide FOV (\(80\times 40\,\text {degree}\) with 60 degree horizontal overlap) together with a high display resolution (\(2200\times 1200\,\text {px}\) in total; \(1920\times 1200\,\text {px per eye}\)). The magnetic 400 Hz head tracker has an accuracy of 0.25 degree.

Figure 9 illustrates the obstacle situation and awareness display in use concerning the hoist operation at the lower access point of an offshore wind turbine. On the right side of the image, you see the base of the wind turbine AV2 [3]. In the background, other wind turbines of the offshore wind park Alpha Ventus and the offshore research platform FINO1 are visible. The scene is rendered by the game engine Unity [21]. The developed HWD symbolics superpose the scene. On the left, you see the exocentric orthogonal coplanar 360 degree view, enabling the pilot to observe the location and distance to the wind turbine pole, while the helicopter is approaching the target hover position abeam the wind turbine pole. The current flight state is displayed by 2D symbolics in the center within the egocentric perspective view.

Fig. 9.
figure 9

HWD OAWS in case of conducting a hoist operation at the lower access point of an offshore wind turbine

Full size image
Fig. 10.
figure 10

HWD OAWS in case of landing on the research offshore platform FINO1

Full size image

Figure 10 shows the developed HMI concept applied to a landing on the research offshore platform FINO1 [22]. The helipad is depicted in front at the lower border of the picture. Furthermore, you see a lattice tower to the right. This tower is 101 m high above the sea and 5 meters beside the helipad border. The helipad is 25 m above the sea surface. Other wind turbines of Alpha Ventus are placed behind in the background. Similar to the previous case, the HWD symbolics overlay the scene. The obstacle is indicated in the inset in parallel to the landing information in the center.

The figures demonstrate the potentials of HWD as well as its drawbacks, e.g., the brightness and contrast of the symbolics to the background or the interaction of the symbolics with the outside world. As discussed, the intents and potentials of the developed display concept has to evidenced in a first concept study. This study is planned for the next months. Helicopter pilots will test the HWD HMI in combination with the two target operations in DLRs simulation and evaluation environment.

5 Conclusion

The contribution presented a new HMI concept for head-worn see-through displays featuring helicopter obstacle awareness and warning systems. The need of obstacle awareness and warning systems for helicopter operations as well as the need for the integration of an exocentric orthogonal coplanar 360 degree top view beside the native perspective view in a head worn see-through display had been motivated. Besides, the background concerning helicopter obstacle awareness and warning systems, situational awareness and orthogonal as well as perspective representations has been pointed out and discussed. Three different options are presented in order to investigate the potential use of the integrated obstacle display beside the primary flight symbolics by the pilot. Finally, the concept was applied and presented to two helicopter offshore operations.