Keywords

1 Introduction

Modern commercial aircraft cockpits are safety critical products with advanced and well-researched user interfaces. As in every other complex product development cycle, adding changes to a mature product causes high costs and design compromises, which leads to an expensive product featuring less than perfect user interfaces. To avoid such changes, knowledge about the final product’s properties should be generated as early as possible. Thus, the development of Human-Machine Interfaces (HMI) in flight deck design requires feedback on human factors, i.e. ergonomics, usability, and cognitive aspects, at a very early stage of the design process [20].

Fig. 1.
figure 1

Simulator continuum for human factors research

Full size image

Depending on the development stage of the product, the HF evaluation methods and prototypes should be chosen carefully in order not to tie up crucial resources [15]. Figure 1 shows different prototypes and their level of fidelity, costs and flexibility. A very low-fidelity prototype can be a paper sketch or a storyboard to walk through sequences of events. If a concept gets more mature, stand-alone prototypes based on desktop computers with more or less complex simulations and interactions can be used. These kind of evaluations are referred to as desktop research. Today, the systems of flight decks are highly interlinked and coupled [24], therefore an integrated comparative evaluation in a more complete flight deck is essential. This simulator research can take place in engineering mock-ups, i.e. fixed based hardware flight simulators specially equipped for engineering needs, or even high-fidelity certified motion based full flight simulators.

From using low-fidelity paper sketches to high-fidelity full flight simulators, the costs increase exponentially whereas the level of flexibility, i.e. the ease of changing, integrating and testing a prototype, decreases. In order to transfer an HMI concept from desktop research to the simulator research domain, a huge additional effort in cost and time is necessary with the consequence of losing flexibility. For human factors research this means that evaluations of HMI technologies can either take place in a flexible low-fidelity desktop-based environment or in a high-fidelity hardware simulator with a product that is already in a rather mature state and is not subject to substantial changes anymore - the broad spectrum that lies between these two possibilities has mostly been ignored so far in the context of HF evaluations.

This research aims to fill the gap in human factors research in flight deck design with the concept of immersive virtual worlds. With the help of Virtual Reality (VR), a subject can experience a three-dimensional space by wearing a Head Mounted Display (HMD) or 3D-glasses with an attached head- and finger-tracking system. This technology is well-known and is common practice for exploring ergonomic aspects of aircraft cockpits [9]. Digital cockpit mock-ups have also been connected to flight simulations for the purpose of pilot training by the Technical University of Darmstadt [5]. Even research on the usability of such a system has been conducted [13]. Another system, the Airbus Helicopters Enhanced Virtual Environment (EVE), is a virtual cockpit that focuses on training [3] and is used in air crash investigations [19]. The University of Istanbul also presented a low-cost Virtual Reality Flight simulator for rotary wing aircraft in 2009 [25].

The research presented in this paper also uses a virtual cockpit mock-up that is connected to a flight simulation. In contrast to the existing systems, this simulator focuses on the rapid prototyping of HMI components and their evaluation. Therefore, cognitive human factors assessment methods have been integrated in this Virtual Reality Flight Simulator. Four user studies that range from basic cognitive research to HMI evaluation and development are presented to show the capabilities of this system for HF research - in particular usability and cognitive aspects - and the validity of the integrated HF methods.

2 The Virtual Reality Flight Simulator

The prototype of the Virtual Reality Flight Simulator that is presented in this paper extends a commercially available flight simulation software by VR capabilities. Here, three distinct features are needed: A tracking system, an output device, and the possibility to interact with the flight simulation and the digital mock-up [6]. The overall architecture of the VRFS is shown in Fig. 2.

Fig. 2.
figure 2

The architecture of the VRFS

Full size image

The optical tracking system delivers six degrees of freedom information of the user’s head and the position of his/her right hand and each finger. This data is then transformed into the local coordinate system of the flight simulation. The transformed hand tracking data drives a geometry that resembles a human hand. The transformed head tracking data is connected to the virtual camera inside the three-dimensional cockpit in the flight simulation [1]. In order to achieve stereoscopic vision, the head tracking data is sent to two synchronized instances of the flight simulation with an optical transformation for each eye. The output of the video data can be presented to the subject by using a head mounted display or a 3D-capable projector.

Different methods have been implemented to give the user control over the simulation and to interact with cockpit elements. The interaction can be fully virtual as a collision detection system is implemented (see Fig. 3): If a collision between a virtual button geometry and a virtual finger geometry is detected, a command is sent to the simulation [1]. In this case, the user has no haptic feedback which makes the usability of these control elements challenging. With mechanisms to prevent dual-activation and by adapting the size of the collision volumes, the interaction with fully virtual elements can be enhanced [1, 13].

By placing simple plywood or acrylic glass plates at the spatial position of virtual buttons haptic feedback can be generated. Frequently used levers and rotary buttons should even be provided as hardware elements for a fast and intuitive use. With this method a so-called mixed mock-up is created, as illustrated in Fig. 3.

Adding hardware controls to the digital mock-up is a balancing act between a fully flexible cost effective environment with no hardware elements and one with costly hardware elements but decreased flexibility. In an extreme case, this could lead to a full hardware mock-up which undermines the purpose of a flexible VR environment. On the other hand, the product maturation process leads to an increasing product feature determination. By equipping the VRFS with this mature hardware, the simulation platform can grow with the project’s progress from a flexible rapid prototyping environment to a high-fidelity engineering mock-up. Regardless at which point in the development process virtual reality is used to evaluate HMI components, the right human factors method must be chosen on a case-by-case basis.

Fig. 3.
figure 3

Mixed mock-up and full virtual interaction

Full size image

3 Human Factors Methodologies

Workload (WL) and Situation Awareness (SA) are commonly used and accepted entities when it comes to the evaluation of flight decks. Another human factors metric is the distribution of visual attention. Especially for the design of Head-Up Displays (HUD) and glass cockpit screens, this is an important factor which influences the interface design. The HF methods for assessing the visual attention as well as workload and situation awareness and their integration into the VRFS are presented in the following sections.

Eye Tracking. In order to evaluate the pilot’s overt information gathering, an eye-movement analysis is necessary. This is of particular interest for the evaluation of displays like Primary Flight Displays (PFD) or head-up displays [14]. In such systems, the gaze provides information on the pilot’s visual perception (L1). With eye fixations and transitions in combination with other data (e.g. physiological data), conclusions about the pilot’s workload can be drawn. Thus, an eye tracking system is mandatory for many human factors evaluations of modern flight decks and is therefore integrated in the VRFS using a small camera attached to an adjustable ring around the HMD’s eye piece [23].

ATTENDO. The ATTENDO method is used for assessing the distribution of visual attention. It is based on the secondary-task-paradigm, i.e. measuring the reaction of a visual stimulus by the pilot [11, 12]. The visual targets are small rectangles or circles and are pseudo-randomly displayed spatially and temporarily in the subject’s field-of-view for a short period of time (e.g. 200 ms). The primary task, e.g. flying the aircraft, has to be executed and monitored by the pilot permanently. Whenever a target is visible for the pilot, he should react by pressing a button. If the pilot reacts more often to targets in a certain area, this area has a higher degree of visual attention.

Physiological Data. A parameter that is easy to measure is the heart rate. Among others, this information can reflect psychological concepts like ‘mental load’ and ‘effort’ [14]. In the VRFS this data source is collected by using a pulse oximeter with a sensor attached to the pilot’s ear. The tracking system delivers data of the head and hand movements. With this data, conclusions on the physical workload can be drawn.

Questionnaires. Questionairs like the The NASA Task Load Index (NASA-TLX) for evaluating workload [10] or The Situation Awareness Rating Technique (SART) for evaluating situation awareness [21] can be applied using a paper questionnaire after a task is completed in the VR. In order not to interrupt the immersive Virtual Reality experience, i.e. removing the HMD or glasses, questionnaires can also be displayed inside a virtual environment [7].

The Situation Awareness Global Assessment Technique (SAGAT) is a disruptive method for assessing situation awareness which was presented by Endsley [8]. This method is based on freezing a task and querying the subject’s understanding of certain aspects of the situation. In the VRFS, the operator can freeze the situation manually. At the same time some flight instruments are hidden and the subject is asked about the current state of those instruments [18].

4 Experimental Research

With the VRFS and the integrated human factors methods, several user studies have been conducted. In this section four of these studies will be presented that show the practical application of the presented methods as well as indications on the validity of the conducted HF research.

4.1 (1) Advanced Head-Up Display Evaluation

Head-up displays present information in the pilot’s head-up field-of-view. Head-up guidance systems (HUGS) provide additional information on the optimal flight path and energy management, enabling a pilot to stay head-up in all phases of flight and improving situation awareness. Nevertheless, when using such a system in a standard Instrument Landing System (ILS) approach, according to procedures in certain aircraft, a pilot has to gather information regarding the final checklist items from head-down displays. In consequence he/she needs to re-accommodate to the outside world [2]. As part of the ALICIA (All Condition Operations Infrastructure) project, an advanced head-up display symbology was developed that eradicates the procedural need for head-down information gathering in a standard ILS approach  [6]. This HUD has a single indication, a simple green disc, for all system states that have to be checked in this phase of the approach.

This so-called ‘Green Disc Concept’ was integrated into a conventional head-up display. To exclude confounding factors stemming from the VR environment a comparison of this novel HUD with a conventional one was chosen. The scenario for both experiments is a low-visibility CATIIIA approach. Starting with the initial approach fix, eventually, this approach ends in a Go-Around (GA) because visual contact with the runway cannot be established due to the low visibility conditions. The experiment supervisor also takes the role of the co-pilot to ensure conformity with operational procedures. Eleven pilots from different German airlines participated in the trials. To asses workload and situation awareness, a SART and a NASA-TLX questionnaire were used after each approach. Head movements and eye gaze, acquired through the built-in eye tracking system, have been used to distinguish between head-up and head-down phases.

The results show that the subjectively measured workload decreased. At the same time the subjectively measured situation awareness increased as illustrated in Fig. 7 [6]. This can be attributed to the fact that the head-up percentage was increased significantly (p =.03789) because of the introduced ‘Green Disc Concept’ as illustrated in Fig. 4.

A similar scenario using a conventional head-up guidance system was used in the research conducted by Bandow in 2006 [2]. For the presented user study in the VRFS, the same operational procedures have been applied as well as a similar conventional HUD. Figure 4 shows that the average head-up percentage in the full flight simulator study (\(86.70\,\%\)) is similar to the conventional scenario in the VRFS (\(87.55\,\%\)). This indicates that the operational behavior in virtual reality is comparable to the full flight simulator environment.

Fig. 4.
figure 4

Head-up percentages during a low-visibility CATIII approach [2, 6]

Full size image

Regarding physiological data, the heart rate was captured in both user studies. Figure 5 shows an illustration of the average heart rate across all subjects. The data is normalized based on the resting heart rate [4]. The time axis is shifted to align with the go-around event and covers a range of five minutes before and 30 seconds after that event. The heart rate in the full flight simulator is substantially higher than in the VRFS, as well as the increase shortly before the go-around. This increase, albeit at a much lower level, is also visible in the heart rate data of the virtual reality scenario. With a section of the heart rate gathered in the virtual reality trials a discontinuity regression analysis [22] was conducted as illustrated in Fig. 5. This method shows a discontinuity of the data 30 seconds before the go-around. This leads to the hypotheses that this increase of heart rate is significant and triggered by the go-around event. Further research is necessary to investigate this effect.

Fig. 5.
figure 5

The normalized heart rates in the VRFS and the full flight simulator study and a regression discontinuity analysis [2, 6]

Full size image

4.2 (2) Calibration of Online SA Assessment Systems

Another study that relies on the observation of the pilot’s gaze in a HUD approach was presented by Mamessier et al. in 2014 [18]. Here, the goal was to calibrate a predictive model of situation awareness. Very few existing models for simulating and predicting situation awareness of expert agents interacting with dynamic systems have been validated with experimental data. As a proof of concept, a computational human performance model developed by Mamessier et al. [17] was calibrated using online data from a simulator trial with the VRFS in combination with SAGAT queries.

The scenario is a low-visibility, semi-automated ILS approach. The pilot has to control the airspeed manually. To vary the workload, new airspeed clearances are verbally communicated to the pilot every 60 seconds. The eye tracking system is used to monitor the pilot’s scanning of the altitude and airspeed indicators and feed the computational model with real-time data. At pseudo-randomly generated points in time, the simulation is frozen and SAGAT-like questionnaires are used to sample the pilot’s knowledge and confidence about the current flight parameters. A comparison between the SAGAT measurements and the predicted situation awareness enables the calibration of the model.

The model predicted both the pilot’s current belief and confidence about the altitude as a consequence of visual scanning, current workload and the pilots assumed understanding of the aircraft dynamics. The data collected enabled the calibration of parameters describing the impact of workload on the registration of monitoring events as well as the pilot-specific confidence interval [18].

The embedded eye tracking capability allowed distinguishing between specific elements of the pilot’s visual scanning and feed the computational situation awareness model with accurate and realistic data. A more comprehensive design of experiments is still needed to further validate and calibrate the model. The VRFS not only shows its suitability to evaluate computational predictive models but also opens the way to affordable situation awareness based design.

In study (1) and in study (2), the T-scan pattern, which is typical for pilots trained in using head-up displays, could clearly be observed in the eye tracking analysis for the primary mode as illustrated in Fig. 6. When switching to the A3 mode in study (1), this wide scanning pattern is replaced by a strong attention focus to the center of the display as expected [2]. Study (3) deals with the so-called mental tunneling effect that stems from this behavior.

Fig. 6.
figure 6

The head-up display and the PVHD and eye-scanning patterns.

Full size image

4.3 (3) Peripheral Horizon Display Evaluation

In the final approach phase, head-up displays equipped with Head-Up Guidance Systems (HUGS) get highly de-cluttered. This means that essential information is moved to the center of the display. This shifts the pilot’s attention to the runway center-line and the central flight guidance symbology. As a side effect, a mental or cognitive tunneling effect may occur [2, 12]. To counter this effect, Bandow proposed a Peripheral Vision Horizon Display (PVHD) [2] that is based on the Malcom Horizon [16]: This system consists of two artificial horizons that are integrated into the struts of the cockpit (See Fig. 6). This peripheral stimulus should expand or even eliminate the mental tunnel created by the de-cluttered and centered HUD/HUGS symbology.

As in this study the influence of the peripheral stimulus of the PVHD is the subject of research, a high field of view and stereoscopic vision is necessary. Therefore the pilot is placed in front of a projector screen with 3D shutter glasses. The pilot has to follow pre-programmed roll and pitch commands as exactly as possible while maintaining the airspeed. Simultaneously, the ATTENDO targets are displayed randomly. In this study, eleven licensed pilots and twelve non-pilots participated.

The results indicate no significant difference in the distribution of visual attention between the flights with, and the flights without Peripheral Horizon Display. Despite this negative result, further studies on the influence of the PVHD should be undertaken to rule out confounding factors stemming from the experimental setup and the virtual environment. In particular, an experiment with licensed commercial pilots and a peripheral vision HMD should be undertaken for that purpose. Yet, the trials showed that the ATTENDO method itself is feasible in virtual reality environments.

4.4 (4) Overhead Panel Evaluations

Another user study - unrelated to head-up displays - aimed at simplifying the ‘Manage Systems’ task in the flight deck. Most of the aircraft systems are controlled via the Overhead Panel (OHP). In consequence, changes in aircraft systems are directly reflected in the design of the overhead panel. The goal of this study is to evaluate the impact on the human performance when conducting system management tasks using a conceptual overhead panel. This OHP stems from the comparison of existing aircraft systems and their HMI and the development of a hypothetical generic aircraft, including simplified HMI concepts and further system automation.

Similar to study (1), a comparative approach has been chosen. Here, the system management procedures and the OHP of an Airbus A320 are compared with the novel system management approach and the respective OHP. Two malfunctions that are followed by rather complex system management tasks have been chosen as scenarios. After each scenario a NASA-TLX and a SART questionnaire is completed by the subject. During the scenario, eye tracking data, the heart rate and video data, used for a behavior analysis, are captured. This user study took place right after study (1); Therefore the same 11 airline pilots participated. The study showed that the situational awareness could be increased significantly (p = 0.021) with the new OHP, despite the further automation. The workload could be decreased significantyl (p = 0.012) in comparison to the legacy system management. Figure 7 shows these results.

Fig. 7.
figure 7

Results for workload and situation awareness for study (1) and (4)

Full size image

The system management tasks that had to be executed in study (4) required fast and numerous interactions with cockpit elements. Besides, these tasks are also interlinked with the aviate task, i.e. flying the aircraft. Despite the constrained VR environment all tasks could be completed by the pilots in time.

5 Discussion and Conclusion

This paper presents a flight simulation environment that uses immersive virtual reality technology for human factors research. The presented use cases show the general usability of this system for conducting human factors user studies. Study (1), a head-up display evaluation, and study (4), a system management task, stem from realistic approach and cruise scenarios. In study (1), a significant heart rate peak at the go-around event and the procedural gaze behavior, i.e. head-up percentage and instrument scanning patterns, indicate that a sufficient level of simulation fidelity, in comparison with a similar study in a full flight simulator, can be achieved. Study (4) with its complex system management tasks was a challenge in terms of usability in the virtual reality environment. Tasks like flying the aircraft and pressing numerous virtual and non-virtual buttons had to be completed in a time critical scenario. Although, no comparable study has been conducted in a full flight simulator, the experiments revealed that the system offers a sufficient level of usability to fulfill these tasks. It also showed certain limitations like the degraded ability to aim for fully virtual, non-haptic buttons. In consequence the pilot’s movements are much slower and the task completion takes more time. Judging and quantifying this effect in a virtual flight simulation environment is an aim for further research. More limitations are a limited field of view, restricted freedom of movement and a low level of wearing comfort of the VR equipment. However, in the future these issues can be addressed with novel head mounted displays and finger tracking devices which are in development or already available on the market. In order to avoid simulation sickness the time in the virtual reality should be limited and sufficient resting phases between multiple experiments in the VR should be provided.

The integration of HF methods into the system is easier than in a hardware simulator in some cases. The fully virtual environment offers the possibility to include ‘augmented’ objects anywhere in the cockpit like in the ATTENDO method presented in study (3) and easy hiding or replacing of cockpit elements as used in study (2) and (4). The integrated eye tracking system benefits from the virtual environment as well. With the simulation view, not visible to the subject, additional information, relevant for the eye tracking system, like augmented reality markers, can be used without obscuring the subject’s view. Parts of the cockpit or even the complete environment can be changed in an instant - a task that can take hours or days in a hardware mock-up. This flexibility makes the VRFS perfect for comparative studies of two different systems on the same day with the same subjects and their respective psychological and physiological condition on that day.

Immersive virtual worlds are a valuable addition to the available prototypes in the flight-deck design process. The scope of the VRFS lies between desktop research and a full flight simulator and cannot replace either of them. The presented studies show that the system can provide an environment with a high level of fidelity and it is possible to gather reliable information on human factors aspects of the interaction with cockpit HMI components in comparative studies.The influence of the virtual environment on the outcome of the different human factors methodologies will be a subject of further research, with the goal of transferring evaluation results from the virtual environment to hardware mock-ups or even real flight decks.