Keywords

1 Background

Aerial refueling is one of the most demanding and dangerous activities faced by pilots. Even under high visibility conditions with clear skies, refueling is a complex perceptual challenge—pilots must maintain their flight profile with respect to the tanker while monitoring tanker refueling operations. Air crews must maintain aircraft control with high precision, under tight physical safety envelopes, against the backdrop of accurately perceiving and appropriately responding to moving connection points with the fuel tanker. Under these challenging conditions, the pilot must work hard to communicate with the tanker, navigate into the tanker’s slipstream and stabilize the aircraft, and then perform a detailed spatial alignment task as they lineup with the boom. Even under high visibility conditions with clear skies, refueling represents a complex perceptual challenge. This complexity is further compounded under reduced visibility (e.g., night, adverse weather) where naturally available and compelling perceptual cues specifying relative orientation and motion through space are degraded.

Many displays attempt to overcome the loss of natural sensory cues by taking critical state information and presenting it digitally over foveal vision displays. However, this format, frequently presented as text or display bugs on scales, is only insufficiently compelling to compete with other “strong-but-wrong” preattentive sensory cues as it must be visually extracted, translated, and interpreted, which is a cognitive, but not perceptual task. As such, these display-mediated cockpit cues do not support direct ecological perception and response. For this reason, such mediated displays do not support direct, ecological perception and response [1].

Furthermore, during refueling operations, pilots must focus for long periods of time while looking up and outside the aircraft (“off-axis”), a more difficult task than focusing forward in the direction of flight (“on-axis”), as seen below in Fig. 1. When looking off-axis, pilots cannot see traditional cockpit displays (e.g., attitude indicators) or virtual heads-up displays (VHUDs) that contain key orienting representations. Because refueling operations frequently occur during sustained, banked turns, pilot difficulty perceiving this continued roll can lead to spatial disorientation phenomena, such as “the leans.”

Fig. 1.
figure 1

During refueling pilots are primarily focusing off-axis (up and to the left) during refueling and traditional on-axis cockpit displays are outside of the pilot’s view

Full size image

Given these challenges, display solutions are required to improve pilots’ accurate perception of their orientation and movement during precision maneuvers in challenging visual environments, particularly in supporting pilots’ spatial orientation during aerial refueling.

2 Approach

Our approach began with a formal work domain analysis. As part of this effort, we performed a set of knowledge elicitation (KE) sessions with a team of pilot subject matter experts (SMEs), including Navy and Air Force pilots. During these knowledge elicitation sessions, we discussed a broad range of topics related to aerial refueling operations, spatial disorientation, cockpit symbology conventions, potential displays to support pilot understanding of aircraft attitude during refueling events, and the many complicating factors from environmental conditions and operational considerations that compound spatial orientation challenges. These interviews incorporated sets of scenario-based walkthroughs which were used for the SMEs to “talk-aloud” through key control tasks, interactions, and task sequences.

One key element contributing to spatial orientation challenges during aerial refueling is sub-threshold maneuver rates and sustained banked turns. During refueling, the pilot must constantly work to maintain an optimal position on the tanker. These maneuvers are typically small in magnitude, with bank angles of 40° or less with pitch between −5 and 5°. This constrains the operational problem to consider smaller roll and pitch rotations for effective display design. To effectively support spatial disorientation, pilots need displays that reminds them that they are in sustained banked turns, which is not readily detected by the vestibular system and contribute to the occurrence of spatial disorientation. Further compounding the spatial orientation challenges is that the pilot refueling and the entire group formation may need to maintain their position with respect to the tanker. Because of this operational aspect, aircraft may not necessarily enter level flight to align with the entire formation when finished refueling the aircraft, a key consideration for when designing directive displays providing control guidance to the pilot. In contrast, our preliminary approach under this effort has been focused on providing descriptive information displays that describe the spatial environment to the pilot.

A key orienting element for pilots is the horizon line as enables the pilot to understand aircraft attitude (e.g., pitch and roll) when visibility is good. In poor visibility, the pilot leverages aided vision capabilities to understand aircraft attitude and monitor position relative to the horizon. However, the pilot looks off-axis during refueling, so normal attitude displays (e.g., artificial horizon, pitch ladders) are not visible. When visibility is poor (e.g., at night, bad weather), there is little visual information about aircraft attitude, and the pilot must attend to the tanker’s director lights to maintain optimal refueling position.

Another key orienting element for pilots includes pitch ladders. Pitch ladders are a traditional on-axis display that provide the pilot with roll and attitude information, but are not displayed when the pilot looks off-axis during aerial refueling. Pitch ladders show aircraft pitch using a series of lines distributed at equal intervals parallel to the horizon. Pitches above the horizon are typically represented with a pair of solid L-shaped lines, with the short end of the L pointing towards the horizon. Pitches below the horizon are typically represented with a pair of dashed L-shaped lines, with the short end of the L pointed towards the horizon. Line pairs below the horizon become increasingly angled with each step away from the horizon. The horizon (neutral pitch) is visually differentiated from other pitches with longer lines that are not L-shaped. When visibility is poor, pilots increasingly rely on pitch ladders to understand aircraft climb, descent, and roll.

While pitch ladders are a critical on-axis display, simply placing the traditional pitch ladder display so it appears when pilots are looking off-axis would be misleading as the pitch ladder display elements no longer map to physical real-world element. For example, in a simple addition of the pitch ladder to the off-axis context, the horizon line display would no longer act as a digital horizon line that maps the physical location of the horizon line in the world. Instead, the horizon line would be an abstract representation of the horizon line displayed from the on-axis perspective.

3 Design Concepts

To address these challenges, we designed a broad set of augmented reality display strategies for head-mounted displays (HMDs) that support pilot spatial orientation during off-axis activities, such as refueling. We explored a range of potential design solutions, both extending traditional pitch ladder displays for the off-axis context and designing new displays that convey critical information specifically tailored for the off-axis context.

Our first set of display strategies extends traditional pitch ladder representation symbology. We explored utilizing a range of display methods to show that the pitch ladder is no longer mapped to the physical pitch in the pilot’s focus area, but instead conveys the information about the direction of flight for the aircraft. These methods included modifying the location/size of the pitch ladder, varying the styling and rendering of the pitch ladder (background border, line thickness, etc.), chaining multiple rendering methods together (e.g., both altering location and background style), and removing aircraft and current path indicators.

For example, Fig. 2 shows one design concept that employs traditional pitch ladder symbology, but uses a smaller size, background color, and explicit placement to distinguish it from traditional on-axis pitch ladders. The placement of this display is at the side of the VHUD nearest to on-axis—if the pilot looks off-axis to the left, it appears on the right side of the VHUD; if the pilot looks off-axis to the right, it appears on the left.

Fig. 2.
figure 2

Extended off-axis pitch ladder design. The pilot is looking off-axis, up and to the left. This design extends the traditional pitch ladder by removing aircraft/route indicators, changing the location/size of the pitch ladder by reducing the size and placing it in the lower right corner, and employing a background for additional cueing that this pitch ladder represents on-axis information.

Full size image

As another example, Fig. 3 below shows a “minimal” variation on traditional pitch ladders for level flight. In this example, the aircraft indicator remains, with only the horizon line is provided for a reference during level flight. The representation has also moved from the center of the screen to the right of the screen to further encode that this is an off-axis display concept, not a traditional on-axis pitch ladder. Figures 4 and 5 show two other examples with a slight roll and slight pitch up and slight pitch down variations.

Fig. 3.
figure 3

“Minimal” extension to traditional pitch ladders for off-axis context indicating a slight roll

Full size image
Fig. 4.
figure 4

“Minimal” extension to traditional pitch ladders for off-axis context indicating a slight pitch up and slight roll

Full size image
Fig. 5.
figure 5

“Minimal” extension to traditional pitch ladders for off-axis context indicating a slight pitch down and slight roll

Full size image

We also explored a range of display concepts for display forms that convey critical aircraft orientation information using new visual forms. These display strategies were designed around key information relationships that integrate new display capabilities with aiding information to more effectively support spatial orientation and control inputs. When implemented in a clumsy fashion, poor visual-cue design and data-organization strategies require significant pilot cognitive resources to compensate, creating an explicit design opportunity to turn these currently cognitive tasks into natural perceptual tasks that free the pilot’s attentional resources.

To address these design challenges, we applied mature ecological interface design (EID) techniques [2,3,4] to develop these design concepts. EID is an approach to perceptually grounded interface design that was developed specifically to address the challenges of cognitive work within highly constrained physical systems, such as pilot control of aircraft in flight. We used established EID design techniques to develop a variety of display sets that maximize the perceptual availability of these critical and integrated information sources to pilots. Figure 6 shows a subset of example display concepts that explore representations for depicting the horizon line, altitude, and roll. In these examples the altitude is represented by a vertical bar and the horizon line is represented immediately beneath it.

Fig. 6.
figure 6

New off-axis pitch ladder design. Key orientation information (roll and altitude) is encoded in graphical formats within the off-axis context with the altitude represented by the vertical bar and horizon line by the horizontal line directly beneath it.

Full size image

As part of this approach we explored both egocentric and exocentric representations. Across the visual forms, we also explored a range of conditions with subset of graphical elements maintained static positions for reference points while other graphical elements were dynamic and changed over time.

4 Prototyping and Demonstration

As these designs evolved, we prototyped a subset of promising display approaches to prototypes of increasing fidelity and detail, both to support our own internal iterative review and incremental design process, as well as to support critical evaluations with our team of pilots. We explored a range of augmented reality prototyping options, such as commercial off-the-shelf (COTS) see-through augmented reality helmets, which can support head-tracking displays capabilities. Head-tracking is particularly critical for this effort to sense the transition between on-axis and off-axis displays, such as when the pilot is looking up outside of the cockpit to manage refueling activity.

For our preliminary proof-of-concept prototype we leveraged the Microsoft HoloLens augmented reality device [5] as a primary hardware prototyping platform due to its rich, out-of-the-box support for head-tracking and rapid integration with our in-house flight simulation environment. However, one challenge of note with the HoloLens is the limited field of view. Using these prototypes, we performed initial cognitive walkthroughs and informal evaluations of our display methods and preliminary prototypes with a team of pilots.

5 Conclusions and Future Work

This paper describes a set of preliminary design strategies to support off-axis spatial orientation for pilots during refueling. As part of this effort, we performed preliminary evaluations with subject matter experts on a set of pitch ladder extensions and new visual displays for the off-axis contexts. Based on the results of these evaluations, both methods show promising methods of conveying critical information in off-axis context. Next steps would include medium fidelity testing on both approaches and formal evaluations with representative users.

Key areas of additional investigation include exploring the effectiveness of transition between on-axis and off-axis orientation and display strategies and methods to facilitate effective transition. For example, consideration such as animations, timing, motion relative to the pilot’s head movement may impact awareness between on-axis and off-axis displays. Another area of investigation includes identifying the impact of effectiveness on refueling task and efficiency of interpreting spatial information. Finally, another broader area of investigation is exploring how these off-axis cockpit displays can work in isolation or in coordination with information displays across other modalities (e.g., auditory displays).

We are currently designing a formal, human-in-the-loop experimental plan to investigate the effectiveness of both these displays and transitions between on- and off-axis orientation. In particular, we plan to identify the impact on pilot interpretation of spatial orientation during intensive off-axis mission tasks, such as aerial refueling. Our research plan will test a set of display strategies and investigate their effects on performance for maintaining a manual tracking tasks similar to those for maintaining a long, sustained turn during aerial refueling, and visually-intensive secondary tasks (e.g., watching for changes in a visual display). We will examine a number of metrics to assess performance and spatial orientation.

This research will provide insight into the basic and fundamental question of how we can better understand, develop, and design effective displays for supporting pilots in on- and off-axis contexts. By examining the effect of on-axis and off-axis HMD strategies on users’ task performance and spatial orientation, we will be able to better understand and develop displays that effectively support pilots in both contexts. Our research will shed light on whether there are effects inherent to wearing HMDs, or whether the different physical viewing orientation of a HMD may differentially affect the user. This research is critical for a variety of applications, including military and medical operations; as well as general gaming entertainment use. We believe the results from this study will inform domain-specific displays as well as more broadly augmented reality displays.