Keywords

1 Introduction

In this MUM-T approach, the UAVs shall enable the crew to directly reconnoiter the intended routes of flight, to survey certain areas such as operation sectors or potential landings zones and provide information on other mission-specific conditions. However, since there is no dedicated sensor operator as compared to legacy UAV systems, the HC’ commander has to handle all related tasks including UAV guidance, mission sensor deployment and data assessment. Figure 1 shows the schematic team configuration which is focus of interest in this evaluation study.

Fig. 1.
figure 1

MUM-T approach illustrated in helicopter mission flight simulator

Full size image

To address potential workload increase resulting from the broader task spectrum and higher overall mission complexity, the crew shall be supported by an adaptive, cognitive associate system [1,2,3]. It will provide situation-adapted support by continuous crew supervision and aims to balance the crews’ workload. In [4], three basic requirements for associate system behavior were proposed, to be applied by associate systems for crew support. To reduce the overtaxing MWL, the associate system can involve suitable automation systems that provide context-depended, variable designed support.

Applying these design requirements to a sensor assistant system fosters the idea of situation-dependent crew support by executing automated machine-processes.

The goal is to achieve a solution that enables the crew to guide the UAVs directly from the helicopters cockpit during flight, in which the UAVs automatically supply mission-relevant reconnaissance results from sensor-perceived and pre-evaluated data.

2 Sensor Operation Assistance Concept

2.1 Motivation

When sensor payload operation now is to be automated in the cockpit, two main factors regarding human operators need to be considered:

  • the effects on the crews’ MWL situation [3] during system operation

  • the operators’ “Trust in Automation” [5]

These factors are essential in human machine cooperation investigated in this MUM–T configuration.

The crew commanders’ working situation is mainly affected by the induction of MWL, caused by additional tasks in the field of reconnaissance sensor operation.

Especially perceiving and interpreting reconnaissance data causes additional MWL, depending on data bandwidth and degree of data abstraction. Therefore data bandwidth and data abstraction are the addressed independent parameters to influence the induction of MWL.

Regarding an operators’ “Trust in Automation”, the automation of airborne reconnaissance brings in domain specific conditions. Automated sensor evaluation systems often do not perform in a highly deterministic way, e.g. because of imperfection in sensor data evaluation [6] or varying operation environments. Out of a technical perspective, this circumstance can be addressed by the measure of “trustworthiness” of automated reconnaissance systems. In [7], a performance prediction method for image assessment algorithms is proposed. In this MUM–T application field, the trustworthiness of automated image assessment by algorithms directly affects reconnaissance performance. Figure 2 shows the effect “Trust in Automation” in the reconnaissance systems, besides existing effects that are well known in highly automated and complex cockpit environments (“Out-of-the-loop” [8], “Opacity-effect” [9, 10], “Over-Reliance” [11], “Brittleness” [9]).

Fig. 2.
figure 2

Causal relationship of “Trust in Automation” and inconsistent automation character

Full size image

As a result, decreasing automated reconnaissance performance or low trust in automation could lead to deeper operator involvement and more preference for executing a task manually, which both induces workload and contrasts the aim to reduce workload. In the event of decreasing automated reconnaissance performance, human MWL is required. Concluding, automated reconnaissance performance and MWL are contrary.

2.2 Solution Approach

The two dimensions, automated system performance (represented by trustworthiness) and human mental workload are triggers and dependencies for a variable interaction concept. So, the proposed general approach is to maximize the necessary reconnaissance performance by decomposing and balancing the antagonism of a tolerable workload and automated reconnaissance performance.

The proposed operation mode of assistance aims to purposefully modify the cooperative relationship between the human operator and machine processes. By adapting the reconnaissance system’s automation level, changes in the crews’ MWL-state are expected. According to [4], task transformation to an easier level is one major bullet in crew support, so changing to a higher automation degree is assumed to reduce workload and transform the non-manageable task demand situation to a manageable one. In contrast a forecasted decrease in machine reconnaissance performance would reduce the automation degree to preserve recon performance by involving more crew resources.

2.3 Realization

As described in [6], three major functional subsystems to implement such an assistance concept were introduced, realized by software implementations. They comprise

  • functionalities for data presentation and assessment on different levels of automation,

  • crew observation for workload based trigger generation as well as

  • decision making for automation level selection.

With respect to the first, variable support means by application of the “levels of automation” (LOA) paradigm [12, 13] in the domain of sensor deployment was presented in previous work [6]. Here a repository of tools for reconnaissance sensor operation was realized on several levels of automation. This toolset includes functionalities for automated data preprocessing and evaluation as well the automated control of the payload gimbal.

With respect to the last bullet point, a management component was implemented to select a suitable level of automation. The tradeoff between the crews MWL and a maximized automated reconnaissance performance is solved by a machine decision process [6], which is able to automatically adapt the degree of automation.

As a prerequisite for decision making, knowledge about the crew’s activity is needed. For this a task-model, containing knowledge of the crew’ task load and associated MWL when performing tasks with different automated support, is utilized. This model is embedded in software and linked to an online crew activity determination. For crew activity determination, an external crew supervision system is used to generate the workload measure of the human crew [14] by referring model knowledge of all crew activities. Such activity determination enables an associate system to trigger automation level changes detecting anomaly in task and workload appearance.

This paper focuses on establishing such a task-model.

3 Modelling of Task-Related MWL

3.1 Modelling Principle

In [3], a unified theory for a task representation and context-rich representation of MWL was presented. This concept is applied to a holistic human-machine system. The term task is used to denote a means of communication between a human operator and an associate system as well as an interface between components of the associate system themselves. The task construct was also used as expression to describe mental workload. Also, the operationalization of crew tasks by a task model was presented. As well, the derivation of associated MWL from such a task model was demonstrated and a model instance covering the execution of a MUM-T- transport helicopter cargo mission was presented [3].

According to the method introduced in [3], the structure consists of elements representing mission tasks, tasks, complex tasks, actions, properties and relations (such as alternative, inheritance) between them and evidences for occurrence (Fig. 3). With these elements, structures can be built up, representing human activity and demand resource allocation.

Fig. 3.
figure 3

Legend of model properties used for complex task representation

Full size image

3.2 Domain-Applied Model Generation

Focus of this work is now to create a task model instance representing aspects of sensor guidance from the transport helicopter’s cockpit. By applying the modelling principle, a depiction of different crew demands for collaboration with the automation system is aspired. For each necessary crew task performed on several automation levels, a corresponding demand representation was added to the task model. Beginning from the lowest elementary task type which is the first type to combine actions, a task structure was built up. For each elementary task, corresponding demands are assigned, represented by eight dimensions according to Wickens’ multiple resource theory [15] (Fig. 4). The representation covers demand components of information perception (visual spatial, visual verbal, auditory spatial, auditory verbal), information processing (cognitive spatial, cognitive verbal) and response (manual spatial, vocal verbal).

Fig. 4.
figure 4

Multiple resource model and demand vector according to [15]

Full size image

A tree representation follows which re-uses subtasks and gives a model instance for crew interaction with the automated sensor system (Fig. 5).

Fig. 5.
figure 5

Snapshot of task model describing crew interaction for several levels of automation

Full size image

The model instance explicitly describes all crew activities, system interactions and demands for different automation levels. As can be seen in the model snapshot in, the detailed view for one complex mission task, in this example the reconnaissance of a helicopter flight route, shows that this task can be performed by support of three automation levels or performed manually (Fig. 5). For activity determination, observables are used. As described in [3], so called “evidences” (observable facts) are assigned in the model representation. Different observation channels with sensors like buttons, touch-sensitive displays and eye gaze tracking are used in this application and associated with modelled “Action” elements. By applying online activity determination, crew activity can be distinguished for different automation levels, and an alternative, advantageous automation level can be proposed by the associate system’s workload projection if workload issues seem to occur.

4 Experimental Evaluation of Mental Workload

To investigate crew behavior in MUM-T missions, a dual-seat generic helicopter (HC) flight simulator, equipped with multi-touch displays and free configurable multi-function displays (MFD), was set up (Fig. 1). Graphical user interfaces for crew interaction with the automated reconnaissance system and visual reconnaissance data representation during flight missions were embedded in the helicopters MFDs (visible in Fig. 6).

Fig. 6.
figure 6

User interface with annotations concerning different types of data representation and human interaction (Color figure online)

Full size image

For crew observation, non-invasive, contact-less eye gaze measurement was applied. The used eye tracking system smarteye pro consists of four cameras per seat and provides functionality to capture, monitor and analyze the human operator’s eye gaze movement. Measured values were current gaze positions on displays. Using a geometric model of the current MFD configuration, semantic references to graphical content currently shown on the surface is created. Furthermore, gaze tracking is complemented by synchronously capturing haptic interaction data to get more explicit evidences.

The goal is to determine human demand parameters [15] to be deposited in the task model described above. Therefore, a raw interaction dataset of visual and haptic interaction was gathered in human-in-the-loop experiments by crew observation sensors.

4.1 Interaction Dataset Recording

We defined and prepared separate use-cases of typical task constellations with several UAVs to be evaluated in the MUM–T mockup mission-simulator. For each use-case representing the task execution on several automation levels, we isolated phases containing the performance of reconnaissance task from the cockpit. Within the experiment, the reconnaissance automation levels were applied within a task-based-guidance concept [16]. A data recorder collected all user interactions with the running automation systems and reconnaissance result representation on the MFDs surfaces.

Figure 6 shows the user interface with observed and recorded interaction values and types, consisting of visual and haptic user interaction. Red borders in Fig. 6 show the interaction fields and data types of different visual data representation on different automation levels. Data representation comprises three different types of data; a live video feed, rectified and georeferenced ground images (image mosaicing) as well as tactical symbols.

Observable visual evidences are the operators gaze position on the live video feed, on the rectified ground images and on tactical map elements. Haptic evidences are user inputs by button presses and gestures on the map. These evidences are directly mapped to actions. For each automation level, different user demands exist. For specifying these demands from the raw data record, the data was analyzed and evaluated by an algorithm based method.

4.2 Derivation of MWL

To derive demand parameters of the raw data set, the two indicators time and interaction event amount were chosen. The factor of event quantity per observation time is the bandwidth of evidence observations. The derived demand values are normalized on the maximal human demand (perception visual spatial and response manual) during manual performance. For making different source data bandwidths comparable, the same amount of reconnaissance result to be presented was configured.

The following section illustrates the automated processing cycle for parameter extraction by pseudocode:

4.3 Results

By evaluating the recorded dataset, demand components of information processing and response of the multiple resource model [15] were extracted successfully. For each automation level, the two most important components VS (visual spatial) and RM (response manual) could be determined.

Figures 7, 8, 9, 10 and 11 show the isolated interaction datasets of “complex” task [3] execution for all automation levels and one exemplary given single task of the task model instance. The corresponding demand parameters were reconstructed from the interaction dataset.

Fig. 7.
figure 7

Interaction dataset and observation interval for “unassisted” mode

Full size image
Fig. 8.
figure 8

Interaction dataset and observation interval for “Video Assisted” mode

Full size image
Fig. 9.
figure 9

Interaction dataset and observation interval for “Map Assisted” mode

Full size image
Fig. 10.
figure 10

Interaction dataset and observation interval for “Tactical Assisted” mode

Full size image
Fig. 11.
figure 11

Interaction dataset and observation interval for task “Identify targets”

Full size image

The observation interval in Fig. 7 shows the interactions of “unassisted” manual reconnaissance. From this observation interval, the reference value RM for normalization was reconstructed. The operator’s haptic interaction activity is the highest possible, the sensor was guided fully manual and sensor data had to be evaluated visually.

Figure 8 shows the interactions of the “Video Assisted” automation level. From this observation interval, the reference value VS for normalization was reconstructed. The operator’s visual interaction activity is the highest possible, monitoring moving image is assumed as the most demanding visual activity. Sensor guidance was executed by automation which means that no manual interaction was required.

In the “Map Assisted” mode, the ground images had to be analyzed. The same amount of reconnaissance data could be evaluated in a shorter normative interaction time interval. Manual interaction occurred when shifting and zooming the ground image map. The extracted values were scaled to the VS and RM maximal values.

The interactions in the “Tactical Assisted” mode (Fig. 10) occurred when the highest automation degree produced tactical elements on the map.

Finally, the single task “Identify targets” was performed (Fig. 11). Three simulated objects had to be analyzed and identified in the live video feed.

4.4 Result Interpretation and Reflection

By critically examining the demand parameter extraction and source dataset the fact became obvious, that there is no informational content about the demands of “information processing” components according to [15] (especially cognitive spatial) contained in the record. Another activity measurement type would be needed to retrieve such information content. The CS demand had to be determined by specialist knowledge applied in this domain. All the other information perception demand values did not occur in this task set (VV, AS, AV) and were set to 0, as well as the other response demand RV and the other processing demand CV.

In general, the applied method is able to fill the described parameters of larger models.

Analyzing systematic error sources, the experiment covered only several use-cases, not the complete task model parameter extraction. Therefore, small observation horizons were used that inherit the risk for high variance and normalizations become more prone to inaccuracy.

Operator observation processes, especially the gaze tracking method, is afflicted with measurement noise that may yield to wrong semantic associations of surface elements; such values were discarded in this experiment.

The goal of the campaign was to methodically determine specified model parts; the campaign and method is not representative in general. The results however show that the suggested automation levels reflect different user demands and the assistance approach is applicable for reduction of MWL.

5 Future Work

By connecting and linking the introduced subsystems, a software based chain to evaluate the sensor assistance concept was implemented in our MUM-T helicopter mission simulator. Future work comprises the application of all software modules in a full-mission scenario. A closed-loop-operation for functional demonstration of the holistic HC- associate system applying sensor automation with usage of the proposed sensor assistant system will be realized in the near future.

Experimental evaluation of the proposed concept as well as the effects on crew and mission performance by human-in-the-loop experiments with military transport helicopter crews is aspired.