Keywords

1 Introduction

There exists a keen interest in the utilization of virtual and mixed reality training applications, specifically in the military domain. Increasing availability of technologies capable of integration into current training programs have made realizing these interests increasingly feasible recently. One particular domain area of focus in the present research is Call for Fire (CFF). CFF is a highly complex task required for requesting and directing artillery fire and close air support (CAS) over a specific target area, initiated by a Joint Forward Observer [1]. The CFF task is highly multidimensional, requiring a number of skillsets and proper communication between multiple personnel in order to execute a request effectively.

Current CFF training utilizes classroom and simulated components. Existing simulation hardware allows JFO trainees to learn the proper procedures for executing a CFF in a safe and repeatable environment. JFO training also requires that individuals maintain currency in their training by recertifying every six months. However, the current simulation technologies are difficult to relocate, are cost prohibitive, and lack a sense of engagement and immersion that can benefit training outcomes [2, 3]. One approach for simulation development has utilized Augmented Virtuality (AV) to help mitigate some of the prohibitive factors existing in typical training, as well as provide adequate training outcomes for CFF training programs.

The present research sought to examine the efficacy of training a CFF-CAS task that utilized AV technology for the simulation portion of training. This effort was undertaken in order to assess the overall effectiveness of an AV system for CFF training, as well as to determine the potential value of a more compact and visually immersive virtual environment for CFF training. The evaluation of the system’s training effectiveness followed the Kirkpatrick 4-level model [4, 5]. This model suggests that training may be evaluated along the constructs of learning, reactions, transfer, and impact. While the Kirkpatrick model supported the overall approach to the presented evaluation, the focus of this effort was to assess the viability of an AV-integrated version of a CFF trainer to train a CAS task. Therefore, a complete 4-level assessment was not conducted. Rather, the present evaluation focused on the learning (i.e., task performance) and trainee reaction (i.e., satisfaction) aspects of this model in an effort to address the following research questions:

  • Does AV provide an effective training environment for students undergoing CFF training (i.e., does the simulator promote learning measured by passing grades during a training scenario)?

  • Does AV technology promote high levels of satisfaction as a result of interaction with the simulator?

2 Background

2.1 Call for Fire and Close Air Support Training

A CFF typically requires a JFO to identify, transmit, and coordinate information between multiple parties in order to effectively guide indirect or close air support fires onto a given target. A JFO is a specialized soldier who receives training necessary to request such fires on a particular target within their target area [6]. During a typical mission, a JFO will obtain visual contact on a target or group of targets within a designated area from a nearby observation point. During a CAS mission, the JFO will acquire target information and communicate that information to a Joint Terminal Attack Controller (JTAC). The JTAC will verify the information passed by the JFO and, in turn, handle communication with aircraft pilots on station. During a CAS request, the JFO is responsible for obtaining target location, elevation, orientation, and providing information on the type of mark (e.g., laser designator, talk-on, etc.) for the aircraft to locate and verify targets. They are also responsible for providing feedback regarding the results of ordnance effects and making adjustments for reattacks, if necessary.

Executing a CAS request requires high levels of efficiency and accuracy from the JFO. In order to execute a CAS request, a JFO in the field must first obtain their position and location, identify any targets for CAS, and observe the immediate area for any threats to possible CAS aircraft. The JFO must communicate this information to his or her fires direction center or JTAC. After coordinating information in the previous step, the JFO must collect specific target or target set information, including a target description, elevation, and grid location, as well as direction and distance of friendly forces from targets and any flight restrictions (e.g., final attack heading and no-fly areas). The JFO must then relay this information to the JTAC. Unless the JFO has specialized equipment for marking the desired target, they must conduct a target talk-on with the pilot. This involves communicating directly with the aircraft pilot and guiding them visually to the target’s location. Once the pilot and JFO reach target correlation, the JTAC with authorize the pilot to begin their attack run and the pilot will drop ordnance on the target while the JFO observes and communicates any necessary corrections to munition targeting. Once the CAS mission is completed, the JFO relays battle damage assessment information and continues on the mission.

2.2 Current JFO Training

The Army requires one JFO per maneuver platoon [7]. Through training, the JFO trainee receives the necessary knowledge and skills to request surface-to-surface fires, naval CFFs, AC-130 CFFs, and CAS CFFs with a JTAC. JFO certification requires classroom, live, and simulated training exercises, as well as prior experience in a fires role (e.g., forward observer) before enrolling. Official JFO certification training takes place over two weeks and consists of a mixture of classroom and simulation exercises. Once certified, a JFO must maintain currency by accomplishing 13 live or simulated CFF events every six months. Current pass rate for JFO certification is approximately 75 % of individuals enrolled in the course [8].

Currently, the U.S. Army incorporates Simulation-Based Training (SBT) into the JFO course curriculum. The SBT systems can provide an immersive and engaging mission rehearsal space in a virtual environment. The current simulation system consists of a virtual CFF scenario typically projected onto a screen in front of a classroom. Students sit at a station that consists of equipment used during a CFF mission (e.g., binoculars, compass, map, etc.). Trainees are able to interact with this equipment in conjunction with the virtually projected environment in front of them.

However, this interaction requires the trainee to view a secondary monitor to observe how manipulation of the provided equipment alters the view of the environment. This action sometimes leads to incorrect handling and usage of the JFO’s equipment, as well as serve as a potential distractor from the actual training exercise. While the simulation may provide some level of immersion, the system as a whole lacks realism in terms of equipment usage and expectations [9].

2.3 Augmented Virtuality

While current training systems are able to produce effective JFO personnel, there are a number of potential ways to increase both the perceived realism and immersion of the simulated portion of CFF training. One method of promoting more realism throughout training and authentic interaction with the tools and equipment is to incorporate an AV approach to the SBT portion of training.

Milgram and Kishino [10] provide a reality-virtuality continuum that helps identify different types of mixed reality. At one end of the continuum lies reality, while at the other lies virtuality, consisting of a completely virtual environment; AV lies closer to the virtuality side. AV, by definition, is a type of mixed reality that includes a predominately virtual environment that maintains some form of interactive elements or objects in the real world. For example, a virtual surgical trainer may utilize real-life surgical tools in order for a trainee to interact with a virtual patient in a simulated environment [11]. The goal of such an approach is to both increase presence of the trainee during SBT and increase the realism of actions within a virtual environment (Fig. 1).

Fig. 1.
figure 1

A simplified representation of Milgram and Kishino’s (1994) Reality-Virtuality Continuum.

Full size image

One method of incorporating AV into current CFF training is by utilizing see-through head-mounted display (HMD) technology. The projection of the virtual environment within the HMD and incorporation of real-world, physical equipment (e.g., compass, binoculars, etc.) that map and correlate to movements in the VE should provide for higher feelings of realism and immersion during training. However, the effectiveness of such training within the CFF domain is uncertain. The present research sought to examine the viability of AV as implemented into CFF simulation training during an actual fires officer training course to obtain a more robust dataset to determine if the use of AV technology is effective for positive outcomes in task performance.

It was hypothesized that (1) performance scores for the CFF task would be significantly higher than the cutoff score for receiving a passing grade and (2) satisfaction ratings would be significantly higher than a neutral response on a subjective Training Satisfaction Survey (TSS).

3 Method

3.1 Participants

A total of 42 U.S. Army officers (32 male) undergoing the Basic Officer Leadership Course (BOLC) for Artillery Field Officers at Ft. Sill, OK were recruited for participation. The average age of these officers was 23.67 years (SD = 2.08). Additionally, participants had served an average of 22.19 months (SD = 24.19) in the military. Thirty-seven participants (88.1 %) indicated having experience on the CFF task either through simulation-based training or live training in the field. Additionally, participant data were collected during week 15 of their 18 week Field Artillery BOLC course, which included classroom instruction of the basic CFF and CAS principles. These participants were selected because they were considered a highly representative sample for this experiment as they were currently undergoing field artillery training, which the AV simulator was able to accommodate. Participation was restricted to U.S. citizens between the ages of 18-40 with normal or corrected-to-normal vision.

3.2 Experimental Materials and Design

Simulator and Hardware.

The Call for Fire Trainer-Augmented Virtuality (CFFT-AV) simulation system is a prototype simulation-based training testbed. The CFFT-AV system is a 7’ × 7’ × 7’ cube-shaped frame equipped with an acoustically-based position tracking system overhead, simulated military equipment, a table with a terrain map, and a partially occluded, see-through Head-Mounted Display (HMD). The HMD displayed a virtual environment consisting of a JFO’s observation point on a terrain set that mimicked the Ft. Sill practice range. The HMD allowed participants to view and interact with the physical tools and equipment within the simulator, while simultaneously projecting a digital representation within the HMD’s virtual overlay of the environment. While wearing the HMD, participants had the ability to turn their heads and body around in order to view an entire 360° virtual environment. The view within the HMD updated in real time based on the direction in which the participant was looking.

The equipment inside the simulator consisted of a terrain map, compass, binoculars, laser range finder, map pens, and a radio. Participants were able to physically interact with all objects in the real world. When the participants picked up and held either the compass, binoculars, or laser range finder close to the front of the HMD, as though they were using an actual device, virtual representations populated within the viewable area of the HMD, changing the image relative to the tool being used. Other tools were placed in clear view on top of the table (Fig. 2).

Fig. 2.
figure 2

CFFT-AV hardware and trainee interaction tools for the CAS task

Full size image

Scenario and Task.

The CFF scenario consisted of a Type 2 CAS request. The virtual environment took place in a large, open field with numerous targets blocking main roadways. The scenario took place under clear weather conditions and optimal visibility with no direct enemy fire. The objective was to clear the roadways of targets to allow freedom of movement on the main surface roads. This scenario mimicked one used for the simulation portion of BOLC training.

Dependent variables included performance metrics for the completion of the CFF/CAS task. Performance was evaluated using the official scoring methods and scorecards for BOLC simulation training, which provided a score of out of 50 possible points. A score of 40 was required to receive a “Go” (i.e., pass). Scores were broken down on the scoresheet into smaller subtasks, allowing students to earn partial points based on the subtask completed successfully. There were a total of 6 subtasks with 1-8 components each for a total of 18 items, ranging between 1-5 points each.

Task completion time was also collected via audio recordings of radio transmissions. Timing metrics included completing the task within 20 min of identifying a target or target set, time to provide the JFO target brief, and duration of the target talk-on with the pilot.

The Training Satisfaction Survey (TSS) consisted of 44 items. Participants rated the degree to which they agreed or disagreed with each item on a 5-point scale (i.e., 1 = Strongly disagree; 5 = Strongly agree). The items on the survey provide subscale measurements related to individual satisfaction with the way the training provides or supports information regarding objectives, support for learning, problem solving, feedback, fidelity, and utility [12].

Procedure.

Participation took place within a block of approximately 1 h, with each participant completing the task individually. Participants first reviewed an informed consent for their participation and were given an opportunity to ask any questions regarding the experiment. After providing consent, participants completed an initial set of questionnaires to provide information about themselves, including demographics, military experience, and familiarity with the CFF task. Once completed, participants were briefed by a Subject Matter Expert (SME) regarding the situation and scenario, as well as the objective of each participant acting as the JFO within the simulation. The SME was one of three certified JTACs and highly familiar with the CAS procedures used in the scenario (Appendix A).

Once briefed, participants were provided with a functional overview of the HMD and AV environment, as well as instructed on how to interact with the equipment within the simulator and communicate with the instructor during the experimental session. After becoming familiarized with the simulation equipment, participants began completing the JFO CAS request with the instructor over the radio.

Participants were given approximately 30 min to complete the entire simulation task. Once completed, they were asked to complete a set of post-exposure questionnaires including the TSS. After completion of the final questionnaires, participants were debriefed on their performance in the simulator and the nature of the research, provided with an opportunity to ask questions, and were dismissed.

4 Results

Preliminary data checks were conducted to ensure no extreme outliers were present in the data. Results revealed no extreme outliers present in the data. However, one participant’s data was excluded from the final analysis due to their nominal skills in the area of land navigation and being unable to complete a significant portion of the scenario in the allotted time. Additionally, since multiple SME also served as graders, consistency between ratings was examined to ensure scores between graders were consistent. Results from this comparison revealed no significant differences between graders on task performance scores (t(40) = −.493, p =.625).

4.1 Performance on the Call for Fire Task Scenario

Results for overall simulation scenario performance were compared to the minimum BOLC simulation passing score of 40. Thirty-five out of 42 participants were able to achieve passing scores (83.3 %). The average score for all participants (M = 43.98, SD = 5.29) was significantly higher than the minimum passing score of 40 (t(41) = 4.88, p < .001).

Prior experience with CAS training, either virtually or live, was examined to determine whether this factor had any effect on performance scores. Results showed that 88.1 % of participants had some prior training with CAS. However, this training did not have a significant effect on performance results between groups (Mann-Whitney U = 79.5, p = .612).

4.2 Time Performance

Participants had 20 min from the time they identified a target for CAS to guide ordnance on the target from the aircraft. Per BOLC standards, failure to complete this task within the given 20 min resulted in an automatic failure. A One-Sample t-test revealed that average time from target identification to contact of ordnance on a target (M = 17:18, SD = 4:31) was significantly lower than the BOLC standard (t(41) = −3.88, p < .001).

However, further examination of the data revealed that only 31 out of the 42 participants were able to complete the task within the given 20 min time limit, meaning that 73.8 % were eligible to receive a passing mark on their performance based on the time allotted for scenario completion. Those who completed the simulation within the allotted time also had higher scores on their overall performance within the simulator than those who did not complete the scenario in time. In addition, other time measurements (i.e., Target brief and Talk-on) were also completed significantly faster by those scoring higher on the simulation performance metric. The descriptive data and statistical results are presented in Tables 1 and 2, respectively.

Table 1. Means and SDs for task perfomance measures
Full size table
Table 2. T-test results between groups on time performance
Full size table

4.3 Subjective Satisfaction with Training

Results from the six TSS subscales were compared to a neutral scale score (i.e., 3 = neutral). One-Sample t-tests were conducted to compare scores. All TSS subscale scores were significantly higher than the neutral score, indicating that participants were generally satisfied with the training they received from the CFFT-AV (Table 3).

Table 3. Results from One-Sample t-tests on the TSS Survey
Full size table

5 Discussion

The goal of the present experiment was to determine the overall effectiveness of a CFF trainer that incorporated AV simulation training technology. The CFFT-AV system relies on AV to provide increased realism while training by incorporating an HMD to present a 360-degree virtual world around the trainee rather than using a projector or monitor screen. Preliminary research exploring AV technology has outlined how the technology provides both high immersion and increased realism in procedural training of CFF-related tasks [12]. The findings of this evaluation add to that literature by providing insight into the relationship between incorporating AV into SBT with an HMD and performance outcomes.

It was predicted that the CFFT-AV would provide adequate training for the CFF-CAS request scenario, as well as result in a generally positive reaction from the participants. Performance results indicated that the majority of participants were able to achieve a passing grade on their CAS scenario. Additionally, approximately three-quarters of participants were able to achieve a passing grade within the acceptable time-to-completion from target identification to ordnance on target. Considering that this was the first time many of the participants performed a CAS request with a JTAC, whether live or simulated, these numbers indicate that the CFFT-AV system is an effective training medium for the BOLC course and JFO training. Despite the system being in a prototype stage, trainees were still able to obtain positive performance scores using the system.

Participants responded to the training in a generally positive way, supporting the second hypothesis. BOLC students reported highly on all satisfaction subscales, indicating that they felt as though the CFFT-AV system provides an adequate amount of information and/or support to individuals training the CFF task. In its current form, the integration of the CFFT-AV simulator into CFF-CAS training for BLOC students led to an effective and positive training experience. Increased satisfaction during SBT and learning are linked to increased levels of motivation [14], which are both related to increased levels of training performance and learning outcomes [15]. As such, future iterations of the CFFT-AV system should adhere to a similar functional design as the current system in order to maintain or improve upon current levels of learner satisfaction.

The CFFT-AV simulator was able to evoke high levels of performance and satisfaction ratings from participants. While this supports the overall effectiveness of the system in general, this effort was unable to attain performance scores and subjective satisfaction from the current CFFT program of record simulator for comparison. Based on available literature, the student pass rate appears to be on par with that of the CFFT-AV [7]. Still, future research that examines the two systems independently may yield findings that better define the differences between the overall effectiveness of the CFFT-AV and the current CFFT training systems.