Keywords

1 Introduction

1.1 Overview

The adaptive training research program strives to provide innovative instructional practices to facilitate and enhance the delivery and assessment of learners. At the center of this program is intelligent tutoring via the Generalized Intelligent Framework for Tutoring (GIFT), which, as an Intelligent Tutoring System (ITS), can provide tailored learning content based on learner proficiencies that are assessed through a course [19]. In establishing needs for adaptive tutoring systems to better support military tasks, Sottilare [18] notes that ITSs need to acquire learner data, assess learner state, and select an optimal instructional strategy to meet the learners’ need. Applied to the military domain, the problem space is often complex, ill-defined, and highly subjective in nature. Therefore, this project seeks to work closely and collaboratively with military educators such that the capabilities of GIFT can serve as an augmentation rather than a replacement for existing instructors.

1.2 Background

Within the Army, one of the most significant locales to connect with military educators is the United States Military Academy at West Point (USMA). The adaptive training research program has a long-standing relationship with USMA, and has recently developed a modeling and simulation cell located physically at West Point and staffed by two adaptive training scientists. This provides opportunities to better engage and integrate with the West Point students, faculty, and staff for a mutually beneficial relationship of meeting ARL and USMA goals.

1.3 Collaboration Efforts

Since the interest of this project was the complex military domain, going to the educators who teach military instruction was a natural pairing. The Department of Military Instruction (DMI) aims to provide the necessary military training to ready cadets from a military perspective. Most instructors within the department are active military personnel and as such, the department experiences a frequent turn-over rate. While the benefit of this is that cadets receive instruction fresh off the field, the limitation of this is that instructional design and learning objectives often lack necessary scaffolding and reinforcement from year to year. Therefore, there has been a growing consensus as to the importance of developing a GIFT course that not only encapsulates necessary domain knowledge about military tactics necessary for military preparedness, but is flexible enough that can be adaptable and flexible for authoring modifications by instructors from year to year.

As such, after the development of a successful capstone working with cadets and GIFT to assess squad and platoon level military tactics [1], there was a desire to examine how technologies like GIFT could further impact military instruction. It is from this, that a new partnership was forged to provide the first steps in the use of GIFT in the military instruction curriculum.

Members from the Army Research Laboratory’s (ARL) adaptive training research program began to have collaborative meetings with the departmental faculty of DMI. The goal of these meetings was to help identify a specific area where functionality such as that which exists in GIFT would be able to assist instructors in delivery of their classes. The aim was to find a problem and task which this combined team could investigate together. From the very beginning of these discussions, it was clear that retention was an area that could use some focus. With lesson content in military instruction being spaced across several semesters with summer training breaks in between, there is ample opportunity for cadets to forget content and therefore necessitating refresher training frequently as a part of classes. An area of interest for military instruction is augmenting existing classroom activities such that information is retained better by cadets, reducing the need for refresher training and improving assessment scores.

An additional challenge faced by the military learner is the amount of content expected to be learned. The amount of information is due to the structure of the course and the utilization of scaffolding (i.e., progressing learner knowledge incrementally with more and more complex problem sets), precisely mirroring the format of the Army Operations Order. If a learner (cadet) does not engage the content and instructor at the onset and remain engaged throughout, it will have a negative aggregate effect on their overall course grade. During the Mission Analysis portion of the course, the learners are learning three specific content areas: (1) the conceptual framework (explained later in the document); (2) the placement of the Mission Analysis input into the Operations Order; and (3) the language or formatting of language to generate content to into the Operations Order. One specific aspect of mission analysis is enemy analysis, which is the focus of this research.

The use of GIFT to support this augmenting existing classroom activities is specifically relevant to DMI and the Military Science (MS) instruction series due to changes in the way the class periods are being restructured. The course, which is 1.5 credit hours, has traditionally been organized in 40 class periods with 55 min per period, equating to 3600 min over the course of the semester. This is being modified for the 2019 academic year to reflect 30 class periods at 75 min per class period. It is documented in the literature that there is a correlation between the amounts of time a learner is exposed to content, and the proficiency in accomplishing a task [6].

As mentioned above, there is a large breadth of material that instructors must cover to achieve ultimate course outcomes. These outcomes are represented in the progression for a learner from the previous course (MS100), to the current course (MS200), to the following courses (MS300 and Cadet Leadership Development Training - CLDT). The ability to have an increase in depth of content, and an increase in number of repetitions to facilitate better retention via conventional pedagogical techniques does not currently exist, which drives DMI to look for Live, Virtual, Constructive, and technologically advanced means (like intelligent tutoring via GIFT) to increase the efficacy and customize the learning experience for each student. The approach of using GIFT in this application within a military classroom can be described as an epistemic process of the assessing and understanding individual student, identifying where the problem areas or gaps in knowledge-building exist, and reinforcing them with a technologically efficient means. GIFT is able to support the conjunction of the vast amount of information needed to be learned and the need to retain it over long periods of time, minimizing skill decay of enemy analysis concepts.

2 Skill Decay and Enemy Analysis

2.1 Identification of the Problem: Skill Decay

Skill decay occurs when skills are not used and the ability to execute suffers. The amount of decay varies in accordance to the task and their dependence on cognitive and psychomotor information elements [8, 17]. Deterioration of performance is further compounded if skills have not been reinforced or have been newly learned. Performance is often determined by the level of experience someone has and how frequent they are trained on the task of interest [6].

Major dimensions that are often discussed in skill decay research include: length of non-use period, how much overlearning occurred, characteristics of the skill, testing methods of previous learning, type of retrieval, method of training, individual differences, and motivation [7, 8, 10, 15, 20]. There is well-established research that has shown a direct relationship to the rate at which forgetting occurs and the amount of controlled rehearsal associated with the task [14]. Additionally, skill decay contributes to loss of confidence in performing a necessary skill [7].

2.2 Refresher Interventions

One of the ways to minimize skill decay is the use of refresher interventions. Refresher interventions are techniques that assist in re-attaining skill proficiency after it was lost due to skill decay. It has been shown that different refresher interventions effect skill and knowledge retention differently [10].

Symbolic rehearsal is one such refresher intervention, defined as the visualization of a task without actually performing the task [10, 11]. Practice problems consist of examples of the course material applied to actual learning scenarios. Since symbolic rehearsal has been shown in previous studies [9, 10] to be as effective as practice for knowledge retention but not for skill retention, recent research has proposed including process visualization tasks to provide a procedural component as well [11]. To this end, the domain area of military instruction is well suited to determine whether symbolic rehearsal will mitigate skill decay, within the subdomain of enemy analysis.

2.3 Conceptual Overview of Enemy Analysis

Enemy analysis is an area that has been identified by USMA instructors as being particularly susceptible to skill decay for cadets. Enemy analysis is a complex topic because it requires a multistep process to be successful.

Enemy analysis requires the learning of many different symbolic aspects of the battlefield and representation of the enemy, as well as incorporating other elements, such as terrain and weather, as elements that need to be woven into a cohesive narrative as a part of their assignment, which is essentially a simulated briefing. This briefing is executed by cadets after approximately four weeks of instruction in the form of an operations order, which is graded and assessed by their instructors.

The briefing task revolves around building an operations order to support enemy analysis. The operations order contains information on maneuver, which provides information on each the enemy unit’s task and purpose. The second part of the operations order consists of purposes, priorities, allocation of, and restrictions for fire support. It also contains information on intelligence, supplies, commander’s intent, and protection. These are known as the warfighting functions. And it is these warfighting functions that cadets must not only achieve content mastery and retention to accomplish their classroom task, but they must carry this domain knowledge with them as they proceed to their next year, next level of military instruction coursework.

There are 6 warfighting functions: mission command, movement and maneuver, intelligence, fires, sustainment, and protection:

  1. 1.

    Mission Command: The mission command warfighting function that allows a commander to balance the art of command and the science of control in order to integrate other warfighting functions.

  2. 2.

    Movement and Maneuver: The movement and maneuver warfighting function moves and employs forces to achieve a position of relative advantage over the enemy and other threats.

  3. 3.

    Intelligence: The intelligence warfighting function assists in understanding the enemy, terrain, and civil considerations.

  4. 4.

    Fires: The fires warfighting function provide collective and coordinated use of Army indirect fires, air and missile defense, and joint fires.

  5. 5.

    Sustainment: The sustainment warfighting function provides support and services to ensure freedom of action, operational reach and prolong endurance.

  6. 6.

    Protection: The protection warfighting challenge preserves the force so the commander can apply maximum combat power to accomplish the mission.

According to Army Doctrine Reference Publication (ADRP) 3.0, warfighting functions are related tasks and systems united by a common purpose or objective that allow commanders to accomplish mission or training goals [5].

2.4 Assessment of Enemy Analysis

Currently, the enemy analysis content module spans over three lessons coinciding with three formative assessments and one summative assessment. The first assessment, cadets are provided a snapshot of an enemy squad on a given area of terrain with a specified task and purpose. Cadets are individually assessed on their ability to provide a doctrinal template of how that squad is comprised (also known as composition and depicted in the form of a line-wire diagram) via numbers of personnel and equipment.

Additionally, they have to demonstrate their understanding of how to graphically depict the same information on a map with four required components: breakdown of the squad-sized element into two team-sized elements; provide a task and purpose for each team, demonstrating that they are mutually supportive of one another (adhering to the concept of nesting) and graphically depicting the coinciding tactical mission task for each respective element; a depiction of the key weapon systems of each team and providing their proper orientation (i.e. RPG Variant, RPK, and PKM); and all labeling adhering to the standard as prescribed by ADRP 1-02 [4].

During the second assessment, they are paired in groups of two, given an entire mission analysis prompt (including terrain, light & weather, and enemy analysis) and asked to conduct analysis on each respective input, brief it to the instructor in front of their peer cohort, and receive constructive feedback (with a weighted grade) in an attempt to allow them a complete deliberate repetition prior to receiving the mid-term assessment with a 20% course grade weight value. Finally, with the current course design, the content transitions to offensive operations during the latter half of the semester with a culminating summative assessment that encapsulates enemy analysis, forcing cadets to demonstrate retrieval.

3 Collaborative Methodology

3.1 Understanding the Problem Space

After a series of meetings and sharing of documents between ARL researchers and USMA instructors, qualitative observations were conducted in CPT. Robert Davis’s cadet classes. From this step in the methodology, we elicited the declarative knowledge elements associated with the content. For the purposes of enemy analysis, these include the baseline understanding of four fundamental tenets: composition, disposition, strength, and capabilities, which are defined below:

  1. 1.

    Composition: describes how an entity is organized and equipped - essentially the number and types of personnel, weapons, and equipment.

  2. 2.

    Disposition: refers to how threat/adversary forces are arrayed on the battlefield/battlespace. It includes the recent, current, and projected movements or locations of tactical forces.

  3. 3.

    Strength: is described in terms of personnel, weapons, and equipment. The most important aspect of strength when evaluating a regular force is to determine whether the force has the capability of conducting specific operations.

  4. 4.

    Capabilities: an analysis that must determine what the enemy is capable of doing against a friendly platoon during the mission. Such an analysis must include the planning ranges for each enemy weapon system that the platoon may encounter.

Members of the ARL research team took this content and translated it into a basic prototype GIFT course to demonstrate proof of concept (see Fig. 1).

Fig. 1.
figure 1

Example of declarative knowledge course flow in GIFT

Full size image

3.2 Understanding the Technology Space

The next step in the methodological process was determining how actual content was being delivered as part of a lesson. To accomplish this, qualitative observations were conducted in CPT. Davis’s class. During this evaluation we identified the key concepts, domain knowledge, and learning objectives of enemy analysis, where these observations helped identify common patterns of misconceptions as well as help requests that cadets articulated during class time. The purpose of these observations was to help guide the design of the GIFT course in a more dynamic and adaptive manner.

In its current form, GIFT provides adaptive lesson capabilities based on individual differences (e.g., prior knowledge, motivation, grit, etc.) and real-time embedded assessments. GIFT applies a domain-agnostic pedagogical approach that is based on David Merrill’s Component Display Theory (CDT [12]), where content is structured in a way to support the presentation of material (i.e., in the form of general rules of a domain and specific instances of those rules applied as seen by an example) and the assessment of material through dedicated question banks that are configured on a concept by concept basis. This theoretical approach was used in the design of GIFT’s Engine for Management of Adaptive Pedagogy (EMAP [3]), which uses the CDT as the framework by which an instructor configures lesson material, with the design goal that the EMAP can apply across all notional cognitive problem domains.

In the context of enemy analysis, GIFT’s EMAP provides the framework for an instructor to establish specified content (i.e., Rules and Examples) they want to present on a concept by concept basis, along with the tools to establish assessment questions and scoring parameters that drive performance outcomes. What this development supports is a closed-loop remediation model, where each individual cadet will receive a personalized experience based on their learner model and on the outcomes of the GIFT managed assessments. Following an assessment event, GIFT will either progress a trainee on to the next lesson activity, if all scoring thresholds were satisfactory, or GIFT will initialize a remediation loop that targets the specific concepts that scored below expectation. This model uses a focused-coaching strategy that targets only the concepts that require further instruction, so as to key in on each individual’s strength and weaknesses.

An important note linked to the EMAP is its dependency on assessments to drive personalized remediation loops. For the initial implementation, the enemy analysis assessments in GIFT will utilize question bank approaches to infer the domain concepts that require further attention. In future iterations, there is room to incorporate more focused scenario-based exercises that leverage simulation environments. These practice events extend the assessment space by enabling more focused scenario-based exercises that focus on application of skill, rather than recall of knowledge.

Key Challenges Identified.

In the first round of observations, CPT Davis engaged his cadets in discussions about the six warfighting functions. Importantly, CPT Davis had cadets work in groups to discuss one of the six warfighting functions, including identifying and utilizing information from the ADRP 3.0 doctrine, and describing in layman’s terms what the information meant for enemy analysis [5]. What this activity demonstrated was not only that there should be a consideration for having a functionality in GIFT for collaborative work, but more importantly to consider providing an opportunity for cadets within GIFT to teach each other. This pedagogical approach to learning is well aligned with dialogic teaching along the tradition of Dewey [2] and Vygotsky [21] where students are involved in the collaborative construction of meaning and share control over classroom discourse [16].

Observations also yielded information that there should be a consideration in the GIFT course to incorporate “help” functions in the form of hyperlinks or sidebar content for particularly complex ideas. This “help” function would be well aligned with the adaptive functionality of GIFT in recognition that not all learners need additional reinforcement and help in the same way and with the same frequency. Lastly, observations revealed the importance of incorporating within the enemy analysis GIFT course the need to include dynamic graphical supports such as maps and key legends. These two elements could serve a dual function: as a symbolic reinforcer as well as an additional assessment instrument.

Ultimately the final design of the Enemy Analysis course will be an iterative process that includes using key learning objectives articulated by DMI, a course design informed both by Merrill’s Component Display Theory and educational psychology principles, where the content validity is established by current DMI instructors. Once this phase of the project is executed and validated, a longitudinal empirical study will be conducted to evaluate how refresher interventions improve the retention of military instruction compared to traditional learning methods.

4 Experimental Design

For the purposes of the anticipated experiment, three different cadet sections will be divided into three experimental groups across three time periods: Initial Training (IT), Refresher Intervention (RI), and Retention Assessment (RA) with two weeks of spacing in between – spanning a time from the summer before the year of instruction to the end of the fall semester. The same cadets will be tracked through all three time periods (See Fig. 2).

Fig. 2.
figure 2

Experimental design

Full size image

The experiment will examine how refresher interventions using GIFT can impact cadet performance. The two refresher interventions that will be manipulated will be practice problems (condition 1) and symbolic rehearsal (condition 2). The control condition will not include any refresher interventions.

To further support these refresher interventions in conditions 1 and 2, feedback will also be given via GIFT. One of the consistent findings through ITS research is that it is important to provide training activities that offer refresher interventions that also provide feedback tailored to the student [13]. Learning outcomes will be measured via a pre-posttest design, to determine if the refresher interventions had any impact on skill decay in conditions 1 and 2, independently in the IT, RI, and RA time periods, and longitudinally from IT to RA. Repeated measures Analysis of the Variance (ANOVA) will be used to analyze statistical significance of outcomes and interaction of trait metrics.

4.1 Procedure

To help provide a more concrete idea of how the experiment would be executed, we have created a hypothetical procedure that the experiment would follow:

  1. 1.

    The cadet enters class and opens up a GIFT course, which is presented via the Web.

  2. 2.

    The cadet is shown large amounts of declarative knowledge elements in the form of bolded terms that the cadet will be asked to pay attention to.

  3. 3.

    The cadet will be provided with several example scenarios explaining how the knowledge elements are used. To make the task more challenging, more terminology will be interspaced in the text of the example scenarios.

  4. 4.

    In all conditions, a pre-assessment survey will be administered to establish baseline knowledge.

  5. 5.

    In the control condition, the cadet would receive traditional instructor-based training using PowerPoint slides.

  6. 6.

    In the practice problems condition, the cadet would be presented with a scenario and a series of multiple choice questions as to the proper definitions or courses of action based on that scenario.

  7. 7.

    In the symbolic rehearsal condition, the cadet would be asked to freely recall as many of the previously bolded terms as they can based upon the scenario.

  8. 8.

    In both conditions, the cadet would then receive feedback on the answer they provided with a justification as to why that was or was not the correct answer.

  9. 9.

    Two weeks after the administration of the test, the cadet will receive a surprise pop quiz on the content. The format of the quiz would match their experimental condition.

  10. 10.

    GIFT would then provide customized remediation based on content that the cadet got wrong. Each one of the bolded items from the original course would be tagged with one or more course concepts. Any course concept that a cadet receive more than 20% incorrect answers, he or she would receive remediation on.

  11. 11.

    Two weeks later, all conditions undergo a retention assessment in GIFT.

4.2 Establishing Research Questions

Research Questions.

Based on previous research on refresher interventions and skill decay, the following research questions are proposed:

  1. 1.

    Can the use of refresher interventions improve the retention of military instruction content in comparison to traditional learning methods?

  2. 2.

    How do different types of refresher interventions impact assessment performance by cadets?

Hypotheses.

These questions will determine whether we can reject or accept the following hypotheses:

  1. 1.

    The experimental conditions will have higher rates of retention than the control condition (supporting research question 1).

  2. 2.

    The practice experimental condition will have higher rates of retention than the symbolic rehearsal condition (supporting research question 2).

4.3 Next Steps and Future Work

The near term next steps include determining what resources and capabilities are needed to facilitate the research as well as an experimental research protocol to lay out specifics and responsibilities by the participating organizations. It will most likely require additional support in the creation of GIFT content to meet DMI’s curriculum need.

In the longer term, the specific content being created will have to be developed incrementally. The first type of content is the declarative knowledge assessment that can be presented in GIFT and easily captured via the GIFT survey system. The second type of content is the ability to produce diagrams that are captured on paper. GIFT currently does not have existing functionality that can capture these diagrams, therefore an effort will be made to investigate the translation of line wire diagrams to GIFT course objects. Once the declarative knowledge elements and the line wire diagrams can be combined into a GIFT course, it will be more feasible to represent existing classroom activities, especially with an emphasis on group collaboration.

5 Conclusion

Although this project is still in its infancy, the synergy between ARL’s Adaptive Training research program and USMA’s Department of Military Instruction serves as a promising classroom use case of GIFT to assist in the mitigation skill decay. It will provide insight in the use of different types of refresher interventions and inspire new research questions for future investigation. It assists in identifying a path for longer range retention studies using GIFT and enhancing GIFT’s capabilities to support military learners at USMA. An added benefit of this collaboration will be to identify ways that GIFT can assist the instructors to have a better sense of what is occurring within their classes with the goal of providing more efficient and effective learning for all.