Keywords

1 Introduction

1.1 Behavior Cue Analysis

Behavior cue analysis techniques provide the perceptual skills necessary for Warfighters to maintain their situation awareness in a combat environment and mitigate the occurrence of a critical incident [4]. Training in behavior cue analysis prepares Warfighters to proactively detect threats among the human terrain along various axioms of human behavior such as physiological indicators, body language, and socio-cultural behavior patterns [5]. The scope of the behavior cue analysis tasks in this experiment included the detection and classification of non-verbal behaviors. The process of behavior cue analysis begins with establishing an environmental baseline, identifying anomalies, and then selecting a course of action [6].

Traditionally, behavior cue analysis and related strategies are trained in classroom-based and live training settings. Currently, instructional design efforts are progressing towards Simulation-Based Training (SBT) of behavior cue analysis type tasks in virtual settings. However, SBT lacking in constructive pedagogy is both ineffective and inefficient, and has shown to elicit negative training [7]. Behavior cue analysis is a largely perceptual task, therefore, selected SBT strategies should improve perceptual ability.

1.2 Instructional Strategies

Literature on training and education provides evidence to support the idea that implementing instructional strategies into SBT can effectively improve the perceptual skills of military personnel [8]. In this experiment, two strategies were chosen for further investigation due to their relevance to improve perceptual skills–Massed Exposure and Highlighting. Massed Exposure, also referred to as massed practice, consists of presenting a high volume of stimuli within a reduced time period [8]. Highlighting refers to directing the attention of the learner to significant training content utilizing a non-related content element [9, 10]. Previous empirical research suggests that utilization of the Massed Exposure or Highlighting strategies during virtual behavior cue analysis training may improve trainees’ response time for the detection of target behaviors [4, 11]. Although performance outcomes are a critical consideration in the design of effective SBT, considering the trainee experience may improve SBT efficiency. There is increasing interest among the instructional design community to incorporate SBT strategies that support a scaffolding model of instruction through the maintenance of a trainee’s Zone of Proximal Development (ZPD).

1.3 Zone of Proximal Development

Scaffolding involves the adaptation of instructional delivery methods or content to guide the learner through a gradual acquisition of the knowledge or skill until mastery and independence are achieved [1, 2]. In SBT, scaffolding models may bridge the gap between trainees’ immediate knowledge and skill and their potential level of understanding when scaffolded support is provided. This gap represents the optimal level of learning referred to as ZPD. The goal of ZPD is to provide learning opportunities that encourage and advance the ability to accomplish a task or conduct a skill independently [12]. During instruction, ZPD may be maintained dynamically through the adjustment of instructional support and challenge levels [3]. Theoretically, the optimal level of learning in the ZPD range may be achieved through pedagogical methods that offer enough challenge to extend and develop a learner’s knowledge and proficiency without overwhelming them [2]. Constructs that relate to ZPD are stress state and intrinsic motivation. High levels of stress during a task may indicate that an individual’s processing of cognitive information is being overloaded due to the delivery method of instruction. In addition, the task at hand may be too demanding, which inhibits skill acquisition. In contrast, task engagement and motivation may decrease if the challenge level is too low, therefore, underutilizing the learner’s cognitive resources.

1.4 Stress

Within the stress state construct, task demands are classified into three dimensions: task engagement, distress, and worry [13, 14]. Task engagement includes feelings of arousal, interest, motivation, and concentration, while distress is characterized by an unpleasant state of tension indicative of low confidence and low perceived control. Worry refers to feelings of low self-esteem, cognitive interference, and feelings of task-induced self-consciousness.

1.5 Intrinsic Motivation

Intrinsic motivation is herein defined as the internal desire to perform well on an external task as is influenced by the perceived level of effort, improvement, competence, pressure, tension, interest, and enjoyment during a task [15]. Evidence suggests that intrinsic motivation positively impacts performance outcomes [16], and low levels of intrinsic motivation have been correlated with indifference [17, 18].

1.6 Research Objective

The objective of this study was to assess the effect of the instructional strategies on stress and intrinsic motivation experienced during training as well as the relationships between these constructs in a SBT context for the behavior cue analysis task. Ultimately, the goal of this investigation was to take the initial steps towards exploring the efficacy of leveraging stress state and intrinsic motivation to measure ZPD during SBT and monitor the effectiveness of scaffolding strategies.

2 Method

2.1 Participants

A total of 123 participants from the University of Central Florida and the surrounding community were used for this experiment. The following inclusion and exclusion criteria were used to recruit participants: between 18 and 40 years of age, U.S. citizenship, and normal or corrected to normal vision. In addition, to adhere to the U.S. Army vision requirements and previous experimentation, full color vision according to the Ishihara’s Tests for Colour Deficiency was also a requirement [19]. A total of eight participants were omitted, due to technical difficulties, voluntary discontinuation of participation, and failure to meet the proficiency requirements. Data from 115 participants, 56 female and 59 male, were analyzed. The ages ranged from 18 to 33 years (M = 22.05, SD = 3.02). Monetary payment or class credit was provided as compensation.

2.2 Experimental Testbed

Virtual Battlespace 2 (VBS2) version 2.0 software was used to develop the virtual environment scenarios in this experiment. The scenario terrains were Middle Eastern and Culturally Agnostic. Culturally Agnostic is a non-geotypical environment. Twelve distinct virtual 3D models of four skin tones (i.e., fair, light, medium, and dark) were used to display a total of eight human behavior cues, four target and four non-target. The virtual models were arranged in groups of four alternated between the left and right sides of the route.

2.3 Experimental Design

This experiment assessed the effectiveness of two instructional strategies (i.e., Massed Exposure and Highlighting) against a Control condition by conducting a between groups design.

2.4 Measures

The Dundee Stress State Questionnaire (DSSQ) is a validated measure that was used to measure task related stress by three 0-32 point scales that comprise task engagement, distress, and worry [13, 14].

The Intrinsic Motivation Inventory (IMI) consists of individually validated scales selected for their relevance with the experimental task domain. The selected scales contain statements related to four aspects of intrinsic motivation including: Effort and Improvement, Pressure and Tension, Perceived Competence, and Interest and Enjoyment [15]. Participants rated their agreement with each statement along a 1-7 point Likert scale with “not at all true” and “very true” as the minimum and maximum scale anchors, respectively.

2.5 Procedure

Each participant was given the Informed Consent document to read and sign. The document provided the experiment’s purpose, tasks, minimal risks, and benefits, as well as the participant’s rights as a volunteer. To verify whether the participant fulfilled the study restrictions, the pre-experimental questionnaire and color deficiency test were then administered by the researcher. Afterward, the participant filled-out a paper-based demographics questionnaire and was randomly assigned to one of the following conditions: control, massed exposure, and highlighting.

To acquaint the participant with the virtual environment and experimental task of monitoring the virtual UGS, selecting the classification button, and clicking on the detected target, an interface training presentation was then provided. Following the presentation, the participant practiced the target detection and classification task by completing an interface training scenario. To avoid priming effects, the targets used were yellow and red colored barrels which were unrelated to the stimuli in the succeeding scenarios. To continue the experiment, the participant must have received at least 75 % detection accuracy at the end of the scenario. All participants were provided up to two opportunities to obtain the proficiency score.

Then, a slide presentation explaining the task for the pre-test scenario was given. The participant was asked to monitor the virtual UGS display, as well as detect and classify targets that appeared to be exhibiting aggressiveness or nervousness according to the participant’s personal experience. After completing the pre-test scenario, the participant was given a five minute break.

Next, another slide presentation was provided to present the training content. This gave an overview of behavior cue analysis and the Kinesics domain, as well as described each target behavior cue and the corresponding classification. Examples of each target behavior cue were supplied in the form of photographs. If the participant was assigned to the highlighting or massed exposure conditions, content on the instructional strategy as it applied to the training scenario was also included. Then the participant completed two training scenarios. After each training scenario, the participant completed a computer-based DSSQ.

Following the training phase, an additional presentation was provided on the practice scenarios that followed. Once again, the participant completed the computer-based DSSQ after each practice scenario. After finishing the scenarios, a five minute break was offered. After the break, a final slide presentation was given introducing the post-test followed by the post-test scenario. Upon completion of the post-test scenario, the participant completed the computer-based IMI. Finally, the participant was debriefed and dismissed.

3 Results

Two-way mixed ANOVAs with scenario type (training 1, training 2, practice 1, and practice 2) as the within-subjects factor and instructional strategy (Control, Massed Exposure, and Highlighting) as the between-subjects factor were conducted to assess the main effects and interaction effects for the Distress, Engagement, and Worry scales of the DSSQ. All reported degrees of freedom for the interaction effects and main effects of scenario type reflect the Greenhouse-Geisser correction for sphericity in SPSS.

There was a significant main effect of scenario type for the Distress scale, F (2.49, 279.27) = 6.83, p < .001, η 2 P  = .057, with decreases in distress from the first to the second training scenario followed by an increase in distress for the first and second practice scenarios (Fig. 1). This was qualified by a significant interaction effect between scenario type and instructional strategy for the Distress scale, F (4.99, 279.27) = 2.95, p = .013, η 2 P  = .050.

Fig. 1.
figure 1

Individual group means for the DSSQ Distress scale per scenario

Full size image

There was also a significant main effect of scenario type for the Engagement scale, F (2.54, 284.23) = 16.09, p < .001, η 2 P  = .126, with a trend toward decreasing task engagement over time in the Control and Massed Exposure conditions (Fig. 2). The Highlighting condition had a decrease in task engagement from the first to the second training scenario, an increase in task engagement for the first practice scenario, and another decrease in task engagement by the second practice scenario. There was also a significant interaction effect between scenario type and instructional strategy for the Engagement scale, F (5.08, 284.23) = 6.76, p < .001, η 2 P  = .108.

Fig. 2.
figure 2

Individual group means for the DSSQ Engagement scale per scenario

Full size image

For the Worry scale, there was a significant main effect of instructional strategy, F (2, 112) = 2.95, p = .040, η 2 P  = .056. This was qualified by a significant interaction effect between scenario type and instructional strategy for the Worry scale, F (4.80, 268. 56) = 2.78, p = .020, η 2 P  = .047. There was trend toward increased worry in the Massed Exposure condition. Although the Worry scores were significantly lower in the Control compared to the Highlighting group, both conditions had relatively stable levels of worry across scenarios (Fig. 3).

Fig. 3.
figure 3

Individual group means for the DSSQ Worry scale per scenario

Full size image

One-way between groups ANOVAs revealed a significant effect of instructional strategy for each IMI scale including: Effort and Improvement, Perceived Competence, Pressure and Tension, and Interest and Enjoyment (Table 1).

Table 1. Between groups ANOVAs results for the IMI scales.
Full size table

Post hoc analyses using Bonferroni correction revealed significantly lower scores (p < .01) in the Highlighting condition compared to the Control and Massed Exposure conditions for all IMI scales (Fig. 4). However, the Control did not differ significantly from the Massed Exposure condition for any IMI scales.

Fig. 4.
figure 4

Individual group means for IMI scales. Error bars indicate standard error

Full size image

Pearson’s r correlations were conducted to assess the relationship between IMI scales and DSSQ scales per scenario type. Across conditions, there were significant positive correlations between each IMI scale and the DSSQ Distress score per scenario suggesting that increased distress was associated with greater motivation (Table 2). The strength of the correlations for the training scenarios were stronger than those for the practice scenarios.

Table 2. Correlations between DSSQ distress and IMI scales across conditions
Full size table

Between groups, only the Massed Exposure and Highlighting conditions revealed significant correlations between worry and intrinsic motivation (Table 3). In the Massed Exposure condition, the Effort and Improvement, Perceived Competence, and Interest and Enjoyment scales of the IMI were positively correlated with the DSSQ Worry scale in the first training scenario only. In the Highlighting condition, the Effort and Improvement and Interest and Enjoyment scales were positively correlated with the DSSQ Worry scale in the first and second training scenarios. Additionally, the Perceived Competence scale was positively correlated with DSSQ Worry in training scenario two. The Pressure and Tension scale of the IMI did not correlate significantly with the DSSQ Worry scale. Further, the DSSQ Engagement scale did not significantly correlate with any IMI scales.

Table 3. Correlations between DSSQ worry and IMI scales for the massed exposure and highlighting groups.
Full size table

4 Discussion

In all three conditions, the level of distress decreased by the second training scenario indicating that repeated exposure to a task during training may gradually reduce trainee distress overtime, regardless of the type of instructional support provided. However, while the level of distress in the Control and Massed Exposure conditions remained relatively stable through the practice phase, the Highlighting condition revealed a nearly three point increase in distress from the second training scenario to the second practice scenario. This considerable increase in the Highlighting condition suggests that the removal of explicit, highly indicative instructional support, such as the non-content feature used to signal targets, may increase trainee distress when completing the task independently. Due to the target detection assistance provided by the non-content feature in the Highlighting training scenarios, participants in this group were ultimately required to only classify the target behavior cues. The Highlighting participants may not have employed perceptual skills for target detection, such as pattern recognition [8], during training scenarios, and thus, the amount of distress may have increased because the participants expended greater effort to conduct the entire behavior cue analysis procedure (i.e., detection and classification) during practice scenarios. The correlations between distress and the level of effort and improvement reinforce this phenomenon.

Participants in both the Control and Massed Exposure conditions experienced a gradual decrease in the level of task engagement from one training or practice scenario to the next. The decrease in task engagement in the Control and Massed Exposure groups may be attributed to boredom or disinterest in the task due to the similarities in the presentation of content between the training and practice scenarios. The Highlighting group indicated more fluctuation in task engagement between scenarios. Perhaps the novelty of the non-content feature peaked the participants’ engagement in the task during the initial exposure, but their engagement waned by the end of the training phase. Likewise, the initial absence of the non-content feature in the first practice scenario may have prompted participants to re-engage in the task, yet, the level of engagement dropped once again by the end of the practice phase, possibly due to disinterest.

Per condition, the level of worry remained fairly stable with approximately a one to two point fluctuation from one scenario to the next. The Highlighting group revealed the highest level of worry overall. During training scenarios, perhaps participants receiving the Highlighting strategy were concerned that they were not familiar with the detection aspect of the task because targets were indicated for them. Conversely, perhaps during the practice scenarios when target detection assistance was not provided, the participants were worried that they may have missed some targets. Additionally, although the change was not significant, the increase in worry during the practice scenarios in the Massed Exposure condition may indicate that participants were worried that they were missing targets because the task may have seemed simpler due to the reduced target probability compared to the training scenarios.

Intrinsic motivation was lowest in the Highlighting condition, while the higher levels in the Control and Massed Exposure conditions were similar. The positive correlations of distress and worry with the IMI subscales suggest that participants’ level of stress may have impacted their intrinsic motivation, regardless of condition. Combined with the two-way ANOVA results, it is possible that the level of challenge elicited by each instructional strategy affected participants’ perception of their own ability to conduct the task successfully as well as their motivation to remain engaged in the task. Perhaps the training scenarios applying the Highlighting strategy were too easy. Interestingly, distress was low and worry was high during the Highlighting training scenarios. Distress is more closely related to the task experience, while worry is more aligned with external influences [13, 14]. Therefore, low distress may be an indicator that the instructional support did not provide enough challenge and high worry may relate to a lower level of intrinsic motivation. Conversely, the greater distress and lower worry in the Massed Exposure condition may indicate that the Massed Exposure strategy offered a preferred level of challenge during training, and, consequently, greater intrinsic motivation. However, the IMI was administered only once upon the conclusion of all scenarios. The single instance of the measure may have reduced its sensitivity to adequately assess intrinsic motivation. In order to further delineate the impact of intrinsic motivation on stress state and vice versa, future experimental designs should administer the IMI in the same manner as the DSSQ after each scenario.

5 Conclusion

Ultimately, the intention of this effort was to expand the body of instructional design research related to the measurement and monitoring of ZPD and the effectiveness of scaffolding methods in SBT. Clearly, it is evident from the results of this experiment that the Highlighting strategy provided the lowest level of challenge and reduced intrinsic motivation, while the Massed Exposure strategy provided a greater level of challenge and increased intrinsic motivation. From a scaffolding perspective, incorporating the Highlighting strategy into SBT may diminish the capability of the training platform to maintain a trainee’s ZPD. Although further empirical research is necessary, according to the results herein, the Massed Exposure strategy may be the preferred option to maintain a trainee’s ZPD during SBT.