Keywords

1 Introduction

“Simulation will help learning” is a popular mantra these days among many decision-makers; many times their action, unfortunately, is to run to purchase the latest technological toy with the hopes that learning will occur better. In reality, learning is a journey toward understanding the processes, procedures, backgrounds, culture, nuances, etc. of the particular topic under study, and is best accomplished with a spectrum of technological tools. In this paper, we discuss the basis for understanding how well designed simulations utilize multiple modalities as they support the learner on their journey, from the specific standpoint of learning purpose, training success, use of immersive technologies and confirming impact models.

The learner’s journey today and into the future has the opportunity to be supported by robust tools for creating and delivering interactive learning experiences, where dynamic simulations drive engaging scenarios. These experiences, when designed well, allow the learner to set parameters and drive events through decision-making so that the learner can scaffold experiences toward deeper understanding. Quality experience design uses multiple types of technology; with each doing what it does best for learning impact. From reading established text, to live discussions, to data-driven computer training exercises, to immersive simulation to test transfer to the operational environment, all work together so that each learner is successful in today’s complex workplace; a workplace that is increasingly interdisciplinary, telepresence-based, and intergenerational. That last element has promise to help illuminate foundations for personalizing learning.

Today’s learner has global connections readily available, and learning in today’s connected society means that the notion of intergenerational teaming is being tested regularly. We no longer always know the age/race/gender of the people we are interacting with online. We do not actually care; we care about the quality of their interaction with us, and the contextual quality of the value exchanged between us. Online, you would never know that “Tom”, the paleontologist that has been helping you understand Jurassic sea creatures, is 12 years old. How many of us would have even read Tom’s answer if we had known in advance his age? That simple example parallels thousands of transactions between humans and technology every day. Simulation allows the learner to explore based on the level of those exploring with them, or test ideas and concepts with those on a more advanced level. The psychographics of the people we interact with take priority over the demographics; the value is in the shared island of expertise. This holistic view begins to illuminate how we might support each learner, what prerequisites, additional challenges, even remediation, will uniquely help them progress toward mastery. What is their ability to handle workload? What motivates them to keep working through the difficult moments? How can they best be rewarded?

As that journey progresses the ability of a system to capture data on the learner’s behavior begins to build a personal learning library, based on unique choices by learners. As time progresses, the library can be analyzed to see progression toward performance metrics, or topic mastery, or situational awareness if targeting soft skills.

In 2012, a training project for the Orange County Fire Rescue Department looked at improving individual mastery of fire command for every lieutenant in the county fire division. Success in the training was the demonstration of mastery by each trainee. As will be described in another section of this paper, the training curriculum completely redesigned based upon an enhanced understanding of the necessary performance metrics for success. The final segment of the training was a series of sessions in an immersive simulator, mirroring a live fire scene in one of seven environments found in Orange County, Florida [1]. The training chiefs set up scenario parameters, and events unfolded as the trainee made and communicated decisions to the virtual firefighters on the scene. The system design provided the capability to capture decisions made or not made by the trainee, establishing a data file on their emerging command mastery of a complex multi-layered experience.

As the fire command project unfolded in development, visitors to the research facility from all different ages and backgrounds were given the opportunity to become fire professionals and try the simulation. It quickly became apparent that both adult and adolescent learners responded to the immersive simulation, and were quickly motivated to make decisions and change them based on the results. These observations led researchers to discuss a concept for a virtual world to host learning activities that focus on critical thinking and problem-solving, using performance-based design scaffolding from naïve to advanced understanding in real world environments. Researchers believe this initiative will have important information for the experience design field on using technology to help learner empowerment at all levels of competence, and build quality learning and training systems for all learners that leads to the ability to demonstrate mastery. In the next section, we discuss performance metric development as a precursor to technology choices.

2 Performance and Their Metrics as Foundation of Impact

The focus and the reason for training, especially when using advanced learning technology like simulation, are really to improve human performance. Whether this is to prepare a soldier for combat or a surgeon to perform surgery, training should always focus on enhancing the abilities of a human to perform some behaviors directed toward their job, home, or recreational pursuits. However, recent years shows a shift in focus from improving performance to understanding content and classroom presentation. This shift is partially due to remnants of the efficiency and the process improvement ideas of the industrial revolution, as well as the incomplete transition from traditional journeyman/apprentice training that relied on subject matter experts for content development.

The very issue is that technology is not the sole solution to improved performance outcomes after training. The learning content must accurately reflect the analysis of expert performance, which then can translate into a delivery system that improves the learner’s ability to process that information. The result of which is a positive change in behavior leading to successful execution of tasks. Thus, the role of the media delivery tool is to enhance the learning or training material for effective and efficient transfer of that information to the learner/trainee.

Human behavior, especially expert behavior, is both complex and hard to elicit. There are behavioral measures and methods for modeling expertise: however, it takes much more time to capture and translate this information to appropriate learner material and activities. Introducing technology simply adds a new method to an old and most often incomplete process.

In the next section, we describe an example of how to elicit expertise by using methods of ethnographic reporting followed by a breakdown of candidate behaviors into their sub-components and associated assessment measures of success in achieving those behaviors. These behaviors and measures translate into organized learning components for the training/learning material. This translation represents the Design phase of the Instructional System Design process. The model combines expert testimony and behavioral measures that account for several types and levels of learning, which requires different instruction design methods to meet the learner’s needs [2].

We demonstrate the use of technology as an enhancement to the training material in the Orange County Fire Rescue Department Incident Fire Command simulation-based training system. The training was targeted for new Company Officers (CO) who needed to lead a team to determine how best to put out a fire under different environmental and material conditions. To target gap areas in the current training, designers performed “ride-alongs” to acquire an ethnographic report of context, tools, interactions, communications, and others behaviors necessary for successful Incident Command (IC) performance.

The intent of the ethnographic report is to objectively describe the activities, personnel, tools, and behaviors across a series of different incidences. The designer organizes and analyzes the detailed activities and presents that information to an IC expert. The IC expert assists in capturing key cues and decisions, along with metrics of success at the more detailed levels. This process identifies gaps in performance between the senior IC personnel and the incoming CO.

The next aspect of this design process raises the level of IC contextual information by interviewing other firefighters and simulating other types of field related fire activities. Flow charts described steps in the IC process from arrival reports, to diagnosing the type of fire based on “reading smoke” and the type of building construction, to selection of different tactics and how to communicate tactics to the fire fighters. Comparisons between the current training content and the new training model identified gaps among successful outcome performance, trainee’s prior knowledge and current training methods.

The result of this approach was a detailed blueprint of IC training that began with 11 h of pre-training via the web. Desktop training provided guided lecture using case study incidents that demonstrated application and practice with feedback of successful IC resolution. Once the CO reached a level of competence, he or she was brought into an immersive simulation that required the CO to perform their job in a Fire Scene situation in which they had to be the IC and command the avatar firefighters in properly engaging the fire. IC experts critiqued the CO’s performance based upon criterion identified as critical in the initial phases of the design.

The success of this training implementation did not depend on a single training delivery platform. Each level of successful performance outcomes necessitated the correct technology that matched the purpose of the training. The mapping or transfer of the performance outcomes to the appropriate technology depended upon the accurately and systematically defined, detailed description of successful performance delineated through the ethnographic report, behavior deconstruction, along with subject matter expert input. By focusing on the performance outcomes and then choosing the types of technologies that best delivered that content, we achieved a high level of success in training a very complex and dangerous task. The ultimate test of training and learning is transfer from the training environment to the operational environment, and it is at this stage in the learning journey that emerging immersive technologies can efficiently and effectively assess transfer into application.

3 Value of Immersive Technology

As discussed, successful learning and training should use a range of technologies best suited for success toward a measurable training or learning goal. Just the way the abacus allowed a visual representation of numbers for merchants and students, so do computer graphics and virtual worlds allow learners today to visualize learning environments, people and events. Simulations allow us to interact in real-time within the context of learning environments. The actions and reactions are real; the movements, procedures, tactics and strategies are directly applicable to real world circumstances, with the benefit that simulations can collect rich data.

By designing learning environments with clear performance metrics and learning objectives as the foundation, the interactivity becomes seamless and intuitive, leaving only learners and mentors working toward a common goal. Along with individual content knowledge gathering, group discussions both face-to-face and online, and even computer-based small scale simulations, we now have rich emerging technologies with high levels of immersion, that allow us to more intuitively navigate the virtual worlds that we create. As the way we interface with the technology begins to mimic our natural impulses and motions, the complexity and types of learning environments we can achieve become more applicable to the diversity and complexity seen in everyday life.

To demonstrate application of training or learning as an outcome metric to the journey, we move to using the tools of the real world environment rather than the trappings of the training environment. Immersive simulations allow us to proceduralize tasks. Research has shown greater engagement with immersive VR and MR technologies because the training environment seems real to the trainees. Head mounted displays allow us to use our natural senses to experience virtual content. We move our heads and eyes to experience a virtual space as we would in reality. Tracking devices allow us to interface spatial information between the physical space and the virtual elements.

Mixed Reality (MR) engages the learner in direct, first person interaction with a real-time environment containing both physical and virtual assets and agents. Because one attains the highest level of realism and immersion, MR spaces can reduce costs and increase productivity. One cost saving aspect of MR is that the design no longer necessitates a large number of physical prototypes of learning environments. Engineers can now review their data in 3d physical space the same way they could review a costly physical mockup. Using MR they can walk around an object, and intuitively move it with their hands. Interacting with a life-size representation of the object allows one to understand the total system and how complex parts with their forces work together or in opposition. In a time when our virtual worlds are melding with the physical world on many levels, training and learning need to move effortlessly between the two realities.

This information exchange between the physical world and the learner, whether through technology or real world experience, must account for the active learning process of the human brain. The change to the individual during learning is an internal process demonstrated through external action. As we establish best practices of how to design learning experiences with these emerging technologies, we need to understand the impact of the technology and the learning models on the learning process itself. We explore this opportunity in the next section.

4 Effectiveness Measures in Training and Education

The previous sections discuss the journey of the learner through their lifespan and across different learning spaces (e.g., college courses or specific training). Each of these learning spaces has the potential to enhance the learning experience and the applicable knowledge outcomes of that experience using technology. From a neurobiological perspective, learning by definition is the process that constructs memories, while memory is the outcome of learning [3]. This operational definition of learning provides a foundation to: (1) determine effectiveness measures that can predict human performance and (2) define gaps and solutions for directly measuring the learning process. Currently, we determine the success or failure of learning by assessing an indirect measure of the outcome of the learning process: behavior or the memory guided appropriate action to accomplish a set of tasks. However, what a person actually knows may not be observable in behavior. Is this behavioral approach enough to determine the effectiveness or how well the learning transfers to the real world?

The contention of this section is that along the history of defining best practices in instructional design, we have also run parallel in defining how the brain learns across the life cycle. Many instructional design strategies are abstractions of common knowledge of how the brain functions. For example, cognitive load theory derives from the assumption that working memory is a short-term brain store for material the brain is currently processing and suffers from overloading. Thus, the appropriate delivery of learning content is one that does not burden this brain processor [4]. While researchers search for a means to validate this brain-based instructional design approach (ex. [5]), there is still a reliance on assumptions about how the brain integrates information across sensory modalities to facilitate the learning process. These assumptions are yet another abstraction of the models, and more often metaphors we use to understand the brain. Technology affords the opportunity to apply the close-enough external learning strategies and cues, while stimulating the internal learning processes without knowing all the details.

This “close enough” combined approach of learner appropriate content and compatible technology delivery systems is an intermediary step to better understanding how the brain learns in context, naturalistically during the process of learning. Investigations within military-relevant training suggest that electroencephalography (EEG) can validly assess signatures of attention, memory and workload during the learning process [6, 7]. These EEG measures also offer a reliable means to quantify accurately key aspects of information processing [8].

The field of virtual rehabilitation shows successful examples of choosing the right technology to deliver effective re-training or treatment to the patient [9]. The user-centered design of the therapy environment takes into account the separate and integrated contributions of the technology and therapy content. The use of psychophysiological measures provides a means to evaluate objectively the brain state changes of the patient and to monitor those changes that relate to positive retraining. More importantly, brain monitoring can occur in real world settings under conditions that reflect the learner’s true environment. We extended this method of user-centered assessment of learning content and technology delivery system to serious games for training [10]. Results from this work suggest that brain based instructional design theories are not completely accurate or generalizable across learning spaces.

The current and future challenge then is to understand the learning process within the context of the learning space (e.g., classroom or in the field). To accomplish this, we must go beyond our abstractions of how we think the brain learns and really understand the biological process of learning. In so doing, we can stop adding incomplete theories and assumptions to the decision of what content and what technology are best suited for the learning space.

5 Conclusion

In this paper, we explore the impact of technology and instructional design on the learner. Preliminary results from the Orange County Fire Rescue Department Incident Fire Command simulation-based training system show that this type of immersive simulation allowed any person experiencing the training to build bridges in their knowledge such that they could transfer their experience to the real world, sometimes in novel ways. However, the preliminary web-based training was necessary to build the knowledge base for the Fire Command Officers to improve their job performance in the simulation and the real world. Thus, multiple modalities of technology were needed to achieve successful outcomes. The future question is why and how does this technology coupled with the training context affect the learning process. For this, final step the use of unobtrusive psychophysiological measures is the key. This combined approach will take us from brain-based learning theory to application with true measurable success.