PerFECT: An Automated Framework for Training on the Fly

  • Hari Thiruvengada
  • Anand Tharanathan
  • Paul Derby


Currently available cognitive training systems can highly benefit from more adaptable and encapsulated frameworks that include better performance assessment methods, robust feedback mechanisms and automated mechanisms that reduce the manual intervention and curriculum management required during training sessions. In short, there is an ardent need for an automated human in the loop training system that can effectively train cognitive skills required for military operations. An automated training system would be extremely beneficial if it can be easily coupled with a synthetic learning environment to function autonomously is an entirely data driven manner. Such a system would enable rapid deployment of key training scenarios, skills and tactics to war fighters and help them maintain a superior level of competence in the battlefield. An automated framework for training on the fly also known as performance feedback engine for conflict training (PerFECT) which includes key components for simulating training scenarios, measuring trainee’s performance, providing relevant feedback and dynamic curriculum management is discussed in this chapter. First, the training system comprises of custom plug-in interface that allows components of the training framework to readily interface with a simulated virtual learning environment. Second, it has a “Performance Evaluator” that enables automated, real-time and objective evaluation of a trainee’s performance grounded within an objective framework known as time window and enables run-time evaluation of performance skills based on a skills matrix. Third, PerFECT has a “Feedback System” that can provide contextual and immediate feedback to trainees based on process measures. Finally, PerFECT includes a “Curriculum Manager” that dynamically selects appropriate training scenario from a template library with varying levels of complexity. The selection algorithm for training scenario is based on the trainee’s historical performance scores and complexity of the earlier scenarios. We also present the initial findings from a pilot study which helps illustrate the capabilities of the framework and conclude with future directions in this area of research.


Virtual Environment Training System Automate Speech Recognition System Training Scenario Performance Evaluator 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work supported by DARPA/IPTO under contract# HR0011-09-C-0102. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the DARPA/IPTO. We would like to thank Amy Vanderbilt (DARPA-IPTO), David Montgomery (DARPA-IPTO) and Joseph Traugott (US RDECOM-STTC) for their guidance and mentoring on this project. We would also like to thank Tim Stone (Omega Training, Cubic Corporation) for his support and shaping our understanding of the training provided to the Fire Teams. We would also like to thank members for CATT Lab (University of Maryland) including Michael Pack, Walter Lucman, Michael VanDaniker and Michael Couture for helping us understand the capabilities of the OLIVE Virtual Environment. Second Life is a registered trademark of Linden Research, Inc. OLIVE is a trademark of Forterra Systems Inc (now part of SAIC). All other trademarks used herein are the property of their respective owners.

The views, opinions, and/or findings contained in this article/presentation are those of the author/presenter and should not be interpreted as representing the official views or policies, either expressed or implied, of the Defense Advanced Research Projects Agency or the Department of Defense.


  1. Anderson PL, Rothbaum BO, Hodges L (2001) Virtual reality: using the virtual world to improve quality of life in the real world. Bulletin of the Menninger Clin 65:78–91CrossRefGoogle Scholar
  2. Bainbridge WS (2007) The scientific research potential of virtual worlds. Science 317:472–476CrossRefGoogle Scholar
  3. Balzer WK, Hammer LB, Sumner KE, Birchenough TR, Martens SP, Raymark PH (1994) Effects of cognitive feedback components, display format, and elaboration on performance. Organ Behav Hum Decis Process 58:369–385CrossRefGoogle Scholar
  4. Barnett J (2009) Virtual Environments and unmanned systems: human system integration issues. In: Cohn J, Schmorrow D (eds) The PSI handbook of virtual environments for training and education: developments for military and beyond. Praeger Security International, WestportGoogle Scholar
  5. Bewley WL, Chung GKWK, Delacruz GC, Baker EL (2009) Assessment models and tools for virtual environment training. In: Schmorrow D, Cohn J, Nicholson D (eds) The PSI handbook of virtual environments for training and education: developments for military and beyond, vol 1. Praeger Security International, WestportGoogle Scholar
  6. Billings CE (1997) Aviation automation: the search for a human-centered approach. Erlbaum, MahwahGoogle Scholar
  7. Cannon-Bowers JA, Salas E (1997) A framework for developing team performance measures in training. In: Brannick MT, Salas E, Prince E (eds) Team performance assessment and measurement: theory, methods, and applications. Erlbaum, HillsdaleGoogle Scholar
  8. Cannon-Bowers JA, Salas E (2001) Reflections on shared cognition. J Orgmet Ch 22:195–202Google Scholar
  9. Cannon-Bowers JA, Bowers CA, Sanchez A (2008) Using synthetic learning Environments to train teams. In: Sessa VI, London M (eds) Work group learning: understanding, improving and assessing how groups learn in organizations. Lawrence Erbaum Associates, New YorkGoogle Scholar
  10. Canon-Bowers JA, Burns JJ, Salas E, Pruitt JS (1998) Advanced technology in scenario-based training. In: Cannon-Bowers JA, Salas E (eds) Making decisions under stress: implications for individual and team training. APA, WashingtonCrossRefGoogle Scholar
  11. Charness N, Schultetus RS (1999) Knowledge and expertise. In: Durso FT, Nickerson RS, Schvaneveldt RW et al (eds) Handbook of applied cognition. Wiley, West SussexGoogle Scholar
  12. Cohn J, Nicholson D, Schmorrow D (2009) Integrated systems, training evaluations, and future directions. The PSI handbook of virtual environments for training and education: developments for military and beyond. Praeger Security International, WestportGoogle Scholar
  13. Department of the Army (2002) Battle drills for the infantry rifle platoon and squad, ARTEP 7–8Google Scholar
  14. Dieterle E, Clarke J (2008) Multi-user virtual environments for teaching and learning. In: Pagani M (ed) Encyclopedia of multimedia technology and networking, 3rd edn. Idea Group, HersheyGoogle Scholar
  15. Endsley MR (1995) Measurement of situation awareness in dynamic systems. Hum Factors 37(1):65–84CrossRefGoogle Scholar
  16. Ericsson KA, Krampe RT, Tesch-Romer C (1993) The role of deliberate practice in the acquisition of expert performance. Psychol Rev 100:363–406CrossRefGoogle Scholar
  17. Foltz P, LaVoie N, Oberbreckling R, Rosenstein M (2009) Automated performance assessment of teams in virtual environments. In: Schmorrow D, Cohn J, Nicholson D (eds) The PSI handbook of virtual environments for training and education: developments for military and beyond, vol 1. Praeger Security International, WestportGoogle Scholar
  18. Gordon SE (1994) Systematic training program design: maximizing effectiveness and minimizing liability. Prentice Hall, Englewoods CliffsGoogle Scholar
  19. Green DM, Swets JA (1988) Signal detection theory and psychophysics. Peninsula Publishing, Los AltosGoogle Scholar
  20. Grier RA, Warm JS, Dember WN, Matthews G, Galinsky TL, Szalma JL, Parasuraman R (2003) The vigilance decrement reflects limitations in effortful attention, not mindlessness. Hum Factors 45:349–359CrossRefGoogle Scholar
  21. Grubb PL, Warm JS, Dember WN, Berch DB (1995) Effects of multiple-signal discrimination on vigilance performance and perceived workload. Hum Factors Ergon Soc Annu Meet Proc 39:1360–1364Google Scholar
  22. Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of experimental and theoretical research. In: Hancock PA, Meshkati N (eds) Human Mental Workload. North Holland Press, Amsterdam pp 139–183CrossRefGoogle Scholar
  23. Johnston JH, Smith-Jentsch KA, Cannon- Bowers JA (1997) Performance measurement tools for enhancing team decision making. In: Brannick MT, Salas E, Prince C (Eds) Assessment and measurement of team performance: Theory, research, and applications. NJ: Erlbaum, Hillsdale pp 45–62Google Scholar
  24. Kirwan B, Ainsworth LK (1992) A guide to task analysis. Taylor & Francis, BristolGoogle Scholar
  25. Kluger AN, DeNisi A (1996) The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull 119:254–284CrossRefGoogle Scholar
  26. Kozlowski SWJ, Toney RJ, Mullins ME, Weissbein DA, Brown KG, Bell BS (2001) Developing adaptability: a theory for the design of integrated-embedded training systems. In: Salas E (ed) Advances in human performance and cognitive engineering research, vol 1. JAI/Elsevier Science, AmsterdamGoogle Scholar
  27. Lampton DR, Bliss JP, Morris CS (2002) Human performance measurement in virtual environments. In: Stanney KM (ed) Handbook of virtual environment: design, implementation, and applications. Erlbaum, MahwahGoogle Scholar
  28. Lee J, Moray N (1992) Trust, control strategies and allocation of function in human-machine systems. Ergonomics 35(10):1243–1270CrossRefGoogle Scholar
  29. Lewandowsky S, Little DR, Kalish M (2007) Knowledge and expertise. In: Durso FT, Nickerson R, Dumais S, Lewandowsky S, Perfect T (eds) Handbook of applied cognition, 2nd edn. Wiley, West SussexGoogle Scholar
  30. Loomis JM, Blascovich JJ, Beall AC (1999) Immersive virtual enviornment technology as a basic research tool in psychology. Behav Res Methods instrum comput 31:557–564CrossRefGoogle Scholar
  31. Mantovani F (2001) VR learning: potential and challenges for the use of 3D environments in education and training. In: Riva G, Galimberti C (eds) Towards cyberpsychology. IOS Press, AmsterdamGoogle Scholar
  32. Meister D (2004) Conceptual foundations of human factors measurement. Earlbaum, MahwahGoogle Scholar
  33. Moray N, Inagaki T, Itoh M (2000) Adaptive automation, trust, and self-confidence in fault management of time-critical tasks. J Exp Psychol Appl 6:44–58CrossRefGoogle Scholar
  34. Mosier KL, Skitka LJ, Heers S, Burdick M (1997) Automation bias: decision making and performance in high-tech cockpits. Int J Aviat Psychol 8:47–63CrossRefGoogle Scholar
  35. Nicholson D, Schmorrow D, Cohn J (2009) The PSI handbook of virtual environments for training and education: developments for military and beyond, vol 2. VE components and training technologies. Praeger Security International, WestportGoogle Scholar
  36. Oser RL, Gualtieri JW, Cannon-Bowers JA, Salas E (1999) Training team problem-solving skills: an event-based approach. Comp Hum Behav 15:441–462CrossRefGoogle Scholar
  37. Parasuraman R, Riley V (1997) Humans and automation: use, misuse, disuse, abuse. Hum Factors 39:230–253CrossRefGoogle Scholar
  38. Rasmussen JR (1983) Skills, rules and knowledge: signals, signs, symbols, and other distinctions in human performance models. IEEE trans syst man cybern 13:257–266Google Scholar
  39. Rasmussen J (1986) Information processing and human-machine interaction: an approach to cognitive engineering. Elsevier, New YorkGoogle Scholar
  40. Riva G (1997) Virtual reality in neuro-psycho-physiology: cognitive, clinical and methodological issues in assessment and rehabilitation. IOS Press, AmsterdamGoogle Scholar
  41. Roman PA, Brown D (2008) Games—just how serious are they? Interservice/industry training, simulation and education conference (I/ITSEC). Academic Press, OrlandoGoogle Scholar
  42. Rothrock L (2001) Using time windows to evaluate operator performance. Int J Cogn Ergon 5:1–21CrossRefGoogle Scholar
  43. Salas E, Cannon-Bowers JA (1997) Methods, tools, and strategies for team training. In: Quinones MA, Ehrenstein A (eds) Training for a rapidly changing workplace: applications of psychological research. APA, WashingtonGoogle Scholar
  44. Salas E, Oser RL, Cannon-Bowers JA, Daskarolis-Kring E (2002) Team training in virtual environments: an event-based approach. In: Quiñones MA, Ehrensstein A (eds) Training for a rapidly changing workplace: applications of psychological research. APA, WashingtonGoogle Scholar
  45. Salas E, Cannon-Bowers JA (2001). The science of training: A decade of progress. Ann Rev Psychol 52:471–499CrossRefGoogle Scholar
  46. Salas E, Priest HA, Wilson KA, Burke CS (2006) Scenario-based training: improving military mission performance and adaptability. In: Adler AB, Castro CA, Britt TW (eds) Military life: the psychology of serving in peace and combat. Praeger Security International, WestportGoogle Scholar
  47. Sarter NB, Woods DD (1997) Team play with a powerful and independent agent: operational experiences and surprises on the Airbus A-320. Hum Factors 39:553–569CrossRefGoogle Scholar
  48. Schmorrow D, Cohn J, Nicholson D (2009) The PSI handbook of virtual environments for training and education: developments for military and beyond, vol 1. Learning, requirements and metrics. Praeger Security International, WestportGoogle Scholar
  49. Schultetus S, Charness N (1999) Recall vs position evaluation revisited: the importance of position-specific memory in chess skill. Am J Psychol 112(4):555–569CrossRefGoogle Scholar
  50. Shaffer DW, Resnick M (1999) "Thick" authenticity: new media and authentic learning. J Interact Learn Res 10:195–215Google Scholar
  51. Smithers JW, Wohers AJ, London M (1995) A field study of reactions to normative versus individualized upward feedback. Group Organ Manag 20:61–89CrossRefGoogle Scholar
  52. Sterling BA, Burns CA (2004) Skills required for platoon leaders in the objective force unit of action. Army Research Lab Aberdeen Proving Ground, MDGoogle Scholar
  53. Swets JA (1996) Signal detection theory and ROC analysis in psychology and diagnostics: collected papers. Earlbaum, MahwahMATHGoogle Scholar
  54. Tharanathan A, Derby P, Thiruvengada H (under review) Training for metacognition in simulated environments. In:Human-in-the-Loop Simulations. Springer, LondonGoogle Scholar
  55. Thiruvengada H, Rothrock L (2007) Time windows based team performance measures: a framework to measure team performance in dynamic environments. Cogn Tech Work 9:99–108CrossRefGoogle Scholar
  56. van Buskirk W, Cornejo J, Astwood R, Russell S, Dorsey D, Dalton J (2009) A theoretical framework for developing systematic instructional guidance for virtual environment training. In: Schmorrow D, Cohn J, Nicholson D (eds) The handbook of virtual environment training. Praeger Security International, WestportGoogle Scholar
  57. van Dongen KW, Tournoij E, van der Zee DC, Schijven MP, Broeders IAMJ (2007) Construct validity of the LapSim: can the LapSim virtual reality simulator distinguish between novies and experts? Surg Endosc 21:1413–1417CrossRefGoogle Scholar
  58. Verdaasdonk EG, Stassen LP, Monteny LJ, Dankelman J (2005) Validation of a new and simple virtual reality simulator for training of basic endoscopic skills. Surg Endosc 20:1–9Google Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  • Hari Thiruvengada
    • 1
  • Anand Tharanathan
    • 1
  • Paul Derby
    • 1
  1. 1.Honeywell ACS LabsMinneapolisUSA

Personalised recommendations