Pilot Performance Assessment in Simulators: Exploring Alternative Assessment Methods

  • Pete McCarthyEmail author
  • Arnar Agnarsson
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10906)


Flight crew performance and competency assessment are daunting tasks requiring expertise and training, and still will not be possible without a certain degree of subjectivity. On the other hand, collecting reliable data on flight crew competencies is at the core of Evidence-based Training, a major modernization of training methodology the industry as a whole has embarked on. Data from assessment informs training departments about where pilots seem to be lacking in proficiency, so those issues can be addressed in initial and recurrent training programmes of airlines. The effectiveness of the training hinges on the quality of the data. Accurate interpretation of the data is crucial for the decisions on training to respond to the true needs of commercial pilots. The industry has made a great effort to develop ways to measure crew performance. The role of Human Factors in incidents and accidents has been known for a long time, and the need to assess and train Human Factors has been identified. In recent years, with the introduction of Evidence Based Training (EBT) there has been a shift in focus from task-based assessment to competence based assessment. This study analysed crew performance in 25 videos from simulator sessions in a high fidelity full flight simulator. A checklist of Desired Flight Crew Performance (DFCP) was used to distinguish between high and low performing crews. Then the performance of selected crews was analysed in detail, using Performance Indicators (PI) as developed in EBT. The findings suggest that while the DFCP method was useful for the classification of high and low performing crews, the PI method provided detailed information for the understanding of underlying factors that affected the performance of the crews. The study also considers the value of using PI to understand and emulate well executed flying and problem solving, to change the focus of training from the study of error and accidents, to training best practices and safe operation.


  1. Barry Issenberg, S., Mcgaghie, W.C., Petrusa, E.R., Lee Gordon, D., Scalese, R.J.: Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med. Teach. 27(1), 10–28 (2005). Scholar
  2. BEA: Sx-Bhs, August 2013Google Scholar
  3. Cannon-Bowers, J.A., Salas, E.: Team performance and training in complex environments: recent findings from applied research. Curr. Dir. Psychol. Sci. 7(3), 83–87 (1998). Scholar
  4. Dekker, S.: The Field Guide to Understanding Human Error. Ergonomics, vol. 51 (2006).
  5. Ericsson, K.A., Ward, P.: Capturing the naturally occurring superior performance of experts in the laboratory: toward a science of expert and exceptional performance. Curr. Dir. Psychol. Sci. 16(6), 346–350 (2007). Scholar
  6. EU: Commission Regulation (EU) No. 965/2012. Official Journal of the European Union, 5 October 2012Google Scholar
  7. Field, J.N., Mohrmann, F., Fucke, L., Grácio, B.C.: Flight crew response to unexpected events: a simulator experiment. In: AIAA Modeling and Simulation Technologies Conference (2016).
  8. Flin, R., Martin, L., Goeters, K.-M., Hörmann, H.-J., Amalberti, R., Valot, C., Nijhuis, H.: Development of the NOTECHS (non-technical skills) system for assessing pilots’ CRM skills. Hum. Factors Aerosp. Saf. 3(2), 95–117 (2003).,
  9. Fowlkes, J.E., Dwyer, D.J., Oser, R.L., Salas, E.: Event-Based Approach to Training (EBAT). Int. J. Aviat. Psychol. 8(3), 209–221 (1998). Scholar
  10. Grossman, R., Salas, E.: The transfer of training: what really matters. Int. J. Train. Dev. 15(2), 103–120 (2011). Scholar
  11. Harris, D.: Human Performance on the Flight Deck. CRC Press, Boca Raton (2012)Google Scholar
  12. Helmreich, R.L., Klinect, J.R., Wilhelm, J.A.: Models of threat, error, and CRM in flight operations. In: Proceedings of the Tenth International Symposium on Aviation Psychology, pp. 677–682 (1999)Google Scholar
  13. Helmreich, R.L., Merritt, A.C., Wilhelm, J.A.: The evolution of crew resource management training in commercial aviation. Int. J. Aviat. Psychol. 9(1), 19–32 (1999). Scholar
  14. Hollnagel, E.: Is safety a subject for science? Saf. Sci. 67, 21–24 (2014). Scholar
  15. IATA. Evidence-Based Training Implementation Guide (2013)Google Scholar
  16. ICAO. Procedures for Air Navigation Services - Training (Doc 9868). Planta (2006)Google Scholar
  17. ICAO. Manual of Evidence-based Training (2013).
  18. Kanki, B., Helmreich, R., Anca, J.: Crew Resource Management. Crew Resource Management, pp. 1–5. (2010). Scholar
  19. Kanki, B.G., Greaud, V.A., Irwin, C.M.: Communication variations and aircrew performance. Int. J. Aviat. Psychol. 1(2), 149–162 (1991). Scholar
  20. McDonnell, L.K., Jobe, K.K., Dismukes, R.K.: Facilitating LOS Debriefings: A Training Manual, March 1997Google Scholar
  21. Orlady, H.W., Orlady, L.M.: Human factors in multi-crew flight operations. Aeronaut. J. 106(1060), 321–324 (2002)Google Scholar
  22. Osgood, C.E.: The similarity paradox in human learning: a resolution. Psychol. Rev. 56(3), 132–143 (1949). Scholar
  23. Rankin, A., Woltjer, R., Field, J., Woods, D.: “Staying ahead of the aircraft” and Managing Surprise in Modern Airliners. In: Proceedings of the 5th Resilience Engineering Association Symposium, pp. 209–214 (2013).
  24. Rosen, M.A., Salas, E., Wu, T.S., Silvestri, S., Lazzara, E.H., Lyons, R., Weaver, S.J., King, H.B.: Promoting teamwork: an event-based approach to simulation-based teamwork training for emergency medicine residents. In: Academic Emergency Medicine, vol. 15, pp. 1190–1198 (2008). Scholar
  25. Salas, E., Rosen, M.A., Held, J.D., Weissmuller, J.J.: Performance measurement in simulation-based training: a review and best practices. Simul. Gaming 40(3), 328–376 (2008). Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Cranfield UniversityBedfordUK
  2. 2.Iceland AirReykjavikIceland

Personalised recommendations