Measuring Expert and Novice Performance Within Computer Security Incident Response Teams

  • Austin SilvaEmail author
  • Glory Emmanuel
  • Jonathan T. McClain
  • Laura Matzen
  • Chris Forsythe
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9183)


There is a great need for creating cohesive, expert cybersecurity incident response teams and training them effectively. This paper discusses new methodologies for measuring and understanding expert and novice differences within a cybersecurity environment to bolster training, selection, and teaming. This methodology for baselining and characterizing individuals and teams relies on relating eye tracking gaze patterns to psychological assessments, human-machine transaction monitoring, and electroencephalography data that are collected during participation in the game-based training platform Tracer FIRE. We discuss preliminary findings from two pilot studies using novice and professional teams.


Cybersecurity Training Teams Visual search Eye tracking EEG In situ testing Measuring individual differences Psychological measures 



Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under Contract DE-AC04-94AL85000, Sandia Report SAND2015-1796 C.


  1. 1.
    Stevens-Adams, S., Carbajal, A., Silva, A., Nauer, K., Anderson, B., Reed, T., Forsythe, C.: Enhanced training for cyber situational awareness. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) AC 2013. LNCS, vol. 8027, pp. 90–99. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  2. 2.
    Reed, T., Nauer, K., Silva, A.: Instrumenting competition-based exercises to evaluate cyber defender situation awareness. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) AC 2013. LNCS, vol. 8027, pp. 80–89. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  3. 3.
    Benet-Martínez, V., John, O.P.: Los Cinco Grandes across cultures and ethnic groups: multitrait-multimethod analyses of the big five in Spanish and English. J. Pers. Soc. Psychol. 75(3), 729–750 (2013)CrossRefGoogle Scholar
  4. 4.
    Cacioppo, J.T., Petty, R.E., Feng Kao, C.: The efficient assessment of need for cognition. J. Pers. Assess. 48(3), 306–307 (1984)CrossRefGoogle Scholar
  5. 5.
    Cacioppo, J.T., Petty, R.E., Feinstein, J.A., Jarvis, W.B.G.: Dispositional differences in cognitive motivation: the life and times of individuals varying in need for cognition. Psychol. Bull. 119(2), 197–253 (1996)CrossRefGoogle Scholar
  6. 6.
    Scott, S.G., Bruce, R.A.: Decision-making style: the development and assessment of a new measure. Educ. Psychol. Measur. 55(5), 818–831 (1995)CrossRefGoogle Scholar
  7. 7.
    Loo, R.: A psychometric evaluation of the general decision-making style inventory. Personality Individ. Differ. 29(5), 895–905 (2000)CrossRefGoogle Scholar
  8. 8.
    Spicer, D.P., Sadler-Smith, E.: An examination of the general decision making style questionnaire in two UK samples. J. Manage. Psychol. 20(2), 137–149 (2005)CrossRefGoogle Scholar
  9. 9.
    Thunholm, P.: Decision-making styles and physiological correlates of negative stress: is there a relation? Scand. J. Psychol. 49(3), 213–219 (2008)CrossRefGoogle Scholar
  10. 10.
    Silva, A., McClain, J., Reed, T., Anderson, B., Nauer, K., Abbott, R., Forsythe, C.: Factors impacting performance in competitive cyber exercises. In: Proceedings of the Interservice/Industry Training, Simulation and Education Conference (I/ITSEC2014) (2014)Google Scholar
  11. 11.
    Matzen, L.E.: Effects of professional visual search experience on domain-general and domain-specific visual. Paper presented at the HCI International, Los Angeles, CA (2015)Google Scholar
  12. 12.
    Haass, M.J., Matzen, L.E.: Using computational modeling to assess use of cognitive strategies. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) FAC 2011. LNCS, vol. 6780, pp. 77–86. Springer, Heidelberg (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Austin Silva
    • 1
    Email author
  • Glory Emmanuel
    • 1
  • Jonathan T. McClain
    • 1
  • Laura Matzen
    • 1
  • Chris Forsythe
    • 1
  1. 1.Sandia National LaboratoriesAlbuquerqueUSA

Personalised recommendations