Skip to main content
SpringerLink
Log in

Cyber Situational Awareness Testing

  • Chapter
  • First Online:
Combatting Cybercrime and Cyberterrorism

Abstract

In the cyber security landscape, the human ability to comprehend and adapt to existing and emerging threats is crucial. Not only technical solutions, but also the operator’s ability to grasp the complexities of the threats affect the level of success or failure that is achieved in cyber defence. In this paper we discuss the general concept of situation awareness and associated measurement techniques. Further, we describe the cyber domain and how it differs from other domains, and show how predictive knowledge can help improve cyber defence. We discuss how selected existing models and measurement techniques for situation awareness can be adapted and applied in the cyber domain to measure actual levels of cyber situation awareness. We identify generic relevant criteria and other factors to consider, and propose a methodology to set up cyber situation awareness measurement experiments within the context of simulated cyber defence exercises. Such experiments can be used to test the viability of different cyber solutions. A number of concrete possible experiments are also suggested.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 129.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    To know “what is going on” is a phrase used by Endsley [12] in order to provide an informal and intuitive definition of the situational awareness concept.

  2. 2.

    In this paper we use the terms “situation awareness” and “situational awareness” interchangeably.

  3. 3.

    http://www.zdnet.com/article/symantec-data-stealing-hackers-use-ddos-to-distract-from-attacks/.

  4. 4.

    In military theory, the hierarchical war levels consist of the (lowest) tactical, operational, strategic, and political (highest) levels.

  5. 5.

    https://www.honeynet.org/.

  6. 6.

    http://www.telegraph.co.uk/finance/personalfinance/bank-accounts/12129786/HSBC-online-banking-fails-again-after-succumbing-to-cyber-attack.html.

  7. 7.

    http://www.cnet.com/news/cybercrooks-use-ddos-attacks-to-mask-theft-of-banks-millions/.

References

  1. Artman, H.: Team situation assessment and information distribution. Ergonomics 43(8), 1111–1128 (2000)

    Article  Google Scholar 

  2. Bedny, G., Meister, D.: Theory of activity and situation awareness. Int. J. Cogn. Ergon. 3(1), 63–72 (1999)

    Article  Google Scholar 

  3. Brynielsson, J.: An information assurance curriculum for commanding officers using hands-on experiments. ACM SIGCSE Bull. 41(1), 236–240 (2009)

    Article  Google Scholar 

  4. Carroll, L.A.: Desperately seeking SA. TAC Attack 32(3), 5–6 (1992)

    Google Scholar 

  5. Dekker, S.W.A., Hummerdal, D.H., Smith, K.: Situation awareness: some remaining questions. Theor. Issues Ergon. Sci. 11(1–2), 131–135 (2010)

    Article  Google Scholar 

  6. Dennehy, K.: Cranfield situation awareness scale: users manual. Technical report 9702, Applied Psychology Unit, College of Aeronautics, Cranfield University, Bedford, United Kingdom, January 1997

    Google Scholar 

  7. Durso, F.T., Hackworth, C.A., Truitt, T.R., Crutchfield, J., Nikolic, D., Manning, C.A.: Situation awareness as a predictor of performance in en route air traffic controllers. Technical report DOT/FAA/AM-99/3, Office of Aviation Medicine, Federal Aviation Administration, U.S. Department of Transportation, Washington, District of Columbia, January 1999

    Google Scholar 

  8. Endsley, M.R.: Design and evaluation for situation awareness enhancement. In: Proceedings of the Human Factors Society 32nd Annual Meeting, Anaheim, California, pp. 97–101, October 1988

    Google Scholar 

  9. Endsley, M.R.: Situation awareness global assessment technique (SAGAT). In: Proceedings of the IEEE 1988 National Aerospace and Electronics Conference (NAECON 1988), Dayton, Ohio, pp. 789–795, May 1988

    Google Scholar 

  10. Endsley, M.R.: A survey of situation awareness requirements in air-to-air combat fighters. Int. J. Aviat. Psychol. 3(2), 157–168 (1993)

    Article  Google Scholar 

  11. Endsley, M.R.: Measurement of situation awareness in dynamic systems. Hum. Factors 37(1), 65–84 (1995)

    Article  Google Scholar 

  12. Endsley, M.R.: Toward a theory of situation awareness in dynamic systems. Hum. Factors 37(1), 32–64 (1995)

    Article  Google Scholar 

  13. Endsley, M.R.: Theoretical underpinnings of situation awareness: a critical review. In: Endsley, M.R., Garland, D.J. (eds.) Situation Awareness Analysis and Measurement, pp. 3–32. Lawrence Erlbaum Associates Inc., Mahwah (2000)

    Google Scholar 

  14. Endsley, M.R.: Situation awareness misconceptions and misunderstandings. J. Cogn. Eng. Decis. Making 9(1), 4–32 (2015)

    Article  Google Scholar 

  15. Endsley, M.R., Rodgers, M.D.: Situation awareness information requirements for en route air traffic control. Technical report DOT/FAA/AM-94/27, Office of Aviation Medicine, Federal Aviation Administration, U.S. Department of Transportation, Washington, District of Columbia, December 1994

    Google Scholar 

  16. Endsley, M.R., Selcon, S.J., Hardiman, T.D., Croft, D.G.: A comparative analysis of SAGAT and SART for evaluations of situation awareness. In: Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting, Chicago, Illinois, pp. 82–86, October 1998

    Google Scholar 

  17. Europol: Hackers deployed to facilitate drugs smuggling. Intelligence Notification 004-2013, European Cybercrime Centre (EC3), Hague, Netherlands, June 2013. https://www.europol.europa.eu/sites/default/files/publications/cyberbits_04_ocean13.pdf

  18. Federal Bureau of Investigation: Fraud alert - cyber criminals targeting financial institution employee credentials to conduct wire transfer fraud. Press release, Financial Services Information Sharing and Analysis Center (FS-ISAC) and the Internet Crime Complaint Center (IC3), September 2012. http://www.ic3.gov/media/2012/fraudalertfinancialinstitutionemployeecredentialstargeted.pdf

  19. Flach, J.M.: Situation awareness: proceed with caution. Hum. Factors 37(1), 149–157 (1995)

    Article  Google Scholar 

  20. Franke, U., Brynielsson, J.: Cyber situational awareness - a systematic review of the literature. Comput. Secur. 46, 18–31 (2014)

    Article  Google Scholar 

  21. Franke, U., Buschle, M.: Experimental evidence on decision-making in availability service level agreements. IEEE Trans. Netw. Serv. Manage. 13(1), 58–70 (2016)

    Article  Google Scholar 

  22. Gorman, J.C., Cooke, N.J., Winner, J.L.: Measuring team situation awareness in decentralized command and control environments. Ergonomics 49(12–13), 1312–1325 (2006)

    Article  Google Scholar 

  23. Hauss, Y., Eyferth, K.: Securing future ATM-concepts’ safety by measuring situation awareness in ATC. Aerosp. Sci. Technol. 7(6), 417–427 (2003)

    Article  Google Scholar 

  24. Hill, J., Carver, C., Humphries, J., Pooch, U.: Using an isolated network laboratory to teach advanced networks and security. In: Proceedings of the 32nd ACM SIGCSE Technical Symposium on Computer Science Education, Charlotte, North Carolina, pp. 36–40, February 2001

    Google Scholar 

  25. Hogg, D.N., Follesø, K., Strand-Volden, F., Torralba, B.: Development of a situation awareness measure to evaluate advanced alarm systems in nuclear power plant control rooms. Ergonomics 38(11), 2394–2413 (1995)

    Article  Google Scholar 

  26. Holm, H.: Baltic cyber shield: research from a red team versus blue team exercise. PenTest magazine 2(5), 80–86 (2012)

    Google Scholar 

  27. Holm, H., Sommestad, T., Franke, U., Ekstedt, M.: Success rate of remote code execution attacks: expert assessments and observations. J. Univ. Comput. Sci. 18(6), 732–749 (2012)

    Google Scholar 

  28. Jacobson, D.: Teaching information warfare with lab experiments via the internet. In: Proceedings of the 34th ASEE/IEEE Frontiers in Education Conference, Savannah, Georgia, pp. T3C/7–12, October 2004

    Google Scholar 

  29. Jeannot, E., Kelly, C., Thompson, D.: The development of situation awareness measures in ATM systems. Technical report HRS/HSP-005-REP-01, European Organisation for the Safety of Air Navigation (EUROCONTROL), Brussels, Belgium, June 2003

    Google Scholar 

  30. Kirwan, G., Power, A.: Cybercrime: The Psychology of Online Offenders. Cambridge University Press, Cambridge (2013)

    Book  Google Scholar 

  31. Libicki, M.C.: Conquest in Cyberspace: National Security and Information Warfare. Cambridge University Press, Cambridge (2007)

    Book  Google Scholar 

  32. Maloof, M.A., Stephens, G.D.: elicit: a system for detecting insiders who violate need-to-know. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 146–166. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  33. Marusich, L.R., Bakdash, J.Z., Onal, E., Yu, M.S., Schaffer, J., O’Donovan, J., Höllerer, T., Buchler, N., Gonzalez, C.: Effects of information availability on command-and-control decision making: performance, trust, and situation awareness. Hum. Factors 58(2), 301–321 (2016)

    Article  Google Scholar 

  34. Matthews, M.D., Beal, S.A.: Assessing situation awareness in field training exercises. Research report 1795, U.S. Army Research Institute for the Behavioral and Social Sciences, Alexandria, Virginia, September 2002

    Google Scholar 

  35. Matthews, M.D., Strater, L.D., Endsley, M.R.: Situation awareness requirements for infantry platoon leaders. Mil. Psychol. 16(3), 149–161 (2004)

    Article  Google Scholar 

  36. McGuinness, B.: Quantitative analysis of situational awareness (QUASA): applying signal detection theory to true/false probes and self-ratings. In: Proceedings of the 2004 Command and Control Research and Technology Symposium (CCRTS), San Diego, California, June 2004

    Google Scholar 

  37. McGuinness, B., Foy, L.: A subjective measure of SA: the crew awareness rating scale (CARS). In: Proceedings of the First Human Performance. Situation Awareness and Automation Conference, Savannah, Georgia, pp. 286–291, October 2000

    Google Scholar 

  38. Mullins, B.E., Lacey, T.H., Mills, R.F., Trechter, J.M., Bass, S.D.: How the cyber defense exercise shaped an information-assurance curriculum. IEEE Secur. Priv. 5(5), 40–49 (2007)

    Article  Google Scholar 

  39. Parasuraman, R., Sheridan, T.B., Wickens, C.D.: Situation awareness, mental workload, and trust in automation: viable, empirically supported cognitive engineering constructs. J. Cogn. Eng. Decis. Making 2(2), 140–160 (2008)

    Article  Google Scholar 

  40. Patrick, J., Morgan, P.L.: Approaches to understanding, analysing and developing situation awareness. Theor. Issues Ergon. Sci. 11(1–2), 41–57 (2010)

    Article  Google Scholar 

  41. Raser, J.R.: Simulation and Society: An Exploration of Scientific Gaming. Allyn and Bacon Inc., Boston (1969)

    Google Scholar 

  42. Rid, T., Buchanan, B.: Attributing cyber attacks. J. Strateg. Stud. 38(1–2), 4–37 (2015)

    Article  Google Scholar 

  43. Romney, G.W., Higby, C., Stevenson, B.R., Blackham, N.: A teaching prototype for educating IT security engineers in emerging environments. In: Proceedings of the Fifth IEEE International Conference on Information Technology Based Higher Education and Training, Istanbul, Turkey, pp. 662–667, May-Jun 2004

    Google Scholar 

  44. Salas, E., Prince, C., Baker, D.P., Shrestha, L.: Situation awareness in team performace: implications for measurement and training. Hum. Factors 37(1), 123–136 (1995)

    Article  Google Scholar 

  45. Salmon, P.M., Stanton, N.A., Walker, G.H., Baber, C., Jenkins, D.P., McMaster, R., Young, M.S.: What really is going on? Review of situation awareness models for individuals and teams. Theor. Issues Ergon. Sci. 9(4), 297–323 (2008)

    Article  Google Scholar 

  46. Salmon, P.M., Stanton, N.A., Walker, G.H., Green, D.: Situation awareness measurement: a review of applicability for C4i environments. Appl. Ergon. 37(2), 225–238 (2006)

    Article  Google Scholar 

  47. Salmon, P.M., Stanton, N.A., Walker, G.H., Jenkins, D., Ladva, D., Rafferty, L., Young, M.: Measuring situation awareness in complex systems: comparison of measures study. Int. J. Ind. Ergon. 39(3), 490–500 (2009)

    Article  Google Scholar 

  48. Sarter, N.B., Woods, D.D.: Situation awareness: a critical but ill-defined phenomenon. Int. J. Aviat. Psychol. 1(1), 45–57 (1991)

    Article  Google Scholar 

  49. Schlenker, B.R., Bonoma, T.V.: Fun and games: the validity of games for the study of conflict. J. Conflict Resolut. 22(1), 7–38 (1978)

    Article  Google Scholar 

  50. Smith, K., Hancock, P.A.: Situation awareness is adaptive, externally-directed consciousness. In: Gilson, R.D., Garland, D.J., Koonce, J.M. (eds.) Situational Awareness in Complex Systems. Aviation Human Factors Series, pp. 59–68. Embry-Riddle Aeronautical University Press, Daytona Beach, Florida (1994)

    Google Scholar 

  51. Sommestad, T., Hallberg, J.: Cyber security exercises and competitions as a platform for cyber security experiments. In: Jøsang, A., Carlsson, B. (eds.) NordSec 2012. LNCS, vol. 7617, pp. 47–60. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  52. Stanton, N.A., Chambers, P.R.G., Piggott, J.: Situational awareness and safety. Saf. Sci. 39(3), 189–204 (2001)

    Article  Google Scholar 

  53. Stevens, S.S.: Measurement, statistics, and the schemapiric view. Science 161(3844), 849–856 (1968)

    Article  Google Scholar 

  54. Taylor, R.M.: Situational awareness rating technique (SART): the development of a tool for aircrew systems design. In: AGARD Conference Proceedings No. 178: Situational Awareness in Aerospace Operations, pp. 3/1–17, April 1990

    Google Scholar 

  55. U.S. Department of Defense: Cyberspace operations. Joint Publication 3–12(R), Joint Chiefs of Staff, Washington, District of Columbia, February 2013

    Google Scholar 

  56. U.S. Department of Defense: Cyber electromagnetic activities. Field Manual 3–38, Headquarters, Department of the Army, Washington, District of Columbia, February 2014

    Google Scholar 

  57. Vidulich, M.A., Hughes, E.R.: Testing a subjective metric of situation awareness. In: Proceedings of the Human Factors Society 35th Annual Meeting, San Francisco, California, pp. 1307–1311, September 1991

    Google Scholar 

  58. Waag, W.L., Houck, M.R.: Tools for assessing situational awareness in an operational fighter environment. Aviat. Space Environ. Med. 65(5), A13–A19 (1994)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. FOI Swedish Defence Research Agency, 164 90, Stockholm, Sweden

    Joel Brynielsson

  2. KTH Royal Institute of Technology, 100 44, Stockholm, Sweden

    Joel Brynielsson & Stefan Varga

  3. SICS Swedish Institute of Computer Science, Box 1263, 164 29, Kista, Sweden

    Ulrik Franke

  4. Swedish Armed Forces Headquarters, 107 85, Stockholm, Sweden

    Stefan Varga

Authors
  1. Joel Brynielsson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ulrik Franke
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan Varga
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joel Brynielsson .

Editor information

Editors and Affiliations

  1. CENTRIC, Sheffield Hallam University, Sheffield, South Yorkshire, United Kingdom

    Babak Akhgar

  2. CENTRIC, Sheffield Hallam University, Sheffield, United Kingdom

    Ben Brewster

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Brynielsson, J., Franke, U., Varga, S. (2016). Cyber Situational Awareness Testing. In: Akhgar, B., Brewster, B. (eds) Combatting Cybercrime and Cyberterrorism. Advanced Sciences and Technologies for Security Applications. Springer, Cham. https://doi.org/10.1007/978-3-319-38930-1_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-38930-1_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-38929-5

  • Online ISBN: 978-3-319-38930-1

  • eBook Packages: Law and CriminologyLaw and Criminology (R0)

Publish with us

Policies and ethics

Search

Navigation