Towards a Coherent Assessment of Situational Awareness to Support System Design in the Maritime Context

  • Francesca de RosaEmail author
  • Anne-Laure Jousselme
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 965)


Information systems in support to Situational Awareness (SAW), such as maritime surveillance systems, are an important family of tools that introduced automation with respect to the human cognitive activities. An integral part of the system design process is the testing and evaluation phase. The formal assessment of the mental state of SAW is a complex task. It appears that SAW assessment in testing and evaluation is often either overlooked by adopting a technology-focused approach or only partially addressed through the use of specific human factors methods. In this paper the authors will discuss how the testing and evaluation of maritime surveillance systems could account both for the system components enabling Situational Awareness and the human element. Furthermore, for such systems a simple and coherent list of key performance indicators and measures of performance is provided.


Situational awareness Systems engineering Testing & evaluation Key performance indicators Maritime surveillance 



This research was supported by NATO Allied Command Trans-formation (NATO-ACT) through the DKOE programme of work.


  1. 1.
    Parasuraman, R., Sheridan, T.B., Wickens, C.D.: A model for types and levels of human interaction with automation. In: IEEE Transactions on Systems, Man, and Cybernetics – Part A: Systems and Humans, Vol. 30, pp. 286–297 (2000)CrossRefGoogle Scholar
  2. 2.
    Endsley, R.M.: The application of human factors to the development of expert systems for advanced cockpits. In: Human Factors Society 31st Annual Meeting, pp. 1388–1392. Human Factor Society, Santa Monica, CA (1987)CrossRefGoogle Scholar
  3. 3.
    Endsley, R.M.: Toward a theory of situation awareness in dynamic systems. Hum. Factors 37(1), 32–64 (1995)CrossRefGoogle Scholar
  4. 4.
    Nato Marcom, Marcom MSA Direction and Guidance (NATO UNCLASSIFIED). Technical Report Version 2.0, NATO MARCOM (2016)Google Scholar
  5. 5.
    NATO - STANAG 1241 Standard Identity Description Structure for Tactical Use (2005)Google Scholar
  6. 6.
    Mevassvik, O.M., Veum, K.: Distributed maritime situation picture production and data fusion. In: 4th International Conference on Information Fusion (2001)Google Scholar
  7. 7.
    Stowers, K., Oglesby, J., Sonesh, S., Leyva, K., Iwig, C., Salas, E.: A framework to guide the assessment of human–machine systems. Hum. Factors 59(2), 172–188 (2017)CrossRefGoogle Scholar
  8. 8.
    Taylor, R.M.: Situational awareness rating technique (SART): the development of a tool for aircrew systems design. In: Situational Awareness in Aerospace Operations (AGARD-CP-478), pp. 3/1–3/17, Neuilly Sur Seine (1990)Google Scholar
  9. 9.
    Sarter, N.B., Woods, D.D., Billings, C.E.: Automation surprises. Handb. Hum. Factors Ergon. 2, 1926–1943 (1997)Google Scholar
  10. 10.
    Blasch, E.P., Valin, P., Bosse, E.: Measures of effectiveness for high-level fusion. In: 13th International Conference on Information Fusion (2010)Google Scholar
  11. 11.
    Waltz, E., Llinas, J.: System Modeling and Performance Evaluation. Multisensor Data Fusion Systems, Artech House, Boston (1990)Google Scholar
  12. 12.
    Llinas, J.: Assessing the performance of multisensor fusion processes. In: Hall, D., Llinas, J. (eds.) Handbook of Multisensor Data Fusion. CRC Press, Boca Raton (2001)Google Scholar
  13. 13.
    Theil, A., Kester, L.J.H.M., Bosse, E.: On measures of performance to assess sensor fusion effectiveness. In: 3rd International Conference on Information Fusion (2000)Google Scholar
  14. 14.
    Van Laere, J.: Challenges for IF performance evaluation in practice. In: 12th International Conference on Information Fusion (2009)Google Scholar
  15. 15.
    Costa, P., Laskey, K., Blasch, E., Jousselme, A.L.: Towards unbiased evaluation of uncertainty reasoning: The URREF ontology. In: 15th International Conference on Information Fusion (2012)Google Scholar
  16. 16.
    Blasch, E.P., Breton, R., Valin, P.: Information fusion measures of effectiveness (MOE) for decision support. In: SPIE 8050 (2011)Google Scholar
  17. 17.
    Sheridan, T.B.: Humans and Automation: System Design and Research Issues. Wiley, Santa Monica (2002)Google Scholar
  18. 18.
    Marquez, J.J., Gore, B.F.: Measuring safety and performance in human–automation systems: special issue commentary. Hum. Factors 59(2), 169–171 (2017)CrossRefGoogle Scholar
  19. 19.
    Endsley, M.R.: Situation awareness measurement in test and evaluation. In: O’Brien, T.G., Charlton, S.G. (eds.) Handbook of Human Factors Testing & Evaluation, pp. 159–180. Lawrence Erlbaum Associates, Mahwah (1996)Google Scholar
  20. 20.
    Endsley, M.: Situation awareness and workload: flip sides of the same coin. In: Jensen, R.S., Neumeister, D. (Eds.) Seventh International Symposium on Aviation Psychology, pp. 906–911 (1993)Google Scholar
  21. 21.
    Parasuraman, R., Sheridan, T.B., Wickens, C.D.: Situation awareness, mental workload, and trust in automation: viable, empirically supported cognitive engineering constructs. J. Cogn. Eng. Decis. Making 2(2), 140–160 (2008)CrossRefGoogle Scholar
  22. 22.
    Vidulich, M.A., Tsang, P.S.: The confluence of situation awareness and mental workload for adaptable human-machine systems. J. Cogn. Eng. Decis. Making 9(1), 95–97 (2015)CrossRefGoogle Scholar
  23. 23.
    Endsley, M.R.: Theoretical underpinnings of situation awareness: a critical review. In: Endsley, M.R., Garland, D.J. (eds.) Situation Awareness Analysis and Measurement. Lawrence Erlbaum Associates, Mahwah (2000)Google Scholar
  24. 24.
    Stanton, N.A., Salmon, P.M., Walker, G.H., Baber, C., Jenkins, D.P.: Human Factors Methods: A Practical Guide for Engineering and Design. Ashgate Publishing Company, Brookfield (2006)Google Scholar
  25. 25.
    Endsley, R.M.: Measurements of situation awareness in dynamic systems. Hum. Factors 37(1), 65–84 (1995)CrossRefGoogle Scholar
  26. 26.
    Hogg, D.N., Folleso, K., Strand-Volden, F., Torralba, B.: Development of a situation awareness measure to evaluate advanced alarm systems in nuclear power plant control room. Ergonomics 38(11), 2394–2413 (1995)CrossRefGoogle Scholar
  27. 27.
    Hauss, Y., Gauss, B., Eyferth, K.: SALSA - a new approach to measure situational awareness in air traffic control. Focusing Attention on Aviation Safety. In: 11th International Symposium on Aviation Psychology, Columbus (2001)Google Scholar
  28. 28.
    Jeannott, E., Kelly, C., Thompson, D.: The development of situation awareness measures in ATM systems. Technical Report, EATMP (2003)Google Scholar
  29. 29.
    Durso, F.T., Hackworth, C.A., Truitt, T., Crutchfield, J., Manning, C.A.: Situation awareness as a predictor of performance in en route air traffic controllers. Air Traffic Q. 6, 1–20 (1998)CrossRefGoogle Scholar
  30. 30.
    McGuinness, B., Foy, L.: A subjective measure of SA: the Crew Awareness Rating Scale (CARS). In: Human Performance, Situational Awareness and Automation Conference, Savannah (2000)Google Scholar
  31. 31.
    Matthews, M.D., Beal, S.A.: Assessing situation awareness in field training exercises. Research Report, U.S. Army Research Institute for the Behavioural and Social Sciences (2002)Google Scholar
  32. 32.
    Waag, W.L., Houck, M.R.: Tools for assessing situational awareness in an operational fighter environment. Aviat. Space Environ. Med. 65(5), A13–A19 (1994)Google Scholar
  33. 33.
    Dennehy, K.: Cranfield - situation awareness scale user manual. Technical Report, College of Aeronautics, Cranfield University, Bedford (1997)Google Scholar
  34. 34.
    Matthews, M.D., Pleban, R.J., Endsley, M.R., Strater, L.D.: Measures of infantry situation awareness for a virtual MOUT environment. In: Human Performance, Situation Awareness and Automation Conference (HPSAA II), Daytona, LEA (2000)Google Scholar
  35. 35.
    Sullivan, C., Blackman, H.S.: Insights into pilot situation awareness using verbal protocol analysis. In: Human Factors Society 35th Annual Meeting 1, pp. 57–61. Santa Monica, CA: Human Factors Society (1991)CrossRefGoogle Scholar
  36. 36.
    Sarter, N.B., Woods, D.D.: Situation awareness: a critical but ill-defined phenomenon. Int. J. Aviat. Psychol. 1, 45–57 (1991)CrossRefGoogle Scholar
  37. 37.
    de Rosa, F., Jousselme, A.-L., De Gloria, A.: A reliability game for source factors and situational awareness experimentation. Int. J. Serious Games 5(2), 45–64 (2018)CrossRefGoogle Scholar
  38. 38.
    Jousselme, A.-L., Pallotta, G., Locke, J.: Risk game: capturing impact of information quality on human belief assessment and decision making. Int. J. Serious Games 5(4), 23–44 (2018)CrossRefGoogle Scholar
  39. 39.
    SIAP System Engineering Task Force (SIAP SETF): Single Integrated Air Picture (SIAP) Attributes Version 2.0. Technical Report 2003-029 (2003)Google Scholar
  40. 40.
    Representative measures of a single integrated air picture. Technical Report, NAVSEA 05 (2000)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.NATO STO Centre for Maritime Research and ExperimentationLa SpeziaItaly

Personalised recommendations