Multidimensional Real-Time Assessment of User State and Performance to Trigger Dynamic System Adaptation

  • Jessica SchwarzEmail author
  • Sven Fuchs
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10284)


In adaptive human-machine interaction, technical systems adapt their behavior to the current state of the human operator to mitigate critical user states and performance decrements. While many researchers use measures of workload as triggers for adjusting levels of automation, we have proposed a more holistic approach to adaptive system design that includes a multidimensional assessment of user state. This paper outlines the design requirements, conceptual framework, and proof-of-concept implementation of a Real-time Assessment of Multidimensional User State (RASMUS). RASMUS diagnostics provide information on user performance, potentially critical user states, and their related impact factors on a second-by-second-basis in real-time. Based on these diagnoses adaptive systems are enabled to infer when the user needs support and to dynamically select and apply an appropriate adaptation strategy for a given situation. While the conceptual framework is generic, the implementation has been applied to an air surveillance task, providing real-time diagnoses for high workload, passive task-related fatigue and incorrect attentional focus.


Multidimensional user state Physiological measures Workload Attention Fatigue Performance Real-time assessment Adaptation Augmented Cognition 


  1. 1.
    BEA (2012) Final Report on the accident on 1st June 2009 to the Airbus A330-203 registered F-GZCP operated by Air France flight AF447 Rio de Janeiro – Paris. Accessed 10 Feb 2017
  2. 2.
    Schwarz, J., Fuchs, S., Flemisch, F.: Towards a more holistic view on user state assessment in adaptive human-computer interaction. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, San Diego CA USA, pp. 1247–1253 (2014). doi: 10.1109/SMC.2014.6974082
  3. 3.
    Rouse, W.B.: Adaptive aiding for human/computer control. Hum. Factors 30(4), 431–443 (1988)Google Scholar
  4. 4.
    Scerbo, M.W.: Theoretical perspectives on adaptive automation. In: Parasuraman, R., Mouloua, M. (eds.) Automation and Human Performance: Theory and Applications, pp. 37–63. Erlbaum, Mahwah (1996)Google Scholar
  5. 5.
    Stanney, K.M., Schmorrow, D.D., Johnston, M., Fuchs, S., Jones, D., Hale, K.S., Ahmad, A., Young, P.: Augmented cognition: an overview. In: Durso, F.T. (ed.) Reviews of Human Factors and Ergonomics, HFES, Santa Monica, vol 5, pp. 195–224 (2009). doi: 10.1518/155723409X448062
  6. 6.
    Fuchs, S., Hale, K.S., Stanney, K.M., Berka, C., Levendowski, D., Juhnke, J.: Physiological sensors cannot effectively drive system mitigation alone. In: Schmorrow, D.D., Stanney, K.M., Reeves, L.M. (eds.) Foundations of Augmented Cognition, 2nd edn, pp. 193–200. Strategic Analysis Inc., Arlington (2006)Google Scholar
  7. 7.
    Steinhauser, N.B., Pavlas, D., Hancock, P.A.: Design principles for adaptive automation and aiding. Ergon. Des. 17(2), 6–10 (2009)Google Scholar
  8. 8.
    Dorneich, M.C., Verver,. P.M., Mathan, S., Whitlow, S., Hayes, C.C.: Considering etiquette in the design of an adaptive system. J. Cogn. Eng. Decis. Making 6(2), 243–265 (2012). doi:
  9. 9.
    Hancock, P.A., Chignell, M.H.: Input information requirements for an adaptive human-machine system. In: Proceedings of the Tenth Department of Defense of Conference Psychology, vol 10, pp. 493–498 (1986)Google Scholar
  10. 10.
    Veltman, J.A., Jansen, C.: The role of operator state assessment in adaptive automation. TNO Report, TNO-DV3 2005 A245, Soesterberg (2006)Google Scholar
  11. 11.
    Akerstedt, T., Mollard, R., Samel, A., Simons, M., Spencer, M.: The role of EU FTL legislation in reducing cumulative fatigue in civil aviation (2003). Paper prepared for ETSCGoogle Scholar
  12. 12.
    Wilkinson, R.T., Edwards, R.S., Haines, E.: Performance following a night of reduced sleep. Psychon. Sci. 5, 471–472 (1966). doi: 10.3758/BF03328474 CrossRefGoogle Scholar
  13. 13.
    Wickens, C.D.: Attentional tunneling and task management. Technical report, AHFD-05-01/NASA-05-10, NASA Ames Reseach Center, Moffett Field CA (2005)Google Scholar
  14. 14.
    Endsley, M.R.R.: Situation awareness and human error: designing to support human performance. In: Proceedings of the High Consequence Systems Surety Conference, Albuquerque NM (1999)Google Scholar
  15. 15.
    Staal, M.A.: Stress, Cognition, and Human Performance: A Literature Review and Conceptual Framework. National Aeronautics & Space Administration, Hanover (2004)Google Scholar
  16. 16.
    Hilburn, B., Jorna, P.G., Byrne, E.A., Parasuraman, R.: The effect of adaptive air traffic control (ATC) decision aiding on controller mental workload. In: Mouloua, M., Koonce, J. (eds.) Human Automation Interaction: Research and Practice, Erlbaum, Mahwah NJ, pp. 84–91 (1997). doi: 10.1518/155534310X522851
  17. 17.
    Kaber, D.B., Prinzel, L.J. III, Wright, M.C., Clamann, M.P.: Workload-matched adaptive automation support of air traffic controller information processing stages. Technical report NASA/TP-2002-211932, NASA, Hampton VA (2002)Google Scholar
  18. 18.
    Parasuraman, R.: Adaptive automation matched to human mental workload. In: Hockey, G.R.J., Gaillard, A.W.K., Burov, O. (eds.) Operator Functional State Assessment: The Assessment and Prediction of Human Performance Degradation in Complex Tasks, pp. 177–193. IOS Press, Amsterdam (2003). doi: 10.1518/001872007X249875
  19. 19.
    Greef, T., Lafeber, H., Oostendorp, H., Lindenberg, J.: Eye movement as indicators of mental workload to trigger adaptive automation. In: Schmorrow, D.D., Estabrooke, Ivy V., Grootjen, M. (eds.) FAC 2009. LNCS (LNAI), vol. 5638, pp. 219–228. Springer, Heidelberg (2009). doi: 10.1007/978-3-642-02812-0_26 CrossRefGoogle Scholar
  20. 20.
    Jorna, P.G.A.M.: Heart-rate and workload variations in actual and simulated flight. Ergonomics 36(9), 1043–1054 (1993). doi: 10.1080/00140139308967976 CrossRefGoogle Scholar
  21. 21.
    Veltman, H.J.A., Jansen, C.: The adaptive operator. In: Vincenzi, D.A., Mouloua, M., Hancock, P.A. (eds.) Human Performance, Situation Awareness, and Automation: Current Research and Trends, vol. 2, pp. 7–10. Erlbaum, Mahwah (2004)Google Scholar
  22. 22.
    Berka, C., Levendowski, D., Lumicao, M., Yau, A., Davis, G., Zivkovic, V., Craven, P.L.: EEG correlates of task engagement and mental workload in vigilance, learning and memory tasks. Aviat. Space Environ. Med. 78(5, Section II, Suppl.), B231–B244 (2007).
  23. 23.
    Veltman, J.A., Jansen, C.: Differentiation of mental effort measures: consequences for adaptive automation. In: Hockey, G.R.J., Gaillard, A.W.K., Burov, O. (eds.) Operator Functional State, pp. 249–259. IOS Press, Amsterdam (2003)Google Scholar
  24. 24.
    Schwarz, J., Fuchs, S.: Efficacy of metrics from a low-cost EEG sensor for multi-dimensional user state assessment. In: Presentation held at 11. Berliner Werkstatt Mensch-Maschine-Systeme, Berlin, 07–09 October 2015 (2015)Google Scholar
  25. 25.
    Schwarz, J., Fuchs, S.: Test-retest stability of eeg and eye tracking metrics as indicators of variations in user state - an analysis at a group and an individual level. In: Hale, K.S., Stanney, K.M. (eds.) Advances in Neuroergonomics and Cognitive Engineering, pp. 145–156. Springer International Publishing, Cham (2016). doi: 10.1007/978-3-319-41691-5_13 Google Scholar
  26. 26.
    Tomarken, A.J.: A psychometric perspective on psychophysiological measures. Psychol. Assess. 7(3), 387–395 (1995)CrossRefGoogle Scholar
  27. 27.
    Diethe, T.: The future of augmentation managers. In: Schmorrow, D.D. (ed.) Foundations of augmented cognition, pp. 631–640. Erlbaum, Mahwah NJ (2005)Google Scholar
  28. 28.
    Barker, R.A., Edwards, R.E., O’Neill, K.R., Tollar, J.R.: DARPA improving warfighter information intake under stress - augmented cognition Concept Validation Experiment (CVE). Analysis Report for the Boeing Team, pp. 1–112 (2004)Google Scholar
  29. 29.
    Parasuraman, R., Sheridan, T.B., Wickens, C.D.: A model of types and levels of human interaction with automation. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 30, 286–297 (2000). doi: 10.1109/3468.844354 CrossRefGoogle Scholar
  30. 30.
    Fuchs, S., Schwarz, J.: Towards a dynamic selection and configuration of adaptation strategies in augmented cognition. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) AC 2017. LNCS (LNAI), vol. 10285, pp. 101–115. Springer, Cham (2017)Google Scholar
  31. 31.
    Schwarz, J.: Benutzerzustandserfassung zur Regelung Kognitiver Assistenz an Bord von Marineschiffen. In: Söffker, D. (ed.) 2. Interdisziplinärer Workshop Kognitive Systeme: Mensch, Teams, Systeme und Automaten, DuEPublico, Duisburg-Essen (2013). doi: 10.17185/duepublico/31351
  32. 32.
    Neerincx, M.A.: Cognitive task load design: model, methods and examples. In: Hollnagel, E. (ed.) Handbook of Cognitive Task Design, pp. 283–305. Lawrence Erlbaum Associates, Mahwah NJ (2003)CrossRefGoogle Scholar
  33. 33.
    May, J.F., Baldwin, C.L.: Driver fatigue: the importance of identifying causal factors of fatigue when considering detection and countermeasure technologies. Transp. Res. Part F: Psychol. Behav. 12(3), 218–224 (2009)CrossRefGoogle Scholar
  34. 34.
    Billings, C.E., Lauber, J.K., Funkhouser, H., Lyman, G., Huff, E.M.: NASA aviation safety reporting system. Technical Report TM-X-3445, NASA Ames Research Center, CA (1976)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Fraunhofer Institute for Communication, Information Processing and Ergonomics, Department of Human-Systems EngineeringWachtbergGermany

Personalised recommendations