Estimating Human State from Simulated Assisted Driving with Stochastic Filtering Techniques

  • Gregory M. GremillionEmail author
  • Daniel Donavanik
  • Catherine E. Neubauer
  • Justin D. Brody
  • Kristin E. Schaefer
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 780)


This work proposes a process for formulating a model and estimation scheme to predict changes in decision authority with a simulated autonomous driving assistant. The unique component of this modeling approach is the use of direct estimation of governing mental decision states via recursive psychophysiological inference. Treating characteristic quantities of the environment as inputs, and behavioral and physiological signals as outputs, we propose the estimation of intermediate or underlying psychological states of the human can be used to predict the decision to engage or disengage a driving assistant, using methods of stochastic filtering. Such a framework should enable techniques to optimally fuse information and thereby improve performance in human-autonomy driving interactions.


Autonomous driving assistants Stochastic filtering Decision-making Simulation 



The authors thank Jason Metcalfe, Amar Marathe, Jamie Lukos, Justin Estepp, Kim Drnec, Victor Paul, Benjamin Haynes, Corey Atwater, and William Nothwang for the simulated driving dataset. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.


  1. 1.
    Phillips, E., Ososky, S., Grove, J., Jentsch, F.: From tools to teammates: toward the development of appropriate mental models for intelligent robots. In: Proceedings of the Human Factors and Ergonomics, pp. 1491–1495 (2011)Google Scholar
  2. 2.
    Hancock, P.A., Verwey, W.B.: Fatigue, workload and adaptive driver systems. Acc. Anal. Prev. 29(4), 495–506 (1997)CrossRefGoogle Scholar
  3. 3.
    Stanton, N.A., Young, M.S.: Driver behaviour with adaptive cruise control. Ergonomics 48(10), 1294–1313 (2005)CrossRefGoogle Scholar
  4. 4.
    Koo, J., Kwac, J., Ju, W., Steinert, M., Leifer, L., Nass, C.: Why did my car just do that? Explain-ing semi-autonomous driving actions to improve driver understanding, trust, and performance. Int. J. Interact. Des. Manuf. 9(4), 269–275 (2015)CrossRefGoogle Scholar
  5. 5.
    Parasuraman, R., Manzey, D.H.: Complacency and bias in human use of automation: an at-tentional integration. Hum. Fact.: J. Hum. Fact. Ergon. Soc. 52(3), 381–410 (2010)CrossRefGoogle Scholar
  6. 6.
    Adams, J.A., Deloach, S.A., Scheutz, M.: Shared mental models for human-robot teams. In: Proceedings of AAAI, Stanford, pp. 99–105 (2014)Google Scholar
  7. 7.
    Awais, M., Henrich, D.: Human-robot collaboration by intention recognition using probabilistic state machines. In: Proceedings of International Workshop on Robotics in Alpe-Adria-Danube Region (RAAD), pp. 75–80. IEEE (2010)Google Scholar
  8. 8.
    Ososky, S., Schuster, D., Jentsch, F., Fiore, S., Shumaker, R., Lebiere, C., Kurup, U., Oh, J., Stentz, A.: The importance of shared mental models and shared situation awareness for transforming robots from tools to teammates. In: Proceedings of SPIE - The International Society for Optical Engineering, p. 838710 (2012)Google Scholar
  9. 9.
    Chen, J.Y.C., Barnes, M.J.: Human–agent teaming for multirobot control: a review of human factors issues. IEEE Trans. Hum.-Mach. Syst. 44(1), 13–29 (2014)CrossRefGoogle Scholar
  10. 10.
    Schaefer, K.E., Brewer, R., Putney, J., Mottern, E., Barghout, J., Straub, E.R., Orlando, F.L.: Relinquishing manual control: collaboration requires the capability to understand robot intent. In: Proceedings of the IEEE 9th International Workshop on Collaborative Robots and Human Robot Interaction, pp. 359–366 (2016)Google Scholar
  11. 11.
    Lazarus, R.S.: The cognition-emotion debate: a bit of history. Handb. Cogn. Emot. 5(6), 3–19 (1999)Google Scholar
  12. 12.
    Matthews, G.: Towards a transactional ergonomics for driver stress and fatigue. Theoret. Issues Ergon. Sci. 3(2), 195–211 (2002)CrossRefGoogle Scholar
  13. 13.
    Xu, A., Dudek, G.: Trust-driven interactive visual navigation for autonomous robots. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 3922–3929 (2012)Google Scholar
  14. 14.
    Takahashi, M., Kubo, O., Kitamura, M., Yoshikawa, H.: Neural network for human cognitive state estimation. In: Advanced Robotic Systems and the Real World Intelligent Robots and Systems 1994, pp. 2176–2183 (1994)Google Scholar
  15. 15.
    Ji, Q., Zhu, Z., Lan, P.: Real-time nonintrusive monitoring and prediction of driver fatigue. IEEE Trans. Veh. Technol. 53(4), 1052–1068 (2004)CrossRefGoogle Scholar
  16. 16.
    Metcalfe, J.S., Marathe, A.R., Haynes, B., Paul, V.J., Gremillion, G.M., Drnec, K., Atwater, C., Estepp, J.R., Lukos, J.R., Carter, E.A., Nothwang, W.D.: Building a framework to manage trust in automation. In: Proceedings of SPIE 10194, Micro- and Nanotechnology Sensors, Systems, and Applications IX, p. 101941U (2017)Google Scholar
  17. 17.
    Donavanik, D., Hardt-Stremayr, A., Gremillion, G.M., Weiss, S., Nothwang, W.D.: Multi-sensor fusion techniques for state estimation of micro air vehicles. In: SPIE Defense and Security, p. 98361V. International Society for Optics and Photonics (2016)Google Scholar
  18. 18.
    Rosencrantz, M., Gordon, G., Thrun, S.: Decentralized sensor fusion with distributed particle filters. In: Proceedings of the Nineteenth conference on Uncertainty in Artificial Intelligence, pp. 493–500 (2002)Google Scholar
  19. 19.
    Van Der Merwe, R., Wan, E., Julier, S.: Sigma-point Kalman filters for nonlinear estimation and sensor-fusion: applications to integrated navigation. In: AIAA Guidance, Navigation, and Control Conference and Exhibit, p. 5120 (2004)Google Scholar
  20. 20.
    Gremillion, G.M., Metcalfe, J.S., Marathe, A.R., Paul, V.J., Christensen, J., Drnec, K., Haynes, B., Atwater, C.: Analysis of trust in autonomy for convoy operations. In: Proceedings of International Society for Optics and Photonics Defense+Security, p. 98361Z (2016)Google Scholar
  21. 21.
    Cacioppo, J.T., Tassinary, L.G.: Inferring psychological significance from physiological signals. Am. Psychol. 45(1), 16 (1990)Google Scholar
  22. 22.
    Just, M.A., Carpenter, P.A.: Eye fixations and cognitive processes. Cogn. Psychol. 8(4), 441–480 (1976)CrossRefGoogle Scholar
  23. 23.
    Poole, A., Ball, L.J.: Eye tracking in HCI and usability research. In: Encyclopedia of Human Computer Interaction, vol. 1. pp. 211–219 (2006)Google Scholar
  24. 24.
    Glaholt, M.G., Reingold, E.M.: Eye movement monitoring as a process tracing methodology in decision making research. J. Neurosci. Psychol. Econ. 4(2), 125 (2011)CrossRefGoogle Scholar
  25. 25.
    Marshall, S.: Measures of attention and cognitive effort in tactical decision making. Decis. Mak. Complex Environ. 321, 332 (2007)Google Scholar
  26. 26.
    Beatty, J.: Pupillometric measurement of cognitive workload. TR-22, Department of Psychology, California University Los Angeles (1977)Google Scholar
  27. 27.
    Schwalm, M., Keinath, A., Zimmer H.D.: Pupillometry as a method for measuring mental workload within a simulated driving task. Hum. Fact. Assist. Autom. 1–13 (2008)Google Scholar
  28. 28.
    Parasuraman, R., Sheridan, T., Wickens, C.: Situation awareness, mental workload, and trust in automation: viable, empirically supported cognitive engineering constructs. J. Cogn. Eng. Decis. Mak. 2(2), 140–160 (2008)CrossRefGoogle Scholar
  29. 29.
    Shi, Y., Ruiz, N., Taib, R., Choi, E., Chen, F.: Galvanic skin response (GSR) as an index of cognitive load. In: CHI 2007, Extended Abstracts on Human Factors in Computing Systems, pp. 2651–2656 (2007)Google Scholar
  30. 30.
    Bethel, C.L., Burke, J.L., Murphy, R.R., Salomon, K.: Psychophysiological experimental design for use in human-robot interaction studies. In: International Symposium on Collaborative Technologies and Systems (2007)Google Scholar
  31. 31.
    Figner, B., Murphy, R.O.: Using skin conductance in judgment and decision making re-search. In: Handbook of Process Tracing Methods for Decision Research, pp. 163–184 (2011)Google Scholar
  32. 32.
    Matthews, R., McDonald, N.J., Trejo, L.J.: Psycho-physiological sensor techniques: an over-view. In: 11th International Conference on Human Computer Interaction (HCII), pp. 22–27 (2005)Google Scholar
  33. 33.
    Montague, E., Xu, J., Chiou, E.: Shared experiences of technology and trust: an experimental study of physiological compliance between active and passive users in technology-mediated collaborative encounters. IEEE Trans. Hum.-Mach. Syst. 44(5), 614–624 (2014)CrossRefGoogle Scholar
  34. 34.
    Schaefer, K.E., Scribner, D.R.: Individual differences, trust, and vehicle autonomy: a pilot study. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 59, no. 1. pp. 786–790 (2015)Google Scholar
  35. 35.
    Haak, M., Bos, S., Panic, S., Rothkrantz, L.J.M.: Detecting stress using eye blinks and brain activity from EEG signals. In: Proceeding of the 1st Driver Car Interaction and Interface, pp. 35–60 (2008)Google Scholar
  36. 36.
    Rozado, D., Dunser, A.: Combining EEG with pupillometry to improve cognitive workload detection. Computer 48(10), 18–25 (2015)CrossRefGoogle Scholar
  37. 37.
    Haufe, S., Treder, M.S., Gugler, M.F., Sagebaum, M., Curio, G., Blankertz, B.: EEG potentials predict upcoming emergency brakings during simulated driving. J. Neural Eng. 8(5), 056001 (2011)CrossRefGoogle Scholar
  38. 38.
    Lin, C.T., Chuang, C.H., Huang, C.S., Tsai, S.F., Lu, S.W., Chen, Y.H., Ko, L.W.: Wireless and wearable EEG system for evaluating driver vigilance. IEEE Trans. Biomed. Circuits Syst. 8(2), 165–176 (2014)CrossRefGoogle Scholar
  39. 39.
    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)MathSciNetzbMATHGoogle Scholar
  40. 40.
    Li, J., Cheng, K., Wang, S., Morstatter, F., Trevino, R.P., Tang, J., Liu, H.: ACM feature selection: a data perspective. Comput. Surv. 50(6), 94 (2017)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature (outside the USA) 2019

Authors and Affiliations

  • Gregory M. Gremillion
    • 1
    Email author
  • Daniel Donavanik
    • 1
  • Catherine E. Neubauer
    • 1
  • Justin D. Brody
    • 1
  • Kristin E. Schaefer
    • 1
  1. 1.United States Army Research LaboratoryAberdeen Proving GroundUSA

Personalised recommendations