Skip to main content
SpringerLink
Log in

Designing an AI-Companion to Support the Driver in Highly Autonomous Cars

  • Conference paper
  • First Online:
Human-Computer Interaction. Multimodal and Natural Interaction (HCII 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12182))

Included in the following conference series:

Abstract

In this paper, we propose a model for an AI-Companion for conditionally automated cars, able to maintain awareness of the driver regarding the environment but also to able design take-over requests (TOR) on the fly, with the goal of better support the driver in case of a disengagement.

Our AI-Companion would interact with the driver in two ways: first, it could provide feedback to the driver in order to raise the driver Situation Awareness (SA), prevent them to get out of the supervision loop and so, improve takeover during critical situations by decreasing their cognitive workload. Second, in the case of TOR with a smart choice of modalities for convey the request to the driver. In particular, the AI-Companion can interact with the driver using many modalities, such as visual messages (warning lights, images, text, etc.), auditory signals (sound, speech, etc.) and haptic technologies (vibrations in different parts of the seat: back, headrest, etc.).

The ultimate goal of the proposed approach is to design smart HMIs in semi-autonomous vehicles that are able to understand 1) the user state and fitness to drive, 2) the current external situation (vehicle status and behavior) in order to minimize the automation surprise and maximizing safety and trust, and 3) leverage AI to provide adaptive TOR and useful feedback to the driver.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Affectiva, (n.d.).: Affectiva AutomotiveAI. http://go.affectiva.com/auto

  2. Arkonac, S.E., Brumby, D.P., Smith, T., Babu, H.V.R.: In-car distractions and automated driving: a preliminary simulator study. In: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications Adjunct Proceedings AutomotiveUI 19 (2019). https://doi.org/10.1145/3349263.3351505

  3. Ascone, D., Tonja Lindsey, T., Varghese, C.: An examination of driver distraction as recorded in NHTSA databases (No. DOT HS 811 216). United States, National Highway Traffic Safety Administration (2009)

    Google Scholar 

  4. Awais, M., Badruddin, N., Drieberg, M.: A hybrid approachto detect driver drowsiness utilizing physiological signals to improve system performance and wearability. Sensors 17(9), 1991 (2017)

    Article  Google Scholar 

  5. Bakker, S., Niemantsverdriet, K.: The interaction-attention continuum: considering various levels of human attention in interaction design. Int. J. Des. 10(2), 1–14 (2016)

    Google Scholar 

  6. Beattie, D., Baillie, L., Halvey, M., McCall, R.: What’s around the corner?: enhancing driver awareness in autonomous vehicles via in-vehicle spatial auditory displays. In: Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, pp. 189–198. ACM, October 2014

    Google Scholar 

  7. Borojeni, S.S., Boll, S.C., Heuten, W., Bülthoff, H.H., Chuang, L.: Feel the movement: real motion influences responses to take-over requests in highly automated vehicles. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI 18) (2018). https://doi.org/10.1145/3173574.3173820

  8. Borojeni, S.S., Wallbaum, T., Heuten, W., Boll, S.: Comparing shape-changing and vibro-tactile steering wheels for take-over requests in highly automated driving. In: Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, (Automotive UI 17) (2017). https://doi.org/10.1145/3122986.3123003

  9. Borowsky, A., Oron-Gilad, T.: The effects of automation failure and secondary task on drivers’ ability to mitigate hazards in highly or semi-automated vehicles. Advances in Transportation Studies. 2016 Special Issue, (1), 59–70, 12 p (2016)

    Google Scholar 

  10. Bueno, M., Dogan, E., Selem, F. H., Monacelli, E., Boverie, S., Guillaume, A.: How different mental workload levels affect the take-over control after automated driving. In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC) (2016). https://doi.org/10.1109/itsc.2016.7795886

  11. Capallera, M., et al.: Owner manuals review and taxonomy of ADAS limitations in partially automated vehicles. In: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications - AutomotiveUI 19 (2019). https://doi.org/10.1145/3342197.3344530

  12. Wang, C., et al.: Designing for enhancing situational awareness of semi-autonomous driving vehicles. In: Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications Adjunct (AutomotiveUI 2017), pp. 228–229. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3131726.3132061

  13. Endsley, M.R.: Toward a theory of situation awareness in dynamic systems. Hum. Factors 37(1), 32–64 (1995). https://doi.org/10.1518/001872095779049543

    Article  Google Scholar 

  14. Solovey, E.T., Zec, M., Garcia Perez, E.A., Reimer, B., Bruce, M.: Classifying driver workload using physiological and driving performancedata: two field studies. In: Proceedings of the SIGCHI Conference on Human Factorsin Computing Systems, CHI 2014, pp. 4057–4066. ACM, New York (2014)

    Google Scholar 

  15. Ferreira, E., et al.: Assessing real-time cognitive load based on psycho-physiological measures for younger and older adults. In: 2014 IEEE Symposium on Computational Intelligence, Cognitive Algorithms, Mind, and Brain (CCMB), pp. 39–48, Orlando, FL, USA (2014)

    Google Scholar 

  16. Forster, Y., Naujoks, F., Neukum, A., Huestegge, L.: Driver compliance to take-over requests with different auditory outputs in conditional automation. Accid. Anal. Prev. 109, 18–28 (2017). https://doi.org/10.1016/j.aap.2017.09.019

    Article  Google Scholar 

  17. Gold, C., Damböck, D., Lorenz, L., Bengler, K.: Take over!” How long does it take to get the driver back into the loop? In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 57, no. 1, pp. 1938–1942 (2013). https://doi.org/10.1177/1541931213571433

  18. Gold, C., Körber, M., Lechner, D., Bengler, K.: Taking over control from highly automated vehicles in complex traffic situations: the role of traffic density. Hum. Factors J. Hum. Factors Ergon. Soc. 58(4), 642–652 (2016). https://doi.org/10.1177/0018720816634226

    Article  Google Scholar 

  19. Grah, T., et al.: Dorsal haptic display: a shape-changing car seat for sensory augmentation of rear obstacles. In: Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 305–312. ACM, September, 2015

    Google Scholar 

  20. Hosking, S., Young, K., Regan, M.: The effects of text messaging on young novice driver performance (No. 246). Monash University, Accident Research Centre (2006)

    Google Scholar 

  21. Johns, M., Mok, B., Talamonti, W., Sibi, S., Ju, W.: Looking ahead: anticipatory interfaces for driver-automation collaboration. In: Proceedings of the 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pp. 1–7, Yokohama (2017). https://doi.org/10.1109/itsc.2017.8317762

  22. Ko, S.M., Ji, Y.G.: How we can measure the non-driving-task engagement in automated driving: comparing flow experience and workload. Appl. Ergon. 67, 237–245 (2018). https://doi.org/10.1016/j.apergo.2017.10.009

    Article  Google Scholar 

  23. Kugurakova, V., Talanov, M., Manakhov, N., Ivanov, D.: Anthropomorphic artificial social agent with simulated emotions and its implementation. Procedia Comput. Sci. 71, 112–118 (2015). https://doi.org/10.1016/j.procs.2015.12.217

    Article  Google Scholar 

  24. Lee, H., Lee, J., Shin, M.: Using wearable ECG/PPG sensors for driver drowsiness detection based on distinguishable pattern of recurrence plots. 8, 192 (2019)

    Google Scholar 

  25. Löcken, A.: AmbiCar: ambient light patterns in the car. In: AutomotiveUI, Nottingham, UK (2015)

    Google Scholar 

  26. Lugano, G.: Virtual assistants and self-driving cars. In: Proceedings of the 2017 15th International Conference on ITS Telecommunications (ITST) (2017). https://doi.org/10.1109/itst.2017.7972192

  27. Mehler, B., Reimer, B., Coughlin, J., Dusek, J.: The impact of incremental increases in cognitive workload on physiological arousal and performance in young adult drivers. Transp. Res. Rec. 2138, 6–12 (2009)

    Article  Google Scholar 

  28. Mok, B., Johns, M., Yang, S., Ju, W.: Reinventing the wheel: transforming steering wheel systems for autonomous vehicles. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 229–241. ACM, October 2017

    Google Scholar 

  29. Patel, M., Lal, S.K.L., Kavanagh, D., Rossiter, P.: Applying neural networkanalysis on heart rate variability data to assess driver fatigue. Expert Syst. Appl. 38(6), 7235–7242 (2011)

    Article  Google Scholar 

  30. Petermeijer, S., Bazilinskyy, P., Bengler, K., Winter, J.D.: Take-over again: Investigating multimodal and directional TORs to get the driver back into the loop. Appl. Ergon. 62, 204–215 (2017). https://doi.org/10.1016/j.apergo.2017.02.023

    Article  Google Scholar 

  31. Petermeijer, S., Cieler, S., Winter, J.D.: Comparing spatially static and dynamic vibrotactile take-over requests in the driver seat. Accid. Anal. Prev. 99, 218–227 (2017). https://doi.org/10.1016/j.aap.2016.12.001

    Article  Google Scholar 

  32. Pettitt, M., Burnett, G., Stevens, A.: Defining driver distraction. p 12 (2005)

    Google Scholar 

  33. Politis, I., Brewster, S., Pollick, F.: Language-based multimodal displays for the handover of control in autonomous cars. In: Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications - AutomotiveUI 15 (2015). https://doi.org/10.1145/2799250.2799262

  34. Rauch, N., Kaussner, A., Boverie, S., Giralt, A.: HAVEit-Highly automated vehicles for intelligent transport. Model of driver behaviour for the assessment of driver’s state. Deliverable D32, 1 (2009)

    Google Scholar 

  35. Rosario, H.D., Solaz, J.S., Rodriguez, N., Bergasa, L.M.: Controlled inducementand measurement of drowsiness in a driving simulator. IET Intell. Transp. Syst. 4(4), 280–288 (2010)

    Article  Google Scholar 

  36. SAE International. Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles (2018)

    Google Scholar 

  37. Serrano, J., Di Stasi, L., Megías, A., Catena, A.: Effect of directional speech warnings on road hazard detection. Traffic Inj. Prev. 12, 630–635 (2011). https://doi.org/10.1080/15389588.2011.620661

    Article  Google Scholar 

  38. Shahid, A., Wilkinson, K., Marcu, S., Shapiro, C.M.: Karolinska sleepiness scale (KSS). In: Shahid, A., Wilkinson, K., Marcu, S., Shapiro, C. (eds.) STOP, THAT and One Hundred Other Sleep Scales. Springer, New York (2011). https://doi.org/10.1007/978-1-4419-9893-4

    Chapter  Google Scholar 

  39. Shakeri, G., Ng, A., Williamson, J.H., Brewster, S.A.: Evaluation of haptic patterns on a steering wheel. In: Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, (AutomotiveUI 2016), pp. 129–136. Association for Computing Machinery, New York (2016). https://doi.org/10.1145/3003715.3005417

  40. Stockert, S., Richardson, N.T., Lienkamp, M.: Driving in an increasingly automated world – approaches to improve the driver-automation interaction. Procedia Manuf. 3, 2351–9789 (2015). https://doi.org/10.1016/j.promfg.2015.07.797. ISSN 2351-9789

    Article  Google Scholar 

  41. Walch, M., Lange, K., Baumann, M., Weber, M.: Autonomous driving: investigating the feasibility of car-driver handover assistance. In: Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications - AutomotiveUI 15 (2015). https://doi.org/10.1145/2799250.2799268

  42. Wang, M., Lyckvi, S.L., Chen, C., Dahlstedt, P., Chen, F.: Using advisory 3D sound cues to improve drivers’ performance and situation awareness. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 2814–2825. ACM, May 2017

    Google Scholar 

  43. Waytz, A., Heafner, J., Epley, N.: The mind in the machine: anthropomorphism increases trust in an autonomous vehicle. J. Exp. Soc. Psychol. 52, 113–117 (2014). https://doi.org/10.1016/j.jesp.2014.01.005

    Article  Google Scholar 

  44. World Health Organization: Global status report on road safety 2018. Switzerland, Geneva (2018)

    Google Scholar 

  45. Wulf, F., Zeeb, K., Rimini-Döring, M., Arnon M., Gauterin, F.: Effects of human-machine interaction mechanisms on situation awareness in partly automated driving. In: Proceedings of the 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013), The Hague, 2013, pp. 2012–2019 (2013). https://doi.org/10.1109/itsc.2013.6728525

  46. Zhang, G., Yau, K.K.W., Zhang, X., Li, Y.: Traffic accidents involving fatigue driving and their extent of casualties. Accid. Anal. Prev. 87, 34–42 (2016). https://doi.org/10.1016/j.aap.2015.10.033. ISSN 0001-4575

    Article  Google Scholar 

Download references

Acknowledgment

This work is part of a project funded by the Hasler Foundation. We would also like to thank our colleagues who helped us during this project.

Author information

Authors and Affiliations

  1. HE-ARC University of Applied Sciences Western Switzerland, Saint-Imier, Switzerland

    Emmanuel de Salis & Stefano Carrino

  2. HEIA-FR University of Applied Sciences Western Switzerland, Fribourg, Switzerland

    Marine Capallera, Quentin Meteier, Leonardo Angelini, Omar Abou Khaled & Elena Mugellini

  3. University of Fribourg, DIUF, Fribourg, Switzerland

    Marino Widmer

Authors
  1. Emmanuel de Salis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marine Capallera
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Quentin Meteier
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Leonardo Angelini
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Omar Abou Khaled
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Elena Mugellini
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Marino Widmer
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Stefano Carrino
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Emmanuel de Salis .

Editor information

Editors and Affiliations

  1. The Open University of Japan, Chiba, Japan

    Masaaki Kurosu

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

de Salis, E. et al. (2020). Designing an AI-Companion to Support the Driver in Highly Autonomous Cars. In: Kurosu, M. (eds) Human-Computer Interaction. Multimodal and Natural Interaction. HCII 2020. Lecture Notes in Computer Science(), vol 12182. Springer, Cham. https://doi.org/10.1007/978-3-030-49062-1_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-49062-1_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-49061-4

  • Online ISBN: 978-3-030-49062-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation