Keywords

1 Introduction

Taking back control from an automated driving system will be a common event for drivers in highly automated vehicles. A driver may decide that he or she wants to re-engage in the driving task or an automated driving system may ask the driver to resume control over certain parts of the driving task because of a degradation of sensor quality or other constraints imposed by the driving situation (e.g. entering an area which is not certified for automated driving). If a driver resumes control of the vehicle there is a shift of responsibility from the vehicle to the driver. The driver has to switch his or her attention back to the driving task, perceive, understand and interpret the driving situation and manually, tactically and strategically control the vehicle in the context of the driving situation. Unfortunately, in many cases the situation in which the driver is asked to resume control will not only be highly complex, but will also feature some sort of time-constraint imposed on the driver, which is determined by the car (c.f. Object and Event Detection Response as defined by NHTSA 2017). In other words, if the car asks the driver to resume control it will often do so because it can only guarantee safe automated driving for a certain amount of time after the take-over request.

Therefore, from an engineering perspective it appears crucial to determine how long a driver will take to resume control and thereby relieve the automated driving system of its responsibilities. Therefore, the amount of time necessary for the driver to take control of the vehicle determines the time that an automated vehicle has to provide in a take-over situation (c.f. Operational Design Domain as defined by NHTSA 2017). A number of studies have investigated the necessary reaction time and found that after a take-over request most drivers will deactivate an automated driving system after approx. 3–6 s on average (e.g. Gold et al. 2013, Vogelpohl et al. 2018a). Researchers have also found that the duration of this automation deactivation seems to depend on several factors: Engagement in a Non-driving related task (NDRT), the current take-over situation, previously experienced take-over situations and the time available to take-over seem to have the largest influence (see review by McDonald et al. 2019) on this “take-over time”.

However, while it is crucial to know how long the average driver will take to take back control from the car, this perspective fails to consider what happens to the car and the driver after the driver has taken back control. It also ignores the drivers who fail to react within the average time frame. For example Jarosch et al. (2019) found that after prolonged driving with automation several participants in a driving simulator study completely failed to react to a take-over request. Similarly, other studies have shown that under certain conditions drivers will sometimes fail to react to take-over requests (c.f. De Waard et al. 1999, Stanton and Young 1998). In a previous study we (Vogelpohl et al. 2018) found that there were large individual differences between reaction times to take-over requests and that the average reaction time may not be representative for a safe transition to manual driving.

From a human factors perspective, as compared to an engineering perspective, it may therefore actually not be relevant how long a driver takes to deactivate the automation. What matters is if and how well the driver deactivates the automation and how well he or she performs the driving task after control has been transferred. If an automated driving system were able to measure in real time how long the driver is taking to deactivate the automation in a specific situation and how well he or she is performing during this transition, it could in turn determine when and how the automation can safely be deactivated. Such an Adaptive Automation (c.f. Corso and Moloney 1996, Kaber and Endsley 2004) would not rely on predefined predictions about how long an average driver takes, but would use the information available in the situation to determine how long a specific driver in a specific situation needs to safely transition to manual driving.

Kircher et al. (2014) speculate that “Drivers do not just seem to react to automation, but rather interact with automation […]” and that “[…] drivers integrate the behavior of the automation into their tactical planning of the whole situation instead of only reacting to the responses of the automation” (Kircher et al. 2014, p. 166). This statement can be also be taken to mean that in turn automation should incorporate the driver into its tactical planning and make the best use of the resources available to both the system and the driver in a take-over situation. If this perspective is applied, an automated driving system would need to answer the following questions:

  • How long does a specific driver need to successfully switch back to the manual components of the driving task (instead of how long the average driver takes to switch back)?

  • How does a specific driver switch from a NDRT to the driving task (regarding this as a dynamic process, rather than a moment in time)?

  • How does a specific driver allocate his or her attention towards relevant objects in the driving scene (again, regarding this as a dynamic process)?

  • How does a specific driver continue to drive after the responsibility for driving has been transferred?

If some or all of these variables are taken into account by the automated driving system, the system could adaptively decide which information to provide to a driver in a take-over situation. It could also predict if and when to hand control over to the driver and how to support the driver after the responsibility for driving has been shifted to the driver.

In summary, we believe that the narrow focus on reaction time in take-over situations falls short of what we actually need to understand to ensure a safe transition of control from the vehicle to the human driver. We may need to look at specific drivers and design the automated driving system to adaptively react to how individual drivers perform after a take-over request in a specific environment. Such a handover assistance system that provides a guided transition to manual driving could supervise the drivers’ actions during a transition to manual driving and decide to adapt the information provided to the driver and the support provided by the automated driving system based on the drivers’ performance in a given situation.

In the next sections we present what we believe to be evidence for a need to go beyond take-over times and to consider attentional dynamics of take-over situations. We discuss findings from our own research and from other researchers and explain why we think adaptive handover assistance systems in highly automated vehicles may improve the safety of automated driving systems.

2 Take-Over Times and Quality After Automated Driving

2.1 Take-Over Time

A number of studies have determined influencing factors on take-over times after automated driving (see McDonald et al. 2019 and Zhang et al. 2018, for recent reviews of take-over times). Such studies largely agree that average take-over times may lie anywhere between 2 and 6 s and that take-over time depends on a number of factors. The level of supervision of the automation, the NDRT before the take-over request, the time available for the transition and complexity of the situation are cited as possible influences on take-over time (McDonald et al. 2019, Vogelpohl et al. 2018a), as well as inner-subject factors such as experience with the automated driving, fatigue or age (e.g. McDonald et al. 2019, Vogelpohl et al. 2018b, Jarosch et al. 2019). Interestingly, one major factor also seems to be the time available for the transition. Specifically, longer take-over time budgets seem to correspond to slower take-over reactions (e.g. Gold et al. 2013, Gold et al. 2018, Payre et al. 2016, Zhang et al. 2018)

2.2 Take-Over Quality

Other studies have focused on the quality of driving after the transition to manual driving, rather than at the duration of the take-over (e.g. Gold et al. 2018, Happee et al. 2017). Studies focused on take-over quality have found that driving may be influenced for as long as 5 min after a transition to manual driving (Brandenburg and Skottke 2014). Among others, variables such as standard deviation of lane departure (e.g. Wandtner et al. 2018, Wiedemann et al. 2018, Zeeb et al. 2016), number of steering reversals (Merat et al. 2014), car following behavior or lane crossings (e.g. Zeeb et al. 2017, Strand et al. 2014, Wandtner et al. 2018) have been found to be influenced by transitions to manual driving.

Again, take-over time budgets seem to influence the quality of driving after a take-over request (c.f. Wan and Wu 2018, van den Beukel and van der Voort 2013, Mok et al. 2015, Gold et al. 2013, Gold et al. 2018): Specifically, lower time budgets will negatively influence lane and distance keeping quality (e.g. Wan and Wu 2018) and will also have an impact on higher level decision making processes (i.e. whether to brake, steer or both; c.f. Gold et al. 2013, Gold et al. 2018). Gold et al. (2016) argue that not only the available time budget may influence the quality of manual driving after a take-over request, but also the time drivers themselves allow for the transition.

The results from the studies on the take-over time and take-over quality have been interpreted to signify that drivers are physically and mentally taken out of the driving control-loop (Endsley and Kiris 1995) during automated mode and that they need time to resume the driving task. From research in other areas where human supervision of automated systems is required, we know that being out of the control-loop can have serious consequences for the human-machine system, such as a decrease in vigilance (Parasuraman and Davies 1977), a loss of situation awareness (Kaber et al. 2006) and mode confusion (Degani et al. 1999). This holds especially true if drivers have been performing immersive or motivating non-driving tasks during the automated ride (c.f. Vogelpohl et al. 2018a), which may make them vulnerable to task perseveration effects (Zeigarnik 1938, Fox and Hoffmann, 2002). Wandtner et al. (2018) observed task perseveration effects after automated driving with NDRTs. At the same time, it is likely that such motivating NDRTs will become more frequent as drivers become more confident with the safety of high automation levels and stop to supervise the automated driving system (e.g. Carsten et al. 2012, Jamson et al. 2013).

2.3 Situation Awareness After Driving with Automation

In addition to the immediate effects of take-over time on the transition to manual driving, effects on Situation Awareness after automated driving have been discussed in the literature. Situation Awareness is largely defined as ‘‘[a] the perception of elements in the environment within a volume of time and space, [b] the comprehension of their meaning, and [c] the projection of their status in the near future.” (Endsley 1988, p. 97). Some studies have identified the complexity of a situation as a predictor for the time that drivers will take to transition to manual driving after a take-over request (e.g. Gold et al. 2016, Naujoks et al. 2014, Radlmayr et al. 2014). The complexity of a take-over situation can be linked to the level of situation awareness required to appropriately react to a take-over request after a drive where NDRTs have been performed (e.g. Merat and Jamson 2009, Merat et al. 2012).

In Vogelpohl et al. (2018a) we define the complexity of a take-over situation in congruence with the definition of Situation Awareness by Endsley (1988): The complexity of take-over situation could therefore be defined by “[a] the number of elements present during the transition to manual driving [b] the difficulty with which meaning can be attributed to the elements and [c] the predictableness of the future status of these elements” (Vogelpohl et al. 2018a, p. 466). This definition is in line with the findings from Gold et al. (2016) who found that participants in their study needed longer to take back manual control if the number of elements (the traffic density) was increased in a take-over situation. The authors argue that “longer takeover times are actually indicative of a better reaction, as participants took the necessary time to regain situation awareness before starting a maneuver” (Gold et al. 2016, p. 7). Vlakveld (2015) notes that we can only be sure that the driver has fully regained situation awareness when we know if driver is able to react to latent threats and hazards in the driving situation, during and after the transition to manual driving. Studies which focused on situation awareness after take-over request have found that eye-tracking measures may be promising to determine a drivers’ readiness for a take-over (Braunagel et al. 2017) and that based on such measures drivers may take up to 7–12 s to obtain situation awareness after take-over requests (c.f. Lu et al. 2017, Vogelpohl et al. 2018a). Lu et al. (2017) suggest that it may take a driver even longer than this to infer predictive information about latent hazards and safety-relevant objects from a complex driving scene.

2.4 What Is a Safe Transition to Manual Driving?

It can therefore be argued, that neither the duration of the time which a driver takes to transition to manual driving after a take-over request nor the immediate quality of his or her driving is a sufficient measure of the ability to perform well in a complex driving situation. Instead of a measure of take-over quality, the take-over time and the driving quality after the take-over are only a function of the take-over situation and the individual driver. In other words, it would be insufficient to infer the readiness of the driver to react to latent threats after a take-over situation from the time provided to the driver to take over. In part this is because drivers will react within the time which is provided to them (as discussed above) and in part this is because drivers do not necessarily deactivate the automation when they are actually ready to drive (as argued in Vogelpohl et al. 2018a). Instead, some drivers will deactivate the automation because the take-over request tells them to deactivate the automation (c.f. automation complacency, e.g. Parasuraman and Riley 1997) or because they fail to perceive the complexity of the situation.

If take-over time is not a sufficient predictor of the drivers’ ability to perform after the responsibility of control has been shifted from the automated driving system to the driver, how can critical situations after take-over requests be avoided? An analysis of driving quality after the transition of control could be an indicator, but can only be measured after the fact. If a driver is found to badly follow his or her lane or is following very close behind a lead vehicle after the automation has been deactivated, this can be taken as an indicator that the transition to manual was unsuccessful or is indeed not yet completely finished. But it cannot be used to avoid such behavior in the first place. If a driver deactivates the automation very quickly after a take-over request this can be taken as an indicator that the driver has failed to carefully inspect the driving situation and will likely miss latent hazards or unexpected events. However, it could also mean that the driver has been monitoring the driving situation before the take-over request and is simply ready to take back control even before the take-over request is issued. To address these open issues, we discuss attentional dynamics as a more encompassing view of driver behavior in a take-over situation.

3 Attentional Dynamics After Take-Over Requests

Attentional dynamics are a concept first proposed for the context of driver distraction and partially automated driving systems by Lee (2014). Lee (2014) distinguishes attentional dynamics from attentional resources and argues that distraction during driving can also be described as a failure to appropriately allocate attentional resources. This includes aspects of task timing, task switching and task prioritization as well as task perseveration. Lee (2014) argues that understanding attentional dynamics, i.e. how, when and where drivers allocate attentional resources, will become increasingly important with higher levels of vehicle automation and that this concept may complement the perspective of attentional resources. We believe that this approach is not only valid in manual driving for frequent switches between the driving task and secondary tasks, but may also be applicable to the more detailed level of switching from a NDRT back to manual driving in higher levels of automated driving. As described by Lee (2014), interruption management becomes crucial to the safety of the driver, if a sudden task switch from the NDRT back to manual driving is required, as would be the case in a take-over situation.

If, as we argue, take-over time and take-over quality provide an incomplete picture of the transition to manual driving or are not sufficient to be used as a predictor for the outcome of a transition to manual, attentional dynamics could provide a new perspective to understand take-over situations. We propose regarding the transition to manual driving as a dynamic process with interdependent granular sub-tasks, which are indicative of the attentional dynamics and attentional resources of the driver. If sub-tasks are neglected or falsely prioritized, this may indicate insufficient situation awareness and a mismatch of attentional resources to the attentional demands of the take-over situation (c.f. Johnson et al. 2017, Vogelpohl et al. 2018a).

The attentional dynamics of a take-over situation could for example be described as follows: If a take-over request is issued while a driver is engaged in a NDRT, the driver is likely to redirect his or her attention towards the roadway in a certain amount of time (task switching), but will then often return visual attention to finish/interrupt the NDRT (task perseveration) or to find a place to store the task if it was performed on a handheld device. While, or after, the NDRT is stored (task prioritization), the driver will direct his or her visual attention towards areas which he or she believes to be relevant for the manual driving task (prioritization of lower priority sub-tasks, as discussed in Johnson et al. 2017). The gaze may be directed towards the forward roadway, it may be directed towards one of the rearview mirrors, the gauges of the car or some other visual area which the driver expects will help to understand the situation (expectancy). Visual attention will then be redirected multiple times in response to the first impression of the driving situation, the information which is gathered through other modalities (e.g. vibration from the road surface, acceleration of the car, approaching sirens of an ambulance) and the previous knowledge that is stored in the drivers’ long term memory (prior knowledge and experience with take-over situations; c.f. Payre et al. 2016). While visual attention is spent on gathering information, simultaneously motor processes are engaged which move the drivers’ hands and feet towards the controls of the car.

In a take-over situation, the visual, motor and other modalities will partially depend on one another and on the information gathered from the environment. If, for instance, visual attention is quickly directed towards the side mirror and a vehicle in the adjacent lane is perceived, the likelihood that the first steering reaction will be towards that lane may be reduced. At the same time, the likelihood that the driver will brake may increase if a previous visual check has detected a braking lead vehicle. In this way, every redirection of attention is partially linked to every previous redirection of attention. Thereby, the attentional dynamics of the driver are the sum of all observable reactions from the driver and their temporal relation to one another after a take-over request has been issued. The prioritization and allocation of attentional resources after a take-over request is moderated by prior experience with take-over situations. The allocation of attention before a take-over request is influenced by the trust placed in the automated driving system (which is in turn influenced by prior experiences with take-over situations). We have visualized an example for the process of disengaging from a NDRT and transitioning back to manual driving as related to the theory of attentional dynamics in Fig. 1.

Fig. 1.
figure 1

Example model of attentional dynamics in a take-over situation.

Full size image

How can we use the attentional dynamics during a take-over request to increase the safety of transitions from automated driving to manual driving? We argue that an automated driving system should be aware of the attentional dynamics and resources of the driver in a take-over situation. By observing and tracking the drivers’ reactions and their temporal relation the automated driving system may be able to predict in real time how attentional resources of the driver are allocated and how this will influence the drivers’ reactions to the take-over situation. In consequence, such a system would be able to react to what the driver is seeing and doing and improve the information and support to the driver accordingly. This would make the automated driving system and “adaptive automation”.

4 Adaptive Automation and a Handover-Assistance System

Adaptive automation has been widely discussed in contexts where humans supervise complex automated systems and where time critical decisions may be required from the human operators (e.g. power plants, aircraft; air control centers; c.f. Parasuraman et al. 1996, Parasuraman et al. 1999). In contrast to static automation, adaptive automation will react flexibly and context dependent to the requirements of a human operator. It will take into account what is known to the operator and provide information about the current system state in such a way that an optimized performance is achieved by the operator-machine team. Adaptive automation has been shown to reduce some of the negative effects of automation on human operators by improving situation awareness and avoiding overreliance and skill degradation (e.g. Hilburn et al. 1997, Scallen et al. 1995, Scerbo 1996). The goal of adaptive automation is to increase the performance of the human-automation system and to mitigate the effects of excessive workload as well as task induced fatigue.

This can be achieved by adapting the information provided to the operator to a specific context or situation or by adapting the tasks allocated to either the operator or the automation. For example, during normal operation a power plant operator may be provided with a scaled down overview of the most important variables of the plant, which will allow him or her to monitor all variables at once and to identify unusual patterns if they occur. However, during an emergency situation the variables most important to the current situation will prominently be provided to the operator to allow him or her to focus on restoring normal operation. Additionally, decision support may be provided during such high workload situations to guide the identification of potential solutions to the emergency. During emergencies the automation may also decide to take over tasks which are performed by the operator during normal operation. This can temporarily free up attentional resources which the operator needs to handle the current emergency.

Such an adaptive system could also improve the transition from automated driving back to manual driving. Based on the variables discussed in Sect. 2, some pre-conditions would apply to enable a system to actively guide a driver through the transition to manual driving and to adaptively provide support to the driver. The handover assistance system which we propose should:

  • Track the driver availability (e.g. Marberger et al. 2017) and monitor the driver state (e.g. fatigue) during the automated drive.

  • Track the drivers’ hand, feet and eye movements to allow predictions about motor readiness and visual attention allocation.

  • Include a sophisticated model of driver behavior which can quickly predict driver reactions based on the available variables (e.g. predict impending lane changes from glance behavior, see. Henning et al. 2008, Salvucci and Liu 2002).

  • Include a sophisticated model of driver attention and driver workload which allows predictions about the current and future status of the drivers’ attention allocation and workload.

  • Include a sophisticated environmental model which interacts with the driver model (for example drivers may be less likely to use steering to avoid a crash if the road is slippery; drivers may be more likely to react by braking if the traffic density is high).

  • Enable minimal risk maneuvers (e.g. Reschka and Maurer 2015) from complex driving situations as a fallback option in every take-over situation if the driver is judged unavailable or if the driver fails to appropriately react in a take-over situation.

Such a hypothetical system which is able to track the drivers’ actions and reactions and his or her estimated allocation of attentional resources during a transition to manual driving could then use this information to:

  • Direct the drivers’ attention towards potentially relevant objects and controls.

  • Improve the drivers’ attention prioritization by providing guidance on “what to do next” and “where to look next”.

  • Avoid task perseveration effects by quickly removing NDRTs.

  • Warn the driver about hazards which he or she may not have detected during the transition to manual driving.

  • Decide whether to provide braking and/or steering support to the driver in a complex take-over situation.

  • Decide if and when a shift of responsibility to the driver is safe or whether a minimal risk maneuver should be engaged.

  • Decide when the transition to manual driving has been completed.

  • Decide if a take-over request is likely to be beneficial to the safety of the driver, or whether a minimal risk maneuver should be engaged without trying to reengage the driver (e.g. if the driver is exhibiting strong signs of fatigue).

A handover assistance system based on attentional dynamics and attentional resource allocation has the potential to not only ensure that a transition to manual driving is safely performed. It could also render the transition more efficient by reducing the uncertainty of the driver about a situation (thereby improving situation awareness in complex situations, c.f. Johnson et al. 2017, Vogelpohl et al. 2018a) and by adapting to drivers who are experienced at take-over situations or perform well in a take-over situation to quickly transition to manual driving. On the other hand, inexperienced or slow drivers would be provided with the optimum level of assistance which could help to build trust in the automated driving system and to improve learnability of the system. We have summarized a first draft for an adaptive automation based on the prediction of attentional dynamics and attentional resources in Fig. 2.

Fig. 2.
figure 2

Draft for an adaptive automation for a handover assistance system

Full size image

5 Existing Research on Handover Assistance Systems

Some efforts have already been made to define and build handover assistance systems and human-machine interfaces which support the driver during transitions to manual driving after driving with automation. Walch et al. (2015) designed and tested a handover assistant in a driving simulator setting. However, the handover assistant in the study by Walch et al. (2015) did not meet the criteria of an adaptive automation in that the information provided to the drivers and the drivers’ input after a take-over request was not adapted to the reactions of the drivers. The authors note that the majority of the participants “took over as soon as they were alerted, and did not report trying to perceive the situation before they intervened” (p. 17). Therefore, Walch et al. (2015) propose a system monitored handover during which the drivers’ steering and braking inputs are monitored and may be adjusted by the automation to avoid impulsive reactions from the drivers. The handover assistance system proposed in Fig. 2 is different from such a system in that it could not only avoid inappropriate driver reactions, but actively improve the reactions of the driver to render the take-over more efficient. Also, the hypothetical system proposed in our paper goes beyond driver inputs such as steering and braking by using driver gaze analysis and hand/feet tracking to estimate visual attention allocation. Similarly, Braunagel et al. (2017) have proposed a measure to determine driver readiness for take-overs through the analysis of the drivers NDRT engagement, the current traffic situation and the drivers gazes at the roadway. The authors showed that the aggregation of this information could successfully be used to issue pre-warnings to drivers who displayed a low driver readiness before a take-over request.

Clark et al. (2018) propose a speech based handover assist for planned, non-critical transitions of control. In situations where the car is able to rely not only on its sensors, but also on information from a navigation system or car-to-x data, take-over requests may be issued much earlier to allow the driver to transition to manual driving at his or her own speed (e.g. highway exits). Similar to time-critical take-over situations, researchers have found a wide range of reactions and reaction speeds to non-critical take-over requests (e.g. Eriksson and Stanton 2017). Therefore, for such non-critical transitions it would be equally important to guide drivers through the transition, thereby rendering their behavior more predictable and their driving more stable. Clark et al. (2018) based the design of their handover assist on checklists from other domains where handovers of control frequently occur, such as air traffic and medical care. The drivers in their study where guided through the transition of control by a speech assistant. Depending on the guidance strategy used, the drivers in the study took more or less time to transition to manual driving after driving with automation. The authors note, that in the experimental condition where no speech guidance was provided drivers paid very little attention to the environment before the automation was deactivated. This further highlights the need for a guided transition to manual driving. Again, the speech guidance provided in this study was not adapted to the individual needs of the driver.

Adaptive automation for driver assistance systems may be difficult to implement, but if warning signals and control allocation is adapted to the contextual needs of the driver such systems could be significantly improved. As an example, Kaß et al. (2018) proposed an adaptive warning strategy for collision avoidance systems which would be able to reduce unnecessary warnings to the driver. The authors discuss that by adaptively taking into account the time to collision as well as the drivers’ maneuver intentions system effectiveness as well as user acceptance could be increased.

6 Future Research Needs

Much research is needed to determine the correct variables to as input for the driver model, as well as to reliably measure these variables and to improve real-time predictions. The variables discussed in this paper (e.g. gazes to the mirrors, gazes to the gauges, hand/feet movements) can only serve as a first indication of what might actually be needed to reliably model and predict driver behavior in a take-over situation. Individual differences (such as age or driving experience) and driver states (such as fatigue or stress) may also need to feature prominently in the modeling of take-over situations. Finally, in the course of future studies it may become apparent that taking into account individual “live” variables during transitions to manual driving does not significantly increase the predictability of driver behavior in a take-over situation compared to what is already known about average take-over behavior. Empirical testing of adaptive handover assistance systems should determine whether and how they can improve the transition to manual driving.

Additionally, much work needs to go into the definition and technical improvement of minimal risk states for every conceivable driving situation. Due to the high variance in possible reactions after a take-over request, highly automated vehicles should always provide a fallback option for those situations in which the driver does not react to a take-over request or in which he or she reacts inappropriately. The models presented in this paper should only be taken as a starting point which can guide future research on the attentional dynamics of take-over situations and adaptive car to driver handover assistance systems. Future research will need to determine if the transition to manual driving can be improved by adaptive, human centered automation.