1 Introduction

The most traditional way of providing information to the driver is through visualization. Graphical user interfaces (GUIs) are still the most popular method of information mediation in GPS systems, radios, and mobile phone as well as other IVI systems. This is problematic, as GUIs reduce driving safety alarmingly. When a complicated GUI captures driver’s gaze and attention, it demands more than 20 s for the driver to gain awareness of surroundings and take control over driving as defined by NHTSA [1, 2]. In urgent situations, 20 s is too long a delay to prevent collisions or other accidents. Head up displays (HUDs), continuous information mediation in automated driving, and glasses have been proposed to solve the problem as they seem to be superior to manual driving assisted with dashboard-mounted displays [3, 4]. However, the basic problem of visualization capturing user’s attention remains. The advantage of auditory feedback is that information presentation through non-directional audio signals might be less disruptive in shifting gaze from road to the UI. The driver can use speech for input and get audio as a response. Meanwhile, in addition to traditional synthetic speech samples and warning beeps, directional and spatial audio has been implemented in IVI systems as feedback methods. Several studies indicate that such specialized audio cues work better than un-localized sounds. Beattie et al. [5] used audio cues to inform the driver about approaching vehicles. Spatial audio signals made drivers react faster to surprising events than localized non-directional audio cues in comparison to no audio conditions. Politis et al. [6] tested the effectiveness of spoken messages and audio warnings in two experiments. The participant’s task was to play with tablet and take control of driving after receiving abstract audio cues. Abstract audio signals worked only when presented from the center of gaze and attention (in this case, tablet) instead of, for instance, behind steering wheel. However, audio is easily neglected due to environmental noise inside vehicle, and therefore not alone enough for information mediation.

The use of haptics and touch-actuated interfaces has often been considered as an option to overcome shortcomings of both visual and auditory information mediation. Our research in the field [7, 8] shows by simply implementing haptics to current IVI systems, it is possible to reduce the visual workload and decrease reaction time for driving-related tasks. Haptic feedback or clicks have also shown [9] to significantly improve evaluated user pleasantness and task completion times when operating with touchscreen. Moreover, in touchscreen-based IVI systems, surface properties and shape can have drastic effects on interaction. Different textures can be easily identified [10], friction variations can make the same surface feel smooth, slippery, or resisting [11], and by adjusting the surface temperature it is possible to make the haptic interaction more pleasant [12]. Providing haptics through shape-changing in user interfaces like touchscreen can be divided into varying orientation, form, volume, texture, viscosity, and spatiality. Recent developments in actuator technologies and materials science have enabled active morphing or deformation of haptic surfaces and devices [13]. Even though there has been plenty of progress in haptic technologies recently, most used haptic interaction methods still produce primitive vibration in the entire device making haptic feedback prone to environmental noise. Another significant problem of the technology is the driver’s need to touch the device being operated. This means that at least one hand must be taken off the steering wheel to be able to operate the UI, which can be seen as somewhat problematic. Implementing touch-sensitive buttons mounted on the steering wheel or using gestures instead of pressure-based haptic input could solve this problem, but perhaps the most efficient method would be to utilize non-contact haptic feedback.

2 Designing novel multimodal interaction systems

Essentially, IVI systems must be built so that they enable the possibility of using several modalities based on task, environment, and even the driver’s status. Our research [7,8,9] has already started to solve this problem showing, for example, that it is possible to provide driving information and improve driving performance by augmenting current environmental and telemetric information through haptics as well as visual and auditory feedback. Other research groups have shown similar results that information mediation is most effective when several modalities are used concurrently [6]. Burke and Prewett [14], conducted a review of 43 studies comparing the effect of multimodal information exchange and concluded that redundancy in information delivery using different modalities can increase task performance and reduce task completion times (TCT). Adding multimodal feedback, especially haptic actuation for handheld and mobile devices can also improve perception in subjective workload [15] and reduced task completion times [16]; in both cases, improvements were also observed in error rates. Other studies such as Lee and Spence [17] and Serafin et al. [18] explored the driver’s attention within an automotive-themed task. Their findings indicated that reaction and task completion times were both improved when information was provided through visual, audio, and haptic feedback. Moreover, Richter et al. [19] in collaboration with BMW investigated the evolution of a touchscreen interface and how haptic feedback can increase immersion and reduce errors within a driving environment using the 2 + 1 model. They found that participants on average performed less errors and completed the tasks much faster with haptics feedback than without it.

Looking at current research there is substantial evidence that multimodal interaction approach within an automotive environment can greatly enhance in-car interaction. However, most of this research only utilizes vibrotactile signals to provide haptic feedback whereas several different techniques have been used to generate visual feedback (Head-Up projection, center and Instrument cluster displays etc.) and auditory feedback (directional sonification and surround sound audio). As a moving vehicle already has greater environmental (vibration) noise [20,21,22,23], it is important to consider other options for delivering haptic signals to the skin. For this reason, our research explores how haptic feedback (i.e. kinesthetic and pneumatic feedback) can be used for touchscreen and non-touchscreen in-vehicle interaction in combination with other modalities. The research also explores developing actuation devices that provide non-contact haptic feedback using pneumatic actuation. Moreover, the research investigates possibilities of mitigating environmental noise (road noise) by altering the bandwidth of the actuation signal to ensure that these signals remain outside the scope of the environmental noise (providing tangential actuation compared to orthogonal actuation). And lastly, this research explores new interaction techniques of providing secondary tasks (navigation) information using in-seat kinesthetic actuation (skin stretch) along with visual and auditory feedback to reduce driver distraction.

2.1 Pneumatic sub-woofer prototype (PSW) for touchscreen interaction

2.1.1 Altering actuation bandwidth to reduce environmental noise

The main motivation of the pneumatic sub-woofer (PSW) device was to have easily perceivable touch feedback in a wide range of environments and extend the field of contact. Applying air pressure (pneumatic) variations on the screen contact point is not entirely new, but its application within a moving vehicle is novel. In fact, most haptic devices that incorporate pneumatic actuation for touchscreen either employ mechanical systems to generate vibration-based actuation or try to deform the touchscreen surface to create on screen objects, similar to Tactus technologies®. Essentially, very few researchers have tried to directly apply pneumatic variations to the skin contact on the touchscreen. This is interesting as gentle, yet sensible variations of pneumatic pressure can not only stimulate the rapidly adapting (RA) and Pacinian corpuscle (PC) receptors but can also stimulate the thermoreceptors in the skin [20]. As cold sensing receptors are 3.5 times as common as the warmth sensing receptors [24], any compressed or pressurized displacement of air can simulate cooling or evaporation effect on the skin. This would mean that given the right environment conditions pneumatic variations presented to the skin contact maybe useful in providing encoded haptic information (e.g. environmental information for deaf driver). However, considering compressed air feedback, there is a need to pressurize air into a reservoir and ensure the volume is enough. Furthermore, the process of compression as well as release often generates unwanted noise and vibration within the system. For this reason, we decided to forgo the conventional methods of creating pressurized pneumatic variation and utilized existing technologies already available inside current vehicles to achieve the desired result. Hence, we developed the pneumatic sub-woofer (Fig. 1) that utilizes the actuation of the car sound system (two subwoofers) and funnels the air to the touchscreen surface.

Fig. 1
figure 1

The PSW prototype provides pneumatic feedback from the bottom of the touchscreen

Full size image

2.1.2 PSW device parameters and design

The pneumatic subwoofer (PSW), prototype creates pressurized air pulses via 2 hermetically sealed subwoofers in a closed chamber and funnels the air pulses onto the surface of ExoPC tablet’s touchscreen. The prototype provides variable magnitude of pneumatic pulses via a modulated digital sine wave generator, which can be regulated to translate signal amplitude and frequency into pneumatic haptic signals. It consists of two standard (Raptor-6) car-woofers of 140 W each with (2 × 4) 8 ohms load impedance and uses a maximum of (6.5 A × 2) 13 A of current. The signal was amplified using a custom linear full-bridge floating balanced power amplifier (with short circuit protection) that pressurized a limited volume of air and pushed it gently onto the touchscreen surface to provide haptic feedback. The PSW provided suppressed air feedback to the surface of the touchscreen center console by pushing air into a channel that was attached at the bottom of the touchscreen console (Fig. 1) and concealed using a plastic bracket. A frame around the touchscreen display ensured that the provided pneumatic feedback remained unaffected by any environmental noise (wind) that may be present in the moving vehicle. Conventional touchscreen contact based haptic systems are generally dependent on the type and duration of touchscreen contact, as this duration dictates the signal specifications as well as efficiency of information transfer from the point of contact (stiff glass surface) to the skin contact (fingertip). Conversely, while using non-contact actuation such as PSW, it may be possible to provide similar actuation signals in a lower amount of screen contact time as both the approach as well as retreat of the finger from the display can be used as actuation on-time. And lastly, once the user ends screen contact, the fingertip is no longer interacting with the glass surface and hence, is not limited by the dampened skin receptors that occur while interacting with flat ridged surface of the touchscreens [24]. Therefore, the applied pneumatic actuation has the potential of being more sensible.

2.2 Linear screen exciter (LSE) for touchscreen interaction

2.2.1 Overcoming environmental noise in a moving car

As the IVI touchscreen systems are used in haptically noisy environments, there are fundamentally only two ways to improve or enhance tactile feedback. The first is to alter the specifications of the actuation signal by increasing the magnitude, or the frequency to make it easily perceivable. Both approaches could be challenging, as magnitude increases would require more power and or different types of actuation sources, while increasing the frequency may drastically alter how skin receptors sample the applied signal and how touch signals may be interpreted by the human brain [24]. The second method would be to alter the type of signal in such a way that it is no longer presented in the same bandwidth as the environmental (noise) signal. This essentially means that because the motion and suspension of the vehicle translates road noise into vertical orthogonal component of vibrations within the cabin, the ideal method of circumventing this environmental (noise) signal is to change the axis of the applied information signal. For that reason, a tangential signal, which is perpendicular to the motion of the vehicle (lateral motion) to generate skin micro-displacements would be more effective than a similar orthogonal signal applied to the touchscreen device. This is because shear force and tangential micro-displacements of the skin are perceived by several types of mechanoreceptors [25]. Therefore, we are of the option that skin micro-displacements via lateral forces can be used to create the illusion of textural surfaces, by varying frequency and the lengths of micro-displacements of the contact surface.

2.2.2 LSE device parameters and design

To overcome environmental noise, we developed an advance layer of tactile feedback that could reduce the need for continuous visual validation. This device uses a stiff transparent screen overlay that actuates tangentially over the touchscreen to generate skin micro-displacements. This type of tangential or lateral actuation was designed to create better sensible confirmation feedback signals in a haptically noisy environment. The tangential actuation of the screen overlay was created using three 3 Tectonic voice coil actuators (TEAX14C02-8) affixed to the transparent screen overlay with an L-shaped bracket. The overlay was constructed from a Plexiglas frame (10 mm wide) covered with a PET film of 100 μm thickness. The touchscreen device used for this setup was an Intel ExoPC running Meego IVIs operating systems with a custom IVI skin designed for the HapticAuto Project [7]. The overlay covered the entire screen of the ExoPC and sat almost flush with the screen (Fig. 2).

Fig. 2
figure 2

(Left) Functional design of the LSE device and (middle) mockup as well as the final device (right)

Full size image

The overlay, was slid into place using horizontally mounted grooves/rails on the top and bottom of the glass display, ensuring the once the actuators were fired the overlay would slide within the rail mechanism efficiently, creating smooth and similar displacement every time. The actuation signal sent to the voice coils was calibrated to create the maximum stable displacement in the overlay. Furthermore, two delimiters were fixed within each railing (on either side of the edge of the overlay), to ensure that the overlay displacement did not exceed the maximum shift of 2 mm, side to side (max 1 mm in any direction). This was done after the pilot testing of the setup, to keep the displacement consistent in a haptically noisy environment and to ensure that the users were not able to slide the screen overlay too much when no actuation was being applied. This also adheres to the perceptual angular threshold identified by Vitello et al. [26] and displacement threshold researched by Placencia et al. [27] in the comparison between the distal and proximal direction. Furthermore, the maximum displacement value of the screen overlay in any one direction (1 mm) correlated with the average excursion of the voice coil actuation under the load of the screen overlay (~ 0.9 mm ± 0.1 mm), therefore the mechanism was quite robust, even in environment with vibrational noise.

2.3 Supplementary system

To ensure that participants were provided consistent visual and haptic information and selection of Meego IVI platform did not affect performance in any of the designed tasks, certain experiment-design specification was enforced in the three studies. These included the development of a universal onscreen keyboard for the ExoPC slate custom IVI software, a custom Head-up Display and a Custom Vibrotactile based Actuation Prototype (CVAP) for the ExoPC tablet. The custom onscreen keyboard (OSK) followed the conventional QWERTY layout with a key-size similar to the native keyboard (as seen in Fig. 3a) as both Studies (I and II) utilized it for visual feedback. The customized Head-Up Display (Fig. 3b) was used to let the driver interact with the IVIS without taking his eyes away from the road. This was developed as a safety option (in Study II) ensuring the participants had a fall back option to use in case of sudden onset of traffic.

Fig. 3
figure 3

a The customized haptic onscreen keyboard utilized during the studies. b The HUD implementation of the IVIS depicting the touch point (cursor on the left) and the interaction (on the right)

Full size image

Along with custom OSK and HUD, we along developed a Custom Vibrotactile based Actuation prototype (Fig. 4), which served as a control mechanism to facilitate a meaningful comparison between the designed PSW (Study I) and LSE prototype (Study II) devices. At the time of this study, there is no standard IVI system specification to provide vibrotactile feedback. For this reason, we needed to develop a standard reference point for testing the PSW and LSE prototypes. To achieve this, we evaluated some of the common tablet devices available in the market (Galaxy Tab) with haptic feedback, by measuring the displacement and acceleration generated by the embedded actuator(s) using Micro Sense displacement sensor (5810) and 5622-LR, 20 kHz probe.

Fig. 4
figure 4

Custom vibrotactile actuation prototype with specifically designed rear mounted actuator shaft for the ExoPC slate tablet

Full size image

Similarly, we measured the acceleration using a tri-axis acceleration sensor (MSR165 series) by PCE Instruments. Using these sensors, we recorded a maximum of 171.2um displacement and a maximum of 4.3 g, 2.5 g and 0.3 g (at x, y and z-axis respectively) acceleration. Therefore, we developed the CVAP to produced twice the maximum displacement (~ 340um) and 1.5 times the acceleration (recorded on the x-axis of the Samsung Galaxy device), at an accuracy of ± 5% of the desired functional baseline. The device was developed using a rotation motor and an eccentric shaft spanning the entire diagonal length of the tablets, and it was actuated by a powerful DC gear motor. The motor was adjusted with a gearing mechanism to ensure the necessary actuation parameters were at 12 V, 1 A power input.

3 User study I: usability and sensibility of conventional vibrotactile actuation

The first study was carried out in a Volvo XC60 being driven in a straight line by professional drivers on the Nokia Tires Track (NTT). The purpose of the study was to evaluate the effectiveness of providing pneumatic actuation on the central touchscreen for text entry and menu-based selection tasks as compared to simple vibration-based feedback.

3.1 Experimental setup

This user study focused on investigating the usability and sensibility of conventional vibrotactile actuation versus pneumatic actuation on the touchscreen in a stationary and moving vehicle. There were a few reasons for including both the stationary and moving car scenarios in the testing. Firstly, although the car was stationary, the engine was left running, which did create some environmental noise in the cabin. Moreover, as most cars and their IVI systems lock-up specific features (which they may consider to be dangerous to carry-out) when the car is being driven, it was important to test these features (text entry, menu, submenu selection and gesture) within both conditions. Vibrotactile feedback was provided via the custom CVAP device, which was affixed at the rear of the 11.6-inch ExoPC slate tablet. The pneumatic feedback was provided using the PSW prototype, which generated pneumatic feedback specific to a certain section of the screen. Both prototype devices were controlled by a dedicated control module linked to the ExoPC Slate via a USB port. Furthermore, the ExoPC Slate used the Meego 1.2 IVI open source operating system. The entire setup was installed in a Volvo XC60 vehicle and designed to function in a moving car using the onboard electronics.

3.1.1 Experimental design and primary (driving) task performance measurements

The study was conducted at the Nokia Tires test track and the participants (N = 6, all males) were all professional test drivers of the company (Fig. 5). They were asked to carryout simple touchscreen tasks related to the 3 use cases (one task each), while carrying out their primary task of driving at 50 km/h on a straight road (about 400 m for a single run) or while the vehicle was stationary. During each run (400 m) participants performed one task/subtask using one of three modalities (visual only, CVAP and PSW). Using average speed of 45 km/h, participants had approximately 32 s to complete the task (Fig. 5). Driving behavior, including speed, steering wheel movements (steering reversals), horizontal vehicle deviation (lateral lane positioning) and head movements were measured and referenced to baseline measurement. Touchscreen interaction, such as task/subtask durations; incorrect/incomplete selection and erroneous repetitive selections were all logged. The data was evaluated for both cases where the vehicle was stationary or moving and the participants were asked to fill out a questionnaire comparing the two types of haptic feedback.

Fig. 5
figure 5

Lateral deviation measurement and driving behavior of participants while driving on the NTT. (Photo was taken in early April however; the testing was conducted in May when the snow had melted away)

Full size image

3.1.2 Touchscreen interaction secondary tasks performance measurements

For the purpose of this study three tasks were considered to assess the functionality of the multimodal application setup (PSW and CVAP). These included “Text Entry”, “Layered Menu Selection” and “Onscreen four directional Gestures” (Fig. 6). These tasks were each performed in two conditions (Still/Moving car) and in the following modalities visual only, CVAP + visual, PSW + visual. The setup and the features of operating each task were introduced to the participants before the start of the test. The two conditions of the vehicle were; the stationary car, in the parking area with the engine running, and the moving car; where, the participants were asked to driver in a straight line down a section of the Nokian Tires Track in Nokia City. The track was market with a 152.4 mm (6 inches) wide white line in the center of the lane, and the participants were asked to keep a leveling scale marker mounted in the dashboard of the vehicle aligned to the white line on the track (Fig. 5). This was done to ensure that the participants had a visual baseline of their expected driving performance on the open-ended track. After each subtask/prototype (modality) was evaluated, the participants were asked to fill-in a questionnaire rating the particular modality against each other. At the end of each test a non-formal interview was conducted to evaluate the overall system and compare it with regular functionality inherent to the original IVI systems.

Fig. 6
figure 6

The three tasks included in the study (gestures multi-layered menu selection and test entry on OSK)

Full size image

3.2 Results and analysis

The test included drivers performing a primary task; driving in a straight line following a white marker on the road, while interacting with UI controls on the center console. Both driving performance (primary task) and UI interaction performance (secondary task) was recorded. Primary (driving) task performance included Time to Complete Task (pTCT), Deviation from Central Line (DCL) and steering reversal rate, while performance in the secondary task was measured through task completion times (sTCT) and errors (sTE) committed during the task. For all measures we conducted a pair sample two tailed T Test (with Bonferroni correction) comparing baseline with all three techniques (visual only, visual + CVAP, and visual + PSW).

3.2.1 Primary (driving) task performance results

For the primary (driving) task, Steering Reversal Rate (SRRs) was calculated as defined in SAE J2944 [28] using a 2.5° angle. The specific angle was chosen due to the smaller distance of each driving run (400 m). Nevertheless, SRRs between the three modalities did not show any statistically significant differences, we believe that this was due to the low sample (6 participants) size. On the other hand, if we compare the Time to Complete the Primary Task (pTCT), we measured a statistically significant difference between the visual only and CVAP (p = 0.035, 0.041, 0.038) as well as visual only and PSW (p = 0.01, 0.021, 0.042) modalities for all three tasks (Text, Menu & Gestures). There was also a trend indicating statistically significance for the “Text Entry” between CVAP and PSW, however, current sample size was too small to validate that (Fig. 7). This shows that both types of haptic feedback improved driver’s ability to complete the run faster, with PSW being better for Text Entry specifically.

Fig. 7
figure 7

Primary task performance (on the left) SRRs (in one round) and (on the right) pTCT (in s) for all nine conditions while the car was moving

Full size image

Contrary to how Standard Deviation of Lane Position (SDLP in SAE J2944) [28] is recorded where drivers are evaluated on their performance regarding lane keeping, we used a non-conventional measurement method, which approximated the deviation from a central line marked on the track. This was done to identify the drift of the vehicle from a central point (painted line) on the road. The thickness of this line was 6 inches, and it was aligned with markers within the car. The markers were attached to the center of the dashboard in such a way that the driver could exactly align them to the line on the track. A camera was used to measure the deviation of the markers from the white line on the track. Deviation of the width of the marker (10 mm) in either direction from the central line was classified as one deviation point. Using the total length of the drive, we took 102 sample points and assimilated the deviation into four intervals (0–2, 3–4, 5–6 and 7–8) and plotted these intervals (Fig. 8) with respect to the time participants spent in each interval (deviation from central line).

Fig. 8
figure 8

Illustration of deviation from central line (in % time)

Full size image

Baseline measurement showed that all participants remained within the first two intervals when no secondary task was allocated. However, participants deviated the most from the central position while conducting Text Entry tasks within visual only feedback modality. There was a statistically significant difference between the baseline and all three modalities as well as individual “Text Entry” tasks for Visual versus CVAP and visual versus PSW. Similarly, the same task yielded the highest amount of time (57%) spent at this deviation interval (5–6).

3.2.2 Secondary (IVIS) task performance results

Comparing primary (driving) task performance (PTP) with secondary (IVIS) task performance (STP) we performed a pair wise T Test (with BF correction) on each of the nine conditions when the car was stationary as well as when the car was moving to evaluate sTE and sTCT. In sTE, comparing “Text Entry” for moving and stationary conditions we see a statistically significant difference between all three conditions (visual only tasks, visual + CVAP tasks and visual + PSW task), where stationary condition yielded fewer errors. For the “Menu Selection” task we were only able to observe a statistically significant difference between the two visual conditions (stationary vs. moving) where stationary condition yielded fewer errors. Moreover, for the “Gesture” task we did not find any statistically significant difference between the moving and stationary car conditions. Comparing CVAP, PSW and visual only conditions with each other in the moving car we only see a trend (p = 0.073) which points towards possible statistically significant difference between visual only and PSW tasks for text entry (Fig. 9), if the participant size was increased. Moreover, the remaining UI tasks did not show any statistical significance between the CVAP, PSW and visual only modalities. Similarly, if we look at sTCT and compare and 9 conditions (3 task × 3 modalities) between moving and stationary environments, we observed a similar trend as with sTE. All 9 conditions demonstrated a statistically significant differences (p < 0.05) in the stationary vs moving pair wise T-test comparison. Furthermore, we see a trend between the “Text Entry” task for Visual only modality and PSW in the moving car, but the sample size is too small to validate the findings.

Fig. 9
figure 9

(On the left) Number of sTE and (on the right) sTCT (in s) for secondary task while the car was stationary and moving

Full size image

We see the same trend continue for task completion times; however, the difference was considerably less. This may be because there was a limited amount of time during the primary task (driving 400 m) in which the secondary task could be completed (Fig. 9). Therefore, if the task took longer the participants were not able to complete it, which was recorded as an error (depending on uncompleted sub-tasks). Nevertheless, in the TCT for secondary task we again recorded that Text Entry was the most time-consuming task. In fact, this was also the case when the car was stationary as well. Continuing the trend, we found that PSW + visual modality performed better in both stationary and moving car.

4 User study II: tangential and lateral actuations approach in a moving vehicle

4.1 Experimental setup

The second study was conducted to evaluate the tangential/lateral actuation approach on a touchscreen device, and it was carried out on a patch of highway within the participants’ own vehicle.

4.1.1 Experimental design and secondary (IVIS) task performance measurements

Similar to study I, we utilized three types of UI interaction; visual feedback, visual + tactile feedback using the LSE prototype and finally visual + tactile feedback using the CVAP device. The CVAP device was identical to the one employed in study one as it provided a baseline parameter for the other two types of interaction. Whereas, the LSE device was an attachment on top of the existing ExoPC tablet mounted on the central stack as seen in Fig. 10. Unlike the fixed setup utilized in Study I, where all participants operated a single vehicle (Volvo XC60) on the test track, in Study II the test setup had to be adapted to the participants’ (N = 10, 8 males, 2 females) own vehicles. The minimum criterion for enrolling participants in the study was that they were city locals, who had driven on that part of highway regularly and had at least 10,000 km driving experience and were not drivers by profession. As illustrated in the sketch in Fig. 10, both prototypes were added to the vehicles, in front of their respective IVI controls. Therefore, although the setups varied slightly in absolute placement, their relative positioning was somewhat similar. Furthermore, as both setups were placed in front of the vehicle’s IVI controls, each participant was familiar with the placement and distance from the driving position.

Fig. 10
figure 10

Attachment schematic (left) and usage of the LSE device (right) in participants own vehicles

Full size image

Moreover, both CVAP and LSE prototypes were supplemented with visual feedback. This included the touchscreen display on the ExoPC slate as well as custom designed HUD implementation, the latter of which was considered by all 10 participants as unnecessary and hence was switched off during the testing. Somewhat similar to Study I, participants were asked to carryout simple UI tasks related to the 3 use cases scenarios introduced in Study I, while driving on a straight road (~ 1400 m at 70 km/h) on the highway and in a stationary car. These tasks included “Text Entry”, “Layered Menu Selection” and “Onscreen four directional Gestures” (Fig. 6) users had approximately 26 s to complete each task.

4.1.2 Highway driving and primary (driving) task performance measurements

The driving task was a little more challenging as the participants were given the primary task of driving down a straight section of the highway on the outskirts of town. The drivers were asked to complete one task per round and were instructed to only utilize the specific modalities in question, to complete the afore-mentioned task. To further ensure the user study was as secure as possible the testing was conducted in the early part of the morning (between 5 and 6 am) on Sunday mornings in the month of July, which took 3 weekends to complete. Driving behavior, including Mean Driving Speed (MDS), steering wheel movements (steering reversals, SRR), and head movements were measured and referenced to baseline measurement. Touchscreen interaction tasks i.e. task/subtask durations; incorrect/incomplete selections and erroneous repetitive selection were also logged. The data was evaluated for both cases where the vehicle was stationary or moving and the participants were asked to fill out a questionnaire comparing the two types of haptic feedback. Furthermore, after each task (modality), the participants were asked to fill out a questionnaire rating the particular modality against each other. At the end of each user test a non-formal interview was conducted to evaluate the overall system comparing it with regular functionality of the user’s vehicle.

4.2 Results and analysis

As with Study I both driving performance (primary task) and UI interaction performance (secondary task) was recorded. Primary (driving) task performance (PTP) included MDS and SRRs (Fig. 11), while performance in the secondary task (STP) was measured through task completion times (sTCT) and errors committed during the task (Fig. 12). We conducted a pair wise T test comparison on all the data, comparing each modality (visual only, CVAP + visual and LSE + visual) and task (Text Entry, Menu Selection and Gestures) for both stationary and moving environments.

Fig. 11
figure 11

(On the left) Steering wheel reversal rate (SRRs per min) and (on the right) mean driving speed (MDS in km/h) across all nine conditions

Full size image
Fig. 12
figure 12

(On the left) Number of sTE and (on the right) sTCT for secondary task (in s) while the car was stationary and moving

Full size image

4.2.1 Primary (driving) task performance results

For PTP, SRRs was calculated as defined in SAE J2944 [28] using a 1.5° angle. The specific angle was chosen as it provided the necessary data during the pilot runs. The result of the SRRs data illustrate that there was a statistical difference between the SRRs for baseline condition and all other modality across all other tasks. Furthermore, there was also a statistical difference between visual only versus CVAP and LSE modalities for all modalities across all tasks illustrating the added advantage of haptic feedback to PTP (Fig. 11).

We also recorded a statistical difference between CVAP and LSE for all tasks (Text Entry, Menu Selection and Gesture) compared to visual only feedback. With reference to MDS, we recorded some interesting results. As expected, we saw a statistical difference Baseline condition and all other modalities across all the three tasks. Besides this, we recorded a statistical difference (of MDS) between visual and LSE modalities for the “Text Entry” task as well as CVAP and LSE modalities in the “Gesture” task. This result shows (Fig. 12) that participant focused more on the primary task while the performance in the secondary task suffered considerable. Even so, we still see a trend that LSE Text Entry influenced the primary (driving) task performance the least, making it a safer choose compared to the other modalities.

4.2.2 Secondary (IVIS) task performance results

For STP we evaluated sTE and sTCT. In sTE comparing stationary and moving environments we recorded a statistically significant difference between all tasks, showing participants performed poorly when the car was moving. We also found a statistically significant difference between modalities, where both CVAP and LSE where faster than visual only modalities for all task (Text Entry, Menu Selection as well as Gestures). We also establish a weak statistical difference between CVAP and LSE for “Text Entry” while the car was moving. Similarly, looking at sTCT we recorded a statistically significant difference (p < 0.05) between all tasks in their corresponding modalities across the two environments (moving vs stationary car). We also recorded a statistically significant difference between visual modality versus CVAP and LSE, where visual only was statistically slower for all three tasks. Furthermore, we also observed a statistically significant difference between CVAP versus LSE for all the three tasks, where participants were statistically faster in completing these secondary tasks with LSE (Fig. 12). This clearly shows that LSE improved secondary (IVIS) task performance while the car was moving.

5 Discussion

Driver distraction is a widespread, yet complex issue [2, 29, 30]. The diverting of attention from activities critical for safe driving has been proven to cause accidents and loss of life. Though, existing research [15,16,17] explored how secondary task information should be communicated to the driver with minimum distraction, most of the approaches used have primarily been focused on adding additional redundant feedback cues for audio, visual or haptics modalities. Adding more complex or redundant cues has the potential of further distracting the driver [14], especially when the primary task of driving is more complex. In this research we approach the problem by complimenting exiting interaction techniques and providing simplified natural haptic signals. Our haptic cues do not require complex decoding to identify alerts or events and the devices utilizing these cues can easily be developed and utilized in most vehicles. Moreover, we focus on creating haptic signals using techniques that reduce possible environmental attenuation. We try to achieve this by moving away from conventional entire device vibration and exploring pneumatic actuation or tangential vibrotactile actuation. Our approach also mitigates that some secondary driving tasks can be time specific in different scenarios (i.e. navigation) and, hence, may be prioritized over other secondary or even some primary driving tasks. Therefore, an IVI setup needs to be dynamic w.r.t both output and input for secondary tasks (an ideal example is the Tesla Model 3 Center Stack, which can dynamically be used to perform primary and secondary driving tasks).

In the first user study we tested a novel pneumatic actuation device that created air pulses on the touchscreen. Compared to traditional vibrotactile cues used to create haptic confirmation cues during touchscreen interaction, pneumatic actuation has rarely been used in a car. Whereas vibrotactile actuation can get drowned-out is a noisy environment, such as a vibrating car, pneumatic actuation can be much more efficient and usable to rely important secondary task information. Our research proves this hypothesis by showing participant who were professional drivers, were more comfortable and performed better in both primary and secondary task when using the prototype PSW device, as compared to conventional, yet more powerful vibrotactile actuation device (CVAP). Both primary (driving) and secondary (IVIS) task performance (task completion times and task errors) was overall better compared to no haptic feedback, but trends indicated that at least primary task performance remained more consistent using the PSW device (see Figs. 7, 8, 9). If we try to compare PSW with current haptic actuation techniques, we can see that most commercial IVI systems do not have any haptic feedback for touchscreen interaction. Though, the research done by Richter et al. [19], and Serafin et al. [18] utilized vibrotactile devices, the efficiency of such actuation can vary considerable depending on environmental noise and type of interaction (i.e. contact time with touchscreen, skin sensitivity and use of gloves etc.). By using pneumatic actuation signals it is possible to reduce most of these issues bypassing environmental vibration noise and eliminating the need to direct contact to relay actuation signals. Nevertheless, follow-up studies are needed to show the exact measure of improvement each haptic device may have over the other in various scenarios, study I revealed, through a statistically significant difference that visual + PSW feedback can improve primary (driving) task performance (task completion time) and can be more useful compared to visual only feedback for secondary tasks in a moving car.

Results of User Study 1 showed that custom designed vibrotactile actuation may be used in a noisy environment if the noise threshold can be mitigated by varying the frequency bandwidth of actuation. As it would have been irresponsible to utilize untested actuation technology for IVIS tasks in a public driving environment, Study 1 was conducted in a closed environment (Nokia Tires Track). However, as the results for Study 1 suggested that both PSW and CVAP were consistently perceivable is moving car we transitioned the testing environment to a more nature setting (on the highway in the user’s own vehicles).In Study II we used the novel linear/tangential actuation device (LSE) to create vibrotactile actuation outside the scope of the environmental noise threshold. We also used the vibrotactile actuation device (CVAP) to understand how both devices impact on driver performance in a moving car. As hypothesized, the linear actuation device (LSE) operated mostly outside the environmental noise spectrum of the vehicle (vertical road vibration noise). Authors recorded improved primary (driving) task performance for the conditions where the LSE device was used as the feedback mechanism over visual only and CVAP feedback. Steering reversal rates (a measure of driving performance) as well as Mean Driving Speed (MDS) both showed lower additional workload on the driver as specified by existing standards [1, 28] while performing text entry tasks for LSE and CVAP compared visual only feedback. For secondary (IVIS) task performance we also recorded an improvement in Text Entry, by measuring a statistically significant difference between TCT of LSE and CVAP devices. However, results from Total Task Errors Task Completion Times for secondary task revealed no difference between LSE and CVAP. Nonetheless, when we compare LSE to conventional vibrotactile actuation for touchscreen devices the advantages can be obvious. Unlike CVAP, most vibrotactile actuation devices are not powered by dedicated high performance actuation motors/transduces that is why none of the major automobile manufactures provide reliable touchscreen vibrotactile actuation, to date. Furthermore, in more noisy environments (rough terrain, higher speeds), the tactile sense of conventional vibrotactile actuation (actuation provided in the same orthogonal axis as the main component of environmental noise) may either require a lot of energy or may not be perceived at all. Although further research may be required to specify the magnitude of improvement LSE may bring, results from Study II indicated the overall usefulness of tangential actuation in car environment.

6 Conclusion

A distracted driver is an inefficient driver, irrespective of the type or modality of distraction [31]. Our studies show that visual distraction can be reduced by implementing driver centric multimodal interaction systems for both primary and secondary tasks. Car manufactures and IVI systems that try to limit drivers’ ability to perform secondary tasks are often circumvented by users who in-turn adopt far riskier modes of interaction [30]. The results of this research indicate that some tasks can be extremely difficult to perform within the driving environment by simply using visual interaction, depending on driver’s ability to multitask [32]. These tasks include ‘Text Entry’ as well as ‘Multi-Layered Menu Selection’ on a touchscreen-based device as suggested by Kujala and Grahn [33]. As these tasks cannot be completely removed from the systems, the interaction methods need to be carefully developed to reduce driver’s visual and cognitive distraction.

The results of Studies I and II show that non-traditional actuation techniques (i.e. pneumatic or tangential actuation) can also play an important role in complementing such visually intensive touchscreen-based tasks by providing the necessary haptic feedback. Moreover, touchscreen haptic implementation techniques such as pneumatic feedback, as well as tangential vibration can be designed to be more sensible than conventional z-axis vibration. Additionally, this opens up the technology debate of how to create haptic feedback in a vibration-noisy environment, which is the main purpose of the research. Up till now vibration-based actuation has been the most popular method of creating haptic feedback, however, our research shows that other techniques can also be useful, especially within the automotive context. For this reason, it is important to explore different technologies and implementation new techniques for creating haptic feedback.

Moreover, using sensible and natural haptic cues for confirmation-based interaction (subtasks that include item selection, item differentiation and traversal), may further reduce the visual distraction of the driver. Using this method, it is possible to have the drivers visually identify onscreen targets, navigate and select them by getting confirmation through haptic cues supplemented by audio-based feedback; thereby reducing the necessary visual screen time for the driver. This method of breaking down a complex and potentially distractive task into sub-tasks that can be individually supplemented by various modalities ensures that the driver performs the potentially difficult task without overloading any one interaction modality. Therefore, drivers only need visual interaction for a fraction of the entire task, freeing up important resources for driving. In our future research we plan to test this hypothesis further and evaluate participants’ subjective understanding of the hypothesis and how it can be utilized within autonomous driving scenarios.