Touch, Movement and Vibration: User Perception of Vibrotactile Feedback for Touch and Mid-Air Gestures

Abstract

Designing appropriate feedback for gesture interfaces is an important aspect of user experience and performance. We conduct the first investigation of users’ perceptions of vibrotactile stimuli during touch and mid-air gesture input for smart devices. Furthermore, we explore perception of feedback that is decoupled from the smart device and delivered outside its operating range by an accessory wearable, i.e., feedback delivered at arm-level. Results show user perception of vibrotactile stimuli up to 80 % accurate, which we use to recommend guidelines for practitioners to design new vibrotactile feedback techniques for smart devices.

1 Introduction

Gesture interfaces are today’s standard for interacting with smart mobile devices in the form of touch and accelerated motion and, recently, mid-air gestures. Related research has shown that delivering appropriate feedback to users during gesture input can help with gesture training [7], increase recognition accuracy [8], and improve overall user experience [5]. However, providing feedback beyond visual and audio cues is still subject of technical development and investigation of user perception of vibrotactile stimuli [1, 9, 11]. Nevertheless, prior art has reported many advantages of vibrotactile stimuli for user feedback, such as more intuitive indication of body part positions than delivered by visual or audio feedback [11], reduced errors and improved learning rate for motor training tasks [9], and increased accuracy for mid-air finger gesture articulation [1]. However, no work has compared users’ perceptions of vibrotactile stimuli across the various gesture input modalities enabled by today’s smart devices, such as touch, accelerated hand motion, and mid-air finger and arm gestures. Furthermore, at the advent of new miniaturized wearables (e.g., smart watches and interactive jewelry) that enable new interactive contexts, vibrotactile stimuli will likely play an important role for delivering user feedback. These new devices are worn on various body parts and communicate with the user’s smartphone, which is emerging as the central unit of a distributed on-body network of devices [2]. Although prior art has investigated vibrotactile feedback delivered on the smartphone itself [7], no work has examined users’ perceptions of feedback decoupled from the smartphone and delivered on the body by an accessory wearable during gesture input with the primary device.

The contributions of this work are: (1) we conduct the first exploration of users’ perceptions of vibrotactile feedback for touch, accelerated motion, and mid-air gestures performed on smartphone devices with decoupled vibrotactile stimuli delivered outside the operating range of the smart device, i.e., feedback delivered at arm-level; (2) we report good levels of user perception of vibrotactile stimuli up to 80 % accuracy with minimal training; (3) we recommend guidelines for practitioners to design vibrotactile patterns for similar multi-device prototypes. We hope that this first exploration into user perception of vibrotactile feedback during gesture input will inspire the community to further investigate cross-device feedback for gesture-controlled multi-device prototypes and to promote new approaches for gesture articulation guidance.

2 Experiment

We designed an experiment to understand users’ perceptions of vibrotactile feedback at arm level for various interaction contexts involving touch and mid-air gestures.

Participants.

Eleven participants volunteered for the experiment (5 females). Participants were young adults with ages between 21 and 25 years old (mean age 22.7 years, SD 1.04 years). No participant had previous experience with vibrotactile feedback.

Apparatus.

For the purpose of this study, we developed a wearable vibrotactile device with actuators powered by a control unit attached to the arm near the wrist. The unit was implemented around a Spark Core v1.0 board driving two Precision Microdrives vibration motors (of type 304-108) small in size (4 mm diameter) with short rise and stop times (50 ms and 76 ms, respectively), and a vibration speed of 10,000 rpm, a design that we adopted to optimize mobility and perception at skin-level receptors [3]. The two actuators were encased in an ABS housing to prevent direct contact between their rotational mass and the skin, and a spring was added to decouple the housing from vibrations to maximize the vibrating effect on the surface of the skin. The vibrotactile device weights approximately 6 g/55 g (actuator/control unit) and can be worn effortlessly on the arm at any location (Fig. 1 shows our experimental setup). Feedback delivered by the actuators was generated by a custom software application implementing our experiment design that ran on a smartphone (HTC One S), which communicated with the control unit via a wireless connection.

Fig. 1.

Left: Apparatus employed during our experiment composed of a control unit driving two vibrotactile actuators controlled by a smartphone via a wireless connection. The two actuators were placed around the wrist at a distance of 10 cm. Right: Close-up of one actuator.

Design.

Our experiment was a within-subject design, with the following factors:

• (1) Pattern, ordinal variable, with $6$ values: no-feedback (control condition – participants did not receive any vibrotactile feedback, but were still asked afterward what they felt), short-pulse (a continuous, 250 ms constant-amplitude pulse), long-pulse (continuous, constant-amplitude pulse of 750 ms), simple-pattern (sequence of two short and long pulses with 250 ms pauses in-between), complex-pattern (sequence of 3 pulses – short, long, and short – with 250 ms pauses in-between), and line (amplitude decreases for one actuator as it increases for the other, giving the sensation of a moving point, total duration of 1750 ms).

• (2) Intensity, ordinal variable, with 3 values: low, medium, and high. The high intensity condition corresponds to typical operating amplitude (0.85 G) and voltage (3 V) for our actuators, which was found close to the upper threshold of the comfortable stimuli range during pre-tests. The low intensity condition corresponds to 1/3 of the high intensity (0.2 G at 0.9 V), which was found easily detectable during pre-tests. The medium intensity level was selected between high and low at an amplitude of 0.55 G and 2 V voltage.

• (3) Hand-Mode, nominal variable, with 5 conditions: rest (hand is resting on the table or the leg), hold (hand holds the smartphone), hold-and-move (hand performs a touch stroke gesture on the smartphone), finger-motion (fingers move in mid-air), and arm-motion (arm moves in mid-air at low velocity).

• (4) To prevent participants from becoming familiar with the gestures they performed in the hold-and-move and arm-motion conditions, we also varied Gesture type, nominal variable, with 3 conditions: circle, swipe-left, and swipe-right.

Task.

Participants sat in a comfortable chair, watching instructions delivered on a large display about the hand states and movements to perform according to the Hand-Mode and Gesture conditions. During this time, a vibrotactile stimuli was delivered to participants’ arms according to the Pattern and Intensity experimental conditions. In total, there were 90 trials (= 6 Pattern × 3 Intensity × 5 Hand-Mode) randomized across participants. After each trial, participants were asked to recognize both Pattern and Intensity of the applied vibrotactile stimuli. Before the experiment, participants were familiarized with our vibrotactile prototype, and they were presented with the patterns and intensities for several times until they confirmed good understanding. During the experiment, we had participants wearing headsets and listening to music to prevent them to hear the actuators, which would have affected positively participants’ capability to discern patterns and intensities by relying on audio information. The experiment took about 35 min per participant to complete.

Measures.

We are interested for this study in users’ capabilities to recognize vibrotactile patterns applied to arm level, as well as the intensities of these patterns and, therefore, we measure user performance at three distinct levels:

• (1) Pattern-Accuracy represents the percentage of correctly identified vibrotactile patterns, regardless of the intensity at which they were applied.

• (2) Intensity-Accuracy represents the percentage of correctly identified intensities of the vibrotactile stimuli, regardless of the actual pattern being applied.

• (3) Overall-Accuracy represents the percentage of correctly identified both vibrotactile patterns and intensities at which patterns were applied.

Hypotheses.

We set the following hypotheses to verify in our experiment:

• (H1) The Pattern type of the applied vibrotactile stimuli will affect participants’ Overall- and Pattern-Accuracy.

• (H2) The Intensity level of the applied vibrotactile stimuli will affect participants’ Overall-Accuracy and Intensity-Accuracy.

• (H3) The Pattern type will not affect participants’ Intensity-Accuracy, nor will the Intensity level affect participants’ Pattern-Accuracy.

• (H4) The Hand-Mode will affect participants’ recognition accuracy of vibrotactile stimuli for all accuracy measures.

3 Results

Overall, our participants were successful at recognizing combined vibrotactile patterns and their intensities in only 55.4 % of all trials (Overall-Accuracy). Although the overall performance is modest, Pattern-Accuracy and Intensity-Accuracy were significantly higher than the overall performance, as indicated by McNemar tests (79.7 % and 55.4 %, respectively, $\chi_{{\left( { 1 , {\text{N = 990}}} \right)}}^{2} = 2 3 9. 0 0 4$, $p < . 0 0 1$ and 67.8 % and 55.4 %, $\chi_{{\left( { 1 , {\text{N = 990}}} \right)}}^{2} { = 121} . 0 0 8 , p { < } . 0 0 1$, respectively). These results show that our participants managed to recognize either the vibrotactile pattern or its intensity with acceptable rates, but their overall judgment of the multiple characteristics of the vibrotactile stimuli failed significantly more often. To understand more, we performed additional tests for each experimental factor. Figure 2 illustrates participants’ recognition accuracy computed for the Pattern, Intensity, and Gesture conditions.

Fig. 2.

Average user Accuracy of recognizing vibrotactile feedback measured Overall (left), per Pattern (top), and for each Intensity level (right). Error bars show 95 % CIs.

Cochran’s Q tests showed a significant effect of Pattern on Overall-Accuracy ($\chi_{{\left( { 5 , {\text{N = 165}}} \right)}}^{2} = 2 1 1. 3 4 5 , p < . 0 0 1$), Pattern-Accuracy ($\chi_{{\left( { 5 , {\text{N = 165}}} \right)}}^{2} = 2 2 6. 0 6 5 , p < . 0 0 1$) and Intensity-Accuracy ($\chi_{{\left( { 5 , {\text{N = 165}}} \right)}}^{2} = 9 2. 9 6 2 , p{ < } . 0 0 1$), see Fig. 2a, d, and g. Participants had no problems detecting the no-feedback condition (98.2 %), which confirms an appropriate level for our low intensity, and also alleviates concerns regarding a potential vibrotactile after-effect. The simple and complex patterns were recognized with 95.2 % and 92.1 % Pattern-Accuracy (n.s. difference), and were followed by short-pulse with 86.1 %, see Fig. 2b. Confusion matrix analysis showed that long-pulses were often mistaken for short-pulses (18.2 % of the time) and as lines (22.4 %), while lines were repeatedly perceived as long-pulses (34.5 %), see Fig. 3, left. When we removed the no-feedback condition from the analysis, no significant effect of Pattern was present any longer on Intensity-Accuracy ($\chi_{{\left( { 4 , {\text{N = 165}}} \right)}}^{2} { = 1} . 5 0 7 , {\text{n}} . {\text{s}} .$), but the effect was still there for the other two measures. These results validate hypothesis H1 and the first part of H3.

Fig. 3.

Confusion matrices for the Pattern and Intensity conditions. Cell values show percentages of associations between actual and perceived vibrotactile feedback patterns.

Cochran’s Q tests showed a significant effect of Intensity on Overall-Accuracy ($\chi_{{\left( { 2 , {\text{N = 330}}} \right)}}^{2} { = 11} . 3 8 5 , p{ < } . 0 0 5$) and Intensity-Accuracy ($\chi_{{\left( { 2 , {\text{N = 330}}} \right)}}^{2} = 9. 0 2 5 , p{ < } . 0 5$), but no effect on Pattern-Accuracy ($\chi_{{\left( { 2 , {\text{N = 330}}} \right)}}^{2} = 2. 7 7 4 , {\text{n}} . {\text{s}} .$), see Fig. 2b, e, and h. These results validate hypothesis H2 and the second part of H3. Follow-up post hoc McNemar tests (Bonferroni corrected at the $p = . 0 5 / 3 = . 0 1 7$) revealed a significant difference only between the Intensity-Accuracy of medium and high ($\chi_{{\left( { 1 , {\text{N = 330}}} \right)}}^{2} { = 7} . 3 5 8 , p{ < } . 0 1$, $\phi { = } . 1 5$), see Fig. 2h. Confusion matrix analysis revealed that low intensities were often perceived as medium (36.7 % of the time), and high as medium (28.2 %), see Fig. 3, right. At the same time, medium intensities were perceived more as being high (33.0 %) rather than low (14.2 %) by our participants.

We found a significant effect of Gesture on Pattern-Accuracy ($\chi_{{\left( { 2 , {\text{N = 195}}} \right)}}^{2} = 7. 0 5 3 , p{ < } . 0 5$), but not on Intensity ($\chi_{{\left( { 2 , {\text{N = 195}}} \right)}}^{2} = 0. 0 1 9 , {\text{n}} . {\text{s}} .$) nor on Overall-Accuracy ($\chi_{{\left( { 2 , {\text{N = 195}}} \right)}}^{2} = 1. 8 4 2 , {\text{n}} . {\text{s}} .$), see Fig. 2c, f, and i. Post hoc McNemar tests (corrected at $p = . 0 5 / 3 = . 0 1 7$) revealed significant differences only between circle and swipe-left (84.3 % and 75.5 %, $\chi_{{\left( { 1 , {\text{N = 195}}} \right)}}^{2} = 5. 9 2 1 ,p { < } . 0 1 7$, $\phi { = } . 1 7$).

We found no significant effect of Hand-Mode on participants’ Accuracy, neither Overall ($\chi_{{\left( { 4 , {\text{N = 198}}} \right)}}^{2} { = 5} . 0 9 5 , {\text{n}} . {\text{s}} .$), nor at the Pattern ($\chi_{{\left( { 4 , {\text{N = 198}}} \right)}}^{2} { = 2} . 4 0 8 , {\text{n}} . {\text{s}} .$) and Intensity levels ($\chi_{{\left( { 4 , {\text{N = 198}}} \right)}}^{2} { = 1} . 9 8 1 , {\text{n}} . {\text{s}} .$). This result invalidates hypothesis H4, and informs us that small-velocity movements performed on or above the smartphone are not likely to influence users’ accuracy of interpreting vibrotactile feedback decoupled at arm-level. However, it is possible that larger amplitude or faster movements (such as those performed during physically-demanding video games) might lead to different results but, for our specific context of mobile interaction, we have not detected any such effects. (Consequently, we have not illustrated Hand-Mode results in Fig. 2). There were no significant differences between the performance of women and men, as shown by Mann-Whitney U tests (all $p{ > } . 0 5 , {\text{n}} . {\text{s}} .$).

4 Design Guidelines for Vibrotactile Feedback

Our data so far enables us to recommend a number of 4 easy-to-apply guidelines for designing vibrotactile feedback to be delivered by an accessory wearable device at arm level during touch and mid-air gesture input with the primary device:

• (1) Design vibrotactile patterns that vary significantly in their time duration. In our experiment, we found that Pattern type affects significantly users’ recognition accuracy (hypothesis H1). However, the most elementary and simple patterns were not always the most easily recognized ones. For instance, our participants achieved a modest accuracy for long-pulse rather than for our complex pattern design. Our data indicates that it is difficult to recognize patterns’ duration accurately even when they vary by as much as 500 ms, which is supported by the frequent unidirectional confusion between long and short pulses (Fig. 3, left), even when their time duration varied by factor of 3 (i.e., 750 ms vs. 250 ms). Furthermore, long-pulse and line were often confused by our participants, despite that line took twice as much time as long-pulse (i.e., 1750 ms vs. 750 ms). Consequently, we advise practitioners to employ time duration with caution by ensuring enough difference in the duration of vibrotactile stimuli to prevent misrecognition. Our results above suggest rough guidelines of what this difference might be, but we leave this design decision for practitioners, which will ultimately adapt our results to their specific application context.

• (2) Exploit pauses in the design of vibrotactile patterns. The remarkable accuracy achieved by the complex pattern suggests that series of discrete pulses are easier to recognize by users when they appear in conjunction with simple, continuous patters, as users can exploit the pauses between pulses to validate and inform their guesses about the applied vibrotactile pattern. Designs with discrete pulses likely determined complex and line to be discriminated accurately, even if they took the same amount of time to complete (1750 ms both).

• (3) Limit the intensity levels of vibrotactile patterns to at most two. According to van Erp [3], four is the maximum number of levels that should be designed for vibrotactile feedback when using intensity to encode information in the interval ranging between detection and pain. Our experiment shows that three intensity levels were difficult to discriminate (i.e., 67.8 % average accuracy, see Fig. 3, right). Considering that we did not calibrate intensity per participant, nor did we evaluate the perception of intensity alone (i.e., an easier task), this result is not surprising. However, the low and high intensities were rarely mistaken one for the other (there was less than 8 % error rate in each direction). Consequently, we recommend the use of two intensity levels for designs of vibrotactile patterns that are easily recognizable with minimal training conditions. Also, we observed a tendency in our participants to overestimate intensity, which might be an indicator that our medium and high intensities might have been too intense overall. Some participants even complained that high was close to uncomfortable and its intensity should have been reduced, which recommends calibration of intensity levels per participant during training. Our data informs us to recommend normalized vibration amplitudes below 0.85G for devices, which don’t have much mass themselves and are attached tightly to the lower arm/wrist.

• (4) Exploit both duration and pattern type for vibrotactile feedback. Validation of hypothesis H3 shows that feedback can be successfully encoded as both intensity and pattern type, without significantly affecting the users’ accuracy of perception of either. For our applied patterns, no temporal enhancement [3] has been observed, which could have affected the perceived intensities for the simple and complex patterns. Practitioners are encouraged to explore the design space of Pattern × Intensity for feedback options combining both these characteristics.

5 Conclusion

We examined in this work users’ perceptions of vibrotactile feedback delivered at arm-level during touch and mid-air gesture input. As more and more miniaturized wearable devices and sensors will become available in conjunction with the smartphone, such investigations are mandatory for the community to build up the required knowledge to design proper feedback during gesture input with these smart devices. It is our hope that this first investigation on decoupled vibrotactile feedback will inspire the community to examine more such scenarios for gesture interaction with smart devices.