Keywords

1 Introduction

Gaze and mutual gaze are important in the development of trust and deeper relationships [1]. A common face-to-face conversation can contain a wealth of gazes and mutual gazes, which the sighed people take for granted in their daily routines. For example, a sighted speaker consciously or unconsciously uses gaze or eye contact to communicate with the conversation partner. Through the conversation partner’s eyes, she can sense interest, engagement, happiness etc. Gaze signals are frequently used by the sighted in social interactions as visual cues. However, these signals and cues are inaccessible for the blind and hardly accessible for low-vision people. White et al. [2] interviewed 8 visually impaired expert users and a social communicative problem was indicated. The problem was that it was often difficult for them to meet people because they could not see and make eye contacts with the sighted people. McNeill also emphasizes that nonverbal cues such as gazes are integral to a conversation and that ignoring them means ignoring part of the conversation [3]. This becomes more serious when one of them is disabled (e.g. people with visual disability) and the others are not trained to interact with the specific disabled population [4].

In this paper, Tactile Band, a wearable device, was developed to help the blind person feel attention (gaze signals) from the sighted conversation partner in face-to-face communication. Tactile Band tries to map gazes to tactile signals, and let the blind person perceive them in real-time. 30 volunteers (15 blindfolded participants) were invited to evaluate the prototype in the preliminary experiment.

2 Related Work

One related research area is about gaze behaviors. A number of studies have investigated the importance of gaze behaviors of sighted people in social occasions. Argyle studied that in dyadic (two-person) conversations, about 75 % of the time people are listening coincides with gazing at the speaker [5]. Kendon suggested that seeking or avoiding looking at the face of the conversation partner has important functions in dyadic conversations, to regulate the flow of conversation and to communicate emotions and relationships [6]. Vertegaal et al. used an eye tracker to measure gazes at the faces of conversational partners during four-person conversations. The result indicated that gaze was an excellent predictor of attention in conversations [7].

The other relevant area is to make visual cues accessible to the blind people in face-to-face communication. Krishna et al. developed a wearable Social Interaction Assistant to help the blind and visually impaired people to know who was approaching and allowed them to choose whether to initiate a conversation [8]. ur Rehman et al. developed a haptic chair for providing facial expression information to the blind people. Nine vibrators were located in the back of the chair which indicated some specific facial features [9]. Krishna et al. also provided an assistive technology for accessing facial expressions of interaction partners. His research prototype was a vibrotactile glove worn by the blind person and it could convey the conversation partner’s seven facial expressions such as happy, sad and surprise with different vibration patterns [10]. Finocchietti et al. proposed ABBI, an audio bracelet for the blind person’s social interaction, aiming to rehabilitate spatial cognition on where and how the body was moving [11]. These examples were all about the assistive devices for accessing nonverbal signals such as facial expressions and body gestures. However, none of them was about providing blind people with access to gaze signals that are important in nonverbal communication.

In previous user study, we proposed a concept to help the blind people access and react to gaze signals in face-to-face communication. 20 blind and low-vision participants were interviewed to evaluate the features of this concept for their usefulness, efficiency and interest. We reported the evaluation in a conference paper [12]. Based on the evaluation, the concept is further developed into a prototype, namely Tactile Band, to aim at testing the hypothesis that tactile feedback can enable the blind person to feel attention (gaze signals) from the sighted, enhancing the level of engagement in face-to-face communication.

3 Experiment

3.1 Tactile Band Design

The Tactile Band was designed to exam the hypothesis that by enabling the blind person to feel attention (gaze signals) from the sighted, the tactile feedback can enhance the level of engagement in face-to-face communication. In our concept, SMI eye trackerFootnote 1, worn by the sighted, can detect her gazes on the blind person. Gaze signals are mapped to vibration signals of an actuator embedded in the Tactile Band, worn by the blind person on her forehead. The blind person perceives a slight vibration from the Tactile Band as a signal of the sighted looking at her face. Two vibration patterns are used to map basic gaze behaviors: the glance and the fixation. In the glance pattern, the sighted has a quick glance at the blind person’s face to instantly trigger a slight vibration of the Tactile Band. If the sighted shortly looks away, the vibration stops in real-time. In the fixation pattern, the sighted gazes at the blind person’s face for a while and looks away. In this process, the first fixation to the blind person’s face triggers a slight vibration of the Tactile Band and lets her know the sighted is looking at her. If the sighted is still gazing at the blind person’s face, she can feel a slight vibration with an interval in a loop until the sighted looked away. The reason of using intervals is to avoid continuous vibrations, which possibly becomes annoying.

A within-subject design was conducted and it included one independent variable with three levels (no Tactile Band, Tactile Band without vibrations and Tactile Band with vibrations) and one dependent variable (engagement in a conversation). In our preliminary experiment, blindfolded but sighted (hereafter blindfolded) participants were invited to the experiment as an alternative for the target blind users [13]. The level of engagement in a conversation was measured using questionnaires with two subjective measures: relationship quality and partner closeness. Besides the questionnaires, gaze information was collected through SMI eye tracker to help measure the sighted participants’ engagement in conversations. A qualitative analysis on the results from a post-experimental questionnaire (five open questions) was performed to investigate participants’ subjective attitudes towards the Tactile Band and to collect suggestions for further improvements.

3.2 Wizard-of-Oz Setup

The Tactile Band system used Wizard-of-Oz (WOZ) (Fig. 1) to simulate the final system’s behavior as closely as possible: a human “Wizard” simulated the system’s response in real-time, interacting with the users just like the envisioned system [14]. In the Wizard-of-Oz set up, two participants (A1: the blindfolded; A2: the sighted) had a conversation in Room 1, while a wizard situated in Room 2. A2 wore the SMI glasses – a wearable eye tracker C1. A1wore the Tactile Band on her forehead. The wizard observed the real-time eye tracking video from C1 and controlled vibration actuator of the Tactile Band accordingly. The video with gaze information (recorded by the eye tracker C1) was used for the attention analysis after the experiment. Camera C2 captured the entire scene.

Fig. 1.
figure 1

Wizard-of-Oz environment

Full size image

In the Tactile Band system, Eye Tracking Glasses (ETG) connected to an ETG-Laptop and detected gaze signals of the sighted participant in real-time. Wizard observed the real-time gaze video from iView ETG 2.0 (the controller and eye tracking software) installed on an ETG-Laptop. If the gaze hit the facial region of the blindfolded participant, a slight vibration was triggered by the wizard. If the gaze was still in the facial region, slight vibrations with equal intervals were triggered by the wizard. The vibration stopped when gaze was out of the facial region. The moment of each vibration signal was recorded in the wizard computer (accurate to millisecond). Figure 2 shows an overview of the Tactile Band system.

Fig. 2.
figure 2

Overview of the Tactile Band system

Full size image

3.3 Participants

The participants were thirty student volunteers from Eindhoven University of Technology (11 females, Mage = 29.73, SD = 5.69; 19 males, Mage = 28.16, SD = 2.17) with ages ranging from 21 to 42. They were divided into pairs to have dyadic conversations and one of each pair was blindfolded. All the participants had normal or corrected-to-normal vision and were allowed to wear their contact lens, but not allowed to wear glasses due to the inconvenience to wear the eye tracker and the blindfold. We tried to recruit strangers as much as possible as a control for participants’ previous familiarity. The participants were paired randomly: 17 participants never met before; 9 participants knew each other but had rarely or never had conversations; only 4 participants knew each other and had sometimes conversations.

3.4 Procedure

Two paired participants read informed consents and signed names in the office before being led to the lab. In the informed consent, we told the blindfolded participants: “You are wearing a band that vibrates when your conversation partner looks at your face in the experiment.” After completing informed consents, one participant was blindfolded and we took both participants from the office to the nearby lab. In the lab, the blindfolded participant was taken to sit in the chair in Room 1, where we played some light music for relaxing. In Room 2, we helped the sighted participant wear the eye tracker and did the three-point calibration to accurately catch her eye movements. After calibration, the sighted participant went to sit in the other chair in Room 1, facing the blindfolded participant. After ensuring the blindfolded participant’ comfort to the blindness, we turned off the music. Then we randomly picked one topic in fourteen daily topics from IELTS oral exams [15]. These topics were all about daily lives and easier for the participants to start such as the item “Describe a job you have done”. Both participants were asked to share ideas about the topic. After that, the door was closed between Room 1 and Room 2 and the conversation started. After the average 10-min conversation, the sighted participant completed a post-experimental questionnaire in Room 1 and the blindfolded participant was taken to Room 2 to finish it with the blindfold off. Due to the pictorial measurements used in this process, the blindfolded participant was asked to take off the blindfold to complete the questionnaire. When both participants completed the questionnaires, we blindfolded the participant again and took her back to Room 1. Three conversations were taken place under the following experimental conditions for the blindfolded (I. no Tactile Band; J. Tactile Band without vibrations; K. Tactile Band with vibrations) with counter balancing to avoid carry-over effects. Each conversation lasted around 10 min and after each conversation, participants were asked to answer a post-experimental questionnaire. After three conversations and post-experimental questionnaires, we did a short interview to collect the blindfolded participant’s comments and suggestions towards the Tactile Band. Each conversation was video-taped and the short interview was audio-tapped. The overall experiment lasted approximately 90–120 min.

3.5 Measurements

We measured the level of engagement in the face-to-face conversation with two subjective measures: relationship quality (IMI: Intrinsic Motivation Inventory questionnaire) [16] and partner closeness (IOS: The Inclusion of Other in the Self Scale) [17]. IMI included 45 items, assigned to 7 subscales. We were particularly interested in participants’ mutual relationship in conversations. Therefore, relatedness subscale of IMI was used. It has 8 items, such as “It is likely that this person and I could become friends if we interacted a lot”. IOS Scale was used to measure the closeness. It included seven increasingly overlapping circle pairs, which could indicate the distance of the relationship between themselves and their conversation partners. We also collected qualitative feedbacks from an open questionnaire and the interview. After three tests, we left the blindfolded participant alone to complete the open questionnaire with five questions included the item: “Do you have some suggestions for improving the Tactile Band?” After finishing these questions, we did a short interview (average around five minutes) to confirm the answers.

Gaze tracking data from the sighted participants in the tests were recorded and analyzed using the software BeGaze version 3.5, installed in the ETG-Laptop. Results were calculated over 45 conversations and last five minutes of each conversation video were analyzed. The facial region of the blindfolded participant was chosen as the area of interest (AOI) for measuring the fixation duration. This eye metric was selected based on the relevant literature on the attention analysis with eye movements [7, 18]. The AOI can synchronously match the dynamic facial region by setting key frames. When the sighted participant’s gaze hit the dynamic AOI, gaze was registered for the attention analysis. Due to the frequent and strong head movements in tests, the dynamic AOI was not able to accurately catch facial regions of some blindfolded participants in videos. So ten sighed participants’ eye-movement videos were used in the attention analysis and five were excluded. The corresponding eye metrics inside the AOI area were calculated using BeGaze and exported for analysis.

4 Results

4.1 Self-reports

Below we report both quantitative and qualitative results from the experiment.

Quantitative Results.

We used SPSS for the data analysis. Blindfolded participants in 3 pairs out of 15 could not consciously sense vibration signals during the experiment, but they were possibly influenced by vibration signals unconsciously. Therefore, data from these blindfolded participants were not removed from the datasheet. The conversation quality was analyzed using RM-ANOVA with relationship quality and partner closeness as within-subject factors and the role (the blindfolded and the sighted) as a between-subject factor. Table 1 presents mean and standard deviation of relatedness and partner closeness across three conditions. Before running RM-ANOVA, we checked the data for violations of parametric analysis: the sphericity assumption was tested using Mauchly’s test. There were no significant effects of relatedness F (2, 56) = 0.64, p = 0.53, and partner closeness F (2, 56) = 0.20, p = 0.82 in three conditions.

Table 1. Mean and standard deviation of relatedness and partner closeness across three experimental conditions: (I) no Tactile Band; (J) Tactile Band with no vibration; (K) Tactile Band with vibration.
Full size table

Since the blindfolded participants wore the Tactile Band, we analyzed them in three conditions separately. The datasheet was split into two groups: the blindfolded and the sighted. There were no significant effects for blindfolded participants in relatedness F (2, 28) = 0.13, p = 0.88, and partner closeness F (2, 28) = 0.04, p = 0.96 in all conditions. There were also no significant results for sighted participants of relatedness and partner closeness in three conditions (p > 0.05).

Qualitative Results.

We adopted conventional content analysis method that coding categories are derived directly from transcripts [19] to analyze comments from blindfolded participants answering five open questions. In total 70 quotes of user comments were collected and they were merged into three categories: the vibration feedback (20 quotes), the prototype (31 quotes) and suggestions (19 quotes).

We gathered positive and negative comments (Table 2) of the vibration feedback from the result of the question “What do you think about the vibration feedback, when your conversation partner looks at your face?” Two participants (P3, P11) mentioned they could not immediately map the vibration to the gaze signal in conversations. The other participant (P10) explained in the beginning the vibration feedback helped her concentrate on the conversation partner, but after while it became just a subtle clue that she often neglected.

Table 2. Positive and negative comments towards vibration feedback of the Tactile Band
Full size table

We asked participants the question: “Which aspects make you like/dislike the Tactile Band?” (Table 3) Six participants liked the Tactile Band. The example comments were: “The Tactile Band did not feel interfering too much. It was easy to wear and it had subtle cues.” “It used soft material, which was comfortable to the skin.”(P10, P14) Some participants also explained why they disliked the Tactile Band. The primary reason was they disliked having the Tactile Band on the head. The example comment was that “The head feels like a scary location for such direct vibrations. It might also be obtrusive for the conversation partner” (P14).

Table 3. Like and dislike the Tactile Band
Full size table

We received suggestions for improving the Tactile Band in two aspects: try other modalities to map gaze signals and improve the wearability of the Tactile Band (Table 4). As for other modalities, two participants stated temperature changes could map to gaze signals. For example: the soft warmth on eyes indicated a kind of close feeling (P15). Other participants mentioned cue tone, soft touch and different intensity of the vibration. For the wearability of the Tactile Band, participants gave many suggestions and the top three were: at hand, around the arm and using the mobile device, where were more invisible during the conversation.

Table 4. Suggestions about the modality and the position of wearing the Tactile Band
Full size table

4.2 Gaze Signals

Fixation duration area was the facial region of the blindfolded participant. Mean and standard deviation of fixation duration were calculated for three experimental conditions (I, J & K) (Table 5). RM-ANOVA analysis revealed that, the main effect of fixation duration was not significant in all conditions, F (1.41, 39.40) = 2.15, p = 0.14.

Table 5. Mean and standard deviation of fixation duration (in milliseconds) across three conditions: (I) no Tactile Band; (J) Tactile Band with no vibration; (K) Tactile Band with vibration.
Full size table

5 Discussion

The quantitative results were not able to demonstrate the effect of tactile feedback on the engagement between the blindfolded and the sighted in face-to-face communication, but we gained valuable insights while running the studies. We realized our experiment has certain limitations and we also get some useful implications for further improvements in both the design and the experiment.

From observations, we found that: (1) the vibration signals were too subtle for three participants to sense them; (2) other participants could sense vibration signals, but with the engagement in the verbal communication, they started ignoring them. Simply increasing the intensity of the vibration may not be a good solution since it may become annoying in the conversation. We hope to improve our design and experiment as follows: improve the prototype such as the wearability, redesign the scenario in the experiment and give more time to the participants to get used to mapping between gaze signals and tactile signals.

According to the observations and user comments, we need to improve the wearability of the Tactile Band. For example, it could be worn on the wrist, which is less visible than on the forehead. The intensity of the tactile feedback could be fine-tuned. Other types of tactile feedback can also be explored besides vibration, such as a sense of pressure by changing the shape of the material. Since the auditory and tactile signals were two primary nonverbal signals for the blind people to sense in face-to-face communication [20], we also consider using auditory signals to map gaze signals. The scenario of dyadic conversation is mainly verbal communication, which is easier to cause conflicts with other auditory signals. Mapping gaze with auditory signals is far from a perfect solution in our case, but it may be possible under a certain condition. For example, one participant proposed to wear the ear phone in the conversation, mapping gaze signals with different cue tones from the ear phone. It can avoid extra auditory interfering to the conversation partner.

Besides the improvements of the prototype, redesigning the scenario in our experiment is also needed. In interviews, some blindfolded participants expressed several alternative contexts in which they would find them to be more useful. For example, a slight vibration (gaze) signal from the conversation partner predicts the start of the conversation to help them be more concentrated. We also consider in turn-taking, eye gaze plays an important role as it indicates where the speaker’s focus of attention is directed [21]. An alternative scenario can be that, one sighted speaker discusses with two blindfolded participants in triadic (three-person) conversations. The sighted stops talking and gives her turn to one of two blindfolded listeners by the gaze signal.

Spending more time in learning the mapping between gaze signals and tactile signals may be helpful. The blindfolded participants knew the importance of the gazes and they had the direct and clear understanding of gaze behaviors. However, gaze is a visual cue in their perception. It will take some time, even a long-term training for them to map gaze signals to tactile signals, which is unnatural for them. As for the blind people, we found they tend to have the indirect and fuzzy understanding of eyes and gazes [20]. They knew the importance of gazes from descriptions in novels or by others. Mapping gazes with tactile signals is a new experience for them, which is likely to require more time for practicing to get used to.

6 Conclusion

In this paper, we focus on presenting an experiment with Tactile Band, a wearable device, which enables the blind person to feel attention (gaze signals) from the sighted and aims to enhance their mutual engagement in face-to-face communication. We set up a Wizard-of-Oz environment to conduct the user experiment with thirty participants in pairs. The results of the user experiment did not significantly demonstrate the effect of tactile feedback on the engagement between the blindfolded and the sighted in face-to-face communication, but we get many useful insights and design implications: (1) the prototype needs to be improved with the wearability with fine-tuned intensity for the tactile feedback. Other feedback can also be explored such as the cue tone or the sense of pressure by the shape changing of material; (2) an alternative scenario could be used in the experiment, which emphasize turn-taking in communication; (3) longer time in learning is necessary for better understanding of the mapping between tactile feedback and gaze signals. In our future work, we will improve the prototype and involve some target blind users in the evaluation.