Keywords

1 Introduction

Visually impaired users of smartphones have to be able to interact with their devices without visual cues. Current mobile operating systems already offer assistive options that make this task possible (Talkback for Android and VoiceOver for iOS). However, existing assistive options still require the user to interact with the device’s screen even though it cannot be seen. For blind users, this usually requires both hands (one to hold the device, another to touch the screen), which limits their use of these devices in some mobile contexts. For example, a blind person using a cane or assisted by a guide dog will have at least one hand occupied. In these situations they will usually delay their interaction with the device until they reach a place where they feel safe and are confident enough to use both hands [7]. One alternative is speech based interaction. However, this also raises issues in mobile contexts. For instance, the user might shy away from providing private information via speech when not confident of being alone [7].

Another alternative for interacting with smartphones is non-screen based gestural interaction, or body based interaction. By interacting via gestures that resort only to our body parts, without requiring a foreign interaction surface, the device’s screen is no longer needed, and the blind user does not even need to pick or touch the device. Also, if the gestures are inconspicuous enough they can be private to the user. To be used in a mobile context, this solution requires a wearable gesture recognizer that is capable of recognizing gestures made by hands or arms, sensitive enough to recognize inconspicuous gestures. Solutions have emerged in recent years that make this a technologically viable alternative now. Wilhelm et al. [21] present a ring with a transmitter on it. Whenever the fingers move there is a change in the electromagnetic field around the ring thus enabling finger gestures recognition. Zhang et al. [23] enable gesture recognition via a ring with an oscillator built-in which emits an active signal while a sensor band with four pairs of electrodes, placed on the user’s wrist, is used to capture and measure the signal. Rossi et al. [17] use 4 sEMG sensors placed on the user’s forearm to recognize hand and finger gestures.

However, being a novel interaction paradigm for these users, we still need to understand how it can support people with visual impairments [12] and what variables are important for the adoption of such a solution from the users’ perspective. Without this knowledge, we risk designing solutions that, even if technologically sound, will not be adopted by the target users.

Our main objective is to create an interaction model that characterizes how visually impaired users might use body based interaction to interact with mobile devices, and then explore this model to increase the accessibility of these devices. To create this model there are different dimensions that must be taken into account (e.g., technology requirements and limitations, actions that can be performed with this type of interaction, user characteristics). In this document we want to address the social dimension, i.e., the human behavior and interactions in society. More specifically, we want to answer two research questions:

  • RQ1: Are users with visual impairments willing to use on-body interaction to control their smartphones?

  • RQ2: What are the main factors that affect, positively or negatively, the acceptability of on-body interaction by people with visual impairments?

To understand people’s behaviors and attitudes towards on-body interaction we conducted a qualitative study with visually impaired users. The next section presents work related with body based sensing technology, its use on ameliorating the accessibility and social studies regarding body interaction. The methodology followed in this study is described after. This is followed by a results section presenting the topics identified in the study and a discussion section where the main findings are described. Finally, the main conclusions and future work are stated in the last section.

2 Related Work

2.1 Body Based Interaction and Technology

Starting in the past decade, the idea of using our body as a platform for interaction is present in a rising number of research projects. Harrison et al. paved the way with several well known works such as Scratch Input [4], Skinput [6] or Armura [5]. More recent research in on-body interaction is found on SkinTrack [23], the virtual trackpad [10], iSKin [20], ThumbSlide [1] and many others.

A plethora of different kinds of sensors are presented in these works (piezoelectric films [6], modified stethoscope [4], infrared camera [5], capacitive sensors [20, 23], electromyography [10] and photo-reflective sensors [1]), which are able to recognize multiple points of touch in several parts of our body (fingers [20], hand [6, 20, 23], forearm [6, 23] and legs [4]) and mid-air gestures also made by different parts of the body (fingers [1], hand [5, 10], arms [5].

Additionally, the increasing miniaturization and low-power consumption of sensing technology makes interaction with mobile devices possible. Games [1, 23], hierarchical navigation [6], picking up calls [20], music players [20] and text input [20] were the main focus of some projects when using smartphones [4, 6, 20] and smartwatches [1, 23].

2.2 Body Based Interaction and Accessibility

Other works have looked at how on-body interaction can improve the accessibility of interactive systems. They have focused on leveraging proprioception, gesture recognition and micro-interactions. Virtual Shelves [9], an interaction technique that leverages proprioception capabilities to support eyes-free interaction by assigning spatial regions centered around the user’s body to applications shortcuts, compared the performance of this technique between sighted users and visually impaired users. Although sighted users had significantly better performance, people with visual disabilities were still able to launch shortcuts correctly 88.3% of the time. Costa and Duarte [3] developed a gesture recognizer with the goal of removing unnecessary calibration procedures and at the same time being robust for a mobile environment. Oh et al. [13] also studied the potential of using on-body input for micro-interactions in mobile devices for people with visual impairments. Findings suggest that gestures should be location independent (e.g., swiping the hand or the leg results in the same action) mainly because it is easier to learn and remember the gestures than recalling where a gesture should be performed on your body.

2.3 Social Acceptability

It is already known that VI people are concerned with looking “normal” and avoiding strange-looking assistive technologies [18] which can cause embarrassment [2]. Body based interaction techniques, which are a possible technology addressing those concerns, were already subject of several social acceptability studies. The willingness to perform these gestures will largely be dictated by how appropriate those actions look and feel when performed in public. Profita et al. [14] and Rico et al. [15, 16] findings show that there is a significant relation between audience and location with the willingness to perform certain gestures. Findings also show that users are more fond of subtle movements, movements that are similar to what already exists in current technology, and movements similar to the ones used in our everyday lives and enjoyable movements. On the other hand, participants stated that uncommon, large or noticeable movements would look weird in public settings.

Williamson et al. [22] developed an application based on in-air gestural interaction. Despite being considered subtle by the authors, it was observed that some gestures were considered unacceptable in certain settings by the study participants. Consequently, the participants developed new ways of performing the same gesture. To address this issue, such systems much be flexible and develop correction mechanisms in the recognition process. Additionally, the authors found that the willingness to perform gestures in a public setting depends on the characteristics of the audience, such as if they are a sustained spectator (e.g. other passenger on a bus) or a transitory spectator (e.g. a person walking by).

Focused on our target population, Oh et al. [12, 13] investigated preferences for and design of accessible on-body interaction. The authors’ findings show that location specific gestures rose concerns regarding social acceptability in public settings. The least preferred areas were the face/neck and the forearm, while locations on the hands were considered to be more discrete and natural. The findings also suggest that participants may prioritize social acceptability over ease of use and physical comfort when assessing the feasibility of input at different locations of the body.

3 Methodology

We sought to collect rich opinions of potential end users of an on-body interaction based solution for mobile interaction. We conducted a qualitative study focused on interviews, as they provide richer data than questionnaires or online surveys. With an interview we can work directly with the person which gives us the opportunity to probe or ask follow up questions and enrich our findings.

To understand in what conditions people with visual impairments are willing to perform body based gestures we chose a set of different factors that may affect their decisions. Here we present the main factors we saw fit to discuss and why:

The environment where the person is located is a factor that could decide whether the person feels it is opportune or not to perform on-body gestures (e.g., home might provide a safer place for gestural interaction than a bus).

Living in a society means to get along with other individuals. The degree of relationship between two or more individuals restrains or encourages certain actions thus we see it as a factor when considering performing on-body gestures.

Humans can assume different stances, i.e., walk, run or sit. Those are associated with different actions and can affect the predisposition to perform gestures (e.g., seated stance can enable inconspicuous gestures while walking not).

In spite of the possibility of several negative factors deterring a person from doing certain actions (e.g., security, embarrassment, discomfort), there are occasions when the emergency of the situation can overcome these factors.

While we discussed these factors in the interviews, we also explored possible gestures. The gestures explored first on the interview ranged from mid-air gestures to on-body touches. We provided a wide variety of examples during the conversations so that interviewees could have a better understanding of the possibilities. These examples did not take into account technological limitations associated with gesture recognizers. We opted to exemplify gestures with different characteristics in terms of amplitude, body parts involved, meaning, etc., with the aim to generate discussion around the subtleness of gestures.

Still on this topic, we asked participants to rate the subtleness of specific gestures with different characteristics such as finger, hand and forearm amplitude and conveyed meaning. The gestures from that list were categorized based on a two-level taxonomy (movement and style) similar to the one presented in [8]. Gestures can be static, i.e., do not change over time; Or they can be dynamic, i.e., may involve multiple strokes and the whole movement conveys the gesture’s meaning. More specifically, we seek to understand if the gesture’s movement characteristics affects its subtleness.

In what concerns its style, gestures can be defined as: Semaphoric (any gesturing system that employs a stylized dictionary of static or dynamic hand or arm gestures), Pantomimic (gestures that simulate performing an action), Iconic (gestures that are used to clarify a verbal description of a physical shape or form through the use of gestures that depict those characteristics), Manipulative (gestures with a close relation between the gesture movement and an object with the intent to control said object) and Deictic (gestures that involve pointing to an entity to be established as the target subject of an action). With this set of gestures we hope to know if and what characteristics can affect the perception of subtleness by people with visual impairments.

We also wished to explore the social aspects of using speech commands as an alternative interaction approach to standard accessible mobile interaction options. Similar to on-body gestures, speech commands also raise concerns regarding social acceptance. Thus, we want to compare the main differences and similarities between the two modalities.

In order to answer our research questions we followed a qualitative approach supported on semi-structured interviews. We chose this type of study based on all the aspects pointed out previously. Conducting an interview with baseline topics where we are able to generate a discussion will be more advantageous to collect meaningful conclusions supported by the reasoning of the participants. Additionally, it can generate new, unexpected, topics for discussion.

3.1 Interviews’ Procedure

The interviewer took notes and the interviews were audio recorded for posterior analysis. The interviewer firstly asked for authorization to record the interview and then proceeded to gather demographic information and cellphone usage habits from the participants. The interviewer then explained the goal of the study and how the session will proceed.

During the interviews we presented participants with different situations and asked them if they would perform gestures to interact with their smartphone in those situations and then discussed the motivations for their answers.

Table 1. Gestures exemplified to the participants and their classification
Full size table

As we seek to understand what factors impact the willingness to perform body based gestures to interact with their smartphones we selected a set of domains as basis for the situations to be discussed in the interviews: (1) Gesture subtlety [subtle, not so subtle and unsubtle]; (2) Environment context [home, street with passersby, street with people standing by and workplace]; (3) People around [alone, family, stranger and colleague]; (4) Posture [seated, standing up and walking]; (5) Emergency of the action [emergency, no emergency].

Table 2. Main themes generated after the interview analysis.
Full size table

We also wanted to understand what the participants consider to be a subtle gesture. Therefore, we described orally the gesture being discussed. If they could not understand and reproduce the gesture we assisted them in performing it until they correctly understand it. In this part of the interview we discussed a total of 10 gestures (see Table 1). Finally, we asked how they feel regarding speech commands in some of the situations described before.

3.2 Data Analysis

The interviews were transcribed manually from audio to text for further analysis. We then carried out an inductive category analysis [19]. Two coders were, individually, in charge of reviewing notes and discover common factors that affect the acceptability of body based interaction for controlling smartphones in different conditions. The coding task was processed in the online tool QCAMAP [11]. In the next stage, coders shared their interpretations of the interviews and proceeded to obtain a single list of codes. The two coders then reanalyzed the interviews. The results of this analysis will be described in the following section.

3.3 Participants

We recruited 18 visually impaired participants from an institution supporting visually impaired people. However, one participant was excluded from the analysis for not being able to conclude the interview. Each session with a single participant lasted around one hour. Due to problems with the audio recording we could not analyze the recording of 10 participants. For those participants only the interviewer notes were considered in the analysis.

The participants ranged from 25 to 64 years of age (\(M = 46.88, SD = 13.27\)), 8 were female (9 male) and all had some kind of visual impairment (5 participants reported to have some residual sight, the remaining were blind). On average, participants lost their vision since their teen years (\(M = 15.92, SD = 14.84\)). Of the 17 participants, eleven use smartphones (4 iOS) and use VoiceOver or Talkback as their main assistive technology.

4 Results

Expectedly, the analysis of the transcripts and interviewer’s notes revealed themes aligned to the dimensions that grounded the interviews. However, new themes emerged from the analysis and new factors were found regarding the acceptability of on-body interaction. We now discuss the themes presented in Table 2.

4.1 Gesture Acceptability

When discussing on-body interaction some of the participants initially showed no concerns about performing gestures. P10 said “I do not have any issues with touching my body or in my arm” and “We, blind people, already raise attention with our canes, it would not be because of doing gestures. What about the deaf and mute people? Don’t they already perform gestures to communicate?” P15 stated “I am imagining myself on the subway if necessary doing a movement to reject a call, I think the gesture would be similar to other gestures that others do in other devices.” However, some concerns were raised about the type of gestures they would be doing. The same participant, P15, now said “I am imagining being on a restaurant [...] it would depend on the gesture’s characteristics.” P18 supported this by expressing “The gestures must be well thought.”

Frequent gestural interaction, and its consequences, was a legitimate concern. P1 exemplifies this with “if it is a frequent interaction maybe I would not do it, it would be tiring.”

Some participants expressed their concerns about gestures that already have a meaning (e.g., waving hand for “Hello”, thumbs up for “OK”). P13 stated “I think it would be a little weird, others would think that I am waving at them.” P18 added “It depends on the gestures, right? If I do this [performs a waving gesture] others would think I am talking to them.”

Participants also showed strong concerns about gestures having offensive meanings. P11 said “You should not point at others, it is rude.” P14 exemplified it with “if we do this [performs offensive gesture] others might get offended, right?” And P17 summarized it with “as long as the gestures are not offensive to anyone, it would not affect me.” Still on this topic, when shown a specific gesture – V sign – P18 raised a cultural concern when performing gestures: “This sign could be offensive in the UK.”

According to participants, comfort also plays a role on the willingness to perform gestures to interact with their smartphones. On this subject, P15 said “It would depend [..] if it would be more comfortable to hold the cellphone on my hand or keep it on the pocket.”. When the interviewer asked participants to replicate a specific gesture (scissors), this factor was evident together with concerns for injuries or limitations. P11 said “ The right hand does not want to cooperate [...] this one is bad for the tendinitis.” P17 added “[...] it is fine, but there are people with motor impairments, it could be harder for them.”

Gesturing being something that is natural for humans also contributes to the acceptability of this interaction. P10 said that it is more natural than using speech input, “[...] more natural than talking with and listening to a machine.”

One major acceptability factor for this interaction is what it allows users to do. This could offset some concerns and drive its adoption as can be seen in the comments of P4 (“If it brings advantages to me I cannot feel embarrassed”), P10 (“Everything that eases the interaction with my cellphone is not an inconvenient to me. Don’t people talk already with gestures normally?”) and P16 (“[...] if it could ease my life, why would I care about what others think?”).

4.2 Perceived Advantages

During the interviews the participants stated some perceived advantages from using on-body interaction to control their smartphones. Participants mentioned several times the ability to interact without having to take the phone out of the pocket. This was more important when discussing specific tasks.

When discussing walking outside P13 said “it suits me better as I do not have to take out the phone, stop and accept the call. I could walk [...] without having to hold the phone in one hand [...] having a cane this would be beneficial.”

Some participants mentioned advantages when cooking. P15 stated “[...] or I am cooking, it is very interesting to me!” and P17 said “[...] it would be advantageous, I could be gutting the fish at the same time for instance.”

When discussing desk work, P16 mentioned “If I have the phone on the table and I am working I would not have to move to grab the phone, it would help.”

However, the same participant raised the concern of performing gestures that could interrupt a conversation: “as long as it does not interrupt my conversation with another person by doing the gesture [...]”.

Other participant saw advantages concerning privacy issues. P16 exemplified it with “If I am seated, I can do the gestures under the table” and “if I do it like this [demonstrates gesture], no one else would notice.”

Others stated that it would be a great advantage regarding security concerns if they could keep the cellphone in their pocket. For example, P4 stated “[...] this way I would not run the risk of being robbed.”

Participants also stated that using on-body interaction to control their cellphones is a technology that not only helps them but also can help everyone. Examples of this came from P10 (“it is an easier way [to interact with his phone], I think it would be better for everyone.”) and P15 (“[...] I became aware that this is not only good for persons with visual disabilities, but for everyone else too!”)

4.3 Gesture Subtlety

Gesture subtlety is a major factor in the acceptability of this interaction technique as could be seen in the comments of most participants when faced with different levels of gesture subtleness.

If gestures were subtle and the interviewee was assured that no one else would notice it, the overall reaction is very positive. For example, P1 said “I would do it in any circumstance as they are subtle gestures.” In the same line, P13 stated “I would feel comfortable doing gestures, no one would notice so.”

Being on public environments (e.g., street), makes this factor even more important as could be seen in several comments. For instance, P2 mentioned “For me the benefit of using subtle gestures would be even better in this situation.”

When discussing not so subtle gestures that could be noticed by others the reaction was less positive. For example, if asked to do these gestures on the street, P11 commented “Right, I do not know if I would feel very comfortable.”

However, not all participants expressed the same level of concern. P4 stated “Even being less subtle, people also scratch their shoulders and no one notices it [...] there are always ways to avoid people noticing me doing gestures.”

When presented with gestures that surely would be noticed by others, the reaction grew more negative. Even P4 stated “it is more complicated to do it. [...] it is a little embarrassing” and “I would prefer to pick up the phone than using gestures.” Embarrassment was a concern expressed by most participants. P8 saw no point in doing such gestures to interact with his cellphone specially in front of family or strangers, “I would feel embarrassed [...] I do not feel necessary to do such gestures with big movements to control the phone.” P10 outrightly stated that “At home alone I would do it, but on the street I would not.”

Some participants expressed additional concerns about the dangerous aspects of performing such extreme movements being blind. P10 exemplified this by stating “[...] we can not see if we will hit someone.” P14 also discussed this when saying “Most of the times we have to have perception of our surroundings, If I am in my house I would be able to do it. However, for instance if we are talking about raising our arm in front of us, I could be sitting on a table and a bottle of water could be on the table, we have to be aware of what we are doing. I could be surrounded by people and hit someone.”

Near the end of the interviews we discussed what makes a gesture subtle in their opinion. This is important since we had discussed the effect of the subtlety of gestures. To this end we defined a set of ten gestures (Table 1) and we asked what they thought about them.

In general all gestures were well understood when described aurally by the interviewer. However, three gestures had to be explained physically. The “ok” and “Pinch” signs where not understood by 4 participants while the “Victory” sign was not understood by 3 participants.

All except one gesture were generally perceived as subtle (Fig. 1). With this result we could not find any difference between static and dynamic gestures.

Fig. 1.
figure 1

Gesture subtlety perceived by the interviewees.

Full size image

Further analyzing, we observed that the two iconic gestures exemplified to the interviewees were the most perceived as subtle (15 out of 17 participants). One of the reasons stated for the iconic gestures being subtle is that those gestures are known and used in our daily lives. Referring to the scissors gesture, P16 said “these are gestures that we have used for a long time.” The point forward gesture was seen as not subtle by the majority of participants (15 out of 17). P11 even reported it as being offensive, “You should not point, it is rude.”

Gesture subtleness is not only categorized by the characteristics of the gesture but also by external factors such as the person’s stance. While seated, gestures could be performed in a more inconspicuous way as they can be hidden. P16 said “If I am seated I would do it under the table, or I would do it like this [demonstrates the gesture].” Differences between walking and standing were referred regarding the subtlety of the gesture. For instance, P13 stated “as I said before, it depends if we are walking, if we are stationary”, referring that while walking it would be noticed.

Other factor is how the person performs the gesture. Where the gesture is done, i.e., on air or close to body as well as how fast it can be performed were the two main reasons pointed out by the interviewees. Gestures closer to the body and without abrupt movements are perceived as subtle. P8 said “It depends on the gestures’ speed and if you do it on air or on your leg.” P18 stated “It depends on how you do the gesture, if I do it like this [performs gesture closer to the body] [...] This way it is ok [subtle].”

4.4 Effect of Gesture’s Complexity

Not only the subtlety but also the complexity of the gestures affects the willingness to interact with gestures. P2 said “These more complex gestures, I would perform them only inside.” P13 stated “If the gestures are simple yes [I would do it].” P18 said “one will pick a simple gesture to pick up calls”, “I think in any situation I would opt for simpler gestures.”

P13 stated that the issue is not the embarrassment of doing it in front of other people but the quickness when compared with the traditional way of interacting, “I would not have any problems but probably I would not do it because I would have more work than getting the phone out of my pocket.”

4.5 Effect of Location and People Present

The location where the person is also affects the willingness to perform gestures. P10 said “At home alone I could do it but in the street it would be more complicated.” P18 stated that “in situations where I am in closed environments [home], I would not have any problems. However in the street they would not be so practical to do.” P11 states that “At home I would feel much more willing.”

When discussing gesturing in public places, P11 said “I would do it, if the place is not crowded” and P9 stated that “It depends if there are a lot of people and confusion like rush hour, I would do it, but in other cases I would refrain from using gestures with others observing me.” Others were even less concerned about performing gestures in these locations, as can be perceived by the following comments. P5 stated “I do not know if others would notice me doing gestures, but I would do it anyway. Furthermore, if it was an urgency I would really do it.” P1 said “People are too busy to notice” regarding passers-by or people standing by. P8 said that “Shy people would feel embarrassed but not me.”

While in public places, the presence of family is something that can affect positively the willingness to perform gestures. P6 said “Perhaps I would feel more comfortable if I were with my family” and P11 stated “yes, if I was accompanied with my family it would be easier.”

In the presence of strangers the subtleness of the gestures plays an important role in the willingness to perform them. When gestures are subtle P13 says “I would feel comfortable, the gestures are so imperceivable.” If the gestures are not subtle, P7 instead states that “I would avoid doing gestures around strangers.” P9 shares this opinion when expressing “I would feel constrained to do it in front of others.” Nevertheless, it is important to mention that this attitude is not shared by every participant. P15 is uncomfortable with performing gestures in front of strangers, independently of the subtleness, as expressed by “In the case of being with a stranger I would think twice.”

We also discussed in the interviews other degrees of acquaintances between family and strangers. For instance, work colleagues may affect the willingness to perform gestures. P3 stated that only if he trusts the colleague he would perform gestures: “Only if I trust him because I do not want him to tell my boss I am doing gestures.” P4 said that he would do it in front of colleagues but not in front of his boss adding “if it is a colleague, I would explain it to him [why he is doing gestures].” P9 stated that he would only do gestures in the presence of a work colleague if he understood his situation: “Taking into account that the other person knows I need help for being blind.”

4.6 Effect of Stance and Free Hands

We also discussed the impact that stance might have on the willingness to perform gestures. In particular we considered seating, standing and walking. There were not many comments on this subject. A few participants would prefer to avoid performing gestures while walking, like P11 who said “[...] If I was stationary I would in fact do it, without any problem.” P15 related the stance with the subtlety of gestures when stating “Maybe if I was seated it would be more discreet wouldn’t it?”

Given that blind people when walking outside usually have at least one of their hands occupied with a cane or a guide dog’s leash, we also discussed what effect this could have. P10 and P18 said respectively, “No [issues with doing gestures while walking on the street], if I was only using one hand” and “[...] it could be solved by just changing the hand holding the cane, that would not be a problem.” P2 also said “[...] using my non-dominant hand or I could use my index finger while holding the cane.”

While gestures made with one hand only, or more inconspicuous ones that could be done only with fingers, raised no concerns if one hand is free, more ample gestures originated a different reaction. P1 summed it up when stating “Normally I have both hands occupied. I would stop to interact.”

4.7 Effect of Other People Noticing

In many situations participants commented how they would react when noticed by others doing gestures to interact with their smartphones. P14 said he would just ignore offensive comments by others, “[...] people could sometimes say this guy must have nervous twitches and I would say yes yes.” P15 stated that he would quickly forget that others are observing him (“Probably others will look but I will stop noticing that, I do not care”) although he also stated that “I have the tendency to explain everything to others, to make them aware of our reality.”

The willingness to explain what is going on can be seen in the comments of several other participants. P4, P8, P10, P16 and P17 mentioned this explicitly in their interviews. For example, P4 said “If anyone asked me what I was doing, I would explain it to him.” P10 contextualized with the action being done when commenting “[...] I am accepting a call [using] a new easier way to do it.”

4.8 Effect of Gesture Familiarity

For those participants who owned a smartphone and were familiar with Talkback and VoiceOver gestures, we discussed what they think about using similar gestures to what they already use but done in-air or on the body.

Almost all participants agreed that using gestures they already knew would affect positively their willingness to use on-body interaction. P4 said “If we were using the same set of gestures we use on the touchscreen, I would feel more comfortable to do it in any situation.” P9 stated “I would prefer the ones I already know, it would be easier and I would be more comfortable.” P13 said “I would prefer Talkback’s gestures for being simpler.”

Other participants were indifferent to it (P5) or emphasized not the gestures but how they are performed. P8 said “If all of [the gestures] would be performed close to the body it would be the same [as Talkback]. On air they would not.”

4.9 Effect of Task Urgency

During the interviews we discussed, especially with who was less prone to perform gestures in public settings, about their willingness to perform non subtle gestures (eventually with large amplitude of movements) when they urgently needed to use the smartphone (e.g., to call an ambulance, or to answer an important call).

The opinions covered the whole spectrum. Most said they would do it in an emergency context. For example, P2 stated “Yes I would do it even if someone would call me crazy!” while P6 said “If I were afflicted I would do it.” Some showed to be less keen to do it. P4 and P5 said if it was the only option then they would do it. Finally others were not willing to do these gestures even in extreme situations. P9 said “[...] I think I would prefer to use the traditional way” while P14 stated “I would try to avoid doing it.”

4.10 Effect of Ethics and Security

When discussing the use of gestures for interacting with a smartphone when at the work place, we initially identified a common concern that is not related with gestures. Many participants refrain from using their smartphone in the workplace, as can be seen in the words of P5: “On the workplace I always keep my phone away [...] I would only use it if it is an emergency.” From the interviews we concluded that for those that do not use their phone at work, not even the gestures being subtle would make them use it. For those that use it, then on-body interaction would not change it. For example, P14 said “[...] Yes I think I would do it, as long as it has no consequences.”

The ethical concerns extended to outside the workplace, as can be seen in this statement from P17: “Of course we can not obstruct the sidewalk [while doing the gesture] [...] and [I would] move to one of the sides to do it.”

Another factor that was frequently raised during the interviews was security, and how on-body interaction has the potential to make them feel safer. Various participants expressed their fears of having their smartphones stolen from them. P10 said that “I do not like to pick up my cellphone, I am scared that someone near me takes it.” P14 stated “Because we can be on the street and someone takes my phone of my hand.”

4.11 Speech Recognition

At the end of the interview we discussed what did the participants think about using speech to control their smartphones. Participants were asked to comment on this subject based on all the topics that had been discussed previously.

The major concern raised by participants regarding the use of speech was privacy. P1 said “Only at home and with people I really trust.” P2 stated “I would prefer to write over talking [...] I would even try to learn the touchscreen gestures rather than using voice.” When discussing the public location with a stranger setting, P14 said “[I would not use voice] because I do not know that person and I would not trust her.”

Other interviewees are used to and currently use voice commands. These had a different opinion. P4 said “I use it on the bus, for example, to search contacts.” P7 stated that “When I had Internet [on my phone] I usually used it [...] I would use voice commands in any situation even in public.” Other participants (P10) accepted using voice commands due to its familiarity and popularity. P11 said he would prefer voice over gestures.

Besides privacy, other participants showed ethical concerns regarding the use of voice in crowded spaces. P8 said “At home or on a place alone I would use voice [...] In public transportation or places with a lot of people I would avoid it, so I don’t bother them.”

Similar to what happened with on-body interaction, security issues were also mentioned. P10 said “I would avoid it maybe ... and the cellphone would have to be in my hand?” P13 stated that “for instance, my cellphone can be ringing and other person hears it and says accept and I did not want to answer it” referring to other people issuing commands on his behalf.

Another aspect that was not specific to on-body interaction is embarrassment. For a subset of the participants, speech is potentially more embarrassing than on-body gestures. P9 said “If I were to use it, it would never be on the street. I think it is worse than using the gestures we discussed.” P10 stated that “I would do it but I think gestures would be better.” Also, P13 stated that for certain situations gestures are also better than speech: “I would prefer gestures over speech but the street is where it is really advantageous.”

5 Discussion

This study explored the acceptability of on-body interaction by visually impaired users as a potential interaction technique to increase smartphone’s accessibility.

Our findings supplement previous studies that investigated gestures and touches on our body as a means of interaction for people with this impairment [12]. Past studies limited gesture creation to finger gestures and body touch in three different body locations (hand, forearm and face). While in our study we did not explore gesture creation, participants were given examples of gestures and touches that showed the freedom of on-body interaction which is not restricted only to fingers and hands. We believe that in this way participants could be more creative when confronted with the situations described during the interview (e.g., P15 suggested to use the tongue or foot taps for subtle gestures).

While our end goal is to understand what are the design implications for an on-body based interaction model for users with visual impairments, in this document we provide more insight on the social acceptability aspect and what are the factors that impact the willingness of the users to interact with gestures.

The themes that emerged in this study were in accordance with the literature. Rico and Brewster [15] studied the social acceptability based on a survey where a set of 18 gestures were described to participants. They concluded that demographics or previous experience with technology usage are not the only factors that impact social acceptability. There are a lot more factors than embarrassment or politeness. In [16], the same authors state that it can range from appearance, social status, to culture. In their studies, the authors concluded that audience and location are factors that affect the willingness to perform body based gestures. Authors also reported gesture subtleness; similarity to existing technology; similarity to everyday actions/uncommon movements, enjoyable/uncomfortable movements and interference with communication are important factors as well. Our study focused on the acceptability by visually impaired people, unlike Rico and Brewster’s study which did not consider this target group. Nevertheless, it is interesting to register that our findings were similar.

Regarding the location where the individual is at, it was evident that home was the most acceptable location to perform gestures and public places were the least accepted locations. This is, of course, a result of public locations being coupled with an audience that can perceive the gestures being made [16]. However, we learned that being at home brings an added confidence to perform gestures even when around strangers. By being in a “safe place” our participants are more confident of their knowledge of their surroundings and feel more at ease to explain what they are doing.

Besides the audience factor that is associated with public locations and that potentially can cause embarrassment as explained by some of our interviewees, we uncovered other reasons attached to the willingness to perform gestures in certain locations. Among this, the security of the blind person was often mentioned. Our participants felt that a solution that removes the need to take the phone out of a pocket or bag to be able to interact with it would address their fears of having the phone stolen from them while holding it in a public place. Another acceptability factor that was discussed is the ethics of using a smartphone in a public setting. Several participants revealed that in their workplace they refuse to interact with their smartphone. Even if using inconspicuous gestures to interact with the smartphone, our participants maintained their unwillingness to use the devices in the workplace if that was their initial position.

As aforementioned, audience plays a major role in social acceptance of on-body interaction. When alone, most participants accepted doing gestures, some even in the street or at their workplace. The type of audience can affect positively the users to perform gestures in public locations. Family presence comforts and increases user’s confidence to interact with gestures. Work colleagues offered a mixed experience. Participants reported that the confidence in using on-body interaction is based on trusting that their colleagues will not get them in trouble.

The main causes that affect the acceptance in front of others have to do with embarrassment and privacy issues. Gesture subtleness is a major contributing factor to the embarrassment or lack of privacy of the interaction, thus it is a major factor impacting the willingness to perform gestures. While subtle gestures were welcome and even seen as a great feature and advantage of on-body interaction, attention grabbing gestures were not welcome at all. The perceived disadvantages of these gestures weighed more heavily on the participant’s willingness to use them than positive factors like the location and type of audience. Some participants were not willing to do these gestures in their home or when alone in public places. This was more evident in public places when they could not be sure that there weren’t any observers around (e.g., train station). In this situation almost no one wanted to perform these gestures although the presence of family still affected positively some of our participants.

The more ample the gestures are, the more embarrassment they cause. However, this was not the single concern raised. Participants also mentioned safety reasons (e.g., hitting someone while doing the gestures) for not wanting to perform those gestures. However, in emergency situations, such as needing to call an ambulance, the subtleness factor loses importance to some of our participants.

Hand occupancy was also discussed by Oh and Findlater [12]. Our results are in agreement with theirs. During our discussions the interviewees declared that when holding a cane or a guide dog’s leash they would prefer to do gestures with the non-dominant hand or, if with the dominant hand, they need to be allowed to do gestures or touches while holding the cane (e.g., index finger tapping the cane while holding it).

We also investigated if stance would affect the willingness to perform on-body interaction. However, participants shown almost no concerns when talking about different stances (seating, standing and walking), except for the walking stance. The majority of participants did not point any differences in being seated or standing regarding, for instance, the ability to disguise the interaction. Only one participant mentioned this situation. This perception applied irrespectively of the location where the interaction was taking place. The walking stance impact was mentioned only in those situations where the cane is needed, i.e., on the street. At places that are well known, e.g., home or at work, participants feel that they will have no problems performing the gestures while walking.

The tasks users have at hand were also identified as a factor impacting the acceptability. As Rico and Brewster [16] state, there are gestures that may interfere and disrupt a conversation with another person. Gestures that already convey a communication meaning should be avoided. When navigating from one point to another in the presence of obstacles or other factors that might be dangerous for the person, this type interaction was not welcomed by some participants. However, in this setting any interaction with the smartphone is avoided, which means that is not a concern exclusive to on-body interaction. Interestingly enough, there are tasks where both hands are occupied and participants were enthusiastic with the idea of using gestures (e.g., cooking).

Gesture familiarity also contributes for on-body interaction acceptance. Replicating on-screen gestures or other gestures already popular via current technology (e.g., Wii remote control) might help to ease the acceptance for this novel type of interaction [12, 16]. The use of these well-established gestures might also contribute to reduce the embarrassment of doing such gestures.

Gestures’ characteristics and meanings also affect their social acceptance when the user might be observed. We already discussed the importance of avoiding gestures with a meaning. For example, waving hands could mean a communication starter. Doing these gestures “alone” while on a public place would lead to misunderstandings. Gestures may not only convey offensive meanings but they are also culturally dependent.

All the factors above contribute to answer RQ2. Besides these, we received multiple statements suggesting that if this technology is well implemented and in fact brings accessibility advantages for the user, they will use it despite other factors (embarrassment) that would normally deter them from accepting this interaction. This contributes to answering RQ1.

One additional aspect we investigated is how on-body interaction compares with speech interaction. Regarding speech recognition the results suggest that it is a better accepted technology due to its familiarity and popularity among all population. However, it also shares acceptability concerns with on-body interaction. Embarrassment, privacy and security are some of them. When comparing speech with on-body interaction, gestures were mostly preferred probably because if subtle they can overcome privacy and embarrassment.

6 Conclusions

On-body interaction is an interaction technique with the potential to increase the accessibility of mobile devices for visually impaired people. For this potential to be realized, we must first understand if this population is willing to use on-body gestures and in what conditions. To contribute to this understanding we conducted a qualitative study with 18 interviewees with visual impairments to acknowledge what are the main factors that can impact the willingness to use on-body interaction. In this paper, we present a set of factors and conditions that affect positively or negatively their willingness to use on-body interaction.

To answer RQ1, we can state that people with visual impairments are indeed willing to use on-body interaction to control their mobile devices. This statement is even more reinforced if this new interaction technique proves to be faster, more robust and easier to use than standard mobile interaction techniques. Furthermore, the participants saw immediate advantages over their current assistive technologies regarding the practicality and safety aspects of not having to hold their phone on one hand.

In what concerns RQ2, findings also showed that regarding location, home is their most acceptable location to do gestures and public places are the least preferred for this kind of interaction. However, the advantages of on-body interaction are more valuable in public places, with on-body interaction being seen as more practical in this situation than standard interaction. Audience also plays a role in the acceptance: when alone, this population feels more willing to perform gestures even in public places. When in less desirable places to interact, family helps to diminish the embarrassment. The stance is not as important for adoption and only walking was perceived as a stance that can affect negatively the usage of on-body interaction. However, the task the user is performing is a factor that brings up one of the advantages of on-body interaction (i.e., not being required to hold the phone). The characteristics of the gesture is an important factor when deciding to use on-body interaction. The gesture’s subtleness is prioritized over location and audience by people with visual impairments. Additionally, using well established gestures (e.g., similar to what is used in Talkback and VoiceOver) can improve the acceptability of on-body interaction.

Although with this study we reached interesting findings that helped us in answering RQ2, we can not at this stage conclude which ones are the most important factors or which ones are only relevant when combined with others. For that to happen we have to address one of the study’s limitations: the interviewees did not get to try the technology. When using such technology, the opinions might diverge from current conclusions, which was already shown to happen especially when experiencing non-hypothetical situations [16]. To address this limitation, we have developed an accessibility service for Android smartphones based on gestures made with the hand or fingers (thus making them fairly inconspicuous). The development of this prototype was inspired by the findings reported here. In our future work we plan to use this prototype in a longitudinal study with visually impaired people in order to get feedback from real life situations and compare their user experiences with the findings from this study.