Interaction Techniques to Promote Accessibility in Games for Touchscreen Mobile Devices: A Systematic Review

Abstract

Games for touchscreen mobile devices have become a part of popular culture, reaching beyond the limits of entertainment. However, while touchscreen devices have become one of the most far-reaching gaming platforms, there are very few studies that consider accessibility issues for People with Disabilities (PwD). In this scenario, this work presents the results of a Systematic Review (SR), which allowed to identify interaction techniques/strategies that are being applied in touchscreen devices, in order to promote accessibility of motor-coordination PwD. From the results of the SR, not only interaction techniques that promote accessibility were identified, but also low-cost and short development time adjustment parameters that can improve the interaction of motor-coordination PwD in 3D VEs. We noticed that promoting accessibility adjustments to meet different player profiles considering their limitations in motor coordination can be a differential in the player’s experience.

1 Introduction

Videogames have become a part of popular culture, reaching beyond the limits of entertainment. Government agencies, the military, hospitals, corporations and schools at all levels are using games for training and teaching within different fields of knowledge [1]. With the proliferation of mobile devices, games have broken the barriers of videogame controllers and computer desktops to become commonplace on tablets and smartphones [2].

The number of games employing touchscreen interaction has been steadily increasing for at least a decade [3]. However, while touchscreen devices have become one of the most far-reaching gaming platforms, there are very few studies that consider accessibility issues for People with Disabilities (PwD) [3]. These users still face challenges to interact with the applications and games by using touchscreen [4], due to their individual impairments/disabilities, such as limited motor coordination or loss of upper limbs (arms and hands) [5], forcing them to seek other alternatives to interact with such devices, such as the use of the feet or the mouth.

It is important to highlight that interacting with mobile devices is more than merely tapping the screen, since it may also involve swiping, sliding, repeated taps, multi-finger tapping, and multi-touch gesture, all of which require good motor control. This creates barriers for the accessibility of many users, especially for people with motor disabilities [6].

A solution presented by Kim et al. [3], proposes to increase the accessibility of PwD based on the personalization of the input controls (physical or virtual), according to each user category.

Therefore, this work, which is part of a larger context that investigates accessibility aspects and interaction techniques for developing games on touchscreen platforms that are accessible to PwD, aims to present the results of a Systematic Review (SR), which seeks to identify interaction techniques/strategies that are being applied in touchscreen devices, especially in the gaming field, to promote accessibility of PwD, more specifically those with motor coordination impairments.

2 Methodology

The methodology applied in this study was the SR process following the PRISMA model (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) [7], based on the searching strings combination applied in the IEEE and ACM database, aiming to find answers for the following research questions:

• (1) What interaction techniques/strategies are applied in games for mobile devices to promote accessibility of people with motor coordination disabilities?

• (2) What accessibility evaluation parameters are being applied in games for mobile devices?

The SR followed three steps: planning, conducting and extracting results. In the planning stage, an SR protocol was established with guidelines that were followed throughout the review. In the SR conduction stage, primary studies from the last five years (2012–2016) were sought for in order to find new approaches. In the phase of extraction of results, the answers to the research questions were looked into in the studies included in the final selection of the SR.

Figure 1 presents the number of articles included and excluded in the SR process.

Fig. 1.

Distribution of studies included in and excluded from Systematic Review.

As can be observed in Fig. 1, 188 studies were identified by applying search strings. In the preliminary screening phase, 25 articles were selected, out of which 13 articles were included in the final SR selection. These articles sought to answer the research questions posed in the SR protocol.

In relation to the first research question, the main aspects of each work were extracted, such as: interaction type, interaction techniques, which devices were applied in the study and if they allow the customization of control commands.

Concerning the second research question, we identified some parameters to evaluate the PwD user experience in touchscreen devices such as the execution time of a certain task in the game, the rate of errors and hits made by the players and the status of the game at the end of the experience, i.e. whether the player was able to complete the task or gave up in the face of the difficulties. These parameters can help in assessing accessibility and proposing improvements in applications, providing a greater technology inclusion.

Section 3 presents the results of the Systematic Review for each research question investigated in this study.

3 Results of Systematic Review

Regarding the first research question, from the results of the SR, two types of interaction were identified: (a) direct interaction - when the adopted technique requires touching directly on the device screen [4, 6, 8,9,10]; (b) indirect interaction - when it requires the use of external resources (i.e. controllers, sensors, assistive technology) to aid interaction [11,12,13]. It should be noted that some of the studies included in the SR present both types of interaction [3, 5, 14,15,16].

Table 1 shows types of interaction (direct and indirect), interaction techniques and the mobile devices used in the experiments of the reviewed studies.

Table 1. Main interaction techniques identified

Direct interaction techniques included adjustments in the size of the buttons as well as applications that reproduce the touch virtually. On the other hand, indirect interaction techniques do not require direct contact with the mobile device’s screen, opting for external devices.

An important feature that can be used in external devices that require communication with the device itself is to develop such devices so that they support the use of On-The-Go (OTG) USB cable as suggested by Aigner et al. [12]. OTG cables allow you to connect external devices such as mouse, keyboard and joysticks to Android devices, in addition to being inexpensive. However, some external resources require more electric power and the power supplied by the OTG cable is not enough for its operation. In these cases, some other source of power will have to be used, reducing the mobility that mobile devices provide, or choose to use another approach, such as devices that communicate over Bluetooth.

The following sections describe the contributions of each work presented in Table 1 seeking to highlight the main objective, results obtained, which techniques and interaction strategies were used, and which parameters were used to verify accessibility issues.

3.1 Direct Interaction

More and more users are using mobile devices with larger screens, often making it difficult to reach the farthest regions on the screen. Considering this problem, Lai and Zhang [8] developed and evaluated the ExtendedThumb virtual finger, to make the use of these devices easier using only one hand. ExtendedThumb allows users to configure virtual finger distance in reference to the location of the actual finger touch.

Thirty-six participants were selected for the evaluation process, in which they had an initial time for get acquainted and familiar with the three interaction techniques (including direct touch, MagStick and ExtendedThumb). In the test, a 15 min time was established for the task execution in a game where small white rectangles were randomly highlighted on the screen for the user to select. The features that were analyzed to evaluate the interaction techniques were: difficulty in reaching specific regions, time to complete the task, selection errors and the users’ own perceptions. ExtendedThumb obtained the fewest errors when compared with other techniques; however, an average time result was reached, between touch on the screen (faster) and MagStick (slower), considering task completion. Figure 2 shows the ExtendedThumb interaction technique with different distance adjustments of the virtual finger compared with the real finger. Figure 3 shows the test environment that was used in the user experiment [8].

Fig. 2.

ExtendedThumb.

Fig. 3.

ExtendedThumb test environment.

The study of Zhong et al. [4] sought to reduce the difficulty caused by hand tremor in people with some kind of motor impairment, and they developed an experimental screen assistive system known as Touch Guard, which is an application service that runs in the background of devices providing enhanced touch screen techniques and, once installed, users can enable it and grant the permissions required for the feature to work. Touch Guard was designed for Android OS and is compatible with any application installed on it. In this study, the tests were performed with motor disabled users in a laboratory, in which eight participants with hand problems that affect accuracy were recruited; it is worth mentioning that none of the participants used Assistive Technologies (AT) in their own devices. In the tests, everyone used their own fingers to interact with the screen and the sessions lasted 90 min for each participant.

The experiment started with a brief introduction and an interview to understand the needs of each participant, followed by pre-defined tasks and, ultimately, user feedback for improvements. The data analyzed in the experiment were: time to perform the tasks and error rate. The authors observed that Touch Guard has the potential to solve accessibility issues for people with motor impairment in their hands on their own mobile devices, avoiding the need to purchase and use expensive hardware [4].

Trewin, Swart and Pettick [6] examined physical contact on smartphones with touchscreen. Using interviews and observations, they found that participants with motor coordination deficiency found such smartphones useful and usable; however, tablets offer several important advantages. The study sought to examine the use of touch-screen mobile devices by a group of people with motor disabilities and who are regular users of their devices; the observations focused on screen gestures to identify usability success and failure rates. Finally, the work provides a basis for guiding the development of new techniques to improve physical access to touchscreen mobile devices.

Considering that touchscreen devices are designed to respond to predefined interaction parameters such as the response time of the recognizers of users’ gestures, Montague, Nicolau and Hanson [9] analyzed which variations occur in the interaction of users with motor-coordination disabilities, since due to hand tremors or little control in their movements, this interaction technique can be a challenge. Sudoku was used in the test, in which the following parameters were collected during the four-week period: Touch Location, Touch Offset, Touch Duration, Absolute Touch Movement, Straight-line Touch Movement, Relative Touch Movement, Movement Direction Changes and Target Offset. As a result, the authors identified that not only interaction performance varies significantly between users, but also that an individual user’s interaction skills were significantly different between the test sessions. Finally, the authors proposed and evaluated a new gesture recognizer to accommodate individual variables in touchscreen interactions.

A proposal to analyse accessibility issues is presented by Pelegrino et al. [10], suggesting a framework as an interaction technique for games on touch-sensitive screens, verifying how this framework could improve game accessibility and even increase the gameplay experience of users with disabilities or some limitations in motor skills. The proposal presents the use of virtual button controls, as shown in Fig. 4, which adapt to the different contexts of the game, allowing customization of the position and size of the buttons. The work presents how game developers can create specific controls for their games and how to reuse those controls by modifying the layout according to the context of the game. This feature can be used in different ways, thus helping users and changing the layout of buttons.

Fig. 4.

Basic interface on the left and the adapted interface (right) when just buttons A and B are required [10].

3.2 Indirect Interaction

On the other hand, when physical contact with the screen of mobile devices is impossible for the user, external devices have been used to ensure user interaction. The following studies present the use of some external device as an interaction resource accessible to people with motor-coordination disabilities.

Yadav et al. [11] highlight the growth in the use of touchscreen-based devices, as well as the advances that are emerging day by day. However, the biggest difficulty still is to make these devices fit for the variables of the user’s physical abilities. This study focuses on users with motor deficiency and/or difficulties in the movement of the arms.

To understand the interaction challenges, a transversal grid technique was adopted. The technique does not require direct contact with the interfaces but can be controlled using an external switch and minimal arm movements. The accessibility data analyzed in this study were: ease of software use; if the technique has made a touchscreen device more accessible; if the devise was used without much discomfort; if it was possible to navigate the device easily and if the software is an engagement model [11].

For a better evaluation of the method, meetings and interviews were conducted with real users with motor disabilities and their physicians. These interactions aimed to find real problems faced by this group of users and how this technique would be useful in their activities. The method was implemented for the users and their evaluations were recorded for analysis. One of the participants, an attending physician from a well-known neurological department approved the technique and was suggested that it would be helpful for people suffering from Parkinson’s Disease and for some cases of paralysis and spinal injuries. Another participant, a physician with experience handling individuals with neurological disorders, considered that the method was significantly valuable and that the design of the switch could increase ease of use [11].

Considering that PwD are often unable to purchase an AT device or the resources available do not fully meet their accessibility needs, Aigner et al. [12] developed an alternative input device known as FLipMouse. The input device enables people with limited motor skills to use computers, smartphones or other electronic devices using their lip or finger gestures. The FLipMouse is based on a joystick and a blow sensor. In addition, two external switches can be connected as additional input resources. The device has a graphical user interface for configuration, offering flexibility in individual settings according to user needs and it is compatible with Android devices via On-The-Go (OTG) USB. The study verified the feasibility of FLipMouse in different user scenarios (games, desktop control, smartphone control and playback of a musical instrument) with people with different motor abilities; a high level of user satisfaction was confirmed, several of the test subjects continued to use the device on a daily basis. According to the authors’ qualitative analysis, FlipMouse offers a high degree of adaptability not provided by similar AT products available on the market or described in their related searches.

Manresa-Yee, Roig-Maimó and Varona [13], present a head tracker for mobile devices, based on computer vision techniques, which detects the user’s nose, captures its movements and turns them into a position on the screen of the mobile device. The authors performed an initial assessment with four users with multiple sclerosis (MS), central nervous system disease whose symptoms may include paralysis, numbness, spasticity, abnormal sensitivity or visual disturbances. It is worth noting that all users with MS received help in their basic daily activities due to their motor and sensory limitations. Users had control over the movement of their heads, although they had limitations in the extent of their movements and they used wheelchairs. The task completion time is displayed in the results.

3.3 Direct/Indirect Interactions

In the work published by Kim et al. [3], the authors point out that although the gaming platforms currently occupy a position among the most widespread in the world, there are few studies that examine accessibility issues in games. In the study, initial findings were identified from a survey and through a qualitative analysis of popular touchscreen games for IPad/IPhone devices, seeking to find relevant accessibility factors for people with motor coordination disabilities.

The study analyzed approximately 100 games for touch screen mobile devices, collecting information such as target size, gesture type, gender, game speed and penalties (how the game reacts to input errors). The analysis was carried out through a codebook, which was developed using an interactive process described by Kim et al. [3] apud Hruschka et al. Finally, it was possible to identify possible problems or barriers related to accessibility for users with motor coordination disabilities. Few of those games allowed users to customize accessibility resources: 24% require the use of both hands (especially action games), 50% require complex gestures such as swiping or two-finger pinch, and 10% make use of gesture capture. In this study, the authors were able to identify game features that were more accessible, such as the Free Flow game that requires only one hand for interaction and does not require interaction speed [3].

The study of Anthony, Kim and Findlater [14] emphasizes that many of the studies on usability for touchscreen interactions with people with motor impairment have been concentrated at study laboratories with few participants. Although this study did not involve game interactions, it aimed to contribute to use cases, interaction challenges and some home solutions that physically-disabled users are adopting or discovering when using touchscreen mobile devices. The data were collected by analyzing YouTube public videos and forms filled out by people with motor disabilities.

The above-mentioned author identified two types of interaction in which 91% of the participants used direct interaction (using fingers, hands, or feet), and about 8% interacted indirectly by means of an intermediary device (header pointer). One of the videos demonstrated interaction that uses both touch and intermediary devices [14].

One of the objectives presented in this study Grammenos and Chatziantoniou [5] is related to the inclusion of users with motor and visual impairments as potential players. Next, the authors present an electronic puzzle game that provides support for one player and they describe the main design features of the user interface.

The process of creating the game was carried out during various design and development phases, including evaluation sessions with players with various profiles to assist in creating the user interface, improving gameplay in terms of accessibility, usability and fun. It is worth mentioning that the game consists of main attributes that can be adjusted to better fit the user’s abilities [5].

For motor-coordination disabled users, two approaches that employ hierarchical scanning techniques were developed. In both approaches, the first step is to select a jigsaw piece and then make sequential manual changes between the puzzle pieces (using switches) or automatically (at fixed time intervals) and then press another switch when the desired item is in focus [5].

Naftali and Findlater [15] conducted two studies to investigate how smartphones are being used on a daily basis and to discover the true experience of motor-disabled users in the handling of mobile devices.

In the first study a survey was carried out with 16 interviewees. The second study was a little more thorough, including a case study involving four users with disabilities in motor coordination, in which the following techniques were applied: initial interviews lasting 30 min in which information on the following issues was collected: demographic information, medical conditions, model of smartphones that they use, frequency of use of the smartphones and AT that they use; daily sessions on smartphones (10 min), semi-structured interview with small demonstrations (1.5-2 h) and outdoor activities (at malls and other public places). Regarding AT, two users use features of the smartphones themselves to improve accessibility such as: screen magnification and text reader (Android) and a Siri personal assistant as AT [15].

The work of Read et al. [16], describes how children with multiple disabilities use tablet games in their homes. A study was carried out in which 20 children participated in their families; data were collected using daily questionnaires, interviews and observations. A group of six people described some of the difficulties faced by some of the children.

In addition, the authors present six challenges involving children with disabilities for game designers to work on:

“(1) Games that employ ‘side-by-side’ tablets so parents and children can play together – or games that ‘switch’ control from parent to child (2) App games as well as web games - web games were underplayed. (3) Games that can be customized in two directions – in terms of ‘suitably aged’ characters and ‘suitably aged’ game play (4) Forums are needed for parents of children to be able to use to suggest suitable games for other/similar children (5) Increase the amount of parental choice for simple games to support repetitive gameplay mechanics but with varying content (6) Games should be playable offline” [16].

Interaction techniques for the motor-coordination disabled people were identified such as the head mounted device (used by a child with cerebral palsy), subtle screen touches and others. The study brought gameplay experiences to children with disabilities in the broader context of family and home life. In addition to demonstrating several difficulties faced by these families and identifying a number of challenges for game developers. The study highlights the importance of the industry and academia working together to make a difference [16].

Summarizing, there are several interaction techniques and personalization parameters to promote motor accessibility, which will be considered along the investigation’s continuation.

Regarding the second research question, some studies established a set of tasks the users should perform within a specific time frame and then they evaluated parameters such as error rate and task execution time [3, 4, 8].

Interface evaluation methodologies, such as cognitive course, usability test and observation, were adopted in order to verify user-friendliness [6, 11, 14, 15]. It was observed that, although PwD have difficulties in interacting directly with a touchscreen, studies found in academic literature offer interaction techniques/strategies that can render this interaction accessible and extend the gaming scope.

In summary, it should be noted that most studies allow for adjustments and that promoting inclusion is highly relevant as it allows users to customize interaction features.

4 Discussion

From the results of the SR, not only interaction techniques that promote accessibility were identified, but also low-cost and short development time adjustment parameters that can improve the interaction of motor-coordination PwD in 3D VEs, especially in mobile devices. The identified adjustment parameters are shown below.

• Interactive objects adjustment (speed, touch, size, position, time)

Users must be allowed to adjust the movement speed of the virtual object in the navigation scenario, so that their experience is not affected. This is because some users may not have enough precision in their movements in order to control, select, or manipulate objects.

It is worth remembering that speed adjustments can be made for both vertical movements (forward and backward) and horizontal movements (character rotation).

In order to evaluate whether the adjustment is appropriate to the needs of the user, we recommend preparing a task such as the user walking in a virtual scenario up to a certain point/location within a pre-set time.

Regarding touch sensitivity, users should be able to easily touch objects in the VE, with the inclusion of features such as an object touch area and response time that is appropriate to the user’s needs.

We suggest that, after the first touch of the virtual object, the system allow for some time for the object to become interactive again thus avoiding errors caused by multitouch or inaccuracies.

Regarding the size of the virtual objects, Manresa-Yee, Roig-Maimó and Va-rona [13] recommend adopting a minimum size of 76 pixels for any interactive object present in the VE interface.

To evaluate this parameter, we suggest using virtual testing environments similar to those presented by Lai and Zhang [8].

Regarding the position of the objects, it may be necessary to adjust the positioning of virtual controls. For example, in virtual joysticks, it is interesting for the user to choose the side at which the control should be available (right/left) in the interface, since users may present different degrees of accuracy in their left and right hands, or they may even lack one of their limbs. Observation may be one way of assessing the adjustment needs of this parameter.

In relation to time, it can be related to completing a task or a phase of the game, or as interactive objects showing in the scenario not long enough time for the user to interact. In this case, observation may also be the best way to evaluate this parameter.

• Adjusting control of external devices

Considering that virtual environments may include interaction resources by means of external devices such as hand control and wearables technology, it is important to predict possible interaction adjustments according to the needs of the user, in order to promote accessibility in the VE.

Considering that the parameters for accessibility adjustments presented must service the expected target audience for the VE, we recommend starting a 3D VE project, based on the 3D VE Design Model for PwD proposed by Conceição Junior et al. [18], and adapted as shown in Fig. 5. The model is represented by an Inverted Pyramid, which indicates the levels to be covered during the design phases of a 3D VE. The Inverted Pyramid organizes the levels according to their degree of importance, and the target audience is at the most important level due to the user-centered design methodology.

Fig. 5.

(adapted from [18]).

Proposal for 3D User-Centered VE Design Model

We adopted the 3D VE Design Model [18] presented in Fig. 5, including a customization step for the specific needs of each user. Mockups are tested in Participative Design sessions to verify through experiments with final users with disabilities which parameters could be adjusted to best serve those users. Mockups are adjusted and presented in new PD sessions until the ideal 3D VE solution is reached. It is important that the user participate in experiments in each development cycle, to verify if the VE is meeting the needs of the user. It is also worth remembering that involving the final user in the design of 3D VE and identifying customizable parameters can provide positive results, such as avoiding having to make adaptations after the 3D VE delivery.

In order to illustrate the application of Inverted Pyramid levels, we present a case study of an 11-year-old child with cerebral palsy, called L. In this case, the Target Audience corresponds to L: child with cerebral palsy, with motor and speech impairment.

Regarding Human Factors, we have as positive human factors the fact that L uses the left hand well, and he writes his own name as well as some other words, writes in block letters beginning with the last letter (backward writing), shows motivation to overcome challenges and loves going to school and learning new things; he also enjoys drawing. He shows a good level of understanding, speaks isolated words. His mother reported that the child likes to play on the cell-phone and tablet and is skilled at using touchscreen interaction resources. In these devices, the child usually plays Candy Crush, but also enjoys strategy and simulation games, most especially the Pool game.

On the other hand, we can consider as negative factors that L is wheelchair-bound, has limited reading skills, limited vocabulary, and he is not able to form sentences. He pointed out that he does not like to play video games because he finds it difficult to handle the remote control.

When details are included in the levels of the pyramid, the Approach to be adopted is based on an educational game to raise awareness of the choice of the best food for a healthy diet. Additionally, the game seeks an approach that stimulates cognitive, motor and social inclusion development.

Regarding Input/Output Devices, the use of virtual joysticks was indicated because of the familiarity and ability that L showed in tests performed with some types of controls.

As for Interaction strategies, it was necessary to adjust the virtual joystick speed parameter, the size of the virtual control and its position in the interface (left or right), so that L navigated in the VE with greater accessibility considering his limitations.

5 Conclusion

The SR allowed to identify several interaction techniques that are used as accessible alternatives for games of touchscreen devices. In addition, some studies have brought about evaluation parameters, such as how to verify the effectiveness of adjustments and accessibility techniques in 3D VEs.

We noticed that promoting accessibility adjustments to meet different player profiles considering their limitations in motor coordination can be a differential in the player’s experience.

From the results achieved, it is possible to recommend accessibility adjustment parameters and relating with the 3D User-Centered VE Design Model for games adapted from [18]. It is worth mentioning that it is important to involve the end user in all stages of game development, avoiding adaptations after the product is delivered.

As for future research, we intend to carry out PD sessions to develop a serious game accessible to PwD in motor coordination in partnership with CRIDAC (Dom Aquino Correa Rehabilitation Center) in the city of Cuiabá, Brazil, in order to validate the interaction techniques and accessibility adjustment parameters that were identified in the SR.

We also intend to investigate whether the error rate in the presented studies can be adopted to verify if the interaction technique reaches an acceptable accuracy rate. Additionally, we can use this data and compare it with the performance of users who do not have motor coordination deficiency in order to achieve a quality rating.