1 Introduction

Protecting the lives and the rights of the visually impaired people and promoting their social participation is a paramount principle today. Especially for visually impaired people, information accessibility is the important issue under this digital society, so improvements of the interface to make information more and more accessible for visually impaired people are indispensable. This paper introduces our two research projects. One is to propose the graphic representation method with touch and sound as the universal designed touch screen interface for visually impaired people. Another is to investigate the good design of the collaborative work environment of the visually impaired.

In this paper, Sect. 2 is the universal designed touch Interface and its’ application. The digital divide problem of visually impaired people tends to be focused on the access difficulties of graphical information still today. We have been developing a new method which visually impaired people can intuitively recognize the graphical information using audio and touch panels. The method is universal-designed to enable not only the visually impaired people but also the sighted people to enjoy using interactive digital graphical contents together. The proposed method and interfaces are basic techniques for developing plug-ins which help blind people to use ordinary mass-produced computer devices with touch screens, such as smartphones and iPads. Our idea is so simple that musical scales enable users to trace graphics by their fingers and to memorize their position on the touch screen. Our recent progress including digital textbook application for visually impaired children is also reported.

Section 3 is the design of the collaborative work support environment. The collaborative work of visually impaired people and sighted people on equal ground plays a significant role for visually impaired people’s social advance in society. Our developed collaborative application of music composition has a beautiful tangible interface that would attract the attention of both visually impaired and sighted people, and multiple functions that are likely to induce collaborative communication among users. After evaluating this application, we focused our interest on the moment in which the visually impaired were having fun, and on the factor of the excitement and concentration. This paper introduces our experimental system, which is a shooting game application without visual information, to investigate the factor of the excitement and concentration of the collaboration between visually impaired people. Recent analysis results of the collaboration are reported.

2 The Universal Designed Touch Interface and Its’ Application

2.1 Basic Graphical Representation with ONE OCTAVE SCALE INTERFACE

This section introduces our research experiences on developing universal-designed interactive contents on touch panel for visually impaired people. We have been proposing “One Octave Scale Interface (abbr. OOSI) as a graphical representation interface on touch panels for visually impaired people. The OOSI is based on the view that all shapes of graphics are able to be divided into start/goal/relay points and line/curve segments. Each line/curve is divided into eight parts to be linked to a musical scale as showed in Fig. 1 [1]. When a user successfully traces a line/curve, continuous musical scale sound is played depending on the finger position. Proposed methods and interfaces are basic techniques for developing plug-ins which help blind people to use ordinary mass-produced computer devices with touch screens, such as smartphones and tablet computers.

Fig. 1.
figure 1

The overview of the One Octave Scale Interface.

Full size image

For improving the performance of the OOSI as the single-touch screen interface, several experiments with visually impaired people were done as showed in Fig. 2 (left) for investigating the node number effect, the stereo sound effect and the node regulation effect [2, 3]. Figure 2 (right) is the recognition results of the figures in three types of node regulation by eleven visually impaired people. In this figure, ‘S’ means the single-touch experiment, ‘B’ means the blind person, and ‘L’ means the masked low vision people. Despite these efforts of improving, the low recognition rate of curves still remained as unsolved problem. Looking at this problem from different angle, we decided to introduce the multi-touch screen in our development.

Fig. 2.
figure 2

Left: Single touch display experiment (Symbols were not displayed on the screen.), and Right: Recognition results of the single-touch experiment drawn by participants

Full size image

2.2 Improvements and Evaluation

To enrich the OOSI as a multi-touch screen interface, we thought that using multi timbre would provide useful clues to find the location of each finger on a displayed graphic for users [4]. We introduced the timbre of eight musical instruments into the OOSI’s representation of lines and curves as shown in Fig. 3(a). To evaluate the OOSI’s multi-touch function, an application was written by Cocoa for iPad, which allowed us to develop an eleven finger multi-touch application. The application consists of a training graphic in Fig. 3(b), and twelve graphics in three patterns as shown in Fig. 3(c). The size of each graphic is 10 cm × 10 cm as same as the previous experiments. Six blind people and five low vision people, who were staff members or former/recent students of braille training courses of Japan Braille Library, participated in our evaluation. People with low vision wore eye-masks, and all participants were given warm encouragement but no feedbacks about correct answers from examiners while evaluating. All figures were displayed in random order. Maximum time for examining one figure was strictly fixed at four minutes. After examining, the recognition result was presented by the participants with fingertip drawing on a paper, and was traced by examiner with a pen. The recognition results by all participants are shown in Fig. 4 (right). An example of touchng display is shown in Fig. 4 (left). By comparing the recognition results of the same figure “∩” in two experiments shown in Figs. 2 and 4, the number of the perfect matching was zero in the single-touch evaluation and four in the multi-touch evaluation. The curve recognition ratio of single/multi-touch screens is compared in Table 1.

Fig. 3.
figure 3

(a) Sound mapping of figures, (b) A figure for training and (c) Figures for evaluation in three patterns

Full size image
Fig. 4.
figure 4

Multi touch display experiment (Symbols were displayed on a iPad.), and Right: Recognition results of the multi-touch experiment drawn by participants

Full size image
Table 1. The curve recognition ratio
Full size table

2.3 The Recent Research on E-learning Application

As a result of continuing to improve OOSI based on these evaluations, we are now developing an electronic textbook system using OOSI as shown in Fig. 5. An iPad application for students was implemented and evaluated by eight blind junior high school students to solve the problem of finding the length ratio of line segments and solving the problem of seeking congruent figures. As a result, it was verified that OOSI was useful as an interface of electronic textbooks and that it could be used as a learning content substituting Braille textbooks. An application for teachers to create learning contents using OOSI on the iPad was developed and evaluated by 7 university students in the teacher training course and 4 teachers of the blind school. The evaluation consisted of the quantitative evaluation on current application design as to whether contents creation of beforehand prepared ten questions can be done without a burden, and the interview how to improve the application in order for the users to create the content freely. As a result, it was shown that the proposed application can be used as the content creation system of electronic textbook for visually impaired students. We continue to refine and implement the electronic textbook system.

3 The Design of the Collaborative Work Support Environment

3.1 The Research on the Music Application

We started this research from interviews of blind people and a teacher who developed and researched assistive applications for visually impaired students. Agendas for designing the music application from the interviews were as follows; (1) composing music without memorizing or reading musical scores is a much-needed application for visually impaired people contrary to our expectation, (2) user friendly interface without making harsh sounds helps visually impaired people to enjoy the music application with others without embarrassment. For helping intuitive understanding of composing mechanism, we employed the method of laying out tangible objects on the table. A vision sensor (Kinect) detects the positions of the objects, and the detection results are directly changed into musical score as shown in Fig. 6. As the tangible objects, daily goods and stationeries were employed. After many trials, clip objects, such as paper clips, binder clips and clothespins with AR marker, and string objects, such as yarn, ropes and chains, to help users to trace and adjust the positions of clip objects were selected for the user evaluation. Single user evaluation and two user evaluation which two male and three female blinds, aged 18 to 33, and one sighted male aged 22 participated in, has been done. The result shows that all users preferred clothespins and chains to others, and that our proposed music composing interface by laying out tangible objects on the table was effective to help users’ collaboration and communication [5].

Fig. 5.
figure 5

The application for the students (left) and the application for the teacher (right)

Full size image

As the 2nd phase system development, we made the workspace independent for each user so as to promote oral communication, as shown in Fig. 7. Two users sit on both sides of a table, create their own melody lines individually in the area A. In the area B, the dice-like shape box with six AR markers has a function to change sounds of musical instrument of the melody users composed. The six AR markers indicate high-pitched piano, low-pitched piano, guitar, drum kit, trumpet and violin respectively, and two users can select each instrument by rolling each box. In the area C, seven AR markers called “base marker” for base sounds are set on the center on the table. Seven sounds are the loop of percussion, drum, dance beat, pop beat, dance base, pop base and shaker. All base markers are laid upside down at the start. Users can add the base sounds to their music by turning over base markers. Six visually impaired and six sighted people participated in the evaluation. Figure 8 shows the appearance in the experiment. The results show that the proposed interface design helps visually impaired people sufficiently to have the initiatives of collaborative works, and that the developed collaborative application design enables the visually impaired people to enjoy composing music [6].

Fig. 6.
figure 6

A gesture and AR markers recognition result on the two users’ collaboration.

Full size image
Fig. 7.
figure 7

Outline of the improved music application

Full size image

3.2 The Research on the Shooting Game

The purpose of this research is to elucidate the collaboration mechanism by manipulating information given to both visually impaired and sighted people and observing the state of changes in collaboration between users under multiple situations. The system configuration is shown in the Fig. 9. Three iPads are connected by Bluetooth, the two controllers iPad operate like a handle to manipulate the spaceship and send the tilt value measured by the gyro sensor to the server iPad. The server iPad moves the spaceship based on the tilt value and uses the vibration presentation device in Fig. 9 [7], which was developed with the vibration type loudspeaker, to feedback the distance between the spaceship and the enemy to the user in real time. Two users cooperate, navigate one spaceship, fire a beam and defeat more enemies. The spaceship will not move smoothly unless the directions of inclining the two iPads are not matched. In order to prevent random firing of the beam, it is set so that the next beam can not be struck for 1 s after the beam is emitted. Vibrations of a magnitude proportional to the square root of the distance between the spacecraft and the enemy are fed back to the speaker on the enemy side among the vibrating loudspeakers worn by the user on the two indexed fingers. When the spaceship is right under the enemy, the vibration will be zero.

Fig. 8.
figure 8

Appearance in the experiment

Full size image
Fig. 9.
figure 9

Overview of the experimental system (Left) and the developed vibration presentation device (Right).

Full size image

Four types of evaluation contents, (Single Enemy(SE), Double Enemy(DE)) × (Double Hands(DH), Single Hands(SH)), were prepared as shown in Fig. 10. In DE case, the spaceship may not move if there is a discrepancy in decision between users. In the case of SH, since the amount of information is insufficient to grasp the position of an enemy by each user, information sharing among users is indispensable to shoot the enemy. The collaboration changes in the order of SEDH, SESH, DEDH, DESH were examined for 7 pairs of severely visually impaired people near blindness and sighted people and 3 pairs of sighted people.

Fig. 10.
figure 10

Four evaluation contents for collaboration analysis.

Full size image

As a result, it turned out that 10 pairs can be classified into the following four groups from the transition of the score and the characteristics of the conversation. There is a group in which the score of SH is higher than that of DH and the score of DE is higher than that of SE, and conversely, there is another group which decreases in both. We define the former group as successful cases in collaboration, and the latter group as unsuccessful cases in collaboration. There was a substantial difference between the number of conversations and the conversation contents of these two groups. The third group is in which two users’ individual differences in the score greatly increases at DH, and is reduced at SH. From the conversational analysis, their conversations during the game play consist of only information sharing and decision making without heart to heart communications such as consideration and encouragement. The last group is whose individual differences in the score is observed at SE, but is reduced at DE. Their total scores of two users did not grow even though they enjoyed talking a lot and playing games because of insufficient conversation about important information sharing and decision making. In all cases, individual scores did not have any differences or characteristics related to the visual impairment. The pair of sighted had less conversation compared with the pair of visually impaired, but there was no fundamental difference in collaboration.

3.3 The Recent Research on Collaboration Analysis

In order to support the cooperative work of the visually impaired people, not only the universal design interface in which visual impairment does not affect tasks, but also the interface that encourages user’s awareness and improves the quality of collaborative work are indispensable. Currently, we are exploring the design of the interface that encourages user’s awareness. It was confirmed from additional experiments that the merely increasing amount of utterance could not improve the quality of cooperation. In the experiments, two smartphone applications that encourage users to speak to cooperators were implemented, and its effect was evaluated by six pair of users whose collaboration were thought to be going to fail by pre-surveys. The applications succeeded to increase the amount of conversation of all pairs, but the quality of the dialogue was never enhanced. In the case of four of six pairs, shooting score did not grow as the amount of conversation increases. It was confirmed that merely increased conversation was unrelated to improve the cooperative work. In order to encourage users’ awareness, what kind of information should be designed to be more tangible is still pursued.

4 Related Work and Discussion

The authors started the research project of OOSI in 2008. After that, the spread of the iPhone and iPad and the realization of the morphing tactile display [8] have caused changes in the technical situation. The authors have confirmed that OOSI can more effectively support the visually impaired people’s graphic recognition on the morphing tactile display than on the flat display like iPhone, iPad. As an academic research on the speech touch interaction, reading support system with interactive audio feedback more effective than VoiceOver has been developed in 2016 [9]. A recent research of the text input technology enables visually impaired users to tap the edges of the touch panel surface for high-speed text input without audio feedback by being assigned letters to the edge of the smartphone surface [10]. In recent years, problems concerning visually impaired people ‘s touch panel gesture learning have emerged, and researches on gesture sonification and corrective verbal feedback also have been done [11]. Research to provide image information to people with visual impairment had been done by conversion of graphics into tactile sensation of a tactile display or a force feedback device. Even in touch panel devices, research on image feature sonification to provide images with vibration or sound has been conducted [12]. However, due to advances in deep learning technology, conversion of images into natural language by automatic graphic annotion has reached a practical use level now [13]. Our proposed OOSI is the support technology of the graphical recognition and understanding of diagrams and charts, which are particularly important for students in science, technology, engineering, and mathematics (STEM) fields. That is a completely different approach from the image feature sonification and the automatic graphic annotion. Attempts have been made to support the use of large touch panels for the visually impaired by using tangible gadgets as a special option [15]. It is also expected as the future work to obtain synergistic effects by combining such a gadget with the proposed OOSI. As with our approach, there are a few latest researches aiming to develop graphic recognition/understanding support technology for visually impaired children, such as applying machine vision to the tactile graphics for realizing the tactile-audio graphics [16], and converting graphics previewed on a computer screen into tangible and scalable freely with the mini refreshable hyperbraille device on the base assembly and the gesture input ring device [17]. However, there is no research including development of e-learning contents and editor for visually impaired children that can be used in popular touch panel equipment like this research. The world wants to develop technology which is easier to obtain and less expensive [18].

The mechanism of collaborative work between sighted and blind users across different modalities should be explicated [19]. But there are few researches even on collaborative work interface. One research example is that haptic virtual environments were evaluated to support the visually impaired children in an inclusive group work in school [20]. There are also researches on collaborative game applications including visually impaired people use case such as jigsaw puzzle playing [21]. There are few and few researches on collaboration analysis between sighted and visually impaired users such as a collaborative software engineering course between sighted college students and high school students with visual impairments [22]. We’ll analyze the collaboration between sighted and blind users across different modalities and propose the ideal design of the collaborative work environment.

5 Conclusion

This paper described over several years’ design and analysis research attempts to improve the touch panel usability of the visually impaired people and the collaborative work experience between the visually impaired people and the sighted. These trials can be summarized as follows: the information should be more and more tangible because the visually impaired people and the sighted people can make their daily life and social life better together. Future efforts will be expected for bringing the progress in this field.