Keywords

1 Introduction

The learning of literacy is one of the most important tasks that the deaf child has to face. Although reading and writing have the potential to provide the deaf child with an alternative mode of communication, which allows them to access a lot of information, a large proportion of deaf children never reach competent reading and writing levels [1]. This is due to unfavorable conditions for learning and communication, causing them to be in a marginal situation.

Deaf children of birth face various barriers in developing language and communication skills; they not only present difficulties in the learning process in specialized and regular institutes, but also they have difficulties in relating to society. Literacy is a means that allows people to integrate and communicate with society. For deaf people, communication is addressed through the use of different mechanisms, such as: sign language and the dactylology alphabet. The last one allows to represent the spelling of a word, which means, the letters that make up the alphabet by the hands [2]; this mechanism is used to support the teaching of literacy to deaf children.

A deaf child who has no hearing aid has the sign language as the first language, and uses the spelling of each word through the dactylology alphabet as a way of knowing the writing of a word. In addition, the non-appropriation of the dactylology alphabet (which is necessary to establish a link between sign language and spelling relations) [3] by the rest of the citizens causes a communicative barrier, bringing effects such as partial or total marginalization or exclusion of the social, labor and educational life of the children.

Nowadays, interactive systems as videogames are being used not only for entertainment purposes, but also in the educational context to promote the construction of knowledge in a didactic and interactive way, this as a support to the teaching/learning process. For this reason, this research proposes an interactive system that tries to support the teaching process of vowels to deaf children, which is composed of a videogame and an electronic hand glove. The glove allows the child a multi-sensory interaction with the videogame. thanks to being a tangible objects the electronic glove, take advantage of this benefits of constructivist learning through the use of practical experimentation with integrated computer technologies [4]. In addition, the glove (tangible object) takes advantage of the senses and the multi-modality of human interactions with the physical world, providing a multi-sensory experience, fundamental for deaf children [5].

Considering the above, this paper presents the design of the electronic hand glove that allows the child a multi-sensory interaction with the videogame. The glove was made using flex sensors. These detect the movements of children’s fingers and generate a range of values that varies depending on the flexion performed on them. Additionally, this paper presents the preliminary evaluation of the glove through a set of tests with deaf children. During these tests were considered the metrics: time taken to complete a task and number of errors. This in order to performing a quantitative analysis.

This paper is structured as follows: Sect. 2 presents a set of related works. Section 3 describes the proposed interactive system emphasizing the electronic glove. Section 4 presents the activities carried out as part of the evaluative process of the electronic glove. Finally, Sect. 5 presents a series of conclusions and future work.

2 Related Works

In [6] was proposed the development of a haptic glove with vibratory feedback, called Virtual Touch. This glove uses bending sensors to capture at all times the movement performed by the hand. Also, this uses small vibration motors that indicate to the user the contact with a virtual object. This research aims to develop a flexible glove using flex sensors that allow to capture in real time the position of each finger and provide vibrational feedback to the user. This type of sensors was selected as object of study in the present work due to the ease of implementation, the reduced cost and the accuracy that they deliver.

In [7] is presented a system that allows the interpretation of Colombian sign language, this consists of a hardware divided into three components. The first one is a glove made in cloth with five deflection sensors, one for each finger of the hand, an electronic sensor composed by a gyroscope that determines the movement of the hand in each one of the axes and three accelerometers to detect the acceleration of moving objects. The second component is a data acquisition card, in this case implemented with the Arduino Mega 2560, which is responsible for converting the analog signals into digital. Finally, the third component is the algorithm implemented for the recognition of the signals.

In [8] is presented a prototype that is based on a translator of movements of the hand through a glove, which allows to translate the dactylology alphabet, so that in this way, the deaf children can communicate with the rest of the people. The glove consists of eight flexible sensors, which vary their ohmic value when folded. These are located as follows: one in the little finger, one in the thumb and two in the remaining fingers (index, middle and ring). Each data obtained by the glove will be interpreted by a data acquisition card with USB communication. This consists of a microcontroller which will process the data for later sending them to the computer and decrypt them as a symbol representing a certain letter. The letter will be displayed in a graphical interface developed in Matlab.

In [9] is presented a project whose objective is the development of a glove composed of the following elements: (a) five flexibility sensors located in the bottom of the glove to calculate with more precision the resistance exerted by the fingers in each of the sensors, (b) an accelerometer located on the top of the wrist to avoid discomfort to the user, (c) Zigbee as a wireless data communicator; and (d) an Arduino FIO as a microcontroller located also in the upper part of the wrist. The glove elements are use to quantify the rehabilitation process of sports injuries and the sample of advances in therapeutic gymnastics therapies. For the preparation of the electronic glove [9] the standard measure of flexibility sensors on the market was taken into account, which prevented the possibility of scaling the glove size. Therefore, flexibility sensors were manuallyconstructed.

3 Proposed Interactive System

Proposed Solution.

In education deaf children face a complex process of teaching literacy, since it involves the sound-word relationship. But in this case, by omitting the sound, they must learn to extract information visually by relating the concepts to the vocabulary [10, 11].

Based on the above, an interactive system is proposed that supports the process of teaching vowels to deaf children. This system is oriented to interact with unconventional input mechanisms using an electronic hand glove, so the system involves a hardware and software component. In this way, the system will allow the child to interact in a digital as well as a real environment.

The software component refers to a videogame, which is supported in Tablet’s and presents a narrative story that takes place in a kingdom called Las Vocales (The Vowels). This kingdom is ruled by the princess Lectra and remains under the protection of five magicians. Each magician is represented by a vowel. The villain Analfabet steals the powers of the magicians. Then comes a hero who will be incarnated by the child and he will have the mission to travel to different worlds to overcome a series of challenges, and thus to recover the powers of the magicians and save the kingdom.

The hardware component refers to an electronic hand glove (see Fig. 1). The child provides input data to the videogame by using the glove. In this interaction, the child must represent each vowel of a word by the dactylology alphabet to complete the levels proposed in the videogame. This helps the child to memorize both gestural and visually the writing of the vowels in a series of words.

Fig. 1.
figure 1

Hardware and software components of the interactive system.

Full size image

Design of the electronic hand glove.

The User-Centered Design (UCD) was the approach followed for the design and development of the interactive system [12], this in order to knowing and understanding the user’s needs. Among the identified needs, it is important for the child to interact with real objects in a real environment. Therefore, an electronic glove was defined as a hardware device.

For the design of the electronic glove were used average measures of the hands of the children, this decision obeys the following reasons: first, the group of children under study has a different age range, so that the body morphology, especially the hands, varies considerably. Second, the UCD suggests that designing gloves for different user profiles (and not for an average profile) is a good practice [13], however, it was complex because of economic constraints and lack of expertise regarding the use of segmented sensors. Thus, it is considered as future work the design of a glove that fits the size of each child’s hand.

The work intends that the proposed interactive system be used in the Institute of Special Therapy of Senses (ITES) in the city of Santiago de Cali (Colombia). For this reason, the cost of implementing the hardware component should be minimal because most of its Students come from low-income backgrounds. In addition, the economic support that the institute receives from other Colombian organizations is scarce.

The measurements (length and thickness of the fingers) were obtained by an activity conducted with the children, in which each of them shaped the figure of his hand on a sheet of paper. After having the average measures, we then looked for a dressmaker who would make the glove. The glove fabric was selected considering the following criteria:

  • Flexible and adaptable to children’s hand shape.

  • Softness and comfort.

  • Avoid skin irritation due to factors such as humidity and heat.

Related works were an important input for the selection of the sensors and elements necessary in the glove elaboration. For the development of the electronic glove, hardware elements (plates, sensors) of Adafruit Industries were used, which have an adequate quality-price ratio and these meet the needs and requirements raised in the project. The programming was done using the Arduino IDE, which is compatible with the Adafruit plates. Table 1 describes the elements used in the glove specifying its name and a brief description.

Table 1. Electronic glove elements.
Full size table

At the end of the assembly process of all the mentioned elements, the design of the electronic glove was obtained as hardware component of the interactive system. Figure 2 shows the result obtained.

Fig. 2.
figure 2

Electronic glove – hardware component.

Full size image

Interaction videogame–electronic glove.

First, when the child uses the interactive system the challenge screen is displayed (on the Tablet). This screen shows the doors where the vowels are enclosed. Once the child selects a door, a brief introduction will appear on the screen where he will observe the instructions to interact with the videogame. Then, an image associated with the word is presented, and in that word the vowels involved are highlighted (with a green tone). This in order to the child begins to understand the image-word relationship and identifies the vowels. Subsequently, the videogame begins and appears different words to which the vowels are missing (see Fig. 3). Each word is associated with an image so that the child understands the meaning of it. To complete the word the child must represent the gesture of the corresponding vowel using the dactylology alphabet. Then the gesture is captured by the electronic glove and sent wirelessly to the videogame.

Fig. 3.
figure 3

Instruction interface and interaction with videogame.

Full size image

For each word the child completes correctly, the system gives a certain number of stars (from one to three), which are related to the time the child took to complete the word. The stars will allow the child to advance to a new word to complete. Star assignment is the scoring system selected due to the field observation conducted in the classroom of the children, where the stars are awarded for good behavior or for successfully achieving the objectives proposed by the teacher during the class.

The electronic glove allows the child a multi-sensory interaction with the videogame. For the one hand, the use of the proposed videogame stimulates the visual sense due to the use of images and icons, seeking to reinforce the child’s comprehension, integrate new knowledge (organizing, processing and prioritizing new or known information) and identify misconceptions through the representation of words and images. For other hand, the sense of touch is stimulated directly and indirectly. In a direct way, when the child interacts only with the videogame because child must press the vowels that are presented in the screen of the Tablet to complete the different levels. In a indirect way, through the use of the electronic glove because the child reacts to stimuli that mainly include the contact. The proposed interactive system tries to reinforce the gestural abilities of the child, since in order to complete the levels he must use the dactylology alphabet to represent the vowels. This stimulates the appropriation of the dactylology alphabet.

4 Evaluation of the Proposal

This section presents information related to the evaluation process of the electronic glove. For this, we analyzed the results obtained from the interactions of the children with the interactive system considering a set of metrics.

4.1 Definition and Selection of Metrics

To carry out the analysis of results, a set of metrics were defined to objectively measure the results obtained from the conducted evaluations. Thus, the selected metrics are: number of errors and time taken to complete a task. These metrics were selected because they are the measures with the most relevance to the moment that the children interact with the videogame. These metrics provide vital information for the validation of the interactive system. Thus, less time taken to complete a task and fewer errors may indicate an effective understanding of the image-word relationship by the child. Table 2 presents the description of the selected metrics.

Table 2. Metrics description.
Full size table

The selected metrics correspond to basic measures according to the theory of measurement, this indicates that they do not depend on any other measure and whose form of measurement is a method of measurement [14]. On the other hand, the number of metric errors is associated with an absolute scale type [15] since there is only one possible way to measure: counting; while the metric time used to complete the activity is associated with a type of ratio scale, which has a fixed reference point: zero.

At the time of the measurement process, the values of the metrics are not between 0 and 1 (exceed 1), so a standardization table must be used to scale them to values between 0 and 1. After normalizing the values, the metrics generate a real number that is comprised in the interval [0, 1]. Thus, the metrics provide positive evidence if the values are close to 0 and negative evidence if they are close to 1.

4.2 Users Evaluations

Conditions for the evaluation. The evaluation of the interactive system was subject to the following conditions.

  • The tests were done in the classroom. During the tests with the children it was suggested to them to be at rest in order to avoid distractions and difficulties when interacting with the system.

  • Test sessions were recorded using two mobile devices.

  • In tests the devices used are two 7-inch tablets, a Nexus 7 second generation and a Samsung Galaxy Tab 4.

The tests were carried out with the first grade children of the Cali San Fernando ITES Lions Club (Cali, Colombia) and with the accompaniment of the teacher. In each test with the children were taken measures associated with the metrics; Also, as part of the observation process during the tests, annotations were made regarding the children’s comprehension capacity when interacting with the system.

Based on the aforementioned process, once the measurements were made, the normalization of the measures associated with the metrics E and T. The following section presents the results obtained from the evaluation of the interactive system. The following sections present the analysis of the vowel A world due to document extension constraints.

Evaluation of the interactive system.

In the evaluation process a series of abbreviations are defined for the normalization of the metrics. The abbreviations used are: word 1 level 3 (P1 N3), word 2 level 3 (P2 N3), word 3 level 3 (P3 N3). In addition, Tn represents the normalized value of the metric T, En represents the normalized value of the metric E and Ev, represents each of the evaluations performed. Three children in the first grade participated in this evaluation.

The tests performed with the electronic glove reveal that the child works two channels of input to capture the information through the senses, visual and gestural. This serves as a support for memorizing the respective vowels more quickly, as a visual recognition is performed to complete the word accompanied by the sign, which makes use of the dactylology alphabet. Next, Tables 3 and 4 present the values of the metrics T and E for the evaluations of the interactive system.

Table 3. Measures of T metrics.
Full size table
Table 4. Measures of E metrics.
Full size table

Based on Tables 3 and 4, despite the small size of the user sample, the results obtained show a considerable decrease in the metrics T and E. The results obtained are a consequence of the previous interaction of the children directly with the video game. Thus, completing the words through the electronic glove makes it easier for the children, because being familiar with the dactylology alphabet, the time of action to represent the vowel and the number of errors are significantly reduced.

Tables 3 and 4 highlight at each level the highest (red) and lowest (green) registers in the three evaluations performed, based on Table 2, the values that are close to 1 and register Under values close to 0. According to the above, it is observed that child 2 presented the lowest register (0.18), indicating that he understood the contents present in the video game and also how to interact with the electronic glove. On the other hand, it is observed that child 3 presented the highest register (0.79), which indicates that it is best to better out the present information in the video game and adjust the response times obtained by each of the Flex sensors in the Electronic glove, since all children do not have the same agility or the same size of hand to make the representation of each of the vowels in the dactylology alphabet.

One factor that predominates in the difference of averages is the learning process, which is unique in each child and varies from one to another. In the case of child 1, his/her learning process has been constant during the three years he/she has been in the institution, being present from the first school level (pre-school, according to article 11, of the General Law of Colombian Education) until First grade of elementary school, while child 3 only takes one year in the institution and its learning process is beginning. In this sense, it is important to generate stimuli at a young age, since children establish the bases for cognitive, emotional and social development [16]. In addition, it rapidly increases the frequency with which children use signs to refer to objects and actions [17].

In the development of tests, the electronic glove influenced the achievement of high values regarding the metric T, this because the children had no previous experience with a physical device similar to this. For this reason, in the initial stage of the tests the children focused their attention on observing the glove, and not performing the activities suggested in the video game. However, it should be mentioned that aesthetics and ergonomics play an important role since the glove should be pleasing to the sight of the child, comfortable and soft for the child’s hand. It should be avoided so much that the fingers touch the sensors as to leave elements (sensors, cables) in sight, because the children think that the glove could cause them damage.

In the course of the evaluations, it was possible to observe the excellent manipulation of the glove on the part of the children, with which the interaction with the video game and completing the tasks was satisfactory with respect to the metric E. The children had no major complications at the time of Represent the vowels through the glove. However, in some cases confusion occurred regarding the representation of vowels A and E. This confusion is estimated to correspond to the image - word relationship and not to the representation of vowels through the dactylology alphabet. Finally, Fig. 4 indicates that the values of the metric T decrease as the evaluations pass. These results suggest that children, based on the use of the proposed interactive system, achieve an adequate interpretation of the relationship: vocal (represented in dactylology alphabet) - image.

Fig. 4.
figure 4

Average of the metric T referring to the world of vowel A, hardware - software component.

Full size image

On the other hand, based on Table 4 and Fig. 5, it can be seen that the values for the metric E present in 8 of the 9 evaluations the same value, since an average of approximately 0.33 was obtained with a minimum amount of 0 errors and a maximum of 2. This indicates that the previous interaction of the child with the software allowed him/her to understand the image-word relationship, in addition, the use of the alphabet as a tool to complete the activities, the results, due to the excellent mastery that the children have of the dactylology alphabet. For this gives children not only the communication with the environment, but themselves, this being, from a Vygotsky perspective, an indicator of the internalization of the social environment, conducive to constitute the internal language and the child’s thinking [18]. Finally, the correct functioning of each of the components of the electronic glove results in a better experience between the child and the electronic glove, which encourages the children’s motivation for learning and continue to explore this element.

Fig. 5.
figure 5

Average of the metric E referring to the world of vowel A, hardware - software component.

Full size image

However, the average value of the metric E for each of the evaluated words presents an invariable behavior (0.33), which indicates that the evaluated children were able to correctly interpret the vocal relation (represented in the dactylology alphabet) - image, for each one of the words presented regardless of whether they changed and also managed to interpret the mentioned vowels through the electronic glove. On the other hand, the obtained results can be associated to the proximity that the children have with the words used, since previous to the evaluation with the electronic glove, direct tests with the videogame were realized; so, the children had prior knowledge of the words to complete.

Finally, about the ergonomics of the electronic glove, the evaluations allowed to detect some problems during its use. On the one hand, the position of the Flex sensors and their cables at certain times did not adapt to the constant manipulation of the children and yielded to sudden movements of the hand. On the other hand, the position in the upper part of the battery glove, the microcontroller and the Bluetooth module, did not give an adequate freedom for the representation of the vowels, mainly due to the weight that these objects generate in the children’s hands.

5 Conclusions and Future Work

Conclusions.

Through the present work it was possible to obtain an interactive system conformed by a software component and a hardware component. The software component refers to a videogame in which its basic mechanics consists of completing words associated with images by using the vowels. The hardware component is an electronic hand glove for the child to interact with the videogame representing the vowels using the dactylology alphabet to complete a series of words.

This work tries to positively impact society, generating greater inclusion to deaf children. Considering the potential that can be obtained by using an electronic glove as a support for the process of teaching literacy and reinforcing the dactylology alphabet, it could be used to acquire and consolidate knowledge in a non-traditional way generating a better experience and greater motivation for the child. The use of the electronic glove allows children constantly practice the representation of the vowels of the dactylology alphabet. Also, children can reinforce the style of visual learning as they must relate an image to a word.

With the present project an interactive system was developed that contributed to satisfy the needs identified in the tarjet audience. The project allowed the children to interact with the videogame in a non-traditional way (using the electronic glove). It is important to mention that several incidents occurred while the children were using the glove. Although it caught the attention and curiosity of children, it presented drawbacks when it was used, specifically with the Flex sensors and the cables used in the connections. The cables yielded repeatedly due to the manipulation of the children, causing interruptions in the accomplishment of the activities of completing the words. The cause of the problem relates to the selected cables because they did not provide the required resistance.

Regarding to the development of the electronic glove, the activity with greater complexity was to connect and calibrate the installed sensors, because when the sensors were tested as a whole the values obtained presented a high number of variations compared to the individual tests. This complicated the process of representation of the vowels using the dactylology alphabet.

Future Work.

It is desirable to refine elements of the videogame. We hope to improve the designs of the characters, scenarios, images associated with the words used, as well as increase the number of words that make up each world of a vowel.

Regarding to electronic glove, we hope to include improvements in glove making materials for greater ease of use, strength and ergonomics. It is convenient to incorporate a system that allows to adjust the glove to different sizes of hands so that it can be used by a wider group of users. In addition, we will review other technological elements such as: more advanced sensors, accelerometers or gyroscopes, among others, that can be added to the glove. This in order to detect a greater number of characters of the dactylology alphabet and to achieve a greater accuracy in the representation of them.