1 Introduction

According to data from the world health organization (WHO), more than 2% of child population aged between 0 and 6 years old experiences pathological changes that can produce disorders in their development [1]. Developmental disabilities consist of a significant deviation of the natural evolution as a consequence of health or behavior events that compromise the biological, psychological, and social development of a person, estimated to affect 5–10% of children [2]. The developmental delay is a subset of developmental disabilities defined as a significant delay in two or more of the most common developmental domains (gross/fine motor, speech/language, cognition, social/personal, and activities of daily living). The main problem is that there is no specific method to cure it, so any treatment plan would have to take every child’s uniqueness into account and would have to be designed to focus on the child’s individual needs.

Usually, regarding the corresponding developmental disorder, programs focusing on different scopes are used. For example, aspects like comprehension, syntax development, and grammatical elaboration are addressed in the linguistic area; work with several stimuli (visual, acoustic, and tactile) is favoured in the cognitive area; and, in the motor area, reflexes and movement of the different parts of the body are worked [3].

Besides these traditional activities, new technologies are increasingly used to work with this group of children. Computers are a great motivational way to attract children’s attention and to train the visual function with exercises of figure recognition, color identification, and tracking paths [4]. Moreover, within children’s educational context, tangible interfaces offer significant advantages over standard graphical interfaces that depend on a keyboard or mouse [5]; tangible interfaces are more attractive for children, improve collaboration between them, and encourage them to take a more active role in the exploration of the world and its learning. Tangible interfaces are considered ideal for being included in therapeutic and learning applications [6], as they promote collaboration in a shared space, allow free movement, and make flexible the design of the physical interface of the system.

Regarding this matter, our group (the AffectiveLab at the University of Zaragoza, Spain) has developed NIKVision [7], a vision-based tangible tabletop device designed for very young children and children with special needs in which the interaction is carried out by positioning objects on the tabletop surface (Tangible Interaction), allowing children to play with the computer manipulating conventional toys. NIKVision has been tested in nurseries, schools, and special education schools, proving its usefulness when working with this kind of children [8].

Based on our experience over the past years when working with children with special needs, the objective of the work presented here is to develop tangible tabletop activities specially designed for children with developmental problems for the NIKVision tabletop.

The remainder of this paper is organized as follows. Section 2 presents the state of the art in tangible interfaces and applications aimed to work with children with developmental problems. Section 3 presents the environment in which the tangible activities have been built up, namely the tabletop NIKVision and the software KitVision. In Sect. 4, the complete process of development of the therapeutic activities is exposed, including their design, evaluation, and subsequent improvement. Section 5 presents the design and evaluation of a new activity considered very necessary by the therapists, while Sect. 6 elaborates on the changes required in KitVision software in order to be used, in an autonomous way, by the therapists. Finally, in Sect. 7, conclusions and future work are presented.

2 Background: tangible interfaces for children with developmental delays

In this section, several examples of applications and research in the area of tabletops and tangible interfaces for working with children with developmental delays are analyzed.

With regard to works that make use of tangible interfaces, LinguaBytes should be mentioned, a research programme which aims to develop an interactive and adaptive educational environment that stimulates the language and communicative skills of multiple handicapped children aged 1–4 years. Two tools have been developed during the lifetime of the project, namely E-scope and KLEED. E-scope is a tangible device aimed to children with language and communication developmental problems [9]. The prototype consists of a wooden ring-shaped toy with sensors and actuators, a computer with a wireless station and a screen. Using E-scope, the children can listen to stories or play educational games by rolling the E-scope over different pictures that trigger different stories. KLEED [10] is a modular system consisting of exercise mats that can be connected to a central console. In this case, the child interacts with the device by placing physical tagged objects over the mats in order to listen to interactive stories or to realize exercises.

The work carried out by Hunter et al. [11] presents a group of activities to teach spatial concepts and sentence construction to children aged 4–7 years, by using Siftables. Siftables are hybrid tangible-graphical user interface devices that allow wireless communication and that can detect other devices of the same kind. Two main activities can be carried out with these devices: “Make a Riddle” and “TeleStory.” In “Make a Riddle” the child has to make a riddle using three Siftables to create a sentence that will update an image in a fourth Siftable. Not all combinations have an associated image though. That way, the child will feel more motivated to create different combinations in order to obtain an image. In “TeleStory,” the child can influence the story of a cat and a dog that are traveling together. In this case, the child uses the Siftables to change the elements of the story: for example, if the child uses a Siftable that is displaying a picture of the sun, the scene of the story will become daytime. The aim of this activity is to teach vocabulary and to introduce reading to kindergarten children.

In [12], the authors present the “Magic Story Cube,” a tangible AR interactive interface for storytelling that uses a physical foldable cube. Children manipulate the cube and depending on how it is unfolded, segments of audio and 3D animation are played in a pre-defined sequence to tell a story. Also, Sylla et al. [13,14,15] have developed a tangible platform with book appearance for storytelling with children with language disorders.

The interactive fruit panel presented in [16] is a tangible interface designed to help children with communication problems and connect real objects (in this case fruits) with their graphical representation. The game consists of showing a real object or a miniature, which the child has to relate with some pictograms of the object. In this way, children can learn the required association between pictograms in order to have an alternative communication system. In addition to their communication skills, the activity also makes the children work out their concentration, visual activation, and memory.

Finally, Reactable is a tangible tabletop used by Villafuerte et al. [17] to work with autistic children in the creation of complex music pieces, in order to offer them an alternative means of communication. The interaction with Reactable is usually carried out by using special objects called pucks (such as audio-filters, controllers, and generators) but it also allows a direct interaction.

From the examples briefly presented above, it can be concluded that although there are several works that use tangible interfaces to work with children with developmental problems, the use of tabletops in this field is not very widespread. Moreover, the works that make use of tangible tabletops usually focus on working just a definite developmental delay, without offering the possibility of working with children with different problems, so that they are very difficult to adapt or modify. For example, the systems described in [9, 10, 12, 17] work, exclusively, with different language problems. Meanwhile, the systems described in [11, 16] allow to work more varied problems (spatial concepts, concentration, visual attention, memory, etc.) though it is not easy to modify the activities already created. Therefore, the objective of the work presented in this paper is to create a set of tangible tabletop activities that could be useful for children who suffer from a varied range of developmental problems, and that are easy to modify thanks to the software that has been used to create them.

In the following section, an overview of the NIKVision tabletop and the KitVision software, used for creating the different activities, are presented.

3 Environment: NIKVision tabletop and KitVision software

To develop the tangible activities, we used the NIKVision tabletop and also made use of the KitVision software that allows the easy creation of activities through a graphic assistant, and their execution on the tabletop device, through a player application.

3.1 NIKVision tabletop

NIKVision [8] is a tangible tabletop based on the physical manipulation of traditional toys over a table surface. Any toy can be used to interact with the tabletop on the condition that a printed marker (called fiducial) is attached to its base (Fig. 1).

Fig. 1
figure 1

Toys with attached fiducials

Full size image

Once the fiducials have been glued to the toys, children can manipulate them over the tabletop surface (Fig. 2a). There is active image output on the table surface, and a conventional computer monitor (Fig. 2f) adjacent to the table is also used to bring tabletop games closer to the conventional multimedia graphics approach that looks attractive and fun to little children. This last characteristic is optional, and in the actual version of the NIKVision tabletop, the monitor has been removed. Technically, NIKVision uses ReacTIVision visual recognition software [18] to track the position and orientation of the toys placed on the surface (Fig. 2c). An infrared light USB camera (Fig. 2b) captures video from underneath the table and streams it to the computer station which executes the visual recognition and game software. Active image projection on the table is provided by rear projection (Fig. 2d) through a mirror inside the table (Fig. 2e), while the audio of the activities is reproduced by the speakers (see Fig. 2g).

Fig. 2
figure 2

NIKVision tabletop

Full size image

3.2 KitVision software

KitVision is composed of two tools: a graphic assistant, to model the activities, and a player, in charge of running the created activities on the tabletop.

3.2.1 Graphic assistant

KitVision graphic assistant is a desktop application that eases the modeling of tabletop activities. The assistant uses a What You See Is What You Get (WYSIWYG) approach that allows the designers of activities to incorporate graphics, animation, and audio, placing them on the screen in the same way that they will appear on the tabletop surface later.

In order to create an activity, it is necessary to first define the following:

  • A background: the image that will appear on the tabletop surface (Fig. 3a);

    Fig. 3
    figure 3

    KitVision graphic assistant

    Full size image

  • An icon: an image that represents the activity and that will appear on the screen of the tabletop allowing the user to select the activity (Fig. 3b);

  • The areas: positioning a playing-piece in these square-shaped areas will have a specific meaning on the activity. Areas can be associated with the board, as used in board games such as Ludo or Chess, or with a physical object, as used in non-board based games such as the Domino. An area associated with the board is defined in a fixed position on the screen, and that position does not change during all the activity (Fig. 3c). However, an area associated with physical object moves (and optionally rotates) with that physical object, so its position is variable.

  • The playing-pieces: every area of the game has a set of correct playing-pieces and a set of incorrect playing pieces associated with it. As already mentioned, a playing piece is identified by the fiducial that is glued on its base, so to assign the lists of correct and incorrect pieces to each area we use the number of the fiducials (Fig. 1).

  • The feedback: graphic and/or audio elements which show the consequences of children’s actions. For example, in Fig. 3, the feedback is the face that smiles when the child situates a correct toy in the area and that turns sad when the child does not do it right (Fig. 3d).

When the activity is finished, the graphic assistant automatically generates a file that contains the information of the activity and also gathers the resources that are used (images, animation, and sounds). All these files are stored in a folder that the player accesses to display and run them on the tabletop.

3.2.2 Player

KitVision player is the application in charge of loading and running the activities. As described in the previous section, the player reads all the files stored in a specific folder in the computer system. The first thing that the player does is to create a menu with all the stored activities (Fig. 4, left).

Fig. 4
figure 4

KitVision player. Left: menu shown on the tabletop. Right: selectors

Full size image

The way to interact with the tabletop is through two special objects called ‘selectors’. One of the selectors is used to select the activity that we want to run in the tabletop, and the other allows to return to the initial menu while playing an activity (Fig. 4, right). These selectors have a special fiducial attached to their base in order to assure that they are not going to be mistaken for toys when playing the activities.

When an activity is chosen, the player loads the file, recovers the graphic and audio resources that are needed for the selected activity, and displays the activity on the tabletop surface. The player is able to load the most popular image (BMP, GIF, JPG, PNG, etc.) and audio (WAV, MP3, etc.) formats, as well as SWF (Adobe Flash animation) animation.

Once the activity has been loaded, children can play by situating the different toys on the surface, since the player is also in charge of reacting to the manipulation of the toys thanks to the events that it receives from ReacTIVision.

4 Tabletop activities for children with developmental delays

We made contact with the “Centro Base I” of the Aragonese Institute of Social Services (IASS) to show them our interactive tabletop and some previous activities developed by our group [7]. The IASS therapists confirmed the advantages they saw of working with tangible tabletop technology and expressed their interest in developing specific activities for their patients. Therefore, following this meeting, the process of analysing, selecting, and creating the activities began.

4.1 Designing the activities

The first step in this work was to decide on the set of tangible activities for NIKVision that could be useful for the most common problems treated in the “Centro Base I.” Considering the opinion and advice of the therapists, we were able to determine that the set of activities must address the following therapeutic goals: visual attention, psychomotor skills, bilateral coordination, and cognitive skills (see Table 1). A total of 6 activities were selected and developed for the environment described in the previous section. Some of them were adaptations of existing ones while other activities were new.

Table 1 Therapeutic goals of the activities
Full size table

Following, the aim of the activities, their virtual and physical elements, and their therapeutic goals are explained in detail.

4.1.1 Bees

4.1.1.1 Aim of the activity

In this activity, the tabletop surface shows an animation of a tree full of hives with several bees flying around. The animation also shows a beekeeper standing under the tree. After some seconds, the bees stop flying and each one of them disappears into a different hive. Among the bees, there is only one that carries honey. The player has to pay attention, following with the eyes the bee with honey, and place the honey pot toy under the hive in which the bee disappeared.

4.1.1.2 Virtual and physical elements

In this activity, the virtual elements are the scene, the bees that move around it and the beekeeper character that gives feedback to the user (Fig. 5, left). The honey pot toy that the player has to put under the hive is a physical tangible object (Fig. 5, right).

Fig. 5
figure 5

Bees activity. Left: background-animation shown on the tabletop surface. Right: physical honey pot toy to interact with the tabletop

Full size image
4.1.1.3 Therapeutic goals

The wide surface display allows exercising long visual tracking, helping to train the ocular muscles. Also, visual attention is considered, since the difficulty increases not only with the speed of the bee carrying the honey, but also with the growing number of distracting elements (number of bees without honey that move in the display). For those reasons, this activity is useful for children with psychomotor delays, to develop the fine motor skills of the upper limbs, and for children with an attention deficit, since only if the child is paying attention, he/she could complete the activity.

4.1.2 Pac-man

4.1.2.1 Aim of the activity

This activity consists of a maze with obstacles and pac-dots (represented by white dots), in which the pac-man must eat the pac-dots without colliding with the walls. Meanwhile, a ghost roams the maze, trying to catch the pac-man. The activity finishes when the pac-man eats all the pac-dots without being caught by the ghost.

4.1.2.2 Virtual and physical elements

The virtual elements are the maze and the pac-dots (Fig. 6, left). The physical elements are the two characters involved in the activity: the pac-man and the ghost that runs after it (Fig. 6, right).

Fig. 6
figure 6

Pac-man activity. Left: virtual maze and pac-dots. Right: tangible pac-man and ghost

Full size image
4.1.2.3 Therapeutic goals

Since the player has to move the pac-man over the surface without colliding with the obstacles, this activity allows to train fine motor skills, especially of arms and hands. Moreover, in order to eat the pac-dots and not to collide with the walls, it is better to move the pac-man continuously and slowly. Therefore, this activity is also useful to train children with attention deficit disorder since the activity requires patience to follow the path slowly and without getting off.

4.1.3 Plumber

4.1.3.1 Aim of the activity

In this activity, the tabletop surface shows an animation of a pipe composed of segments of different colors. Some of those colored segments present water leaks of the same color: red or blue. The aim of the activity is that the child places his/her hands over the segments of the pipe that have water leaks with the glove of the corresponding color to close the circuit.

4.1.3.2 Virtual and physical elements

The virtual elements are the scene with the pipes that must be covered by the user, and the plumber character that gives feedback to the user (Fig. 7, left). The physical objects are the gloves of different colors that the child has to wear on his/her left and right hands (Fig. 7, right).

Fig. 7
figure 7

Plumber activity. Left: virtual scene shown on the tabletop surface. Right: tangible colored gloves to interact with the tabletop

Full size image
4.1.3.3 Therapeutic goals

This activity helps to develop upper limb coordination skills, bilateral coordination (simultaneous use of both hands), and management of contralateral space (crossing of the body center line with both hands). Therefore, it is useful to work with children with psychomotor delay, especially to train gross motor control.

4.1.4 Fishing

4.1.4.1 Aim of the activity

In this activity, the tabletop surface shows an animation of a pond with a pier where a cat is standing. The player has to use a rod to catch the fish in the lake and give them to the cat (place them on the pier, next to the cat). The aim of the activity is to give the cat the fish that it likes. However, the cat does not eat all the fish: the ones it likes are those whose shapes appear in the pier, next to the cat.

4.1.4.2 Virtual and physical objects

The virtual elements are, again, the scene consisting of the pier and the lake, and the cat character that gives feedback to the user (Fig. 8, left). The tangible elements are the fishing rods of different length (Fig. 8, top-right) and the fish toys of different shapes and colors placed on the tabletop surface (Fig. 8, bottom-right).

Fig. 8
figure 8

Fishing activity. Left: background-animation shown on the tabletop surface. Top–bottom right: physical rod and fish toys to interact with the tabletop

Full size image
4.1.4.3 Therapeutic goals

In this activity, the most efficient posture to grasp the fishing rod is by shoulder co-contraction and smooth movement of the wrist and the fingers. This activity is useful to work with children with psychomotor delays, to train fine motor skills.

4.1.5 Silhouettes

4.1.5.1 Aim of the activity

This activity consists of a farm scene, with several animal silhouettes spread out in the environment. The player must place the toy of each animal over the corresponding silhouette. If the animal is the correct one, the sound of that animal is reproduced.

4.1.5.2 Virtual and physical elements

The virtual element in this activity is the farm scene with the silhouettes (Fig. 9, left). The tangible elements are the toys of the different animals that must be correctly placed in the silhouettes (Fig. 9, right).

Fig. 9
figure 9

Silhouettes’ activity. Left: virtual farm scene. Right: tangible animal toys

Full size image
4.1.5.3 Therapeutic goals

This activity allows to treat simple speech delay, encouraging children to imitate the sounds of the animals, improving the ability to articulate words. Also, in some cases of generalized developmental disorder, the task of association between real objects and silhouettes can help to train cognitive skills.

4.1.6 Supermarket

4.1.6.1 Aim of the activity

In this activity, the player must help the character to do the shopping. Not all the food products can be put in the shopping basket, only those whose pictograms appear in the right part of the scene. The aim of the activity is to choose all the correct food toys and to put them in the basket.

4.1.6.2 Virtual and physical elements

In this activity, the virtual element consists of the scene with the character, the pictograms of the food products, and the shopping basket (Fig. 10, left). The tangible elements are the food toys that the player must put in the virtual shopping basket (Fig. 10, right).

Fig. 10
figure 10

Supermarket activity. Left: virtual scene with the character, the shopping basket, and the food pictograms. Right: tangible food toys

Full size image
4.1.6.3 Therapeutic goals

This activity is aimed at children with generalized developmental disorders, since it is necessary to establish a relationship between the pictogram elements with the more realistic physical corresponding objects (tangible toys). Therefore, the activity is useful to work with children with cognitive delays.

4.2 Evaluating the activities

Once the activities were designed and developed, an assessment with the final users was carried out in order to detect usability problems and to verify the degree of their therapeutic interest. For this purpose, a NIKVision tabletop was installed in the “Centro Base I,” in Zaragoza (Spain). The evaluation sessions were conducted over the course of 1 week, during the therapist consultation hours with children attending the Center.

Five therapists (one speech therapist and four psychomotor therapists) participated in these evaluation sessions. The children that used the different developed activities were selected among the patients by each therapist, considering for whom the tabletop therapy could be more useful. A total of 12 children (seven males and five females) aged between 2 and 5 years participated in the evaluation (see Table 2).

Table 2 Details of the first evaluation session of the activities
Full size table

During the sessions, that lasted an average of 20 min, direct expert observation was used. The therapist selected the more suitable activity according to the disorders of each patient and, while the child played, the expert monitored the session taking notes, writing problems, suggestions, etc. At the end of the session, a brief informal interview with both the therapist and the child was carried out, in order to know their opinion about the session and the activities played.

Each child played at least five activities, the most played being those related to the fine and gross motor skills (bees, plumber, and fishing) since this is the most common disorder treated in the center. The silhouette activity was also played by most of the patients (all except one); though in this case, it was not selected because of its therapeutic goals. Instead, it was chosen due to its ludic component since it allows children to relax after completing other more cognitive-demanding activities.

4.3 Results

After analyzing the data of the assessment sessions, some relevant issues were detected.

In the first place, regarding the use of the activities some general problems emerged. Some of them were usability problems though others were more related to the content of the activities. For example, some activities seemed to be too brief, others did not give enough variability and, in some cases, the difficulty curve between different levels was too steep. Another important problem detected was that several tangible elements were too fragile and could be easily broken. Therefore, the activities were redesigned in order to take all these issues into account (see Sect. 4.4).

Secondly, the therapists detected that none of the activities tackled severe speech delays. Although some activities like “Silhouettes” or “Supermarket” could help to treat simple speech delays, making children repeat the animal sounds or the names of the foods, these activities were not especially focused on treating speech disorders. Therefore, it was decided to create a new specific activity aimed to children with severe speech delays (see Sect. 5).

Finally, the therapists also expressed their difficulty to take notes while the patient was doing the activities with the tabletop, since during the session they also had to select the corresponding activity, give the toys to the child, explain some activities, help the child if necessary, and so on. Moreover, at the end of the session, they had to write down the results of the activities in order to follow and evaluate the progress of each patient. To solve this problem and in order to facilitate the therapists to work with the tabletop in an autonomous way, it was decided to modify the KitVision software so that it could include a new functionality, i.e., a logs management system (see Sect. 6).

4.4 Changes and improvements in the activities

The problems detected in the assessment sessions were analyzed and solved by redesigning the activities and making different changes on the virtual and physical elements. Then, a second assessment with the same methodology and the same children was carried out. The results of this new evaluation showed that the therapists were very satisfied with the new adapted activities.

4.4.1 Redesign of the activities

A thorough analysis of the results of the assessment session showed the following specific problems in the activities:

  1. 1.

    Some activities were too brief. In these cases, the activity was completed very quickly, except in the case of patients with severe cognitive disorders:

    1. (a)

      The silhouette had only one task.

    2. (b)

      The supermarket only asked for three foods.

  2. 2.

    Some activities did not offer enough variability:

    1. (a)

      The paths followed by the bees did not change, so some kids could remember the result.

    2. (b)

      In the supermarket, the kids could remember the foods asked in a precedent session.

  3. 3.

    The difficulty curve between different levels was too steep:

    1. (a)

      The bees activity had two levels (easy and difficult), with a great gap between both of them.

    2. (b)

      The cat asked for different fishes during the game, though only at the beginning, so children often used to forget the aim of the activity.

  4. 4.

    Some tangible elements were too fragile and could be easily broken.

The changes and improvements made for each activity in order to address the previous issues are reflected in Table 3.

Table 3 Improvements made in the activities
Full size table

Since one of the problems detected in most of the activities was the fragility of the toys, after studying different alternatives it was decided to make new ones using wood and vinyl. The new toys are more robust (see Fig. 11), so that the therapists do not need to be controlling the children all the time to prevent them from breaking the toys.

Fig. 11
figure 11

New toys for the activities: a bees, b pac-man, c fishing, d silhouettes, e supermarket

Full size image

4.4.2 Assessment of the improved activities

After finishing the redesign and implementation of the activities, new sessions to assess the changes and improvements were conducted in the center. In the assessment sessions, 8 children (4 male and 4 female) aged between 2 and 5 (see Table 4) participated and 4 therapists (1 speech therapist and 3 psychomotor therapists) attended them.

Table 4 Details of the second evaluation session of the improved activities
Full size table

The activities taken into account in these sessions can be seen in Table 3 with the exception of “Plumber”, which was not modified. We made sure to test the activities that had suffered more changes, i.e., silhouettes and the different versions of supermarket.

Regarding the results of this new assessment, the psychomotor therapists were very satisfied with the improvements introduced in the activities, especially with the new versions of the supermarket, the robustness of the new toys, and with the inclusion of a button that allows to repeat the instructions in the fishing game. All therapists showed their interest in having a NIKVision tabletop permanently installed in their installations.

5 Creation and assessment of a new activity: speech therapy

The results of the first evaluation carried out in the Center revealed that, among the initial six activities, there was not any that directly aimed to work language disorders. Therefore, it was decided to design and implement a seventh activity that handled this particular issue.

Speech-language therapy addresses the evaluation and treatment of language, voice, and communication disorders. It focuses on improving oral and written communication of the patients that suffer from hearing, phonatory or cognitive impairments that cause difficulties in the acquisition and development of the language.

In order to create a new activity for treating this disorder, a study of other existent applications was carried out. On the one hand, in the context of tactile interaction applied to speech therapy, there are several applications dedicated to the treatment of different language impairments. On the other hand, when working with tangible interaction, the most common method of therapy used is through “storytelling,” the interactive art of using words and actions to show images and elements of a story encouraging the listener to use his/her imagination [19]. The main characteristics of storytelling are as follows:

  1. 1.

    It is interactive;

  2. 2.

    It makes use of words;

  3. 3.

    It makes use of actions such as vocalization, physical movements, and/or gestures;

  4. 4.

    It presents a story;

  5. 5.

    It encourages listeners to stimulate their imagination.

In the context of tangible interfaces applied to storytelling, some applications have been already mentioned in the state of the art presented in Sect. 2, and they set the basis of our work. Concretely, in the works of Cristina Sylla et al., “TOK” [13] is a tangible platform with book appearance, in which the kids can create their own stories by using drawing cards, and putting them in order to establish sequences or tales. Also, “t-books” [14] is a TOK evolution, consisting of an electronic book with slots and a set of drawing cards that children can put in the book for interacting and exploring the narrative. According to the cards selected by the children, different animations and audio are executed in the computer. Lastly, “t-words” [15] (“tangible words”) is an interface setup by rectangular blocks in which the children can record audio and can later listen to it. The blocks can be connected together, in order to reproduce the audio sequentially. In this way, the rearrangement of the blocks allows to reorder the audio that was previously recorded.

After analyzing these works and verifying their similitude with the way children manipulate physical objects when interacting with NIKVision, it was decided to create a tangible tabletop activity based on storytelling.

5.1 Design of the activity

The storytelling activity that was designed is explained in the following subsection.

5.1.1 Aim of the activity

The story chosen for the activity was “The Wonderful Wizard of Oz.” A set of sentences related to the story are presented to the child. In the sentences, the prepositions are missing. The child has to choose the preposition that completes the sentence correctly. If the child chooses a correct preposition, an image that represents what it is said in the sentence is showed as prize. If the child completes different phrases correctly, the images that appear as prize are different, so even if there is just one story the child can play the activity several times by completing different sentences in order to see all the different images of the story.

5.1.2 Virtual and physical elements

The virtual elements are the sentences that the child has to complete and the images that are unlocked when the sentences are correctly completed. The physical elements are tokens with the name of the prepositions.

5.1.3 Therapeutic goals

The activity was aimed to be used complementarily to speech therapy sessions. Its main purpose is to help the child understand how to use the most common prepositions. It was discarded that children could create their own stories from scratch since the activity turns very complex and its therapeutic goal was not clearly defined.

In order to choose the prepositions, we consulted with the speech therapists the prepositions used most commonly with their patients. At the end, five prepositions were chosen: “A” (English translation: TO, direction), “DE” (English translation: FROM, OF), “CON” (English translation: WITH), “PARA” (English translation: TO, finality), and “EN” (English translation: IN).

After choosing the prepositions, the toys were made: cards with the preposition name on it, together with an explanatory drawing, so that the youngest children or the children who were not able to read could also play the activity. We decided to use the pictograms of ARASAAC (Aragonese Portal of Augmentative and Alternative Communication) [20] (Fig. 12), since they are mainly used when working with children with special needs.

Fig. 12
figure 12

ARASAAC pictograms of the prepositions “to (direction)”, “from/of”, “with”, “to (finality)” and “in”

Full size image

A second meeting with the speech therapists was carried out to show them a prototype of the activity and the images that were going to appear in the cards. The therapists revised the vocabulary that was used in the sentences to assure that the children were going to understand them, together with the use of the prepositions, so it was clear what preposition had to be used in every case. The speech therapists expressed their satisfaction with the activity, though they commented that the pictograms of ARASAAC could be a little confusing for the children. For that reason, the yellow cages were replaced by the drawing of a relation between an animated character and different objects, in order to represent the corresponding preposition. The character chosen was a dog that, on purpose, has nothing to do with the story so that the preposition cards could be used in future storytelling activities. The final design of the activity and the cards can be seen in Fig. 13.

Fig. 13
figure 13

Storytelling activity. Left: virtual scene. Right: tangible preposition cards

Full size image

5.2 Evaluation of the activity

When the new activity was implemented, an initial evaluation was carried out to detect problems and to suggest improvements. The methodology of the evaluation of this new activity was the same as the one applied in the assessment of the six previous activities, though with less patients. Two sessions of 20 min each were carried out in the “Centro Base I” with two children aged 4 and 6, respectively. This time, only two speech therapists were present.

5.3 Results of the initial evaluation

In the initial evaluation, children did not understand the game well; however, once the speech therapists explained the use of the prepositions to them (with the help of the drawings in the cards) and after reading the sentences aloud, the children began to understand the mechanics of the activity and could complete the sentences.

When we asked the speech therapists for their opinion after this evaluation, they argued that the activity was very useful as a complement for their therapy sessions and they also suggested two improvements:

  • In the audio that sounds when a child completes a sentence, the tone of voice when spelling the preposition should sound louder than the rest of the sentence to make the child focus his/her attention on it;

  • The possibility of repeating several times the same sentence should be added.

To make the first improvement, we simply recorded the audio again putting special emphasis in repeating the prepositions with a louder voice. Regarding the second change, the activity was modified by adding a “back” button that allows to return to the previous taken decision to re-do the sentence or to choose a different one. The flow of the activity can be seen in Fig. 14.

Fig. 14
figure 14

Flow of the storytelling activity

Full size image

5.4 Results of the final evaluation

After modifying the storytelling activity, we carried out a new set of sessions with the speech therapists. This time, eight children of ages between 2 and 6 years participated, all of them with speech disorders.

As in the initial evaluation, 4 of the 8 children that participated in the session needed the help of the card drawings to understand the prepositions. The other four children easily understood the prepositions when the speech therapists read them aloud. Once the activity began, the children understood rather quickly the mechanics of the game. In fact, some of the children repeated the audio that sounded after completing a sentence correctly even if the therapists did not give them any indication to do that.

The speech therapists made good use of the “back” button return to the previous decision, so that the child completed a different sentence or to complete the same sentence in case of having done it wrong. In this new evaluation, no other additional problems were detected.

6 KitVision new functionality: logs management system

From the initial assessment experience with the therapists, the requirements for the log management system were defined, which lead to some modifications and improvements that had to be done in KitVision software. They are described in the following subsection.

6.1 New requirements

A log file, with the information collected during the development of each activity, has to be stored by the system. Each log file will have the “.log” extension and must include the following data:

  • Name of the activity;

  • Date (“dd/mm/yyyy”);

  • Time (“hh/mm/ss”);

  • Time played (in s);

  • Number of correct actions;

  • Number of wrong actions;

  • Whether the activity was completed or not.

To store the information of the child identity, it was contemplated to create an application that allowed the user to introduce his/her personal data before beginning to use the activities or to create a menu similar to the activities menu though this time with users. However, after analyzing the options with the therapists, it was decided that the simplest option was to include in the software the possibility to record speech audio.

This way, the therapists would be able to record the data of the child that they consider more relevant but also to keep any other kind of information that cannot be stored in the text logs (e.g., what activity the child likes most). We also decided to limit the fact that the recording of the audio could just be done through the menu, since recording during the execution of the activity could affect its playability.

Another consideration to bear in mind is how to make the logs (text and audio) available to the therapists. It was decided to save both reports in a file folder corresponding to the current session. We call “session” the time during which a child is performing the activities until the tabletop is shut down. Also, it was decided that the folders stored should be compressed to save memory space in the computer.

Table 5 shows the requirements that the logs management system must fulfill.

Table 5 Logs management system requirements
Full size table

6.2 KitVision modifications

To be able to implement the requirements mentioned in the previous table, it was necessary to perform several modifications in KitVision software.

The first requirement was to decide the way to access the logs of the session (R1). As mentioned in requirements R2 and R3, the logs are stored in a folder that is eventually compressed; however, we had to determine how the therapists could obtain that information easily. After discussing the possibilities, it was decided that the simplest way to extract the information was by using a USB device, since the therapists were familiarized with it. Besides, the tabletop has in one of its sides a USB port, allowing to introduce the device to automatically copy the logs folder (R4 and R5).

Finally, to allow the possibility to record audio (R6), a microphone was installed in the NIKVision tabletop installed in the “Centro Base I.” That microphone was connected to the computer placed inside the tabletop. Regarding requirement R7, that established that the audio recording had to be controlled by the therapist, we decided to include a new tangible selector for that purpose (Fig. 15). The use of the selector is as follows: when it is placed on the tabletop surface (while being in the activities menu) audio speech begins to be recorded, and when the selector is removed (from the menu) the recording stops. At that moment, the recorded audio is stored in the corresponding logs folder (R8).

Fig. 15
figure 15

New tangible selector to record audio in NIKVision tabletop

Full size image

7 Conclusions and future work

After analyzing the use of tangible interfaces and tabletops to work with children with developmental delays, we have created a set of seven tangible activities according to the requirements and characteristics of children with this kind of problem. Thanks to the collaboration of the “Centro Base I” of the IASS, we have been able to design, implement and test the activities with therapists and children. The first evaluation session that was carried out allowed us to detect certain problems in the activities that were analyzed and solved generating new versions of them.

Moreover, the assessment reflected the necessity of developing a new kind of activity aimed to treat severe speech delays. This speech therapy activity was designed and evaluated in the Center with very good results. Finally, the assessment carried out elicited the requirement of including some improvements in KitVision software in order to offer tools for helping therapists in their sessions, for recording and storing information about the sessions, managing logs, etc.

Following the evaluations, the therapists of the Center confirmed the utility of the activities, since the children had felt very motivated to use them. Nowadays, a NIKVision tabletop is permanently installed in the Center.

In the very near future, we intend to continue working on the developed activities and to create new ones that will cover a wider range of developmental issues so they could be useful for more children. A rigorous and long-term assessment must be carried out with more kids to realize the impact of this kind of technology in the improvement of children skills. Also, until now only the therapists’ opinion was taken into account when evaluating the activities. However, we intend to direct the next evaluations to the children to know their perception of their activities besides their progress in their developmental problems.

On the other hand, the therapists of the Center will be trained in the use of the KitVision graphic assistant in order to modify the existent activities and/or create their own without depending on ICT or developers. Therefore, one of the most important problems that must be faced working with tangible technology could be addressed, i.e., the necessity of having computer or technical knowledge [21].

Finally, another future line of work includes carrying out a thorough study and analysis of usability and accessibility, of the use of the KitVision graphic assistant by the therapist to create activities [22], as well as the use of the activities and games by the final users [23] over an interactive tabletop [24].