1 Introduction

Cognitive impairment is an inclusive term used to describe deficits in intellectual functioning, and entails limitations in specific operations, such as conceptualizing, planning, and sequencing thoughts and actions, remembering, interpreting social cues, and understanding numbers and symbols [1]. Although the specific functionality limitations may vary from profound mental retardation with minimal functioning to mild impairment with difficulties in specific operations, a common problem among people with cognitive impairments is self-management and carrying out everyday tasks, such as bathing, eating, preparing simple meals, etc. To this end, cognitive technologies have the potential to help individuals with cognitive disabilities to be more independent and enhance their quality of life [1].

However, accessing a computer and using a typical GUI can be a challenging task for users with cognitive disability. For example, simple interface actions such as double-clicking, using scroll bars, or reading menu items and button labels can present barriers [2]. In this respect, alternative interaction means and multimodal input may benefit cognitive impaired users, while multisensory output may alleviate difficulties in comprehending written text.

This paper presents three systems that have been designed to train children with cognitive disabilities in learning the home environment, money and monetary transactions, as well as preparing simple meals. The systems support multimodal interaction and multimedia output and are intended to be used by trainers of the Rehabilitation Centre for Children with Disabilities in Heraklion (Crete, Greece) to train children with disabilities in the aforementioned domains.

The remaining of this paper is structured as follows: Sect. 2 discusses related work, Sect. 3 introduces the three systems and the process that was followed for their design and implementation, while Sect. 4 describes the requirements elicitation phase that preceded the design of each system. Sections 5, 6 and 7 introduce each one of the three systems. The evaluation that has been carried out for one of the systems is described in Sect. 8. Finally, the paper concludes with discussion and future work in Sect. 9.

2 Related Work

Technology has been claimed to improve performance and independence of individuals with disabilities in comparison to more traditional means, such as pictorial prompts, tactile prompts or auditory prompts [3, 4]. Common assistive technologies include software for reminding and prompting, task guidance, way-finding assistance, remote assistance from caregivers, computer assisted learning and communication [1, 2, 5].

Meal preparation is an important skill to acquire for individuals with cognitive disabilities in order to increase their independence. Several traditional and technological approaches have emerged, while comparative studies have also been carried out. Lancioni et al. [6] introduce a palm-top computer involving pictorial instructions as well as auditory and vibratory mechanisms for preparing four different meals, compare it with a card system and conclude that participants had higher performance with the computer system and preferred it. Mechling et al. [7] studied the use of a self-prompting PDA system using video, picture and auditory prompts to instruct individuals with autism for preparing three cooking recipes and found that it can be an effective assistive medium for performing multi-step tasks. Giroux et al. [8] describe a smart home environment aiming to assist individuals with cognitive impairments in cooking tasks, by presenting the recipe steps through pictures or video and assisting them in locating objects or ingredients in the kitchen.

Money management and purchasing skills constitute another important subject to teach to individuals with cognitive impairments to sustain their autonomy. Lagnone et al. [9] introduce a multimedia environment employing video-based instruction and an interactive CD-ROM providing students with a virtual shopping experience. Several studies have been carried out [e.g., 10, 11] examining the use of the aforementioned computer-based program by students with intellectual disabilities in order to learn fundamental shopping skills, suggesting that computer-based video models can be used to effectively teach functional skills. A suite of serious games has been proposed by Lanyi and Brown [12], including a virtual supermarket to teach students money management skills, where students have to buy virtual products based on a given shopping list and pay using their virtual wallet.

On the other hand, learning the home environment including objects, activities and behavioral rules is a topic that has not been extensively addressed through computer-based media.

The three proposed systems aim at teaching through playing the aforementioned topics which are considered important for daily activities and independent living of individuals with intellectual disabilities. To achieve their objectives the systems employ novel multimodal interaction techniques and deliver multimedia output.

3 The Three Multimodal Interactive Systems

3.1 Overview

Based on the requirements specified by the employees of the Rehabilitation Centre, learning the home environment, learning about money and monetary transactions, and learning how to cook simple meals were selected as fundamental steps in order to enhance the autonomy of children with cognitive impairments and improve their everyday living.

Each one of the three systems has been analysed separately in order to define its main objectives and specific functionality. Furthermore, each system employs a different setup (see Fig. 1) and supports interaction through various input modalities (see Table 1). In more details, the three interactive systems were defined as follows:

  • Learning the home environment (The “Home” game). The system includes a variety of exercises and mini-games aiming to instill which are the main house rooms, which are the most important objects one can find in each room, as well as what activities are usually carried out in each room. Furthermore, the system instructs children regarding dangerous behaviors that should be avoided, as well as good behavior patterns for everyday activities at home, through multiple-choice mini quizzes. Interaction with the system is possible through touch via the provided touch screen, or through printed cards representing objects and activities placed on a large board depicting specific rooms. The system features a touch screen, computer, and a high resolution camera overlooking the board (see Fig. 1a).

  • Learning about money and monetary transactions (The “Money” game). The system’s objectives are to teach children the coins and bills of Euro as well as their value, assist them in learning to identify euros and cents for a given amount, support familiarization with monetary transactions through virtual shopping and promote good behavior regarding shopping and monetary transactions in general. Interaction with the system is achieved through mouse and printed cards (representing coins and bills) placed on a tabletop surface [13]. The system setup consists of a laptop and a high resolution camera (see Fig. 1b)

  • Learning how to cook simple meals (The “Cooking” game). The system includes a variety of exercises and mini games, aiming to instruct children: (i) which foods are appropriate for breakfast, lunch, and dinner, (ii) how to cook simple meals (e.g., bread with butter and honey, lettuce salad, pasta with tomato sauce, etc.), and (iii) fundamental rules of safety and hygiene that should be applied during food preparation. Interaction with the system is achieved through printed cards representing various dishes, recipe ingredients and kitchen utensils, as well as through a custom-made IR pointing device (see Sect. 3.2). The system consists of a computer, a high-resolution projector, a simple wooden table, an RGB-D depth camera, a high-resolution camera, speakers and the custom IR input device [14]. The system will be installed in the actual kitchen of the Rehabilitation Centre. With the aim to support an immersive user experience, the hardware equipment will be hidden inside a kitchen board, leaving visible only the plain wooden table. (see Fig. 1c)

Fig. 1.
figure 1

Representation of the system setups: (a) Learning the home environment (b) Learning about money and monetary transactions (c) Learning how to cook

Full size image
Table 1. Input modalities for each one of the three interactive systems
Full size table

One key characteristics of the aforementioned interactive systems is the ability to frequently update their content to be harmonized with the current educational activities of the Rehabilitation Center. To this purpose, a full-scale content management tool has been developed through which the trainers will be able to update: (i) the “Home” game by adding new or modifying existing rooms, objects and activities, and determining their associations (e.g., object X belongs in room Y, etc.), (ii) the “Money” game, by editing the available products for purchase (e.g., add a new product, change a product’s price, create a new shopping list, etc.), (iii) the “Cooking” game by adding new ingredients, tools, utensils, or even recipes along with detailed directions about their preparation and (iv) all the quizzes used in the games.

Furthermore, with the aim to support user management, the Student Profiler application has been implemented, allowing trainers to register new students, update or delete existing ones, and also to generate student identification cards. Student identification cards are used by the system to identify the current player and adapt each system according to the user’s interaction preferences (e.g., provision of audio prompts, use of symbols along with text, etc.).

3.2 The Custom IR Device

The interaction paradigm of the cooking game could be summarized as follows: a student firstly collects the recipe’s ingredients by laying the respective physical cards on the table’s surface, and then mixes them by interacting with their virtual counterparts. Clearly such process poses some unique requirements: during its first half the interaction mimics the physical actions that someone would perform in a real kitchen, while during its second half the user has to interact in a manner that would not disrupt the natural flow of the whole process.

To that end, a custom IR 3D printed stamp that supports wireless charging (Fig. 2) has been designed, through which students can “tap” on virtual interactive elements. Internally it features a pressure-sensitive bottom which, when pressed, lights up the infrared LED mounted on its topside. Whenever that happens, the computer vision subsystem that monitors the table’s surface through the high-resolution camera (Fig. 1c) propagates an artificial mouse event back to the main game [15]. However, the innovative feature of the stamp is its physical design rather than its functionality, as similar devices can be found in the market.

Fig. 2.
figure 2

A 3D reconstruction of the stamp and its different handles

Full size image

The stamp’s design employs many of Norman’s rules for designing good and usable everyday things [16] in order to accommodate the special needs of the target user group. In terms of materials, the stamp’s main body is made of lightweight plastic, while its bottom side is covered with rubber to satisfy a threefold purpose: (a) increased friction to discourage dragging, (b) minimized noise generation when “hitting” the desk and (c) increased durability. Regaring appearance, its exterior shape features a “pointy” design to implicitly guide the user where to direct the light, while the embedded handle contributes to its stamp-like look and enhances the natural relationship that couples function and control: while holding it a user can only “stamp” interactive elements. Three different handles were built to accommodate different grip sizes (i.e., small, medium, large), while a fourth option is available for students with motor disabilities where the stamp can be strapped on the student’s fist.

3.3 The Design Process

The design and implementation of the three interactive systems has followed an iterative approach, involving multiple evaluations, as shown in (Fig. 3). In more details, for each one of the three systems, functional and non-functional requirements were specified with the collaboration of the Rehabilitation Centre’s personnel (see Sect. 4). The requirements analysis and documentation was followed by an evaluation of the requirements specification by the Rehabilitation Centre experts who will be using the systems together with the children. A detailed design was then produced for each system, which was again evaluated by the Rehabilitation Centre experts. Following, a preliminary prototype was implemented for each system, featuring at least half of the provided functionality. Next, an evaluation experiment involving children of the Rehabilitation Centre will be conducted, with the aim to assess the produced designs as well as the various interaction modalities and system setups. Finally, based on the results of the evaluation, each system will be fully implemented and installed at the Centre for pilot use and evaluation.

Fig. 3.
figure 3

The iterative design process of the three interactive systems

Full size image

Currently, the following phases have been competed for each one of the three systems: requirements elicitation and analysis, mockups’ design, as well as demo implementation. An evaluation experiment has been conducted for the “Cooking” game (see Sect. 8), while the experiments for the remaining two systems are being planned. All three systems are expected to be installed at the Rehabilitation Centre for pilot use within the next six months.

4 Requirements Elicitation

Requirements elicitation has been carried out through questionnaire and interviews with the Rehabilitation Centre’s personnel in order to identify the characteristics and needs of the target users (children with cognitive impairments visiting the Centre for rehabilitation). Furthermore, the detailed functionality of each system has been analyzed through focus groups with the participation of the Rehabilitation Centre’s personnel, as well as system designers and developers. The Rehabilitation Centre participants have professional expertise in various domains, including speech therapist, psychologists, occupational therapists, nursery staff, pediatrician, and a health visitor.

In summary, the users of the interactive systems were defined as children with cognitive age of at least three years old, with a variety of characteristics and skills. More specifically:

  • Children’s physical age ranges from preschoolers (3-4 years old) to teenagers, with the corresponding variations in physique.

  • Reading skills may vary from minimum to excellent. Children with deteriorated skills may recognize short phrases, individual words, texts read aloud, images and symbols.

  • All children can successfully understand when text is read aloud, especially if the reading rate is appropriate and the text language is simple.

  • Upper limb mobility varies from no use of upper limbs at all, to difficulties in separating fingers, to satisfactory gross motor skills, or sufficient mobility.

  • The most common visual problems include color blindness and difficulties in reading small size fonts.

  • Eye-hand coordination problems are common.

  • Children exhibit a variety of individual traits, such as placing objects in mouth, spitting, and carrying out involuntary movements.

As a result, each system should be able to accommodate a large variety of skills, by: (i) supporting information presentation through text, audio and images, concurrently available if needed, (ii) allowing customization of font size, (iii) employing a minimalistic approach towards graphic design, (iv) providing context-sensitive help in each individual exercise, and (v) supporting two levels of content, a simple one for students with severe cognitive problems and a more advanced one, allowing teachers to switch to any of the two levels at runtime.

5 Learning the Home Environment

The system includes nine exercises, namely: (1) Presentation of the primary house rooms (kitchen, living room, dining room, master bedroom, child’s bedroom, bathroom) and secondary house areas (storage room, corridor, balcony, terrace, yard, garage). Users can receive information about a room as well as the most common objects located in it, upon its selection. (2) Locate the room, where children are asked to locate a specific room in the house. (3) Find the correct room, where children are asked to find the room to which a specific object belongs. (4) Locate an object in a specific room, where users have to indicate the most appropriate location of an object (e.g., pot) in a specific room (e.g., kitchen) (see Fig. 4a), for all the primary house rooms. (5) Find mismatches, where users have to indicate which objects should not be found in a specific room (see Fig. 4b), for all the primary house rooms. (6) Place in the correct order the steps of an activity, in which cards representing the steps of a specific activity carried out in the house (e.g., brushing teeth) are used. (7) Place in the correct order the activities of a routine, cards representing activities carried out during a specific time of the day (e.g., what do we do at night before going to sleep) are used. (8) Dangerous behaviors exercise, where users have to select the correct between two behaviors, one of which is dangerous (e.g., I ask an adult to plug a device in the power socket vs. I plug myself devices in the power socket). (9) Social behaviors exercise, where users have to select the correct behavior between two possible answers, one of which is socially inappropriate (e.g., I knock the bathroom door when there is someone else inside vs. I can open the bathroom door when there is someone else inside).

Fig. 4.
figure 4

Indicative screenshots (a) Place the objects correctly in a room (b) Indicate objects that should not be found in a specific room

Full size image

6 Learning About Money and Monetary Transactions

The system comprises seven main thematic areas, as follows: (1) Learn coins and bills, where children are: (a) presented coins and bills and taught about their value (see Fig. 5a), (b) asked to place a coin/bill in the correct location according to its value, and (c) asked to identify a specific coin/bill among three provided options (e.g., “which coin is 1 Euro?”). (2) Order coins and bills, where users are asked to order six cards representing different coins and bills, from the smallest to the largest. (3) Amount comparison, where users are asked to identify the largest (or smallest) of two amounts. (4) Amount analysis, in which users are asked to identify and separate the cents and the euros for a given amount. (5) Amount synthesis, in which users are requested to create a given amount using specific coins and bills (see Fig. 5b). (6) Shopping, where users select a specific store (i.e., super market, bakery, bookstore, toy store, clothes store, shoes store) and are asked to: (a) shop products, pay with virtual money in a virtual wallet and indicate whether they should receive any change back (b) purchase products of a given shopping list, pay with virtual money in a virtual wallet and indicate whether they should receive any change back (c) pay with virtual money the price of a given shopping basket (d) indicate which products they can buy with a given amount in their virtual wallet. (7) Find the correct behavior, where users have to select the correct answer between two options, one of which is inappropriate in a monetary transaction context (e.g., I don’t eat something I took from the shelves unless I pay for it first vs. I can eat it in the store before paying it).

Fig. 5.
figure 5

(a) Learning coins and bills (b) Amount synthesis exercise

Full size image

7 Learning How to Cook Simple Meals

The system features five individual exercises. More specifically: (1) Meal appropriateness, in which students have to select appropriate meals for a specific time of the day (morning, noon, afternoon) among several options. (2) Cooking, which aims to teach students the steps needed for preparing a simple meal. Meal recipes are presented step by step. In each step the user is asked to carry out a specific action (e.g., place the lettuce in a bowl), as shown in Fig. 6a. As soon as a step is completed, the recipe automatically advances to the next one. (3) Collecting recipe ingredients, in which users are presented with various ingredients and are asked to select only the ones that will be needed for executing the recipe. (4) Collecting the utensils needed for a recipe, in which users are presented with various kitchen utensils and are asked to select only the ones that will be needed for executing the recipe (Fig. 6b). (5) Safety and hygiene rules, where users have to select the correct answer between two options, one of which is inappropriate behavior during meal preparation (e.g., I wash my hands before cooking vs. I cook with dirty hands).

Fig. 6.
figure 6

Cooking game (a) Recipe step (b) Collect the utensils

Full size image

8 Evaluation

A first round of user-based evaluation was conducted for the “Cooking” game. A working prototype of the system was set up at the Rehabilitation Centre’s kitchen, where children are actually taught to prepare simple meals. The goal of the evaluation was primarily to examine the overall usability of the system both in respect to its physical attributes and its functional features. At the same time, it also aimed at assessing the overall user experience of children and their teachers. Since this technology is completely new to both user groups, a lot of questions were raised regarding whether the children would find it fun or stressful, distracting or engaging, boring or exciting and whether the teachers would find it conducive and supportive to their teaching strategy. These questions cannot be answered in a single evaluation session per child, therefore a long-term evaluation is planned in the near future.

The chosen evaluation method was user observation. For this purpose, two user experience and usability experts were present during each session and were situated at a distance from the child and the teacher so as not to distract the child. During each session the experts paid close attention to the way both the child and the teacher interacted with the system. A video camera recorded each session to allow further analysis. The experts did not interrupt or speak during the session, allowing for the teacher and the child to use the system freely. Prior to the experiment, the teachers were thoroughly trained on how to use the system and its interaction modes. An observation protocol form was used by the observers to easily mark down specific interaction behaviors and user preferences, as well as the performance of the system features.

A total of nine children participated in the evaluation. The children that were selected to participate had a wide range of cognitive impairments and functional limitations (from level 0-3, with 0 being severe functional limitations and 3 few functional limitations, as shown in Table 2). Five of the children were male and four female. Their parents were asked to sign a consent form. In addition, four teachers participated in the experiment, a speech therapist, a special education teacher, and two occupational therapists.

This phase of the experiment allowed drawing some general conclusions about the system and the user experience in the form of qualitative data. Overall, the system fared well with the children and the teachers. The children’s initial reaction to the system was mainly that of excitement and curiosity, but not to a degree that was distracting them from the focus of the game. The children with more severe impairments (level 0 and 1), had to be guided continuously by their teachers on how to interact with the system, as expected due to the level of their disability. The children with less severe impairments (level 2 and 3) were able to follow what was actually going on with the exercises and could use the pointer device independently. In fact, the design of the pointer device proved to be very intuitive, as all those children immediately grabbed the handle and started moving it around the table and over various elements before any instructions was given by the teacher. At the beginning of the game most children would use unnecessary force to select an element, but gradually applied less force once they realized that the device was responsive even with a gentler tap. These children were able to distinguish the interaction elements that were displayed on the table and understood right away that they had to use the device to select them. However, they did seem to miss a lot of the questions that were given verbally by the system, and as a result the teachers had to often repeat them. This is an area that has to be further investigated in order to identify the source of the problem and whether it is due to speech and voice quality (a synthetic speech engine has been used to automatically transform text to speech) or rather to the combination of visual and verbal cues creating a mental overload for the children. In addition, it was observed that the children with psychomotor impairments had more trouble to accurately point and hold the pointer device over a selection.

Table 2. Disabilities and level of functionality
Full size table

9 Conclusions and Future Work

This paper has presented three innovative interactive systems enhancing the independence of everyday activities of children with cognitive impairments, each employing a variety of modalities along with card-based interaction: mouse and cards, touch and cards, custom IR pointing device and cards. Furthermore, the systems support multimedia output, personalized according to each individual child’s needs and preferences. A preliminary usability evaluation has been conducted for one of the three systems indicating positive results and revealing a few problems with the input device and audio output. Future work will include extensive long-term evaluation of each system, as well as a comparative evaluation of the various interaction combinations.