Keywords

1 Introduction

Autism Spectrum Disorder (ASD) is a persistent behavioral and neurodevelopment disorder manifesting in children at an early age. Over the last decade, researchers have designed and evaluated numerous digital solutions (e.g. games, learning applications, tabletop applications and robotic solutions) for assisting autistic children of different age groups. Earlier studies have shown that factors like cultural background and mother language significantly affect the development of behavioral, emotional, social and communication skills of children. Despite an immense amount of research on autism, the role of culture while designing rehabilitation applications has relatively been understudied. Similarly, most of the research on ASD is limited to group of children from western countries and relatively, less attention has been paid on the children belonging to non-western countries especially from South Asian counties where the prevalence of autism is quite high [1].

Culture can be defined as a way of living for a certain group of people. Recent studies on cross cultural comparisons have shown that social behaviors are affected by religion, socio-economic situation, parent’s way of upbringing, parent’s expectations and level of education [2]. It also shows that socialization practices are influenced by culture [3]. Studies on ASD therapy have also shown that ASD interventions should take into account the customs of the individual society in order to improve behavior and social skills of individuals diagnosed with autism [4].

A variety of digital solutions have been designed for autistic children to address their difficulties with verbal and non-verbal communication and social skills. Picture Exchange Communication System (PECS) is famous example, in which pictures are used as a tool for teaching verbal and non-verbal communication skills [5]. Toby Playpad [6], Proloquo2Go, iComm, and TapToTalkis are also good examples of Speech generating devices which support communication through images and text to voice conversion features. The Let’s Face It! program has been used as a game to teach non-verbal skills including the face recognition skill to autistic children [7]. Avatars have been used in a study to enable autistic children develop understanding about human emotions and expressions. [8]. Touch Story shown [9] is computer game software that deals with development of social understanding in children with autism. Graphical Social Scenarios [10] is another system inspired from social stories and comic strip conversation. However, these applications have been designed primarily for western countries and without keeping cross-cultural differences and similarities in mind. These ‘one size fit all applications might not work well in all cultures because socio-emotional experiences are derived by cultural norms and these applications primarily address these experiences. In this paper, we present the design and of a mobile learning application that is customized based on the local cultural norms in order to improve social and emotional skills of autistic individual from Pakistan. We also present the results of the longitudinal evaluation of these applications performed at an autistic school in Pakistan.

2 Design of Our Applications

The design of our applications is based on the user research and literature review. We conducted a user study focused on observing children, interviewing teachers and parents to gather the user requirements. We observed 8 children in different situations during for about 8 weeks. We also interviewed, 8 teachers, 2 psychologists, and ‘that many’ parents. In addition, we evaluated existing mobile applications for 6 weeks, designed for improving the non-verbal communication and social skills, with a group of autistic children. However, results suggested that children in general, were not able to understand the avatars showing different emotions. In addition, stories used in the applications to depict different social situations were also found to be irrelevant. Based on the observations, feedback from parents and teachers and results of the evaluation sessions with existing applications, we designed two different simple and easy to use tablet applications, (1) Learning Emotions, and (2) Social Learning.

2.1 Learning Emotions and Expressions

Our application, Learning Emotions and Expressions consists of two games. (1) Emotion Finder and (2) Guess Expressions. ‘Emotion Finder’ is a simple game, which enables children with autism to find a particular emotion from a scene. Children were exposed to various emotions in a scene. Later, they were asked to tap/select the asked emotion. ‘Guess Expressions’ is another simple game, which uses natural and spontaneous human facial expressions, elicited in a proper experimental setting, to help autistic children in understanding these expressions. In this game, children are required to choose the correct option from the multiple choices of different expressions.

Keeping in mind the cultural context, actual facial expressions of Pakistani children and elders were used in these games. In case of emotion finder, we used culture-specific backgrounds. The goal was to improve children’s non-verbal communication skills and help them in improving their understanding of basic and complex emotions and expressions (Fig. 1).

Fig. 1.
figure 1

Learning emotions and expressions (left) and social learning (right)

Full size image

2.2 Social Learning

Social learning consists of a simple game that asks children to guess the missing link from a social scene. The design of the app is based on the concept of comic books in which different situations are described with the help of sequence of different images. In addition, the design is also inspired by the core concept of the Touch story [9]. However, the stories designed for our application are according to the cultural norms of Pakistan. Children doing mischiefs story presents a scenario of mischief performed in the cultural settings of Pakistan. Similarly, Saying hello story presents a scenario of how we greet people in Pakistan e.g. shaking hands and hugging. Moreover, it’s a common practice in Pakistan to throw garbage on the road. We designed a story (names as throwing dust story) to help children in realizing that throwing garbage on road is a bad idea.

3 User Evaluation

We designed our applications to address different difficulties faced by autistic children. Our objective was to investigate whether our applications were able to address the problems children face in everyday lives i.e. unable to understand emotions and responding intelligently in different social situations. Past research has shown that autistic children lose interest in therapy applications and do no cooperate. We wanted to see how our game based activities enhanced the engagement of autistic children? How do they socially interact with each other? How understanding about different emotions and expressions was improved? Were they able to understand and connect with different social scenarios presented in the game? Finally, we intended to measure the difference between the app and no app version to justify the need of mobile version of the activities.

Our evaluation approach was to perform one to one and group sessions with autistic children using our applications. The idea was to engage these children socially with each other while they interact with our applications. Furthermore, in order to measure if the application fulfilled the research objective, we implemented a web based feedback system for teachers. At the end of each week, we requested the teachers to give feedback about each child. We asked them about a child’s progress and behavioral adaptions after the sessions. Our approach was inspired by this study [11].

We performed the evaluation at one of the schools at Lahore, Pakistan. The age of the participants was between 8 to 12 years. The sample size of our group was 8 children. All 8 children were high functioning individuals. We designed a longitudinal evaluation plan of 2 months with these applications. We performed a session of 25 to 30 min with pairs of children. We performed a pre and post evaluation questionnaires session with all children at the start and end of evaluations. The ‘Super Skills Profile for Social Discovery’ was used at the start and end of evaluation to measure the difference for various fundamental social and behavioral skills. The teachers were requested to rate these skills on the scale of 0 being “Poor” to 6 being “Excellent”. In order to compare our apps with the non-digital intervention, we designed paper-based apps, equivalent to that of tablet apps. We printed expressions and stories on the cards.

We began to evaluate our applications at a school in Lahore, Pakistan from Monday afternoon starting at the mid of August till the end of second week of October. We conducted activities with a pair of two children on daily basis. Each pair had to work with evaluator and teachers; evaluator was required to conduct a session and we requested teachers to record videos. The role of the researcher was to help individuals understand the applications and to support them performing the activity when required. The children were asked to play the Emotion Finder or Guess the Expression and Social Learning activities in pairs. They were asked to complete the activity by taking turns. They were also encouraged to work together closely to finish the activity.

We performed one activity per day with all eight children. Each session was video recorded. These videos were later used for the analysis to measure the amount of social interaction. We performed a within-subject study. The media type i.e. paper vs. computer was used as an independent variable during the evaluation. We began with the completion of the pre-evaluation questionnaires. After these questionnaires, we began our evaluation with the mobile app for two weeks and then paper-based version of all the activities was used for four weeks. After that, we shifted to the tablet apps for another two weeks. This design helped us in observing the changing behavior of individuals over time and also enabled us to compare the results of independent variables (paper, and tablet). In addition, it enabled us to measure amount of social interaction while using the similar apps in different environments.

4 Evaluation Results

4.1 User Evaluation Feedback Results

The results of the ‘Super Skills Profile for Social Discovery’ questionnaire (See Fig. 2), used at the start and end of evaluation, show that each parameter showed a slight improvement for each child.

Fig. 2.
figure 2

X-axis shows pre and post results for fundamental skills, social skills, getting along with others and non-verbal communication skills of all children and y-axis shows the value parameters.

Full size image

The results of two ‘Learning Emotions and Expressions’ apps showed that overall children were able to understand different emotions. At the end of our evaluation, teachers were very positive about the app and gave very positive feedback. For example one teacher said, “He has been able to understand the new emotions i.e. fearful, disgusted and surprised. He has also been able to imitate them properly”. Another teacher mentioned, “Children’s understanding of the emotions has been greatly improved. He has been able to express and identified all emotions”. Another said, “He has responded very well, initially he did not show any response but as weeks passed, he started imitating the emotions and were able to understand them.”

The overall results for social learning activity were also promising. For example, one teacher said, “He has been learning new words used during each social learning activity session and later repeating them in other activities.” Teachers were also happy that students are applying the knowledge, learnt using the applications, outside the class “He has understood the stories for mischief, dust cleaning, and banana peel. He also uses them outside the class”. Another teacher said, “When i asked him to act out social stories like making the queue, he started doing it without hesitation.” Finally all teachers mentioned “Storytelling app has positive effects on students and we would like to continue with it.”

4.2 Video Analysis Results

All the recorded videos (child’s behavior while performing different activities during each session) were coded for verbal interaction, physical interaction, supportive and discouraging comments. These constructs were coded as numbers. We tested whether the data was normally distributed using Kolmogorov-Smirnov’s (K-S) test. If normal distribution was found, which the case was, we compared activities with app and no app using paired t-test. Two researchers were involved in video coding process. One of the researchers did not take part in the evaluation process.

We focused on coding for social interaction among pairs of children. The videos were coded manually for the following parameters: (1) Verbal Interaction: We counted one verbal interaction when a child spoke during the activity, or repeated after the researcher, (2) Supportive comments: We counted one supportive comment when a child communicated to support during the task, pointed to the characters in the story, asked questions about the story, or depicted the shown emotion or expression, (3) Discouraging comments: We counted one discouraging comment when a child did not like something about the activity or stood up and started moving. In addition, when a child was not comfortable with the partner and verbally expressed displease, and (4) Physical interaction: We counted one physical interaction when child was able to perform turn taking or when a child participated during other child’s turn or when the child imitated the evaluator.

We transcribed all the sessions. We use these transcripts to measure the total number of sentences. All videos were of different length, so we normalized the occurrence of above-mentioned parameters per minute.

Tests showed significant differences between app and no app for no of sentences used (p < .001), for richness of verbal interaction (p < .01), for the use of supportive comments (p < 0.01), for the physical interaction (p < 0.01) and for the use of discouraging comments (p < 0.05). The results can be seen in Fig. 3(a). In all cases, the usage of App had a much better impact than the usage of paper-based intervention. We also used questionnaire to measure differences between level of interaction, likability and involvement between activities performed on paper and tablet. We asked teachers to rank parameters like attention, involvement, likeability, interaction, and enjoyment on a scale from 1 to 5. We calculated the mean value for the app and no app for all these parameters. The results are shown in Fig. 3(b). Again, the app versions performed much better than the no app versions.

Fig. 3a. (left)
figure 3

Left: X-axis represents the mean value of all children; y-axis represents the minimum and maximum value parameters app and no app

Full size image
Fig. 3b. (right)
figure 4

Right: X-axis represents the mean value of all children for app and no app versions; y-axis represents the minimum and maximum value parameters

Full size image

5 Discussions and Limitations

In general, these early results show a very positive and encouraging trend. The result generated by the teacher’s feedback system also gave an indication that children were able to understand complex emotions. Similarly, the understanding about the new social scenarios was also developed. In addition, the transfer of knowledge from the digital environment to the real world i.e. imitating the emotional expressions and stories indicated that social learning, and learning emotions and expressions activities were quite effective. The results of video analysis showed a clear support for the app version because children’s social interaction and communication was significantly improved while performing the activities on a tablet. The results also showed that the level of interaction, involvement and enjoyment was higher in the app version as compared to the paper version. Finally, the higher number of sentences uttered and rich verbal interactions indicate that children showed better interaction skills during these activities. In addition, the higher amount of supportive comments also indicated that children showed interest in the activities.

We think the reason that the app-based intervention worked better than the paper-based intervention is that the children found the digital intervention novel, structured and organized. We know that autistic children feel anxiety when they have to deal with uncertain and unpredictable events. The teachers informed us that interacting with tablets is more predictable and controllable than manual interaction because that much heavily relied on people, which brings more uncertainty and affects the overall structure. This is study is one of the first studies of its kind conducted in the Pakistani context. The evaluation methodology we used for this study is in line with previous studies but unlike previous applications [10], we believe that our design approach is different. We focused on a number of contextual factors while designing emotion training apps and social learning apps and we believe that due to this reasons our Apps worked quite well in this new context.

One of the limitations of this study is the number of participants who participated in this study. In future we would like to run another study with similar apps with more children. Furthermore, we would like to run between-subject design experiment in order to avoid any prior learning effects.

6 Conclusion and Future Work

In this paper, we presented the design and evaluation of two mobile learning applications, which were used as a tool to improve the socio-emotional skills of autistic children. These applications specifically incorporated a number of culture-specific design elements in their design to make them more suitable for Pakistani children. The results indicated that our applications were able to engage children in positive social interactions and their socio-emotional skills were improved after two months. Verbal and physical interaction among children was significantly higher when they used the digital app version of activities as compared to the no app version. The school management also endorsed the effectiveness of our design and they also requested copies of our software, which are currently being used at the school.