Keywords

1 Introduction

Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder characterized by a deficit in social communication and repetitive patterns of behavior [1]. They also exhibit an unusual pattern of attentional behaviors, such as difficulty in sustaining their attention [2]. According to reports from Center for Disease Control and Prevention (CDC) of United States, the prevalence of this disorder is relatively high and has been on the increase, moving from 1 in 110 children in the year 2000 to 1 in 68 children in 2014 [3]. The existing method of ASD diagnosis is conducted by a multi-disciplinary team consisting of specialists in a developmental pediatrician, child psychiatrist, and psychologist. Several instruments have been developed for history taking, play-based observation of the child, such as the autism diagnostic interview, the diagnostic interview for social and communication disorders (both semi-structured interviews) and the autism diagnostic observation schedule (a play based interactive assessment) [4]. Other professionals, such as a speech-language pathologist, occupational and behavioral therapists assess the communication, sensory and behavioral difficulties respectively. Attention skills assessments are using observational methods and are often subjective. Therefore, a need for objectively determining the attentional challenges of children with ASD is warranted.

As a result, studies have explored different technologies for educational and behavioral interventions to support attention span through objective measures. For example, Lahiri [5] found out that the individualized viewing patterns, eye movement, and task performance can be used in the design of a virtual reality application for teaching social skills. Using adaptive robotics for teaching social interaction to children with ASD was achieved by using specific head tilting of the children to make the robot understand their needs and respond accordingly [6]. The need for measuring attention for the design of adaptive learning system is not limited to people with attention deficit, but it is also utilized in the typical population as an experimental evaluation. For instance, Szafir [7] showed that adaptive robotic agent using behavioral techniques improved learners recall abilities and this design improved the learning outcomes in both groups. These interventions work differently for the children with ASD as some may require over-stimulating effect and others prefer the opposite [8] due to their high sensory processing demand as compared to the typical population. Therefore, attention assessment has always been a way of evaluating learning intervention and improving the learning experience.

In human-computer interaction, good learning design influence users’ positive emotions and supports better learning outcome [9]. The evaluation of good learning outcome is usually measured based on the learners’ attention, participation through task accuracy and time taken to finish the task. In this study, we proposed a more objective evaluation of facial expression as a measure of attention during a computer-based attention task in children with ASD and neurotypical peers. Moreover, we discussed how facial movement measure can affect the design of an adaptive learning system for children in the spectrum. This study hypothesized that the existing facial expression for measuring engagement levels by affectiva SDK can apply to typical children and will work differently for children with ASD. This hypothesis tailors our study to 2 research questions:

  • Research Question 1: What facial expressions are exhibited by children with ASD and neurotypical peers during attention task?

  • Research Question 2: Can facial expressions during attention task serve as an indicator to differentiate children with ASD and neurotypical peers?

2 Related Work

Attention assessment can provide great insight into how children with ASD learn as well as how and when their attention needs to be supported. Hence, teachers are keen on observing and taking notes of student’s attention level and interaction objectively. Recently, objective techniques are being explored to automatically detect the attention of children with ASD during educational activities and ways to support their needs accordingly.

Objective approach is commonly used in adaptive learning (intelligent tutoring system) is becoming popular for children with ASD. In recent studies, different objective techniques are being used in the design of learning application with the ability to detect and reorient attention of children accordingly [5, 6, 10]. This approach of design is not limited to children with ASD but it is also utilized in children with hyperactive attention disorders who have similar attention problem with children with ASD [11] as well as typical children [12].

Several studies have looked at the direct measure of attention through the brain, heart rate and skin conductance using EEG (Electroencephalogram) headset and ECG (Electrocardiography) [13,14,15,16,17]. Primarily, these studies have considered obtrusive technology in typical population which gave better attention assessment, but there are chances of interference with the level of attention measured as the learner could be distracted with the thoughts of a foreign object on their body. A recent study conducted by [18] showed an adaptive learning system could be designed for children with ASD using EEG headset. This system gave a better assessment of attention as it measures their attention directly from the brain, but this approach may prove difficult to implement in children with ASD because of their sensitivity to touch and sensory processing disorder [19].

Other studies have considered touch-free technologies to measure attention such a using camera to measure head tilt [20], eyebrow raise, hand raise count [21] and facial expressions [22, 23]. These studies achieved success with the typical population. Recent technology by affectiva [24] identified ten basic facial expressions for measuring engagement levels. However, these expressions are yet to be explored in children with ASD and how these expressions affect design of adaptive learning system.

3 Method

3.1 Participants

This study used 8 participants, 4 children with ASD (3 boys and 1girl) and 4 neurotypical peers (2 boys and two girls) who are within the age range of 7 and 11 years. The ASD participants were recruited through an autism school. The ASD were diagnosed of moderate ASD and this was verified with the reports with the school management to ensure the students are eligible for our study. Further questions were asked by teachers to verify if the participants are not having any form of visual impairment or physical issues that may hinder the participants from taking the experiment.

3.2 Set-Up

Virtual reality continuous performance task (VR-CPT) was used as an attention task which mimics the conventional computerized version of continuous performance task (CPT) used to assess sustained and selective attention [25]. The first VR-CPT was created by [26] which was implemented with head mounted gear to create an immersive effect. The immersive VR-CPT has been successful in measuring attention in different studies for typical and children with ADHD [27]. Our version of VR-CPT was developed as non-immersive to make the experiment bearable for the population of our participants. Majority of the studies reviewed on virtual reality application for children of ASD used the desktop option to avoid the possibility of “cyber-sickness” and unusual head attachment so as not to influence their outcome of the intervention [28, 29]. The desktop VR-CPT presents the simulation of a conventional classroom which presents a teacher in front of the class, other students seated on the chair and desks, ceiling light, windows, a door and a blackboard where alphabets are displayed for 250 ms. Users are expected to interact with VR-CPT through a keyboard. To avoid further distraction through interaction, we improvised for the conventional keyboard with a simplified keyboard as seen in Fig. 1.

Fig. 1.
figure 1

Experimental set-up

Full size image

The testing room consists of two monitors 25 in. and 34 in. for the participants and the researcher respectively. We have used a logi-tech webcam which was attached to the top of the 24 in. monitor for the participants. We conducted this experiment in a dimly light-room environment to prevent interference of the ceiling white light or rays of daylight. In addition, this room was isolated and free from external noise or distractions. The screen-based eye tracker and webcam used for objective attention assessment in this study are less likely to interfere with the research outcome due to its unobtrusiveness.

3.3 Task

There were 4 levels of the VR-CPT experiments, where all letters appear for a period of 250 ms on the classroom board. The participants were expected to focus and sustain their attention on boards irrespective of the actions going on in the virtual classroom. The children needed to click the simplified keyboard only when letter X appears. The first level of the VR-CPT had no distraction. The second scenario had minimal distractions with audio and visual such as coughing and students raising hands mainly from the center of the class. The third scenario had a medium level of distractions which were from the center and left-hand side of the classroom while the fourth level has the highest level of distraction from the left, right and center. All the participants took all the 4 experimental tasks except a participant from the ASD group who took 3 levels attention task.

3.4 Procedures

Parents of all participants were given a consent form which was approved by the institution review board committee. The form gave detailed information about the experiments and the rights of the participants throughout our study. After that, an experimental manual was used to describe how the experiment will go and how they are expected to interact with the software and hardware. We ensured all the participants got the same type of instructions and set-up. At the start of the experiment, the researcher welcomes the participant and caregiver to the room and engaged the child and the parents in a discussion to get the participant settled to the room environment. Then, the researcher gave VR-CPT instructions in form visuals and text on a hardcopy to explain to the participants on how to take the test and number of attention tasks they are expected to complete.

We induced the attention of the participants for an attention and non-attention task by presenting them with a blank screen and a VR-CPT on a desktop. All the participants took a demo of the VR-CPT before the main experiment to ensure they understood what they are expected to do. Each of the participants took a 3 to 4 min break after completing the second level and the total time for the attention tasks was 20 min per participants.

3.5 Data Collection

Facial expressions data collection and analysis was carried with a commercial software tool: iMotions embedded with affectiva [30]. The possibility of extending our research to other biometric measures later in future has influenced our choice of iMotions SDK for conducting our research. Affectiva software utilizes Histogram of Oriented Gradient (HOG) and Support Vector Machine Algorithm (SVM) to classify facial expression based on over 10,000 faces encoded manually across the world [31]. This classifier outputs the quantitative values of the facial expression of the participants using a likelihood value of 0–100% where 0 indicates absence of facial expression and 100 indicates the occurrence of an expression.

The sample data of the 15 and 16 experimental sessions from the ASD typical group respectively (31 tests in total) were collected using Logitech webcam and Affectiva SDK. Data from each experiment was sampled at 16 Hz thereby generating 16 samples of data per second, and each test took 300 s. The total sample for all participants across all the attention tasks taken was 148,800 samples (72,200 from ASD and 76,800 from typical groups). Each sample has 21 features of facial expression, 8 basic emotions and 98 raw features on facial landmarks. We investigated and analyzed 10 facial expressions for all the samples from the two groups of participants. The remaining facial expressions and 8 basic emotions were excluded. A custom python script was used to select the desired features and basic statistical analysis for all samples of each participant. The full description of the facial expressions can be found at [24, 30] while the description of the 10 facial expressions we have analyzed are given in Table 1.

Table 1. 10 facial expressions for measuring engagement level by “affectiva”
Full size table

4 Results

We analyzed 15 experimental scenarios conducted with 8 participants each gets a commission score and omission scores which identify the right clicks (i.e., letter X) and wrong clicks (i.e., other letters aside X) respectively. Then we estimated the missed target (i.e., when the participant did not click letter X). These scores were used to evaluate their attention level. These scores were not shown to any of the participants until they completed the whole experiment in other not to influence the study by their mood especially when they have a low score.

4.1 Participants’ Characteristics and Data Sample

8 children, 4 with mild ASD and 4 neurotypical peers, were enrolled in the study by taking a total of 31 experimental scenarios of attention tasks, where 7 of the participants took 4 experimental tests each except 1 who took 3, i.e., no-distraction, easy, medium and hard. Each caregiver of the participants presented the diagnosis report to confirm they were eligible for the experiment. All the participants are verbal and passed the basic skills test needed for the attention task which was letter recognition. The inclusion criteria for the ASD participants were diagnosis report and who can differentiate letters while the typical children scored below 15 in a Childhood Autism Spectrum Test (CAST) [32].

The average age and the standard deviation of the ASD group were 8.75 and 1.45 respectively, while those of the typical group were 8.25 and 1.30. The t-test conducted to check for the differences between the two groups was p = 0.68. This result indicated that there was no significant difference between the two groups. The summary of the participants is given in Table 2.

Table 2. Participants’ demographics
Full size table

4.2 VR-Continuous Performance Task (CPT)

The scores for correct clicks, omitted letters, and incorrect clicks were saved for the analysis of the participants’ attention level. We have only considered correct clicks scores in this study and excluded the scores for the omitted letters and incorrect clicks based on the scope of this study. The attention performances for all the participants were all above average as seen in Fig. 2.

Fig. 2.
figure 2

CPT scores for ASD and typical

Full size image

In Fig. 2 above, children with ASD performed less than the control group in all the level of the CPT test, and the statistical inference using t-test gave a value of p = 0.02 stating there is a significant difference between the two groups when considering CPT (task performance) for measuring the level of attention.

4.3 Facial Expression

This study examined ten facial expressions that are attached to engagement which are: brow furrow, brow raise, lip corner depressor, smile, nose wrinkle, lip suck, mouth open, chin raise and lip pucker [affectiva]. This claim was based on the facial expression data for 3.2 million people from 75 countries. The hypothesis of this study is to investigate if the facial expressions attached to engagement apply to children with ASD during an attention task and how these facial expressions can influence the design of adaptive software learning for children in this spectrum. This study captured 4,800 quantitative samples of facial expression with over 40 features per participant during each level of the attention task to answer our research questions.

  • Research Question 1: What facial expressions are exhibited by children with ASD and neurotypical peers during attention task?

We identified the facial expression of all the participants from the two groups to determine if there were any differences. The result showed that the level of facial expressions differs in the two groups as shown in Fig. 3a–d.

Fig. 3.
figure 3figure 3

(a) Facial expressions for both groups during attention task level 1 (none level). (b) Facial expressions for both groups during attention task level 2 (easy level). (c) Facial expressions for both groups during attention task level 3 (medium level). (d) Facial expressions for both groups during attention task level 4 (hard level)

Full size image

We calculated the average value of all the facial expression likelihood score for all the participants in each group to understand the hierarchical order of the likelihood score. The result showed that brow raise and mouth open were a prominent facial expression in both groups but in opposite orders. The ASD group showed more of a mouth opened than that of the typical group while brow furrow nose wrinkle is less prominent and other varies differently between the groups as shown in Fig. 4a and b.

Fig. 4.
figure 4

(a) The hierarchical order of affectiva facial expressions in the ASD group. (b) The hierarchical order of affectiva facial expressions in typical group

Full size image

  • Research Question 2: Can facial expressions during attention task serve as an indicator to differentiate children with ASD and neurotypical peers?

The average value of affectiva likelihood score (between 0 and 100) of over 4,800 samples for each participant was taken to identify the frequencies of the expression. Then we chose the likelihood values that were above the median value for each participant as the prominent facial expressions. Lip press is appeared to be common in all the participants with ASD while it is different in the typical group. Although, lip press is common in half of the typical population the remaining half did not exhibit this action while paying attention. Afterward, we identified the facial expression common to each group and sorted them from largest to smallest as seen in Figs. 5 and 6.

Fig. 5.
figure 5

Common facial expressions exhibited among the ASD and Typical groups for the 31 experiments

Full size image
Fig. 6.
figure 6

Comparison of similar facial expression in both groups

Full size image

In Fig. 6 above, children with ASD exhibited more facial expression with a brow raised and mouth open and less with lip suck than the control group when the “affectiva likelihood score” was averaged for all the entire CPT experimental tests. The statistical inference using t-test gave a value of p = 0.40 stating that there was no significant difference between the two groups when considering these prominent facial expressions for measuring the level of attention. However, these findings need to be investigated with a larger sample.

5 Discussion

There is need for an objective approach to measure attention of children with ASD during a learning task using unobtrusive and non-invasive technology for reliable attention assessment with little or no interference of the technology used. This claim is also supported by the study conducted by Aslan [33] where eye gaze and facial expression were used to measure learners’ engagement during a learning activity. In addition, the facial expression had the highest percent (55%) of emotion recognition amongst other parameters such as body language and vocalization [34]. Other studies have also identified facial expression as the fundamental parameter for understanding human feelings objectively [35,36,37]. This study has evaluated the 10 basic facial expressions in measuring the attention of children with ASD and comparing it with their neurotypical peers using a VR-CPT as attention stimuli. The findings of this study are discussed in line with the performance of the participants with the attention stimuli, facial expression during attention task and how these measures differentiate children with ASD from their neurotypical peers. Then we discussed the design implication for an adaptive learning system.

5.1 VR-CPT as a Measure of Attention

CPT is a popular and trusted computer-based test for assessing sustained and selective attention of learners. A step further is the, VR-CPT which provides learners with ecological validity that makes them have the feeling of classroom tasks. All the participants recruited for this study took the VR-CPT version of 4 different levels (No distraction, easy, medium and hard). They were scored based on the number of correct clicks, and all participants’ score were above average in all levels which showed they paid attention. However, the score of the typical group was more than the ASD group at all levels. This finding is similar to claims by other studies that children with ASD have attention deficit and low attention span [2, 38, 39].

5.2 Facial Expression During Attention Task

  • Research Question 1: What facial expressions are exhibited by children with ASD and neurotypical peers during attention task?

The findings from this study showed that 4 out of the 10 facial expressions are common in children with ASD which are: mouth open, brow raise, lip suck, and lip press. Similar facial expressions are prominent in the neurotypical group except for lip press. Hence, 3 facial expressions (mouth open, brow raise, lip suck) are common in both groups during an attention task. This result is consistent with the attention measuring states of learners by Asteriadis [40] where mouth open and brow movement were considered for assessing the attention-related states. Another study by Ross [21] found out that eyebrow raise count and hand raise count are useful for measuring attention level. Our findings are in relation with existing studies shows that among the basic regions of the face which are: eyes, nose, cheek and mouth, the important region of the face that correlates attention in children with ASD and neurotypical peers are the eye region and mouth region.

  • Research Question 2: Can facial expression during attention task serves as an indicator to differentiate children with ASD and neurotypical peers?

According to the result of our experiments with two sets of participants, 3 facial expressions (mouth open, brow raise, lip suck) are common in both groups during the attention task. However, the ASD group exhibited more of mouth open and brow raise than the neurotypical group but the differences in these 3 facial expressions are not significant to differentiate the groups. This finding also negates our hypothesis as we expected the facial expression of children with ASD to be different from that of neurotypical peers based on the findings of existing studies [41,42,43]. However, this finding is only based on mild ASD group who require the least support as compared to severe and moderate ASD. There may be differences in the facial expression of moderate and severe ASD with typical children. Nevertheless, lip press was exhibited by all the participants in the ASD group while half of the neurotypical group showed lip press. This finding may give a clearer picture if this study is repeated with more samples from both groups.

5.3 Design Implication

Assessment of attention deficits in children with ASD is often subjective. This pilot study helps bring objectivity to this measure. Though children with ASD have difficulty inferring other people’s emotions and expressing the same, there are few facial expressions that are common between children with ASD and typically developing children. Further analysis of these common expressions displays the difference between the two groups which helps pinpoint the difficulties children with ASD face with regards to attention.

The facial expressions exhibited by children with ASD during attention tasks have significant educational and behavioral value, both diagnostically and therapeutically. It brings objectivity to the assessment of attention in children with ASD. Based on this, therapeutic programs (for learning and communication) can be tailor-made for children with ASD keeping in mind the facial expressions. Additionally, facial expressions can be used to design adaptive learning software for children with ASD, which can be very helpful in their academics.

6 Conclusion

This pilot study evaluated how existing facial expression parameters can be used to measure the attention of children with ASD and neurotypical peers. We also looked at how facial expressions can be used in the design of adaptive learning software for children with ASD. In line with our findings, we proposed 4 facial expressions: mouth open, brow raise, lip suck, and lip press as parameters for measuring the level of attention in children with ASD during learning. These 4 facial expressions can be embedded in the design of adaptive learning software for children on the spectrum. In the control group (TD), similar facial expression were identified except for lip press which is only associated with ASD group. Although, both groups exhibited almost similar facial expressions during the attention task but differ in one of the expression. Hence, lip press needs to be further investigated as a biomarker for differing ASD and typical children. Considering the small sample size in this study, there is a need for a more detailed evaluation of the effectiveness of facial expressions and the practicality of using the same approach in clinical practice and research. Additionally, the study can be extended to children with Attention Deficit/Hyperactivity Disorder, where attention is the deciding factor.