Keywords

1 Introduction

Neurodevelopmental disorders (including Autism Spectrum Disorders, Attention Deficit and Hyperactivity, Cerebral Palsy, and Down Syndrome) limit individuals’ abilities to process and record information, manage time, regulate affective states, and interact with others. These limitations affect executive functioning and often result in difficulties for the individuals to integrate in society and live independently. Attending classes, finding and sustaining a job, and maintaining relationships are examples of daily activities that can be disproportionately challenging for people with neurodevelopmental disorders.

To successfully perform activities of daily living and better integrate themselves in society, individuals with neurodevelopmental disorders often require support from a personal assistant who continuously monitors them and intervenes when necessary. As these interventions are mostly and traditionally performed through human support, they tend to be costly, obtrusive, stigmatizing, and not scalable. There is a promising potential for technological solutions through wearable applications to support such interventions in a more scalable, unobtrusive, consistent, and less stigmatizing way.

A growing number of emerging technologies have been recently explored to assist individuals with neurodevelopmental disorders [1,2,3, 30]. These solutions include interactive systems, games [4], mobile and wearable apps [5]. The application features include: teaching educational content, providing mood regulation, intervention tools, and exercises to sustain focus and attention [3]. The delivery formats for these assistive technologies range from tablets [6] and smartphones [7], to wearable computers [1] and physical installations [8].

Although the main target population for assistive technologies involves individuals with disabilities, some applications aim at supporting caregivers [9], assistants [26], parents, and teachers [10]. Prior work addressed the following neurodiverse conditions using assistive technologies: Down Syndrome, Autism Spectrum Disorders (ASD), Attention Deficit and Hyperactivity Disorders (ADHD), and Cerebral Palsy (CP) [30]. In the U.S., neurodevelopmental disorders affect 1 out of 6 children [11, 12]. Their prevalence has increased 17% between 1997 and 2008 [11]. Neurodevelopmental disorders have lifelong consequences for individuals. In this context, technology can provide several benefits to them: (1) close contact with users; (2) remote and continuous monitoring; (3) record keeping of data for later analysis; (4) prompt interventions; and (5) reduced stigma.

Despite several opportunities and benefits associated to wearable technologies, they have been underexplored as assistive technologies for individuals with neurodevelopmental disorders. Also, the evaluation of assistive wearables have been carried out mostly in controlled settings during short periods [13]. To shed light on how smartwatch applications can assist postsecondary students with neurodevelopmental disorders in inclusive classrooms, we developed and evaluated WELI (Wearable Life) – a wearable application implemented to support neurodiverse students in mood regulation, interventions, reminders, and communication with their assistants. WELI was evaluated in a long-term IRB-approved study including two field studies in actual classes during two academic semesters.

The studies include data collected from users log (history of user interaction), focus groups, interviews, and questionnaires. The methods include two field studies, each carried out throughout one semester. The first study involved 11 participants – six neurodiverse students and their assistants. The second study involved 10 participants – six neurodiverse students and their assistants. The results show that the assistants find the application helpful to facilitate their job, and are willing to use it in inclusive classes. Additionally, most students find the application beneficial. Also, they feel proud wearing a smartwatch, and would like to continue using it in the future. Still, we noticed that there are situations in which the application may not fulfill specific needs of some users.

2 Related Work

A number of technologies to assist users with diverse abilities have been investigated by the scientific community in the past decades. Assistive wearables specifically, besides having a large potential for applications, also foster user acceptance, engagement and adoption due to their versatility, novelty, and conventional look [5, 14]. Despite their promising potential and increasing popularity, the applications of wearables to assist neurodiverse users remain limited.

Assistive Wearables for the Users with Neurodevelopmental Disorders. Prior research on assistive wearables focuses on helping children with neurodevelopmental disorders for multiple purposes. To improve socialization for the children, TellMe uses interactive robot characters (incorporating microphone, sensors, and actuators) embedded in clothing to treat ASD symptoms of boys by encouraging them to speak out and express themselves while playing with the interactive robot characters on their clothing [2]. EnhancedTouch uses a bracelet with LED lights to provide visual feedback and augment human–human touch events (handshake) for children with ASD [15]. VRSocial uses virtual reality glasses to facilitate the proximity regulation in social communication for children with ASD, which augments the conversations between children with ASD and an avatar with the real-time visualizations of proximity, speaker volume, and duration of one’s speech in VR [16].

Besides socialization, prior research on assistive wearables aimed at helping children in other aspects. For example, WatchMe is a prototype implemented to manage behavioral problems in children with neurodevelopmental disorders (Down Syndrome and ASD). The watch detects hand banging and delivers instructional interventions with visual-haptic cards [17]. CASTT (Child Activity Sensing and Training Tool) is a real-time assistive prototype that captures activities and assists the child in maintaining attention [3]. BlurtLine is an interactive belt aimed at supporting children with ADHD to control their impulsive speaking in classroom settings [18].

Several assistive wearables focus on helping not only children but also adults. For example, ProCom facilitates proximity awareness for individuals with ASD using chest-worn wearables with an infrared sensor module that connects to a mobile app to show the changes of distance and zones of proximity [19]. FOQUS is a smartwatch app that alleviates the anxiety and stress of adults with ADHD by providing tools for time management, guided meditation, and positive message priming [20, 21]. Takt relies on touch and vision to enable individuals with ADHD to tell and read information of a clock using their senses (instead of relying on cognitive abilities) [1]. StretchBand is a wrist-band developed to analyze anxiety levels of adults with Autism by recording the user interaction with a digital stretch band [22].

Recent studies on assistive wearables focus mostly on children as the target population, rather than young adults. Furthermore, most studies support one unique feature at a time, e.g. communication or mood regulation. Most applications do not have a target scenario, and the evaluation is short and performed in controlled settings rather than in the field. Although the form factors of assistive wearable vary, including belts and chest-bands, wrist-worn wearables stand out among the solutions analyzed, possibly due to their conventional looks and versatility in terms of implementation, data collection, user interfaces, and interaction.

Design Considerations for Neurodiverse Users. The design approaches for technologies for special education must ensure a full understanding of the domain through user-centric design, seeking to facilitate the adoption of technology with solutions that are unobtrusive as well as easy to use and to adopt [9]. According to existing research guidelines for work with participants with disabilities [23]: (1) participants with disabilities should be directly involved with research; (2) they should be recruited with clear criteria; (3) collaboration with a group that focuses on the related disabilities should be established; (4) investigators should conduct primarily qualitative studies or a hybrid of quantitative and qualitative research, doing in-depth investigations to deal with small sample sizes; and (5) modifications to research methods for people with cognitive disabilities need to be made based on advice of those familiar with the specific cognitive disability. For Alper et al. (2012), four principles help to address the needs of end-users focusing on their perspectives rather than developers’ mental models, namely: deep engagement, interdisciplinarity, individuality, and practicality [24]. We followed the design considerations and principles mentioned above to guide our field study, to ensure that the solution proposed meet actual users’ needs, suit their individual abilities and preferences, and have a larger potential to be adopted in practical settings in a sustained way.

3 Evaluation Study

Unlike prior work on assistive wearables, evaluated briefly in controlled settings, WELI was assessed in a long-term evaluation in inclusive classrooms with young adults with neurodevelopmental disorders. More specifically, we conducted two field studies over two semesters. The studies were conducted in collaboration with a postsecondary academic program. Aiming at external validity, the studies took place in actual inclusive classes. The first study was carried out in the Fall 2017, and included 11 participants, being 6 neurodiverse students and 5 assistants. The second study took place in Spring 2018, including 10 participants – 6 students and 4 assistants. In the studies, the students and their assistants used WELI, a wearable and mobile application that facilitates assistance and communication in class. Assistants aided the students on demand as they traditionally do; however, instead of intervening verbally, they used WELI on smartphones and smartwatches to deliver the interventions in more unobtrusive and potentially less stigmatizing way.

Fig. 1.
figure 1

WELI: examples of passive interactions are illustrated on the left (A), where assistants can choose in the mobile phone menu and send Focus–related interventions to the students’ smartwatches (b to g); watch notifications are for Focus (b, c), Rewards (d, e, f), Participate prompt (g), and Quiet messages (h, i). Examples of active interaction are illustrated on the right, where the assistants can send assessment questions (K) via the phone, and the student can answer the questions using the smartwatch (l to o).

Full size image

3.1 WELI (Wearable Life)

To attend a postsecondary program, students with neurodevelopmental disorders take inclusive classes accompanied by their assistants. The assistants help the students using verbal interventions and gestures. WELI was designed and developed through user-centered design following 8 user studies with 58 participants [26,27,28]. WELI supports coordination between young adults with neurodevelopmental disorders and their assistants in class. With a mobile and wearable solution [29], WELI (Fig. 1) fosters a more conducive environment for inclusive learning, enabling discreet communication without verbal interventions. WELI provides more unobtrusive assistance to neurodiverse students, minimizing disturbance to classmates and instructors and reducing the stigma of having an assistant visibly coaching students aloud in class. WELI delivers multimodal notifications of text messages and graphics on a smartwatch to students after a quick vibration. Interventions are timely, sent automatically or manually, triggered by the mobile app of WELI by an assistant. WELI has seven main features:

  • Focus: helps students to concentrate in class;

  • Quiet: moderates students’ conversations and voice volume in class;

  • Participate: helps students to be engaged in class;

  • Rewards: gives students a positive reinforcement when they perform well;

  • Assessment: enables students to self-assess their own mood (regulation);

  • Survey: gathers feedback from students about their feelings after class;

  • Take a Break: delivers a countdown reminder before a class break.

WELI includes two intervention styles: passive, without a reply, requiring the student to simply read the notification displayed on the smartwatch, sent by the assistant; and active, requiring the student to reply to the questions sent by the assistant and prompted on the smartwatch. The student answers are delivered to the assistant’s phone. Among all features implemented, Focus, Quiet, Participation, and Rewards are passive, while Assessment, Survey, and Take a Break are active interventions that require a response. WELI has been developed on Android Wear OS, thus it can be conveniently used on different off-the-shelf commercial devices. For the evaluation studies, we employed a Sony Smartwatch 3 and a Moto G5 Plus smartphone.

4 Field Study: Assessing WELI

Field studies are suitable for technologies that are used in specific settings, such as inclusive classrooms. To evaluate WELI, two field studies were conducted in classes of an inclusive postsecondary education program during two semesters. The college-level inclusive classes met up to two times a week for 75 min at most. To avoid disturbing students taking classes and respect their privacy, video was not recorded but usage logs were recorded to collect app data in class. We applied a questionnaire and conducted a focus group after each semester for assessment.

The study participants, students with neurodevelopmental disorders and assistants, are young adults, English-speakers, of diverse ethnicities and genders. They were recruited by purposeful sampling [25] from Mason LIFE, a special education program. The students take college-level courses and special education classes. They understand and command expressive language and are technologically competent. They are able to willingly accept or decline participation in the study. The assistants are graduate or senior undergraduate students, majoring in Public Health, Education, and Psychology. Participation in the study was voluntary.

4.1 Study Protocol

Before the field study, each student and assistant received training to use WELI. In each class, the assistant observed the student and assisted them using WELI on a smartphone. The students wore a smartwatch to receive the haptic, text, and graphic notifications. Each student was paired with an assistant for the semester. The assistants followed traditional work practices to guide and assist students as needed. However, their communication was supported by WELI to assess its impacts in class interventions.

Features Evaluated. To avoid overwhelming the students and assistants in class, 6 features were selected for evaluation: Focus, Quiet, Participation, Rewards, Assessment, and Survey (Fig. 1). The usage of the features was on demand, i.e. based on student behaviors, except for the Survey, which was sent close to the end of each class to get the students’ feedback.

Based on the feedback from the first study, we evaluated all WELI features in a follow-up study, including Take a Break and Customize which were added thanks to the feedback from the first study.

Logs of the User Interaction. To respect students’ privacy, we did not record video in class. We strived to not overload the assistants, respecting their workload that includes observing the students, intervening when needed, and taking notes of the lecture. We recorded logs of app usage in class, including all interventions, messages sent from the mobile app and responses sent from the smartwatch. For each interaction, we logged the notification content and feature category with date and time. To allow data sharing with the investigators after class, we implemented a share function in WELI using gmail. The data was logged to monitor and analyze how the participants used WELI in classroom activities.

Follow-Up Study. We conducted follow-up surveys and focus groups after the field studies at the end of each semester. We employed questionnaires to gather student and assistant feedback on their experiences with WELI in class, and we conducted focus groups with the assistants to discuss WELI and assess its impacts on their workload and benefits to the students. Limitations and opportunities for improvement were also discussed. To evaluate WELI, we analyzed the data collected from all these studies.

5 Evaluation Results

5.1 First Field Study

The first field study occurred in the Fall 2017. Six neurodiverse students and five assistants volunteered to participate. One student withdrew during the first study. The five remaining students (3 male and 2 female, all white) were enrolled in four to six classes when using WELI. Their age ranged from 22 to 26 years old. They were in the 4th year of the postsecondary program. One student had both ID (Intellectual Disability) and ASD, one student had ASD, one had both ASD and ADHD, one had ID and CP, and one had ID. The five assistants were female, being three undergraduate and two graduate students majoring in Public Health and Special Education. We applied a questionnaire for the students and assistants at the end of the semester to evaluate WELI. To gather additional feedback about WELI, a 1-h focus group was conducted with the assistants.

5.2 Results of the Log Analysis

At the end of the study, we analyzed the log files of 24 classes, including 21 classes from the five students who participated in the entire study. The log analysis indicates the features that were used the most, how the intervention messages were used, and how usage varied in time, frequency and nature during classes.

Frequency of Usage per Feature. To assess the usage of different interventions in class, we counted the messages exchanged for each feature for all classes. To clarify, for the Assessment and Survey features requiring a student reply, we only accounted for the number of questions sent to treat it as one single action, without counting the answers repeatedly. In total, 431 messages were exchanged. The operations per feature used in class in order of frequency were:

  • Focus: 162 (38%), Rewards: 133 (31%)

  • Quiet: 57 (13%), Assessment: 37 (8.5%), Survey: 35 (8%)

  • Participate: 7 (2%)

These results indicate that Focus and positive reinforcement (Rewards) are relevant for students, as they exceed 60% of all operations.

Fig. 2.
figure 2

The Notice Rate for each intervention in the first study. Per feature, there were messages successfully noticed (green bars) or ignored (blue bars). (Color figure online)

Full size image

Notice Rate of the Features. Analyzing the logs, we found that the same message was sent consecutively sometimes in less than 1-min interval. In the focus group, the assistants mentioned situations where students did not notice, read, or react to the message on the smartwatch. In these cases, the assistant re-sent the notification until it was successfully delivered and perceived by the student. To identify how the students responded to the interventions, whether they noticed (read the message and reacted to it) or ignored it, we analyzed the ‘notice rates’ of the messages sent. We categorized repeated interaction patterns as follows: when the same message was sent consecutively within the 1-min interval for those passive notifications (not requiring a reply), and when the question did not receive responses for those active features requiring a reply. Otherwise, the interaction was considered successful.

Figure 2 shows notice rates of each feature. First, we analyzed four passive features: 129 out of 133 Rewards (97%) messages were noticed at once. Focus (142/162, 87.7%) and Quiet (50/57, 87.7%) messages had the same high success rate. Only seven Participate messages were delivered successfully, which left few possibilities for the student to disregard it. The reason why the Rewards had a higher notice rate might be because students are keen on receiving the positive reinforcement and barely miss them. For the features asking the reply, Survey got 54.3% (19/35) and Assessment had 51.4% (19/37) of messages replied by the students. The extra effort to provide input on the smartwatch might justify the lower success rate for Survey and Assessment. Survey had a higher success rate probably because it was sent after the class or near its end.

Temporal Analysis of Features. All the students who participated in the study were enrolled in 75-min courses. To analyze the temporal distribution of user interactions, we divided the classes into five intervals of 15 min each. The first interval corresponds to the time period during the first 15 min of each class, the second one ranges from 15 to 30 min, and so on.

To analyze the usage of WELI throughout the classes, for each student we calculated the average number of messages sent in each interval per class. Figure 3 shows the usage of WELI throughout the class intervals per student, and in each class interval it shows the average usage frequencies of two main features (Focus and Rewards) as well as the remaining features. Although we cannot identify any time patterns in WELI usage across all students, we note that student 2 required less interventions than the other four students. Also, we noticed more interventions during the second half of classes for 3 out of 5 students. Overall, Focus was the most common intervention for all students (P1, P3, and P5), except for student 4, who received more Rewards for positive reinforcement. The usage of Focus and Rewards varied across classes per student, but all students received both interventions in all class intervals (except student 2).

Fig. 3.
figure 3

Average usage of WELI features throughout the five 15-min class intervals (D1 to D5) per student in the first study, indicating the usage of Focus (orange), Rewards (green), and other (blue). (Color figure online)

Full size image

Sequential Analysis of Features. To identify potential trends for intervention sequences, we discarded all the unnoticed messages and counted the sequences for each two successive operations. The results show that after a Focus intervention, the following interventions are mostly Focus (55%) again, showing that sustained interventions are needed when assisting neurodiverse students to concentrate. Additionally, 28% interventions after Focus are Rewards, showing the importance of positive reinforcement (after the students focus in class the assistant often encourages them). After a Rewards intervention, 44% are Rewards again, and 31% are Focus, which indicated that assistants try to sustain and encourage students’ engagement.

Fig. 4.
figure 4

The results of the questionnaire with 4 students who participated in the Study 1 concerning app usage, communication, messages, attention, and focus. 

Full size image
Fig. 5.
figure 5

The results of the questionnaire of 5 assistants in the first study. The top part includes 7 positive statements and the lower part shows 5 negative statements. 

Full size image

5.3 Results of the Questionnaires

Students’ Responses. To assess the students’ perceptions and experiences with WELI, at the end of the academic semester, we applied a 5-question questionnaire to the five students who participated in the entire study and received four qualified answers (one was discarded due to extreme answering bias [23]).

Overall, the students’ responses were positive (Fig. 4). They were satisfied using WELI in their classes, and very satisfied with the communication between them and their assistants. They were also highly or mostly satisfied with how they listened to the class with WELI support. Concerning ‘focus on the question the professor asked’ and general ‘messages sent to you by WELI’ questions, one out of four students reported to be ‘somewhat satisfied’. To triangulate the results and avoid acquiescence bias, we also asked the four assistants about their students’ performance in class using WELI. All assistants reported to be very satisfied with WELI concerning how it motivates their students to be more engaged in class; three assistants were highly satisfied with WELI concerning how it helped their students with ‘listening’ skills, improved their ability to complete assignments and to follow the professor’s instructions in class. One assistant reported to be ‘somewhat satisfied’ and ‘not sure’ about those aspects. The results suggest that WELI can improve the students’ engagement and performance in class and that most students were satisfied with the communication and interventions they received via WELI. As expected, there were some variations on responses depending on the student though.

Assistants’ Responses. In the first field study, we also applied a questionnaire to the five assistants. The experience of one assistant was limited to four classes since his/her student was not present in all classes. The questions focused on the assistants’ opinions about using the application in what regards assistance, work impact, the students’ reactions to WELI, as well as the benefits of the features used in class. We grouped questions that received similar responses to facilitate the presentation of the results. We presented 7 questions using positive statements and 5 using negative ones (Fig. 5).

All assistants (strongly) agreed that WELI is easy to use. Four assistants thought WELI made the assistance more efficient and helped their work in class without confusing or distracting them. Most assistants agreed on the aid and usability of WELI. Most assistants (3 out of 5) also wanted to continue using WELI in class and agreed that WELI made their students more independent (one was neutral and one disagreed). Most assistants were neutral about the preference to assist the student with WELI, which may due to the extra effort reported that the smartwatch added to them (3 agreed and 2 disagreed) when comparing it to the traditional assistance.

There were two multiple choice questions about the most and least useful features previously used in class. For the most useful features, Rewards and Focus received five votes, Quiet and Participate received two, Assessment received one. For the least useful features, Assessment and Survey received three votes. This result is nearly consistent with the frequency of feature usage from the log files, except for the Participate feature (2% in the log). Concerning the open questions, the assistants suggested WELI should enable customized messages to be more personalized to the student, including also a favorite page to access frequent messages faster and improve efficiency.

5.4 Results of the Focus Group

To complement, compare and contrast responses from the questionnaire, we conducted a one-hour focus group with five assistants. We discussed the pros and the cons of WELI, whether and how WELI facilitated their work, impacted the students, and asked for suggestions. A coordinator of the inclusive program and two designers facilitated and moderated the session. For documentation, the focus group was video recorded and notes were taken.

The benefits of WELI included making the assistance ‘less obvious’ by not requiring loud ‘talk’ to provide assistance as WELI provided ‘discrete communication’. The ‘buzz’ along with the notification on smartwatch helped wake students up when they were sleepy. The drawback noted is that WELI was ‘a bit distracting’ to one student who was always ‘paying attention to the watch’.

Concerning usability, four assistants judged WELI as ‘user friendly’ and ‘easy to use and learn’. One assistant found initially challenging to ‘find the right message in the menu’. Regarding how WELI facilitates the assistants’ work, they reported that it could further ‘encourage the student’ by providing ‘Rewards’ ‘consistently’ and by providing ‘feedback’ from the student via WELI. For the workload, the assistants mentioned that sometimes the messages were not noticed by the student at first, so that they had to re-send it. For the impact on students, the students were ‘excited’ to use WELI in class, i.e. one student ‘took off his Apple watch and wore WELI watch’ in every class. Another student was proud and ‘showed the watch to his classmates’. One assistant told that her student ‘got better’ with WELI, and she had to send less interventions because her student improved his behavior. Other assistant was able to ‘sit farther away’ from her student thanks to WELI which allowed the student to become ‘more independent’. A student got ‘a little frustrated’ because he used to ‘talk a lot’ and still wanted to ‘talk face-to-face’ to get help in class. ‘Focus’, ‘Rewards’ and ‘Quiet’ were very useful features and the pictures showed in the notifications were appreciated. Sometimes the vibration of the watch was not long enough to notice. Also, the assistants suggested WELI to allow them add the student’s name for each ‘Rewards’ message. Lastly, they wanted to be able to send ‘customized’ messages to the students in addition to the standard ones available in WELI.

5.5 Second Field Study

Customization Feature. Based on the feedback from the focus group in the first study, we extended the vibration, allowed assistants to add the student’s name for Rewards, and added the Customize feature to WELI (Fig. 6). This feature made WELI more personalized, allowing assistants to add interventions that met specific needs of each student.

The second field study was conducted in the Spring 2018. Six neurodiverse students volunteered to participate, along with their four designated assistants. Among them, one student and two assistants had participated in the first field study. For field study 2, one student withdrew due to sleep issues. The remaining five students took five to eight classes with WELI. In total, we collected and analyzed log data from 34 classes. The five students who participated in the study included three male and two female participants, ranging from 21 to 26 years old. Two of them are Asian-American, two are White, and one is African-American. Two of them had ID, one had ID and ADHD, two had ASD and PDD (Pervasive Development Delay). The four assistants included one male and three female, being two senior students majoring in Psychology and Education, and two graduate students majoring in Public Health and Special Education. According to the feedback from the first study, we made the vibration for notifications on watch longer and evaluated two additional features in the second study: Customize and Take a Break.

Analogously to the first study, we applied a questionnaire to students and assistants in the end of the semester and conducted a one-hour focus group with the assistants.

Fig. 6.
figure 6

Customize feature enables assistant to add personalized messages for students (A). Each student has a list of customized messages (B), through which assistant can send notifications (c, d) to students. 

Full size image
Fig. 7.
figure 7

The Notice Rate for each intervention in Study 2. Per feature, the messages were either successfully noticed (green bar) or ignored (blue bar). (Color figure online)

Full size image

5.6 Results of the Log Analysis

Frequency of Usage of WELI Features. In total 818 messages were exchanged, sorted by frequency, the number of operations per feature in class was:

  • Focus: 588 (71%), Rewards: 124 (15%)

  • Quiet, Participate, Customize: 17 (2%) each

  • Survey: 32 (4%), Assessment: 20 (2.4%)

  • Take a Break: 4 (0.5%)

As in the first study, Focus was the most popular intervention for students. Rewards was used oftentimes, and the other features were sparsely used.

Notice Rate of WELI Features. Figure 7 shows the notice rate of each feature. Focus (68.5%, 403/588), Customize (68.8%, 11/16), Assessment (70%, 14/20), Quiet (70.6%, 12/17), and Participate (70.6%, 12/17) had very similar notice rates (around 70%). For the four passive features, the notice rate of Rewards (96%, 119/124) is as high as in the first study, suggesting the students like to receive the Rewards messages. Comparing to Rewards, Focus, Quiet and Participate received lower notice rates. For other features requiring a reply, Survey (93.8%, 30/32), used near or after the end of class, was almost never disregarded. All four Take a Break messages were noticed successfully. Assessment and Survey had much higher notice rates than in the first study, most likely due to longer vibrations.

Temporal Analysis of Features. Figure 8 presents the average usage of WELI features throughout the five intervals for all classes per student. We notice no trends in the usage of WELI among all students along time. However, we notice two patterns in usage. Pattern 1 (students 2 and 3) received fewer interventions than the others. Being that student 2 and 3 received less than 1.5 interventions on average for the first 4 intervals and around 2 messages in the last interval when the survey was sent. Pattern 2 (students 4, 5 and 6) received more than 5 average interventions for all class duration. Compared to the pattern 2, Focus is not the main message received for pattern 1 (students 2 and 3), especially for student 2 who received few focus messages and more other messages including rewards. For pattern 2 (students 4, 5 and 6), focus stands out among all interventions, being more popular than rewards and any other messages. These two patterns might indicate that neurodiverse students who have focus issues need more interventions in class. Similarly to the first study, there is a relationship between the number of focus messages exchanged and overall usage of WELI for the students 4, 5 and 6. In other words, when students received more focus in one duration, they use more WELI in that duration. For Rewards, student 5 and 6 received rewards in all class duration, students 2 and 4 received them in 4 duration, and student 3 only received it in the first duration. Another interesting aspect is that the students 4 and 6 had the same assistant. Although their interventions show different patterns, we notice a similar behavior from their assistant in both cases, since she sent much more messages than other assistants (total of 17 messages for student 4 and total of 14 messages for student 6 for a single duration). While, the maximum number of messages exchanged by the other assistants was 8 in the second study. We note that the interventions varied among different students but there might be a potential similarity regarding the assistant work style.

Fig. 8.
figure 8

WELI usage for the five 15-min class intervals per student in the second study. The average usage of Focus (orange), Rewards (green) and Others (blue) per class interval are shown, ranging from D1 (1st time interval) to D5 (5th time interval). (Color figure online)

Full size image

Sequential Analysis of Features. As in the first study, the results of the second study show that after Focus, 73.7% interventions are again Focus. Also, the second most common intervention after Focus is Rewards (16.5%). After Rewards, 25.2% interventions are Rewards again. Focus (61.3%) is the most likely intervention to follow a Rewards. We note the need for sustained interventions when helping students to focus in class. There are less Rewards sent consecutively than in Study 1, and also a larger number of Focus.

Fig. 9.
figure 9

The questionnaire results of 5 students in the second study. The top part includes 8 positive statements and the lower includes 4 negative statements. 

Full size image
Fig. 10.
figure 10

The questionnaire results of 4 assistants from the second study. 7 questions in positive statements (top), and 5 negative statements (bottom). 

Full size image

5.7 Results of the Questionnaire

As in the first study, we also applied a follow-up questionnaire to students and assistants who participated in Study 2.

Students’ Responses. The questionnaire included 12 questions, responded by five students of the second study. Figure 9 shows the results. For the smartwatch, most students (4 of 5) agree it is ‘comfortable’ and no students judged it as too heavy or too big. All students believe WELI is ‘easy to use’ and 4 students agreed that WELI is ‘cool’ and helped them in class (only one disagreed). Most students were not ‘distracted’ by the smartwatch in class, except one. Most students (3) felt more confident in class with WELI and 2 were neutral. Most students (3) agreed WELI made them more independent and want to continue using WELI in class. There are mixed responses from students on whether WELI made them feel less stressed. Most students liked the smartwatch, and thought WELI helped them in multiple aspects in class, e.g. making them feel more comfortable. Some students have opposite feelings. This requires more in-depth studies to assess whether an excessive number of interventions causes more stress to students.

Assistants’ Responses. In the second study, we applied the same questionnaire as in the first study to the 4 assistants (Fig. 10). All assistants agreed that WELI is ‘easy to use’, without confusing or interrupting their work, but making assistance more efficient by helping them. They also believe that it enabled the students to be more independent in class, and they wanted to continue using WELI. In comparison to the first study, there was overall a higher positive response about WELI on various aspects. However, there are neutral opinions about: the ‘preference’ to assist the student using WELI, WELI making their work ‘less stressful’ or more ‘distracting’ to them. Compared to the first study, only one assistant thought WELI added ‘extra effort’ to their workload. Overall, the assistants gave very positive feedback about using WELI in practice.

For multi-choice questions about the most useful features in class, three assistants chose ‘Focus, Rewards, Quiet’ and one assistant chose ‘Focus, Rewards, Customize’. The Focus and Rewards were indeed the most used features according to the log analysis which validates such preference. Only one assistant chose ‘Customize’ rather than ‘Quiet’ as the third favorite feature. We noticed in logs that indeed one assistant sent more customized interventions than others. Concerning the open questions, three assistants suggested that WELI should allow the student to initiate the communication with them.

5.8 Results of the Focus Group

Analogously to the first study, in the second study we conducted a focus group with the four assistants. Among them, 3 assistants helped 3 students, and one helped 3 students in 3 different classes.

Among the benefits of WELI, most assistants found it supportive to the assistance and to the students, by ‘consistently’ helping the students with ‘Focus’ and with positive reinforcement through the ‘Rewards’ feature. Most students ‘enjoyed’ and ‘loved’ WELI and never felt ‘bored’ or ‘tired’ of it. One student had ‘focus’ issues, but with WELI he ‘would look at the professor and try hard to focus and stay in class more’ rather than ‘gaze out the window or want to leave class’ as before. The student was ‘kind of attached to the smartwatch’ and ‘wanted to have it for his other classes as well’ as a ‘physical reminder’ of behavior moderation even without receiving any intervention. One assistant sent his student customized messages like ‘take your notebook out and leave the phone away’ a few times and then her student remembered these customized messages as she would perform the corresponding actions each class even without receiving them. One assistant told her student became more independent with WELI, because in other classes without WELI the student kept asking her questions and ‘relied more on her’.

Among the limitations pointed, one assistant mentioned that WELI worked well for her student in the first study but not in the second, as her student was neither interested nor responding, due to external factors. One assistant told that it was hard to help her student with discussions in class as the student would be often ‘off-topic’ and she used ‘Take a Break’ but her student did not want to come back, so she had to go out to call him back. While WELI may help assistants to locate their students, it may not help in engaging them in activities that they are not willing to participate in. Two assistants noticed that his/her student would be too excited when receiving ‘Rewards’ messages like ‘you rock’. They suggested sending only ‘thumbs up’ or ‘well done’ as “milder” positive rewards. One assistant mentioned that she was about to use ‘customized’ feature in class, but she was afraid that editing and adding the message would take time necessary to observe and intervene with her student.

For suggestions, two assistants mentioned the students ‘had trouble sliding on the watch’ interface to answer questions for survey and assessment. They suggested to add brief instructions about the sliding direction with the question (an arrow prompt). Three assistants suggested to prepare ‘customized’ messages and add those as generic prompts, since most students will need them. Three assistants also wanted WELI to allow students to initiate the communication with them, so that they could know when the student wanted a ‘restroom’, ‘water’ or a ‘longer break’. As WELI records the number of ‘Rewards’ to send stars, one assistant suggested to design a rule to transfer these stars to actual ‘prizes’ to further encourage students along the semester.

6 Discussion

Triangulation. From the analysis of the data collected from two user studies, triangulating methods was essential. Talking to assistants in the focus groups helped to validate (or to refute) findings based the initial log analysis. The questionnaires provided more comprehensive information, besides complementing the findings from focus groups.

Usage of WELI. Concerning the features, we learned the importance of Focus to help neurodiverse students in class and also of Rewards as positive reinforcement. There was no unified pattern in the interventions’ frequency and time across students. However, there were some patterns depending on the intervention, e.g. Focus was more frequent than other interventions for some students. No unified usage patterns on time also proved that assistance is primarily driven by events and context (not time), and customized per student. It also indicated that the assistant interventions are indispensable, but can benefit from applications like WELI to improve their work efficiency, consistency and accountability, rather than replacing them.

Students. There is a fine trade-off in the assistance provided, leading to mixed results, in other words, the students who performed well and needed less intervention are not the key beneficiaries of some features (e.g. Focus, Quiet), as WELI would prove itself to be less useful for them. On the other hand, students who do not perform very well, e.g. having focus issues in class, get more help from WELI. What we learn is that we should provide two modes in WELI, one offering more functions to intervene and improve students’ performance in class, and another dedicated to functions such as positive reinforcement or self regulation to encourage students to be more independent. The assistant could switch modes for the same student depending on his or her performance in class. As one assistant mentioned, her student “performed very well and focused with the help with app, except that he had bad days”. We need further studies to verify the long term impacts of WELI, hypothesizing that the longer the students use the application, the less they need it, and also to identify the specific contexts in which WELI does not lend the expected results.

Assistant. Besides the diverse needs of each student, we noticed that the assistant style may impact the effectiveness of WELI. Although assistants send interventions on demand, based on the student’s behaviors in class, we noticed certain tendencies of the assistant with the log analysis, Those include, one assistant always sent the student ‘Smiley face’ and ‘Thumbs Up’ at the beginning (the first minute) of the class like a greeting message. Another assistant helped two students for two different classes. Although the help the two students received was different, they received more interventions than all other participants of the study, and that assistant mentioned in focus group that her student ‘took off the smartwatch once, which may due to the delivery of so many interventions to him’. We plan to add some restrictions on WELI and study protocol, and to improve the training, to prevent such situations in the future and to protect students from unnecessary stress.

External Factors. External factors had a strong influence in the evaluation of WELI in class. These factors are not always under the control of the evaluator but should be considered carefully, for instance by triangulating results from alternative methods, carefully selecting participants, and reaching out to a larger number of users. As two assistants informed us, there was one participant whose medication had recently been changed, therefore he/she was still in an adaptation process and was distracted in class. Hence, regardless of the format the intervention, it would not be effective.

7 Conclusion

Overall, the users had a positive response to use WELI. Also, certain students seemed to benefit the most from the wearable assistance than others. We believe that there is a threshold in the neurodiversity spectrum in which the solution not only help to assist the students but also to train them to become more independent in what regards self-regulation and attention. Focus and rewards interventions stood out as most used features for the assistance inclusive class in the wearable format. We did not find a universal temporal pattern of the interventions sent throughout the class intervals, indicating that the next-generation assistive technology for neurodiverse students following inclusive classes should rely neither on time nor on sequence of interventions, but be context as well as event-driven.

Individual profiles of students must be taken into account as well. Unsurprisingly, neurodiverse users have heterogeneous behaviors, therefore the assistance model should be flexible and distinct enough to accommodate for their individual characteristics. While this project helped to unveil the usefulness of wearable assistance to neurodiverse students, there are open questions we would like to explore further in the future. Those include, the implementation and assessment of functions to allow student to trigger the assistance through the watch, to intervene the ‘off-topic’ problem for discussion in class, to design enough messages to intervene in the students’ behaviors not causing distraction or over-excitement, and to add a function to track student’s location during class break by using the built-in GPS in the watch.