Design and Evaluation of Mobile Applications for Augmentative and Alternative Communication in Minimally-verbal Learners with Severe Autism

Abstract

One of the most significant disabilities in autism spectrum disorders (ASD) includes a delay in, or total lack of, the development of spoken language. Approximately half of those on the autism spectrum are functionally non-verbal or minimally verbal and will not develop sufficient natural speech or writing to meet their daily communication needs.

A suite of evidence-based mobile applications, SPEAKall!^® and SPEAKmore!^®, was developed to help these individuals achieve critical speech and language milestones. SPEAKall! and SPEAKmore! enable early language learning, facilitate natural speech development, enhance generalization skills, and expand social circles as students learn. These solutions grow with the learner, enabling better participation in school and community, thus reducing the lifetime cost of care while enhancing chances for classroom success.

Evidence generation for the newly created applications involved: (a) Single-subject experimental designs to evaluate treatment efficacy through repeated measurement of behavior and replication across and within participants; (b) quantitative electroencephalograms to gain information about brain functioning.

The comprehensive approach to evidence-generation facilitated adoption of SPEAKall! and SPEAKmore! in clinical practice. It also allowed identifying critical app features that enhance skill acquisition and contribute to treatment effectiveness.

1 Communication Technology for Autism Spectrum Disorders

1.1 Communication Disorders in Autism

Individuals with autism spectrum disorder (ASD) experience a severe delay or atypical development in the area of communication. Sturmey and Sevin (1994) report that poor communication skills are hallmark symptoms included in most classification systems of ASD. Current diagnostic criteria for ASD are centered around a profound impairment in verbal and non-verbal communication used for social interaction (American Psychiatric Association 2013). The degree of this communication disorder can differ widely in individuals on the autism spectrum. Some learners develop speech and language slowly during the preschool years; it is estimated that about 50% reach phrased speech at the time of entering primary school (Anderson et al. 2007; Howlin et al. 2009). However, a proportion of about 30–50% demonstrate a severe lack in the acquisition of speech and language when making the transition to kindergarten settings (National Research Council 2001; Tager-Flusberg and Kasari 2013). These learners are often referred to as being “non-verbal” or only “minimally verbal” (Tager-Flusberg and Kasari 2013).

Because these learners do not develop sufficient natural speech or writing to meet their daily communication needs they often experience serious barriers for participation in education and society. Communicative abilities of these students may be limited to pre-intentional communication, such as reaching for a desired item, or communication may display intent through simple behaviors such as pointing (Yoder et al. 2001). If speech occurs it is typically limited to unusual or echolalic verbalizations (Paul 2005). Many become candidates for an area of clinical intervention that is known as augmentative and alternative communication (AAC).

1.2 Augmentative and Alternative Communication

Establishing even low levels of communication is an immediate and crucial need for learners with severe, minimally-verbal autism to take part in daily life. A common intervention approach to help these individuals successfully participate in communicative interactions is the use of augmentative and alternative communication (AAC). AAC is defined as the supplementation or replacement of natural speech and/or writing using aided and/or unaided strategies. Blissymbols, pictographs, Sigsymbols, tangible symbols, and electronically produced speech are examples of aided AAC. Manual signs, gestures, and body language are examples of unaided AAC. The use of aided symbols requires a transmission device, whereas the use of unaided symbols requires only the body (Lloyd et al. 1997). The most commonly applied AAC interventions for individuals on the autism spectrum include graphic symbols and speech-generating devices.

Graphic symbol sets and systems are a relatively modern AAC approach for individuals with ASD. As early as the 1980s, practitioners began to explore the potential benefits of graphic symbols because of their non-transient nature (e.g., Mirenda and Schuler 1988). Graphic symbol libraries can be constructed as sets or systems. Sets are collections of symbols that do not present with defined rules for their construction and expansion while systems are tied to an established repertoire of rules (see Lloyd et al. 1997). Graphic symbols most often seen in clinical practice for learners with ASD include: PCS, line drawings, colored photographs, Premack (all sets); Blissymbols, Orthography, Rebus (all systems) (Schlosser and Wendt 2008). Graphic symbol sets and systems that by their nature are more iconic (i.e., they present a better visual resemblance between symbol and referent) seem easier to be learned (Kozleski 1991).

Speech-generating devices (SGDs) represent another potentially viable option for minimally verbal learners with ASD. SGDs refer to a variety of high technology solutions including dedicated electronic communication devices, talking word processors, and handheld multi-purpose mobile devices (e.g., iPad®, iPod®, Android® tablets) loaded with AAC applications (apps). All of these solutions possess built-in technology that enables a learner to communicate via digitized and/or synthetic speech. Digitized speech is produced by using a recording of a human voice and transforming it into an electronic waveform. Digitized speech output can vary in its quality depending on the sampling rate used during the conversion process. SGDs and apps that have higher sampling rates typically produce higher quality speech output compared to those with lower sampling rates. Recording quality may also be influenced by noisy environments, equipment quality, age of the speaker, and characteristics of the speaker’s natural voice (Drager and Fink 2012). Initial research into the potential benefits of SGDs indicates that these newer technologies may have advantages over non-electronic AAC strategies because they provide additional auditory stimuli for the learner via speech output, which can facilitate receptive and expressive language development (Romski et al. 2010).

1.3 New Mobile Technologies

When Apple introduced the iPad in 2010, the toolbox for AAC options changed significantly. Older forms of SGDs, the more expensive, dedicated devices, were no longer the preferred strategy for minimally verbal children with autism. The iPad allowed running various AAC apps to accomplish the same functions that many SGDs provide. AAC intervention via an iPad became more cost-efficient, more socially appealing, and more motivating (Wendt 2014). Because the iPad’s functionality is broader than that of a dedicated device, it can be applied to a variety of activities that help children with autism beyond learning functional communication. In addition, using a tablet with an application for communication may be normalizing and less stigmatizing for a child with autism than a dedicated communication device. Lastly, the fact that the iPad provides many further opportunities for interacting with the device beyond communication purposes (i.e., games, video clips, photos, music, etc.) may provide very strong motivation for its use.

2 Design of AAC Applications for Intervention in ASD

2.1 The Rationale for an “Autism-Friendly” AAC Solution

When designing an AAC application for severe autism, it is critical to pay attention to the specific cognitive and sensory processing characteristics. In recent years, many AAC apps appeared on the market with the sole purpose of turning an iPad into a SGD and simply mimicking the SGD interface. The vast majority of these apps ignored particular behavioral and learning characteristics of individuals with autism. Such apps often came equipped with large sets of graphic symbols that when displayed on the iPad screen overwhelmed the learner with too many visual stimuli. An important sensory processing characteristic in autism is the difficulty to filter out salient and truly important incoming stimuli from a stimulus-rich environment (Minshew and Williams 2007). For the beginning communicator with autism, graphic symbols should be carefully selected and not be presented alongside other conflicting visual stimuli on the interface. Another less ideal feature of existing apps is a hierarchical organization of graphic symbol vocabulary. In order to retrieve a specific symbol, the learner first had to select from abstract and broader themes (e.g., “Food”, “People”, “Places”, etc.). This retrieval strategy increased cognitive load and created a level of abstraction too difficult for individuals with severe autism. Previous apps also included graphic symbols that had very little resemblance with their referents and were not highly iconic, that is, their meaning was not easy to comprehend.

To address the need for an AAC solution that can target critical speech and language milestones while taking into account the learning and processing characteristics of individuals with autism, the research team developed a suite of mobile AAC applications, SPEAKall!^® and SPEAKmore!^®.

2.2 SPEAKall! for Initial AAC Intervention

SPEAKall! facilitates the development of functional communication skills, natural speech production, and social-pragmatic behaviors (see Fig. 1).

Fig. 1.

An example for a communication display on the SPEAKall! application to teach functional communication skills. (Courtesy of the UCF Research Lab on AAC in Autism).

SPEAKall! provides a very intuitive interface for the non-verbal learner. The top portion of the screen features graphic symbols and pictures, and the bottom is a storyboarding strip where users can drag and drop symbols to create sentences. Learners with motor control difficulties can activate symbols with a simple touch gesture. The app also allows caretakers to add photos to the existing bank of symbols, creating relevant and recognizable content of the child’s everyday life. Once a child creates a sentence using the symbols, they can push the “SPEAKall!” icon to hear the sentence out loud. Voice-output can be generated via pre-recorded utterances from caretakers or via Apple’s synthetic Siri^® voice.

2.3 SPEAKmore! for Advanced Language Training

SPEAKmore! teaches advanced language concepts and their generalization to grow vocabulary and extend complexity of utterances (see Fig. 2).

Fig. 2.

An example for a 6 × 6 action-object matrix taught through the application “SPEAKmore!”. (Courtesy of the UCF Research Lab on AAC in Autism).

SPEAKmore! addresses the advanced learner in need of focused language training. The app follows a matrix training format and has three components: (1) An assessment mode to conduct a brief assessment of expressive and receptive language. (2) An intervention mode to present stimuli, elicit responses from the learner, and engage in error correction as needed. Different language learning lessons can be programmed. (3) A generalization mode to test the learner’s ability to extend the new skills learned in intervention to new stimuli.

2.4 Design Considerations

Development and clinical research on SPEAKall! and SPEAKmore! included direct experiences with end users and input from caretakers, clinicians, and teachers using a participatory design approach. This lead to identify subtle, but critical features that in turn influenced app design. This way, the research team was able to develop products that directly relate to the learning characteristics in autism and developmental disabilities. The following features are particularly noteworthy:

Randomization of symbol locations on the selection display forces the learner to truly look at the symbol and develop symbolic comprehension, an important precursor to language development; random placement of symbols prevents the learner from acquiring an automated motor sequence without full semantic understanding of selected symbols.

The user interface is designed with a “less is more” approach: keeping the interface simple and providing “easy access” is important in order to avoid cognitive overload and reduce barriers to successful iPad operation.

Backgrounds, colors, and stimuli are sensory-friendly, that is, the learner can easily distinguish major interface components (selection area versus sentence construction area), colors are calming, and there are no disturbing visual or sound effects that trigger autistic symptoms.

A “hidden lock” button keeps the learner focused on learning activities within the app and prevents avoidance and escape behaviors by not letting the learner browse off to unrelated, other apps.

“Drag and compose” is the default procedure for selecting symbols and moving them to the “sentence strip” to activate voice output; this facilitates construction of simple sentences. For learners with fine motor control issues, symbol activation is possible with a simple touch on or nearby the desired symbol.

Users can choose between pre-recorded speech or a variety of synthetic voices, which some users with autism may prefer over the human voice due to difficulties with auditory comprehension.

3 Clinical Evaluation of SPEAKall! and SPEAKmore!

The research team generated substantial scientific evidence on the effectiveness of SPEAKall! in addition to proof-of-concept data for SPEAKmore! Behavioral data and neurophysiological imaging data demonstrate treatment effects on developing functional communication skills and natural speech at a high level of scientific rigor.

3.1 Behavioral Evidence of Treatment Effects

A series of three different single-subject experiments were carried out to study different approaches to intervention with SPEAKall! Single-subject research designs are one of the most rigorous methods to examining treatment efficacy and are ranked equally to quasi-experimental group designs in evidence hierarchies for AAC (Schlosser and Raghavendra 2004). These designs are typically examining pre-treatment versus post-treatment performance within a small sample of participants (Kennedy 2005).

All study participants met the following criteria: (1) an official diagnosis of autistic disorder (all individuals were re-assessed by using the Childhood Autism Rating Scale, and the Autism Diagnostic Observation Schedule), (2) little or no functional speech operationalized as no more than 5 spoken words, (3) visual and auditory processing within normal levels, (4) adequate hand coordination for graphic symbol pointing on a tablet device, (5) understanding simple verbal commands (e.g., “Sit down”) and responding to yes-no questions.

Target skills and dependent measures included (a) the number of correct requests during a 20-trials session; and (b) the numbers of non-intentional utterances versus intentional vocalizations or word approximations.

Experiment 1 sought to produce replication effects across settings when SPEAKall! intervention took place in clinic, school, and home environments (Wendt et al. 2013a; 2014b). Four students between 10–13 years of age were introduced to the intervention protocol; three of the four mastered all five phases, and one individual achieved mastery of phase 3. All students improved in their ability to produce natural speech: Two students made significantly more intentional vocalizations (e.g., first letter of intended word or a word approximation), and another two students developed sufficient natural speech to produce one-word or two-word spoken requests. For all four students, these intervention effects were replicable across the three settings.

Experiment 2 aimed to replicate the prior intervention results with a cohort of three adolescents and adults between 14–23 years while focusing on response generalization to untrained stimuli (Wendt et al. 2019). All participants improved in their requesting abilities and were able to generalize the newly learned skills to untrained items. Mixed results occurred for targeting natural speech production: one out the three participants achieved the ability to speak simple sentences, two remained minimally verbal, suggesting that the ideal time to start intervention on speech production is at a much earlier age.

Finally, Experiment 3 evaluated the effects of a parent-training protocol for SPEAKall! intervention (Wendt et al. 2013b; Wendt et al. 2014a). This study involved three families with children between 6–8 years of age. Specific emphasis was placed on response generalization to untrained items and procedural fidelity of parent implementation. Results obtained suggest that with proper supervision and training parents can carry-out SPEAKall! intervention. All participants mastered the five phases of the intervention protocol, systematically increased symbolic utterances and spontaneous communication. Response generalization was demonstrated by extending acquired requesting skills from snack to untrained toy stimuli. Two of three children ended up speaking simple sentences without the help of SPEAKall! or their iPad, one child with a dual diagnosis of autism and Down syndrome did not show gains in speech production.

Behavioral Evidence Example.

Figure 3 below shows an example for the behavioral data for three participants from Experiment 2. These participants received SPEAKall! intervention to increase functional communication skills. Performance was measured on the ability to request desired items from baseline to intervention. As soon as intervention began, the participants started to show immediate improvement in requesting skills. This effect is replicated across the three participants, and is being maintained after intervention was completed. The figure shows the effects on developing requesting skills from baseline (pre-intervention) over the course of intervention. The horizontal axis displays the number of intervention sessions. The vertical axis displays the number of correct responses for requesting desired items.

Fig. 3.

Single-subject experiment showing SPEAKall! intervention effects on requesting skills replicated across three participants with severe autism, 14–23 years of age (Wendt et al. 2019).

3.2 Neurophysiological Evidence of Treatment Effects

An emerging clinical application of brain imaging in autism is to conduct a quantitative electroencephalogram (qEEG; Chan et al. 2007). A qEEG measures electrical activity produced by the brain and displays states of neural functioning in the form of a brain map. Such information allows one to pinpoint anomalies in brain function and to document neurophysiological changes over the course of intervention. This technique was used with some of the participants from Experiments 1–3 described above. The following is an example for one of those participants, a 14-year-old male with severe autism and no functional speech. Figure 4 shows the improvement of autistic brain activity from pre- to post-treatment (on the left and right hand sides, respectively).

Fig. 4.

Quantitative electroencephalogram (QEEG) shows resolving anomalies in electrical brain activity pre-and post-treatment for 14-year-old, male participant with severe autism (Courtesy of Autism Parent Care, LLC).

In the upper two rows of the brain maps, anomalies of electrical brain activity appear as red-shaded (too much activity) or blue-shaded (too little activity) areas. Improvement from pre-to post-intervention is demonstrated by a significant transformation of electrical activity to green-shaded areas (normal). The effect occurs on four major areas of neuro-cognitive functioning including emotion and sensation (theta), alertness (alpha), decision-making, information processing (beta), and agitation (high beta). The lower three rows show the ability of the autistic brain to establish connections between different cortical regions. Aberrant connections (over-connectivity) appear as red lines, lacking connections (under-connectivity) appear as blue lines; no lines indicate normal connectivity. Again, improvement is evident as red and blue lines significantly fade out over the course of intervention.

3.3 Proof-of-Concept Data for SPEAKmore!

Two participants, both 12 years old, diagnosed with severe autism and tested as minimally verbal on a standardized language assessment received matrix training through SPEAKmore! Both students were taught action-object symbol combinations on a 6 × 6 matrix as described earlier. The six actions included “point to”, “drop”, “take-out”, “put-in”, “shake”, and “wipe”; the six objects included “ball”, “cup”, “spoon”, “fork”, “apple”, and “car”. From the total pool of 36 possible symbol combinations, the researchers created four different sets of three symbol combinations each that were actively taught. The remaining 24 combinations were tested for generalization effects.

Experimental sessions started with a baseline condition followed by an intervention condition. During baseline, there was no training of combinations, students were presented the 36 video stimuli and asked to activate the correct combination on the screen. The intervention condition involved the active training of the four sets with all 12 combinations taught in sequence. After each intervention session generalization probes were conducted on the remaining 24 symbol combinations. Thus, two strands of data were created for each participant, one to document the acquisition of the training sets, and a second one to document generalization to the untrained combinations.

The extent of the intervention effects for each participant was quantified by calculating effect size estimates for single-subject data. Effect size is an indicator of the magnitude of change or difference between baseline to intervention conditions (Beeson and Robey 2006). One of the most recent effect size metrics for assessing effect size in single-subject data is the Non-overlap of All Pairs index (NAP; Parker and Vannest 2009). A score from 0–65% indicates weak effects, 66%–92% indicates medium effects, and 93%–100% indicates large or strong effects (Parker and Vannest 2009).

Effect sizes for the participants are shown in Table 1:

Table 1. Effect size estimates for two participants receiving intervention with SPEAKmore!

These results suggest that the matrix training intervention implemented via the SPEAKmore! app created an effect large enough to help the participants acquire action-object symbol combinations and generalize to new combinations not previously taught.

4 Conclusions

Generating research evidence to document the effectiveness of newly developed mobile technology is critical for evidence-based practice and decision-making. Valuable clinical lessons were derived from these research experiences:

• 1. Only providing the learner with a sophisticated AAC solution will not automatically result in improved communication. A proper instructional or intervention approach build around the learning characteristics in ASD needs to accompany the technology.

• 2. One single best SGD or AAC app for autism does not exist. Learners with autism present with varying needs and learning profiles and should receive individualized AAC solutions.

• 3. Whenever possible, practitioners should pursue solutions that are grounded in strong research evidence and have documented empirical support. Evidence-based practice is an essential component of successful AAC interventions.

• 4. AAC intervention should not be restricted to one-on-one sessions between learner and clinician but involve a variety of other communication partners, and occur during the entire day.

• 5. AAC does not inhibit natural speech production but facilitates its acquisition process.

• 6. When high technology AAC solutions break down or run out of battery, clinicians need to have a back-up strategy for their learners (e.g., simple communication board, picture exchange book, etc.).

• 7. Technology can be become outdated or outgrown quickly; AAC interventions should be evaluated regularly and technology be modified to meet the changing needs of the learner.

If we remember these principles, we can unlock the communication abilities of our learners with severe autism, maximize the benefits of AAC intervention, and enable better participation across academic and social environments (see Fig. 5).

Fig. 5.

Mother and young daughter engaging in social play facilitated by SPEAKall! (Courtesy of the UCF Research Lab on AAC in Autism).