Electroencephalography-based endogenous brain–computer interface for online communication with a completely locked-in patient
Brain–computer interfaces (BCIs) have demonstrated the potential to provide paralyzed individuals with new means of communication, but an electroencephalography (EEG)-based endogenous BCI has never been successfully used for communication with a patient in a completely locked-in state (CLIS).
In this study, we investigated the possibility of using an EEG-based endogenous BCI paradigm for online binary communication by a patient in CLIS. A female patient in CLIS participated in this study. She had not communicated even with her family for more than one year with complete loss of motor function. Offline and online experiments were conducted to validate the feasibility of the proposed BCI system. In the offline experiment, we determined the best combination of mental tasks and the optimal classification strategy leading to the best performance. In the online experiment, we investigated whether our BCI system could be potentially used for real-time communication with the patient.
An online classification accuracy of 87.5% was achieved when Riemannian geometry-based classification was applied to real-time EEG data recorded while the patient was performing one of two mental-imagery tasks for 5 s.
Our results suggest that an EEG-based endogenous BCI has the potential to be used for online communication with a patient in CLIS.
KeywordsBrain-computer interface (BCI) Completely locked-in state (CLIS) Electroencephalography (EEG) Pattern classification Riemannian geometry
Amyotrophic lateral sclerosis
Common average reference
Completely locked-in state
Event-related spectral perturbation
Fast Fourier transform
Functional magnetic-resonance imaging
Linear discriminant analysis
Left motor imagery
Leave-one-out cross validation
Power spectral density
Spatial covariance matrix
Sequential forward feature selection
Steady-state visual evoked potential
Support vector machine
Tongue motor imagery
Brain–computer interface (BCI) is an emerging technology capable of translating human intentions into control signals, thereby enabling people to communicate with their external environment without any kinesthetic movement . BCI technology has primarily targeted patients with severe or complete motor dysfunction due to various neurological disorders and cardiovascular diseases. Among these target groups, patients with amyotrophic lateral sclerosis (ALS) have the potential to benefit most from BCI technology because they generally maintain unimpaired cognition even after complete loss of voluntary motor function. ALS is characterized by a rapidly progressive degeneration of motor neurons, leading to muscle weakness, respiratory paralysis, and ultimately death . Therefore, patients in advanced stage of ALS usually experience a physical condition, referred to as a locked-in state (LIS) , in which they remain aware of their external environment but lose control of voluntary muscles except for control over eye movements . In the final stages of ALS, the patients even lose the oculomotor function and are isolated from their external environment owing to the lack of communication means . This stage is referred to as the completely locked-in state (CLIS).
Various brain-imaging modalities have been used to implement BCI systems with the ultimate goal of communicating with patients in LIS. Among these, electroencephalography (EEG) has been the most widely used modality because of its portability, non-invasiveness, high temporal resolution, and a reasonable cost compared to other neuroimaging tools, such as near-infrared spectroscopy (NIRS), functional magnetic-resonance imaging (fMRI), and magneto-encephalography (MEG) . Over the last couple of decades, EEG-based BCI systems have been developed with elaborately designed paradigms [7, 8, 9, 10, 11, 12], with greatly promising practical applications. For example, steady-state visual evoked potential (SSVEP) [7, 8], event-related potential (ERP) [9, 10], and motor imagery (MI) [11, 12] are three major EEG-based BCI paradigms. These can be roughly classified into exogenous and endogenous BCI paradigms. The SSVEP and P300 BCIs are representatives of exogenous BCI paradigms. They use external stimuli such as flickering LEDs or auditory beeps to evoke discriminative brain patterns. On the other hand, MI-based systems represent endogenous BCIs as they do not use any external stimuli. If a user of an endogenous BCI performs one of the designated mental tasks, the system will recognize the specific task from self-regulated EEG patterns. The absence of external stimuli is especially advantageous for the patients in CLIS because of their complete lack of motor function. In addition, endogenous BCIs are more straightforward, more convenient, and may be faster than exogenous BCIs. Over the past decade, most EEG-based BCI systems were tested with healthy individuals [13, 14]. Recently, the number of EEG-based BCI studies in patients with ALS has steadily increased, several of which have yielded promising results [4, 5, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25]. Unfortunately, EEG-based BCI systems for online communication with patients in CLIS have been mostly unsuccessful. A recent study demonstrated the potential of using a vibrotactile stimulation-based exogenous BCI paradigm for communication with patients in CLIS . However, no study has yet reported on the successful online communication with CLIS patients using EEG-based endogenous BCI systems, which do not require external stimuli, but instead use neural signals volitionally modulated by BCI users. To the best of our knowledge, there were only two EEG-based endogenous BCI studies with CLIS patients; however, both failed to establish online communication [21, 27].
In this study, we developed an EEG-based online BCI system for the classification of different mental-imagery tasks conducted by a patient in CLIS. An endogenous BCI paradigm based on mental-imagery tasks was designed, and a single patient in CLIS with advanced stage of ALS participated in this study. Offline and online experiments were conducted to validate whether the proposed BCI could be used for real-time communication with the patient. The test–retest reliability of our BCI system was also investigated using training datasets obtained on different days.
A 62-year-old woman with severe ALS was recruited for this study. In November 2012, she was diagnosed with clinically probable ALS according to the Airlie House diagnostic criteria . After diagnosis, her motor weakness rapidly progressed and propagated to all motor subsystems, including bulbar and respiratory functions. In late 2013, mechanical ventilator and transcutaneous endoscopic gastrostomy were needed. Subsequently, all voluntary motor functions including ocular movement and eye blinking were completely lost as of early 2015. Despite of the complete loss of voluntary movement including eye movement, the participant’s event-related potential (ERP) waveform characteristics suggested that her hearing and cognitive functions were preserved. The participant’s CLIS diagnosis was additionally supported by the discovery of meaningful tearing after listening to sorrowful news of her family member’s death. Her ALSFRS-R was zero  and no alternative means of communication with the patient were available at the time of this study. The patient’s husband provided written informed consent prior to all experiments. This study was reviewed and approved by the Institutional Review Board Committee of Hanyang University Hospital (HYUH 2015–11–031-001) and conformed to the tenets of the Declaration of Helsinki.
All experiments were conducted in a hospital setting where the patient had stayed for 2 years. We visited her four times on different days (first visit: 28th Jan 2016; second visit: 4th Feb 2016; third visit: 12th Aug 2016; and last visit: 9th Dec 2016) and performed four different experiments at each visit. All the experiments were performed between 10 am and 12 pm because the patient’s husband told us that she had usually been in a good condition in the morning. During the experiments, all instructions were directly presented to the participant using noise-cancelling headphones.
In the online experiment based on mental-imagery tasks, the pair of mental tasks that showed the highest classification performance in the offline analysis, i.e. the combination of LMI and MS, was selected. Figure 2b and c are schematic diagrams of the training and test paradigms of the online experiment, respectively. The subject participated in 10 training runs without any feedback before the main online test experiment. Each training run was composed of four random, counter-balanced executions of either LMI or MS. The participant then completed four online test runs right after the training runs were complete. Each test run was composed of 10 trials (5 trials for each task). Real-time auditory feedback was provided immediately after each trial based on the classification results (please watch the supplementary video at https://youtu.be/zQWtSQOV50Q). The subject was able to recognize whether the online results were correct or not through auditory feedback.
EEG data acquisition
EEG data were recorded using a multi-channel EEG acquisition system (ActiveTwo, BioSemi, Amsterdam, The Netherlands). For the offline and online experiments, 19 EEG electrodes (Oz, O1, O2, Cz, C3, C4, T7, T8, Fz, FC1, FC2, FC5, FC6, F3, F4, F7, F8, AF3, and AF4) were used. The use of electrodes in the posterior region was avoided because the patient was in a supine position on a bed during experiments. The ground electrode was replaced with two electrodes, a common-mode-sense (CMS) active electrode and a driven-right-leg (DRL) passive electrode, both of which were located in the central region (near CP1 and CP2). The offset voltage between the A/D box and the body was maintained between 25 and 50 mV, as recommended by the EEG device manufacturer. The EEG data were sampled at 2048 Hz.
Analysis of ERPs by auditory oddball test
Raw EEG data acquired via the auditory oddball paradigm were pre-processed to remove unwanted artefacts using MATLAB software (MathWorks, Natick, MA, USA). A pre-processing algorithm used in a previous ERP study  was adopted. The raw EEG data were re-referenced with a common average reference (CAR) that uses the mean of all electrodes as a reference. The DC-level components in all channels were removed by subtracting the mean of the time series for each channel, and the EEG data were band-pass filtered at 1- and 30-Hz cut-offs using a third-order Butterworth zero-phase filter. The pre-processed EEG data were epoched from 100 ms pre-stimulus to 1000 ms post-stimulus. The averaged value of the pre-stimulus interval was subtracted from the selected epochs for baseline correction. We assessed whether the remaining epochs contained significant physiological or environmental artefacts (amplitude exceeding ±75 μV). Because no epoch satisfied this condition, all epochs were used in subsequent analyses.
Mismatch negativity (MMN), the negative peak generated near 200 ms where a human can hear some specific sounds [33, 34], was confirmed to determine whether the patient could hear all instructions presented through the headphones. In this study, the MMN was defined by a minimum amplitude between 100 and 300 ms post-stimulus. There were three different stimuli in our auditory oddball test: white noises (600 times), high-tone beeps (120 times), and low-tone beeps (120 times). In the MMN analysis, the noises and beeps were regarded as standard and deviant stimuli, respectively. The ratio of noises to beeps was 2.5:1. Two hundred forty epochs were randomly selected from 600 noise trials and were compared with the 240 beep epochs. P300 components in ERPs were evaluated on the midline electrodes (Fz, Cz, and CPz) as well as the bilateral temporal lobe electrodes (T7 and T8) to confirm the patient’s cognition. In the ERP analysis, high and low-tone beeps were used as target and non-target stimuli, respectively. A statistical analysis was performed to confirm the significance of the MMN and P300 components. For MMN, each epoch from 100 ms to 300 ms in deviant and standard trials was segmented using a moving window of 62.5 ms with 50% overlap for each channel. For P300, each epoch from 200 ms to 700 ms in all target and non-target trials was segmented using a moving window of 62.5 ms with 50% overlap for each channel. Then, paired t-tests were performed for all pairs of corresponding time segments to find time intervals with a significant difference (p < 0.05) between the target and non-target conditions. The p-values of all pairs were corrected using the Bonferroni correction method for multiple comparisons.
Event-related (de) synchronization analysis
The event-related desynchronization (ERD) and synchronization (ERS) patterns were evaluated for each mental task to validate whether the patient actually conducted the given mental tasks. The raw EEG data for each mental task were filtered using CAR and band-pass filtered at 4 to 45 Hz. Then, each 6-s epoch from − 1 to 5 s was extracted from the filtered EEG signals and was down-sampled from 2048 Hz to 512 Hz. The event-related spectral perturbation (ERSP) was calculated using the function newtimef in the EEGLAB toolbox (https://sccn.ucsd.edu/eeglab/index.php). The function returns estimates of the ERSP across event-related trials for each channel times in the series. As already mentioned, the EEG data from − 1 to 0 s was used as the baseline for the ERSP evaluation. The target frequency bands for each mental task were set differently: alpha (8–13 Hz) and low-beta (13–20 Hz) for LMI and TMI; theta (4–7 Hz) and alpha (8–13 Hz) for MS. The ERSP patterns at different target electrodes were also observed for each mental task: C3 and C4 for LMI and TMI; AF3 and AF4 for MS. Topographical distributions were drawn using the averaged ERSP values from 0 to 5 s for each frequency band.
Analysis of EEG data by the BCI paradigm based on mental-imagery tasks
Raw EEG data obtained during the mental task paradigm were pre-processed using MATLAB software. The raw EEG signals were spatially filtered using a CAR to compensate for common noise components. DC components were removed by subtracting the mean of the time series for each channel, and the EEG data were then band-pass filtered at 4- and 45-Hz cutoffs using a fifth-order Butterworth zero-phase filter. For the pattern classification of different mental tasks, each 5-s epoch (0–5 s after task onset) of all trials for each task was extracted from the pre-processed EEG signals. Finally, each epoch was down-sampled from 2048 Hz to 512 Hz.
The second classification framework was a traditional one using spectral band powers and their inter-hemispheric asymmetry ratios as EEG features. The frequency bands were separated into five sub-frequency bands: theta (4–7 Hz), alpha (8–13 Hz), low beta (14–20 Hz), high beta (21–30 Hz), and gamma (31–45 Hz). In order to extract the power spectral density (PSD) features, each epoch was divided into 1-s segments with 50% overlap, resulting in a total of nine-time segments. Each segment was then transformed into the frequency domain using a fast Fourier transform (FFT) with a Hamming window, and the average spectral power of each electrode at each frequency band was calculated by averaging the spectral powers of the nine-time segments. The asymmetry ratios between the right and left hemispheres were evaluated for each frequency band as (R-L)/(R + L), where R and L represent the spectral powers averaged over the right and left hemispheric electrodes, respectively. Consequently, 95 spectral power features (19 electrodes × 5 frequency bands) and 5 asymmetry ratio features (5 frequency bands) were evaluated for each trial. The LOOCV method was applied to assess the classification accuracy, considering the relatively small number of task trials performed. For each cross-validation, the sequential forward feature selection (SFFS) method was used to select the best feature subset for the current training dataset as well as to reduce the dimensionality of the feature vectors . The 1-nearest neighbor error was used as the criterion function for the SFFS method, and the maximum number of selected features was set to 10 to prevent potential over-fitting of the data. LDA and support vector machine (SVM) classifiers were used to calculate the classification accuracies .
Neurological assessment based on event-related potentials
ERD/ERS during each mental task
Results of the offline experiment
Online performance of our paradigm
We evaluated the test–retest reliability of the proposed mental-imagery-based BCI paradigm. Figure 6b depicts the results of test–retest reliability assessments. In the online experiment, training datasets were recorded immediately before online experimental runs (denoted by Online TR), which resulted in an online classification accuracy of 87.5%, as previously shown in Fig. 6a. During the online experiments, EEG data were recorded and used to confirm the test–retest reliability. We trained the classifier using a training dataset recorded from the same patient about 6 months before the online experiment (denoted by Offline TR). This classifier, trained with the old training dataset, was also applied to the EEG data recorded during the online experiment. Even when the classifier was trained with the old training dataset, the classification accuracy reached 85%, demonstrating high test–retest reliability of our paradigm. In addition, a new dataset was constructed by combining the old training dataset (Offline TR) with a newly acquired training dataset (Online TR). By combining the two training datasets (denoted by Combined TR), the classification accuracy was further improved to 90%, as shown in Fig. 6b. Unfortunately, our patient was unable to participate in follow-up experiments. When we tested the patient four months after the online experiment, we could not find any evidence of consciousness or alertness (see Fig. 3d).
Our EEG-based BCI system showed good performance in terms of accuracy and communication speed. The offline and online average classification accuracies were 95 and 87.5%, respectively, and the time for each trial was just 5 s. Most previous EEG-based endogenous BCI methods have failed to effectively communicate with patients in CLIS [21, 27]. A study by De Massari et al.  proposed a Pavlovian semantic conditioning paradigm, which comprised the presentation of affirmative or negative statements for conditioned stimuli and electrical stimulation of skin for unconditioned stimuli, for the basic communication in CLIS. They reported a few cases with a classification accuracy of up to 70%, but the performance of their protocol was not consistent over all experimental runs. More recently, Chaudhary et al.  tested a binary paradigm for CLIS patients. Their paradigm had known answers and open questions. Their patients were automatically thought of ‘yes’ or ‘no’ for the answers to the questions. According to their reports, the EEG results did not exceed the chance-level threshold for correct communication. These previous EEG-based endogenous BCI studies have used paradigms based on passively generated automatic responses to the given questions. These passive paradigms have been used because the researchers assumed their CLIS patients might not perform active mental tasks due to cognitive decline or dementia originating from the neurodegenerative diseases. We carefully guess this might be a reason for the failure of previous EEG-based endogenous BCIs. If the patients in CLIS can perform active mental tasks such as LMI or MS, the performance of the EEG-based endogenous BCIs may be much improved because more robust EEG features can be extracted. Although previous EEG-based endogenous BCI studies have not yielded promising results for communication with patients in CLIS, recent NIRS-based BCI studies showed potential for use for communication with patients in CLIS [27, 50, 51]. For instance, Naito et al. proposed a NIRS-based BCI paradigm using mental tasks and tested its performance in a sample of 17 patients in CLIS . About 40% of the patients in CLIS included in their study were able to successfully accomplish the experimental task with reliable performance, with an offline classification accuracy of approximately 80%. More recently, Gallegos-Ayala et al. proposed a new NIRS-based BCI paradigm that involved asking a patient in CLIS to think of either ‘yes’ or ‘no’ after yes-or-no questions , resulting in an average accuracy of 76.30%. An extended version of this NIRS study was reported in 2017 , in which the authors tested their NIRS-based BCI paradigm in a larger sample of patients in CLIS and confirmed that the patients could achieve an accuracy greater than 70%, better than that expected serendipitously. Although previous NIRS studies have shown promising results, the major disadvantages of these NIRS-based BCI paradigms are slow communication speeds and low classification accuracies. These paradigms generally require task lengths longer than 15 s for classification. In comparison, our method achieved both good classification accuracy (87.5%) and communication speed (5 s).
In 2017, a study by Guger et al.  demonstrated the potential of using a vibrotactile stimulation-based exogenous BCI paradigm for communication with patients in CLIS. However, the current study still makes an important contribution in the BCI community in that this is the first report showing that EEG-based ‘endogenous’ BCI paradigms can be used for online communication of patients in CLIS. Endogenous BCI is more ideal than exogenous BCI because it does not require any external stimuli in visual, auditory, and tactile forms, but only uses neural signals generated by performing designated mental-imagery tasks. In fact, the performance metrics of the present endogenous BCI were even better than those of the exogenous BCI applied to the patients in CLIS  (classification accuracy: 87.5% vs. 80% (two of three patients); communication speed: 5 s vs. 38 s). The only disadvantage of endogenous BCI paradigms might be the necessity of pre-training sessions to build classifiers (Note that the previous study  adopted an exogenous BCI paradigm, but it required training sessions); however, this study also showed that there were no significant differences in the classification accuracies even when a training dataset that was recorded several months prior to an online experiment was used for the online experiment. This suggests that patients in CLIS might use the proposed EEG-based BCI paradigm without any additional training runs for at least several months.
In 2012, Barachant et al. introduced a classification framework based on RG  that proved to be superior to conventional methods in terms of classification accuracy [35, 36, 37]. However, the performance of RG-based classification frameworks has never been evaluated for patients with ALS. In this study, we confirmed that the performance of an RG-based framework using covariance matrices as EEG descriptors outperformed LDA and SVM using spectral band powers as EEG features.
Brain signals may be the only feasible way for patients in CLIS to communicate with their external environment. In our study, a combination of LMI and MS showed the highest average classification accuracy of 89%, whereas the combination of LMI and TMI exhibited the lowest average classification accuracy of 62.5%. These results are in line with those of previous EEG-based BCI studies, in that the combination of motor and non-motor-imagery tasks resulted in better classification accuracy than the combination of two different motor-imagery tasks [30, 31].
We tested the proposed mental task-based BCI paradigm with only one patient in CLIS because it was extremely difficult to recruit patients who had lost all ability to communicate by conventional methods. Unfortunately, our patient was unable to participate in follow-up experiments after the only online experiment because her physical status worsened after recovering from acute pneumonia. When we revisited her and recorded her ERP signals while applying the auditory oddball paradigm, we were unable to find any evidence of consciousness or alertness (see Fig. 3d). We do not expect that our proposed BCI paradigm will be successfully applied to all patients in CLIS because every BCI paradigm suggested to date has failed in some participants. Nevertheless, we believe that our study is meaningful because our results suggest that EEG-based BCI systems can potentially be used for online binary communication with at least some patients in CLIS.
In this study, we implemented an EEG-based BCI system that can be potentially used for online binary communication with a patient with ALS in CLIS. In the offline experiment, we determined the best combination of mental tasks and the optimal classification strategy leading to the best performance. In the online experiment, we investigated whether our BCI system could be potentially used for real-time communication with the patient. An online classification accuracy of 87.5% was achieved when RG-based classification was applied to real-time EEG data recorded while the patient was performing either left-hand motor imagery or mental subtraction task for 5 s. The offline and online results demonstrate that EEG-based endogenous BCI might be a feasible method for communication with patients who cannot communicate through conventional methods that require intact motor function.
The authors are grateful to Dr. Marco Congedo at CNRS in France for sharing his software.
This work was supported by Institute of Information & Communications Technology Planning & Evaluation grant funded by the Korea government (MSIT) (2017–0-00432, Development of non-invasive integrated BCI SW platform to control home appliances and external devices by user’s thought via AR/VR interface).
Availability of data and materials
All relevant data are available at the following site: https://doi.org/10.6084/m9.figshare.6824798
C.-H. Han carried out all the experiments and data analysis, and also wrote the paper. Y.-W. Kim and D. Y. Kim helped with all experiments. Z. Nenadic and S. H. Kim helped with data analysis and reviewed the manuscript. C.-H. Im designed the experimental protocols and supervised the overall procedures of this study. All authors read and approved the final manuscript.
Ethics approval and consent to participate
Our participant’s husband provided informed consent and were reimbursed for her participation on completion of the experiment. This study was reviewed and approved by the Institutional Review Board Committee of Hanyang University Hospital (HYUH 2015–11–031-001) and conformed to the tenets of the Declaration of Helsinki.
Consent for publication
Our patient’s husband signed an informed consent to publish their data and images.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
- 11.Onose G, Grozea C, Anghelescu A, Daia C, Sinescu C, Ciurea A, Spircu T, Mirea A, Andone I, Spânu A. On the feasibility of using motor imagery EEG-based brain–computer interface in chronic tetraplegics for assistive robotic arm control: a clinical test and long-term post-trial follow-up. Spinal Cord. 2012;50(8):599–608.PubMedCrossRefGoogle Scholar
- 24.Wolpaw JR, Bedlack RS, Reda DJ, Ringer RJ, Banks PG, Vaughan TM, Heckman SM, McCane LM, Carmack CS, Winden S, McFarland DJ, Sellers EW, Shi H, Paine T, Higgins DS, Lo AC, Patwa HS, Hill KJ, Huang GD, Ruff RL. Independent home use of a brain-computer interface by people with amyotrophic lateral sclerosis. Neurology. 2018;91(3):e258–67.PubMedCrossRefGoogle Scholar
- 26.Guger C, Spataro R, Allison BZ, Heilinger A, Ortner R, Cho W, La Bella V. Complete locked-in and locked-in patients: command following assessment and communication with vibro-tactile P300 and motor imagery brain-computer interface tools. Front Neurosci. 2017;11:251.PubMedPubMedCentralCrossRefGoogle Scholar
- 29.Kim HY, Kim SH. Korean version of amyotrophic lateral sclerosis functional rating scale-revised: a pilot study on the reliability and validity. J Korean Neurol Assoc. 2006;25(2):149–54.Google Scholar
- 38.Devijver PA, Kittler J. Pattern recognition: a statistical approach: prentice hall; 1982.Google Scholar
- 46.Lin PT, Sharma K, Holroyd T, Battapady H, Fei D-Y, Bai O. A high performance MEG based BCI using single trial detection of human movement intention. In: Functional Brain Mapping and the Endeavor to Understand the Working Brain. InTech; 2013. p. 17–36.Google Scholar
- 52.Müller-Putz G, Scherer R, Brunner C, Leeb R, Pfurtscheller G. Better than random: a closer look on BCI results. Int J Bioelectromagn. 2008;10(EPFL-ARTICLE-164768):52–5.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.