Advertisement

A cognitive brain–computer interface monitoring sustained attentional variations during a continuous task

  • Antoine Gaume
  • Gérard Dreyfus
  • François-Benoît VialatteEmail author
Open Access
Research Article
  • 62 Downloads

Abstract

We introduce a cognitive brain–computer interface based on a continuous performance task for the monitoring of variations of visual sustained attention, i.e. the self-directed maintenance of cognitive focus in non-arousing conditions while possibly ignoring distractors and avoiding mind wandering. We introduce a visual sustained attention continuous performance task with three levels of task difficulty. Pairwise discrimination of these task difficulties from electroencephalographic features was performed using a leave-one-subject-out cross validation approach. Features were selected using the orthogonal forward regression supervised feature selection method. Cognitive load was best predicted using a combination of prefrontal theta power, broad spatial range gamma power, fronto-central beta power, and fronto-central alpha power. Generalization performance estimates for pairwise classification of task difficulty using these features reached 75% for 5 s epochs, and 85% for 30 s epochs.

Keywords

Brain–computer interface EEG Attention CPT Cognitive BCI 

Introduction

Since its first definition by William James a century ago (James 1890), attention has been generally defined by its cognitive role in the information processing occurring in the brain, rather than by its neuronal substrate or physiological function. This way of defining attention is what makes it so complex and difficult to study (Richard 1980).

The first cognitive models of attention describe it in terms of filters (Treisman 1964; Deutsch and Deutsch 1963; Norman 1968). In these models, percepts are filtered according to their relevance to the organism to be driven towards working memory for conscious processing and possibly long-term storage. Afterwards, the notion of attentional load emerged: attention was modeled as a limited capacity system (Kahneman 1973), where limited mental resources are shared between the different information-processing tasks the brain has to run. Attention becomes responsible not only for selection of relevant activities, but also for inhibition of distracting stimuli. This model includes every cognitive task the brain has to run in his theory of attention and not simply the process of selecting between competing simultaneous stimuli, on which the earlier bottleneck theories had focused. The attentional capacity of a person can vary depending on his arousal (Berlyne 1960), which, in turn, can be influenced by the attended tasks—this is for instance one explanation of driver’s drowsiness (Chen et al. 2018; Zeng et al. 2018). Later on, attention models became modular and more executive (Johnston and Heinz 1978; Posner and Snyder 1975): attention is considered as a flexible mental process, where voluntary control of attention requires more resources than automatic attentional discrimination. Attention was finally modelled from a neurocognitive perspective (Posner and Petersen 1989; Petersen and Posner 2012), as a unified system for the control of mental processing involving a network of anatomical areas independent from the sensory input processing areas of the brain, with three main subsystems of attention: alerting, orienting and detecting.

Attention has several submodalities [e.g. overt or covert attention, focused attention, selective attention, divided attention, alternating attention, vigilance, see Gaume et al. (2015) for a detailed discussion]. We are interested specifically in sustained attention, i.e. the self-directed maintenance of cognitive focus in non-arousing conditions while possibly ignoring distractors and avoiding mind wandering (Roberton and Garavan 2004; Clayton et al. 2015; Gaume et al. 2016). This state of maintaining attention is related to what is called concentration outside the field of neuroscience. Some authors use the term sustained vigilance or just vigilance to designate this state (e.g. Warm et al. 2008), but it is more widely accepted that vigilance refers to a condition of sustained arousal with the purpose of ensuring that important or surprising percepts will not be missed (Gaume 2015) (whereas we are interested in this study in the maintenance of overt attention towards a non surprising or even boring stimulus without mind wandering).

A recent framework for visual attention (Knudsen 2007) defined attention as a competitive selection process. This framework is well-compatible with real-time modelling in cognitive BCI. It introduces an explicit relationship between attentional networks and the direct control of the sensory organs that generate the incoming stimuli. It also integrates the concept of salience filters to describe the set of brain networks that automatically estimate the pertinence of an incoming sensory input (Gaume et al. 2016; Gaume 2015). The present investigation was grounded in Knudsen’s attentional model, with a neuro-cognitive perspective coming from EEG modeling, where attention is considered as the result of three coordinated processes (Stuss et al. 1995; Clayton et al. 2015):
  1. (A)

    Monitoring and evaluation of ongoing cognitive processes, competitive selection process to determine which information gains access to working memory (Desimone and Duncan 1995), therefore linking working memory, attention, and task load.

     
  2. (B)

    Excitation of task-relevant processes.

     
  3. (C)

    Inhibition of task-irrelevant processes.

     
The cognitive control involved in sustained attention tasks is correlated with frontal midline theta-band activity (Cavanagh and Frank 2014; Clayton et al. 2015; Wokke et al. 2017), congruent with [A] monitoring processes. Increased gamma-range oscillations are believed to promote task-relevant activity (Reinhart et al. 2011; Akimoto et al. 2013; Ahveninen et al. 2013; Potes et al. 2014; Clayton et al. 2015) congruent with [B] excitation processes. Alpha power has been linked with inhibition of task-irrelevant sensory modalities in task-irrelevant cortical areas (Makeig and Inlow 1993; Bollimunta et al. 2008; Snyder and Foxe 2010; Toscani et al. 2010; Anderson and Ding 2011; Mazaheri et al. 2014; Clayton et al. 2015; Wang et al. 2016), congruent with [C] inhibition processes.

Brain–computer interfaces (BCIs) are communication systems that enable a direct and real-time exchange of information between the brain and the external world [see Nicolelis (2011) for an introduction to this subject]. The first BCI development attempt, which also served as a proof of concept, was carried out in 1973 by Jacques Vidal and his team in California. Their experiment was called the “BCI project” and was meant to evaluate the feasibility and practicality of utilizing the brain signals in a man–computer dialogue (Vidal 1973). The goal of BCI systems is to create communication pathways that differ from the normal input/output channels used by the brain, namely the sensory organs to capture information about the world and the peripheral nervous system coupled with the muscles to interact with the environment (Wolpaw et al. 2000). The purpose of such alternative pathways is frequently viewed as a means of assisting in the rehabilitation of disabled or paralysed persons, to whom BCIs can be of great help by either replacing a defective sensory input or providing substitute ways to interact with the world. However, many other applications can emerge from the development of real-time brain signals decoding and stimulation techniques. They include applications of neurofeedback, and applications outside of the medical world, such as in BCI games (Marshall et al. 2013), performance analysis (Tharawadeepimuk and Wongsawat 2017), sonification (Sayan et al. 2019), silent communication devices1 or ways to improve cognitive activity.2

Thorsten Zander introduced a slightly different classification in which exogenous BCIs are referred to as reactive while endogenous interfaces are separated into active and passive, respectively if the subject consciously triggers control signals or if the interface passively monitors the user’s brain state (Zander et al. 2010; Zander and Kothe 2011). To further refine this classification, we call sensory BCI a device with a control signal that is a correlate of sensory processing, motor BCI a device that uses activity from the motor cortex, and cognitive BCI a system that monitors cognitive functions (Gaume et al. 2015). The long-term goal of our research is the development of a cognitive BCI able to monitor the variations of sustained attention in real time. This “attentionometer” could for example warn its user immediately and objectively that its attention shifted towards a distractor. It could also be used to determine its user’s ability to pay attention at certain moments of the day, for example to know he is capable of driving. Such a device could also be used to train sustained attention by providing a continuous feedback to the user. More generally, being able to monitor our attention as directly as, for instance, the position of our arm, would probably allow us to learn how to consciously regulate our attention, and to find ways to concentrate easily and comfortably during long periods of time. This is the principle of neurofeedback (Lachaux 2011; Arns et al. 2017; Gaume et al. 2016).

The challenge that is adressed in the present work is to find a neural correlate of the variations of sustained attention that can be monitored in real time and that, ideally, does not require invasive hardware nor the performance of a specific task. Classical approaches to monitor sustained visual attention are based on the Mackworth Clock (Mackworth 1948), and involve long sessions of visual monitoring in search for unusual events. The objective measure used in that case as an indicator of attention is the response time of the subjects to the appearance of targets. However, changes in sustained attention can be measured either as fluctuations, or deteriorations of performances along time (Clayton et al. 2015). Most subjects indicate that their attention fluctuates between targets and that each unusual event recaptures their focus. The discrete response time measurements used in such experiments are therefore more related to deteriorations of attention, while the experiment fails to capture the fluctuations occuring in between the targets. In addition, all subjects who show a significant increase in their response time over the course of the experiment report drowsiness and dozing. Consequently, it is difficult to disambiguate such an experiment from the monitoring of alertness and vigilance (i.e. sustained arousal). Therefore, we deem this experimental approach not optimal for the monitoring of sustained attention. Indeed, it only requires a discrete involvement of attention and only allows a discrete measurement of performance. Consequently, the design of our cognitive BCI involves a continuous task: the subject has to focus continuously, the evaluation of attentional load relies on continuous stimulations, and the performance of the subject is measured continuously.

We therefore introduce an experimental paradigm involving a continuous task for the monitoring of variations of visual sustained attention. Furthermore, we intend to design a task that minimizes the involvement of cognitive functions other than sustained attention. We therefore decided to focus on the attention required to continuously update the visual information we get from our sensory inputs instead of a task based on continuous processing of the information stored in working memory, which would involve working memory load in addition to sustained attention.

Materials and methods

Continuous performance task (CPT)

Continuous Performance Task (CPT) is a test used in neuropsychology for the assessment of sustained and selective attention. Sustained attention is the ability to maintain concentration over time on a given task while selective attention refers to the ability to focus on relevant stimuli in a distracting environment. CPT paradigms generally involve multiple repetitions of a rapid presentation of stimuli with infrequently occurring targets. More details about CPTs can be found in Riccio et al. (2002). The task we developed is different from static CPT but keeps the idea that the evaluation of sustained attention requires a continuous involvement of the subject. Due to the bias-variance dilemma (Geman et al. 1992), if sources of variability exist in the data, it creates a lower bound (the so-called irreducible error) for classification accuracy. We therefore wanted to avoid discrete evaluations of performance, in order to constrain the subject to maintain continuously his attention without fluctuations. Indeed, such fluctuations would induce label noise in our database, which would increase the irreducible error.

Ecological stimuli are of great importance for BCI designs (Huang et al. 2018). On the move estimation of attentional mechanisms (continuous estimation) using video games is more ecological than classical static displays (discrete estimation) (Szalma et al. 2014). In our experiment, the subjects play a driving video game using a joystick: they have to maintain overt visuo-motor cognitive control, involving both the maintenance of cognitive control and visual attention [see e.g. Anguera et al. (2013) about driving video games and attention). It also involves mechanisms unrelated with sustained attention itself, such as arousal, stress regulation, sensory integration of visual cues, and the motor planning and execution of the task. However, these unrelated mechanisms are present in both the low, medium and high difficulty conditions. Our task design was aimed at increasing attention-related functional loads differentially according to task difficulty. In other words, despite the task itself is not measuring only sustained visual attention, the variation of difficulty is more specific to it. This attentional engagement was confirmed by two observations: first, subjects involved in higher difficulties reduced their eye blink frequency, a known correlate of visual attention [see e.g. (Kim et al. 2014)]. Second, they lost the ability to answer direct verbal questions in the hard condition [attentional inhibition typical of attentional tunneling (Wickens and Alexander 2009)].

Our task consists in the motor control of a cursor using a joystick. The concept is simple: subjects of the experiment sit in front of a computer screen displaying a black circle (the target) on a grey background (see Fig. 1 for an illustration). A cursor moves randomly and continuously on the screen and subjects are asked to keep this cursor inside the circle using the right joystick of a joypad (EG-C1036, Targetever Technology Co. Ltd.). The difficulty of the task is adjusted by modifying the speed of the random movement. Performance of the subjects can be monitored in real time based on the correction applied to the random movement. The whole experiment was developed using PsychToolBox-3 for MATLAB (Brainard 1997; Kleiner et al. 2007) and displayed on a 120 Hz screen with a resolution of \(1920 \times 1080\) pixels.
Fig. 1

Illustration of the CPT interface. The subject of the experiment tries to keep a randomly moving cursor inside the target circle using a joystick. The difficulty of the task can be adjusted by changing the speed of the cursor

Experimental procedure

After installation of the electrodes and presentation of the EEG principles and signals, subjects were trained on the CPT several times at low speed (150 px/s) to make sure they understood how to manipulate the joypad.

A calibration session was then performed, during which each subject played the game at 20 different difficulty levels (starting at 75 px/s and up to 1500 px/s, increasing the speed by 75 px/s at the beginning of each sequence). Each level was played continuously for 20 s and subjects controlled when to begin the rounds so they could take breaks in-between. Data from the calibration phase was used to determine the speed of the pointer during the rest of the experiment. This calibration phase lasted less than ten minutes.

The recording session then started. Each subject played a total of 60 rounds of the game at three different difficulty levels (20 “easy”, 20 “medium” and 20 “hard”). The first round was an “easy” round, followed by a “medium” round and a “hard” round. This was repeated 20 times, in other words the presentation order was easy-medium-hard-easy-medium-hard-easy-medium-hard-etc. Each round lasted 30 s for a total duration of around 40 minutes. The subjects controlled when to begin the rounds so they could take breaks in-between. The cursor speed for the “easy” levels was always 150 px/s. Cursor speeds for “medium” and “hard” levels were determined according to calibration results as the speeds for which the cursor would stay 95% and 50% of the time in the circle respectively. Speed ranges were [375, 750] px/s for “medium” level and [650, 900] px/s for “hard” levels, depending on the subject. Last round score, best scores and average scores (percentage of time spent by the cursor inside the circle) for each difficulty were shown to the subject between each round to stimulate his/her motivation.

Experimental conditions

EEG recordings took place in a dark room, where subjects were seated in a comfortable armchair, about one meter away from the screen used to display the CPT. The subjects were shown their EEG activity prior to the recording and explanations were given about muscular artefacts and eye blinks. They were instructed to relax and prevent excessive muscular contractions or eye movements.

Subjects

Seventeen (17) healthy subjects took part in the experiment. Three (3) of them were rejected from the study because the recorded data were too noisy.3 Fourteen (14) subjects remained, among which eleven (11) were males and three (3) females, with an average age of 23.7 (SD 3.9, range 19–32). All had normal or corrected-to-normal vision and none of them had any known history of epilepsy, migraine or any other neurological condition. The study followed the principles outlined in the Declaration of Helsinki. All participants were given explanations about the nature of the experiment and signed an informed consent form before the experiment started.

Data acquisition

EEG signals were continuously recorded at a sampling rate of 2 kHz using 16 active Ag/AgCl electrodes from an actiCap system, connected to a V-Amp amplifier, both from Brain Products. The electrodes were placed according to the 10–20 system with a focus on frontal, parietal and occipital regions at positions Fp1, Fp2, F7, F3, F4, F8, C3, C4, CP5, CP1, CP2, CP6, P3, P4, O1 and O2, as illustrated in Fig. 2. Two additional electrodes were used as ground and reference and were located respectively at AFz and FCz.
Fig. 2

Electrode placement for CPT recordings. Brain activity was recorded using 16 active electrodes (in green), located all over the scalp with a focus on the frontal, parietal and occipital regions which cover several regions involved in attention and visual processing. (Color figure online)

Signal processing

Analyses were performed using MATLAB 2013a. The recorded EEG signals were filtered between 0.5 and 90 Hz using a zero-phase 3rd-order digital Butterworth filter, and the same kind of filter was applied around 50 Hz to remove the power-line noise. Filtering was applied on the raw signals before any segmentation to avoid boundary effects.

Eye blink rejection

Preliminary experiments showed that subjects produced fewer eye blinks when they were engaged in tasks requiring a high level of visual attention than in other tasks. In order to avoid classification of our data based on the number of eye blink artifacts found in EEG signals, we needed either to remove all epochs containing eye blinks, thereby reducing significantly the size of our dataset, or to find a way to remove eye blinks from EEG signals.

Therefore, a Second-Order Blind Identification (SOBI) algorithm from the EEGLAB toolbox was used to decompose recorded EEG signals into independent components. Eye blink activity and strong eye movements artefacts were removed before signal reconstruction from SOBI components. Details about this algorithm can be found in Belouchrani et al. (1993). More information about the eye blink removal using ICA can be found in Jung et al. (2000).

Feature extraction and selection

This section presents results obtained using only spectral features extracted from EEG power spectra. All features were extracted from epochs of 1, 3, 5, 10 and 30 s. Absolute and relative EEG power in the delta (\(\delta\) 1–4 Hz), theta (\(\theta\) 4–8 Hz), alpha (\(\alpha\) 8–12 Hz), low beta (\(\beta -\) 12–18 Hz), high beta (\(\beta +\) 18–25 Hz), low gamma (\(\gamma -\) 25–35 Hz) and high gamma (\(\gamma +\) 35–45 Hz) frequency bands were extracted from each channel of each EEG epoch, accounting for a total of 224 features per epoch (14 features and 16 channels). Absolute power refers to the total power in a given frequency band, obtained using FFT and a Hanning window on a given epoch. Relative power refers to the ratio of the absolute power in a given frequency band to the absolute power in the whole spectrum, taken between 1 and 45 Hz.

We used analysis time windows of 1, 3, 5, 10 and 30 s. These epoch lengths span in an exponential progression the [1–30] s range, so we could test more short windows than large windows. We tested these different time windows in order: (1) to test for stability to parameter changes, and (2) to provide a global overview of the tradeoff between accuracy and speed.

When classification was not based on a single variable, candidate variables were ranked in order of decreasing relevance by Orthogonal Forward Regression (OFR); feature selection was performed by the random probe method, as described in Stoppiglia et al. (2003). Random variables (probes) were added to the feature set and only variables that ranked better than 95% of the probes were kept for classification (35 features out of 224).

Classification

Classification of the EEG epochs was performed using Linear Discriminant Analysis [LDA, see for instance Fukanaga (1990)]. LDA estimates separating hyperplanes in the feature space by seeking the direction, in feature space, such that the projections of the classes on this direction have maximum iner-means distance and minimum variance. Likelihood ratios based on the distance to these hyperplanes are used to assign new observations to the different classes. LDA has the advantages of a very low computational requirement, and a single solution with a given training set. This classifier is simple to use, and has been used with success in a great number of BCI (Lotte et al. 2007). LDA assumes Gaussian distributions, which holds true for our features.

Epochs were labelled as “easy”, “medium” or “hard”, depending on the difficulty of the task. We performed pairwise classification (Knerr et al. 1990) (between all pairs of classes) and three-class classification. Since the datasets used for classification were balanced, expected classification accuracies using random features were 50% for pairwise classification and 33.3% for three-class classification. Feature selection and classifier training were performed using data from all subjects except one and data from the remaining subject were used for validation. This is repeated for each subject and accuracies are averaged (Leave-One-Subject-Out, LOSO).

Results

Single-feature classification

Table 1 shows the evolution of the best accuracy obtained using a Bayes classifier (i.e. a single feature LDA) as a function of the epoch length. For each discrimination, we selected the feature that ranked first in the ranking provided by the OFR method. It can be observed that the best accuracy increases with epoch length for all classification scenarios. However, 30 s epochs seem to be only slightly easier to classify than 1s epochs using a single spectral power feature.
Table 1

Best accuracies using a single spectral power feature for different epoch lengths and for the four classification scenarios described in “Classification” section

Epoch duration (s)

3-Class classifier (%)

“Easy” versus “medium” (%)

“Easy” versus “hard” (%)

“Medium” versus “hard” (%)

Best LOSO classification accuracies

1

40.5

60.1

60.8

54.5

3

42.2

62.7

63.1

55.8

5

42.8

64.3

65.0

57.1

10

44.0

65.5

65.8

58.2

30

46.2

68.8

66.4

60.2

Table 2

Two-class classification results obtained using a single spectral power feature when separating EEG epochs of 10 s recorded during two discriminations among “easy”, “medium” and “hard”

The five features giving the best average accuracies are listed for each classifier. Sensitivities and specificities are given for the best threshold. Accuracies obtained using absolute EEG power for each frequency band and each channel are presented as topographic maps (frontal electrodes are located at the top). Details about frequency bands can be found in “Feature extraction and selection” section

Tables 2 and 3 illustrate the results obtained on 10 s epochs using single feature LDA for respectively three-class classification (“easy” vs. “medium” vs. “hard”) or pairwise classification (between any pair of difficulty levels). We observe that the best accuracies obtained in “easy” versus “medium” (65.5%) and “easy” versus “hard” (65.8%) discriminations come from the same feature (absolute theta power at Fp1) and are significantly higher than the accuracies obtained for “medium” versus “hard” classification. The accuracy maps obtained with absolute spectral power are also very similar in the “easy” versus “medium” and “easy” versus “hard” discriminations, and both qualitatively look like the accuracy map obtained in the 3-class discrimination. We also observe that the features giving the best results are mostly low frequency features (delta and theta band power) in the prefrontal, superior parietal and central cortices. These observations are similar with other window lengths (not illustrated).
Table 3

Three-class classification results obtained using a single spectral power feature when separating EEG epochs of 10 s recorded during “easy”, “medium” and “hard” discriminations

The five features giving the best average accuracies are listed. Accuracies obtained using absolute EEG power for each frequency band and each channel are presented as topographic maps (frontal electrodes are located at the top). Details about frequency bands can be found in “Feature extraction and selection” section

Multiple-feature classification

Tables 4 and 5 show the results obtained using LDA classification with multiple features in the four discriminations described in “Classification” section. All results are given for different epoch lengths ranging from 1 to 30 s. Accuracies are obtained using a leave-one-subject-out cross validation method. Along with the classification accuracies, the best features selected by OFR for the classification of 10 s epochs are listed for each discrimination.
Table 4

Classification results using multiple features for three-class classification (left) and “easy” versus “medium” classification (right)

Accuracies are given for different epoch lengths as a function of the number of features used by the classifier. The best ten features selected by OFR on 10 s epochs are listed for both classifiers

Table 5

Classification results using multiple features for “easy” versus “hard” classification (left) and “medium” versus “hard” classification (right)

Accuracies are given for different epoch lengths as a function of the number of features used by the classifier. The best ten features selected by OFR on 10 s epochs are listed for both classifiers

On average, LOSO classification accuracies increase with the number of features for all discriminations and all epoch lengths. Using 30 s epochs, accuracy reaches 64.8% for the three-class classifier using 26 features; 85.5% in the “easy” versus “medium” discrimination using 14 features; 89.3% in the “easy” versus “hard” discrimination using 26 features and 77.3% in the “medium” versus “hard” discrimination using 25 features. With shorter 5 s epochs, a situation more likely to happen in a real cognitive BCI, accuracy reach 51.8% with the three-class classifier; 75.6% in the “easy” versus “medium” discrimination; 76.2% in the “easy” versus “hard” discrimination and 63.5% in the “medium” versus “hard” discrimination.

It can be noted that the “easy” versus “medium” classification is the discrimination that reaches its accuracy plateau with the lowest number of features, thereby decreasing the risk of overfitting.

The features selected for each classifier are different and, as expected, not directly linked with the features giving the best accuracy when classification is performed using a single input. However, except for the “medium” versus “hard” classifier, which uses several features in the delta range (1–4 Hz), the other three classifiers select similar features among their best ten including:
  • Several features based on absolute or relative theta power (4–8 Hz), taken from multiple locations over the scalp, from frontal to occipital regions,

  • features based on high gamma power (35–45 Hz), taken in the central or parietal regions,

  • a feature based on frontal low beta power (12–18 Hz),

  • a feature based on central high beta power (18–25 Hz),

  • a feature based on centro-parietal alpha power (8–12 Hz).

Subjective feedback

After the experiment, each subject was asked about his perception of the difficulty levels of the game.
  • All subjects found the “easy” rounds very easy. Some reported boredom. On average, 99.8% of “easy” play time was spent inside the target circle.

  • Most subjects reported that the “medium” levels were the most interesting and engaging, as they had the impression of having a real control over the movement of the cursor. 94.6% of total “medium” play time was spent inside the target.

  • All subjects found that “hard” levels were by far the hardest, and most of them reported that it was slightly less motivating than the “medium” difficulty because they felt they did not have enough control over the fast-moving cursor. Some subjects however found this difficulty very challenging and interesting. On average, 56.4% of “hard” play time was spent inside the target circle.

In order to control that the task had an effect on cognitive load, we confirmed on three subjects the effect of variations in task difficulty, using the NASA Task Load Index (Hart and Staveland 1988). The six dimensions (Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration) of the scale were compared between “easy” versus “medium”, as well as “medium” versus “hard” conditions, using a Wilcoxon paired test with Bonferroni corrections for multiple comparisons. Task difficulty change led to significant increase of Mental Demand (\(p < 0.05\) in all subjects and both conditions), there were no significant changes with the other dimensions of the scale (\(p > 0.10\) in all subjects and both conditions).

Discussion

Subjects of the experiment reported that the “medium” rounds and the “hard” rounds required more or less the same concentration. We therefore expected that classification would work better between “easy” and “medium” or between “easy” and “hard” rounds, than between “medium” and “hard” rounds. This is consistent with the results shown on the tables. It is also consistent with the fact that the features selected to discriminate “medium” and “hard” rounds are different from those selected for the other classifiers.

In order to estimate BCI performances, LOSO classification accuracies were used. Classical BCI models are designed and tested within subjects. Instead, LOSO scores are estimates of the generalization performance of the classifier onto new subjects, and it is therefore an estimate of transfer learning (Cheng et al. 2017). The choice of the LOSO validation approach was made to avoid biases in the estimates. Indeed, we might achieve higher accuracies on this database using within-subject analysis instead of cross-subject classifications. However, a within-subject model would increase the risk of modeling subject-related motor and/or cognitive confounders, which would be a serious flaw for a cognitive BCI (Gerjets et al. 2014). Furthermore, a LOSO approach allows us to design a cognitive BCI system which does not need calibration. In other words, it can be applied right away on new subjects without retraining.

Pairwise classification (“easy” vs. “medium” or “hard” rounds) reached 75% even with short 5 s epochs. Accuracies above 85% were obtained using 30 s epochs. Note that these accuracies were obtained without any calibration of the model to the left-out subject. Due to the bias-variance dilemma (Geman et al. 1992), if sources of variability exist in the data, it creates a lower bound (the so-called irreducible error) for classification accuracy. Considering that BCI illiteracy affects 15–30% of the population (Jeunet et al. 2016), the irreducible error is 15%, and therefore accuracies higher than 85% would be suspicious and might indicate a flawed model. These accuracies are therefore close to the optimum, which illustrates the robustness of the BCI model and show that prediction of the current cognitive load required by a task can be monitored using EEG. Such performances are largely sufficient for several biomedical applications. Indeed, neurofeedback and monitoring applications do not need a high accuracy, but a high specificity (above 85%), so that it would trigger only useful alarms. We can achieve such a detection by increasing specificity at the cost of sensitivity, consequently our system can already be used for biomedical applications in its present state.

In the “easy” versus “medium” discrimination, the LOSO classification accuracy estimations reached a plateau when the number of features exceeded ten, thereby proving that the LDA classifiers designed with these features are not very prone to overfit our data. However, in other discriminations, and especially in the “easy” versus “hard” discrimination, accuracy increased with the number of features and did not seem to reach a plateau (see Table 5).

Frontal theta power was a good predictor of the task load, providing a classification accuracy of 66% to separate the “easy” task from higher demand “medium” and “hard” tasks. This result is in accordance with previously reported observations: an increase of frontomedial theta is the best known correlate of sustained attention (Klimesch 1999; Clayton et al. 2015). Joint EEG and fMRI analysis have shown that prefrontal theta oscillations play a key role in salience and executive functions (Mas-Herrero and Marco-Pallars 2016) serving as a common substrate for the engagement of pre-supplementary motor area and the dorsomedial prefrontal cortex. Knudsen’s model does consider salience filters and executive functions as core components of attention (Knudsen 2007). Augmentation of frontal theta has also been widely reported in association with working memory load (Gevins et al. 1998; Jensen and Tesche 2002; Gevins and Smith 2003; Sauseng et al. 2005; Cavanagh and Frank 2014; Hsieh and Ranganath 2014; Fairclough and Ewing 2017).

High beta activity in the left central region provided good classification results. This activity may be linked with motor control, since the task involved the control of a joystick with the right hand. Nevertheless, the high beta frequency band, containing spectral activity between 18 and 25 Hz, does not overlap with the sensorimotor rhythm, and may therefore be linked with cognitive load and attention more than with direct control of motor activity. This is consistent with the work of Wróbel (2000), which supports that the 15–25 Hz frequency band is a general carrier for attention in the brain. Hypotheses about the sources of this frontal activity could be made, however precise reconstruction would require a high number of electrodes (64\(+\)), which is why this method has not been used with our equipment.

Attention is based on brain networks involving coupling of several frequencies (Clayton et al. 2015). In order to target these complex interactions, the feature selection approach which was used in our investigations extracts biomarkers carrying complementary information. Instead of relying on one single marker, the resulting cognitive BCI model combines multiple features (see Table 4). Overall, the attentional load is predicted by a combination of prefrontal theta power, broad spatial range gamma power, fronto-central alpha power and fronto-central beta power. This combination of channel and frequency ranges is coherent with the existing knowledge about the underlying networks of attention: frontomedial theta rythms for attentional monitoring processes, gamma-range oscillations for excitation processes, and alpha power for attentional inhibition processes. As we already explained above, the beta range activity could be correlated with either cognitive load or sensorimotor confounders of the task.

Another potential confounder is the variation of arousal during the task. The session was organized with successive trials of increasing difficulty, which may induce a correlation of mental fatigue with the task condition (the very first trial was always easy, the very last always hard). A randomized order would be a better control for this mental fatigue confounder. However, a randomized order would also introduce an anxiety confounder : not knowing the level of difficulty in advance was more stressful when the difficult condition occurred, whereas anxiety levels were more comparable between conditions when the subjects could anticipate it. Both mental fatigue and anxiety can have an impact on arousal, we chose to prevent anxiety.

Conclusion

To conclude, we presented in this paper an experiment aimed at discriminating low versus high sustained attention states in a continuous task. This continuous task approach introduces a new type of cognitive BCI design, where we attempt to control the cognitive load using an ecological video-game where a cognitive constraint is continuously maintained to prevent load fluctuations. We showed that features such as average frequency band power, which can be estimated continuously and do not require discrete stimulations, provide good classification rates even with short epochs (5 s). A leave-one-subject-out cross-validation approach was used to avoid overfitting. Nevertheless the present investigation is yet insufficient to prevent modeling confounders of attentional load. For instance, event though we removed eye blinks using SOBI, we cannot rule out that some features participating in the classification process may be related to eye movement or electromyographic artefacts, which might act as a potential confounder of sustained attention. Consequently, other experiments will be required to differentiate potential confounders (Gerjets et al. 2014), i.e. the features specific to the motor control task of this investigation, emotional, arousal or stress correlates [see for instance Mühl et al. (2014)], artifacts, and the other cognitive loads that are not specific to sustained visual attention.

Footnotes

  1. 1.

    See for instance Grau et al. (2014) or Rao et al. (2014) for promising brain-to-brain communication interfaces.

  2. 2.

    See e.g. http://dreem.com for an example of BCI for the general public aimed at improving the quality of sleep.

  3. 3.

    The number of electromyographic artifacts made the recording unexploitable.

Notes

References

  1. Ahveninen J, Huang S, Belliveau J, Chang W, Hämäläinen M (2013) Dynamic oscillatory processes governing cued orienting and allocation of auditory attention. J Cogn Neurosci 25(11):1926–43Google Scholar
  2. Akimoto Y, Kanno A, Kambara T, Nozawa T, Sugiura M, Okumura E et al (2013) Spatiotemporal dynamics of high-gamma activities during a 3-stimulus visual oddball task. PLoS One 8(3):e59969Google Scholar
  3. Anderson K, Ding M (2011) Attentional modulation of the somatosensory mu rhythm. Neuroscience 180:165–180Google Scholar
  4. Anguera J, Boccanfuso J, Rintoul J, Al-Hashimi O, Faraji F, Janowich J, Kong E, Larraburo Y, Rolle C, Johnston E, Gazzaley A (2013) Video game training enhances cognitive control in older adults. Nature 501(7465):97–101Google Scholar
  5. Arns M, Batail JM, Bioulac S, Congedo M, Daudet C, Drapier D, Fovet T, Jardri R, Le-Van-Quyen M, Lotte F, Mehler D, Micoulaud-Franchi JA, Purper-Ouakil D, Vialatte FB, The Next Group (2017) Neurofeedback: one of today’s techniques in psychiatry? Encephale 43(2):135–145Google Scholar
  6. Belouchrani A, Abed-Meraim K, Cardoso J, Moulines E (1993) Second-order blind separation of temporally correlated sources. In: Proceedings of the international conference on digital signal processing. Citeseer, pp 346–351Google Scholar
  7. Berlyne DE (1960) Conflict, arousal, and curiosity. McGraw-Hill, New YorkGoogle Scholar
  8. Bollimunta A, Chen Y, Schroeder C, Ding M (2008) Neuronal mechanisms of cortical alpha oscillations in awake-behaving macaques. J Neurosci 28(40):9976–9988Google Scholar
  9. Brainard DH (1997) The psychophysics toolbox. Spat Vis 10:433–436Google Scholar
  10. Cavanagh J, Frank M (2014) Frontal theta as a mechanism for cognitive control. Trends Cogn Sci 18(8):414–421Google Scholar
  11. Chen J, Wang H, Hua C, Wang Q, Liu C (2018) Graph analysis of functional brain network topology using minimum spanning tree in driver drowsiness. Cogn Neurodyn 12(6):569–581Google Scholar
  12. Cheng M, Lu Z, Wang H (2017) Regularized common spatial patterns with subject-to-subject transfer of eeg signals. Cogn Neurodyn 11(2):173–181Google Scholar
  13. Clayton M, Yeung N, Cohen Kadosh R (2015) The roles of cortical oscillations in sustained attention. Trends Cogn Sci 19(4):188–195Google Scholar
  14. Desimone R, Duncan J (1995) Neural mechanisms of selective visual attention. Ann Rev Neurosci 18(1):193–222Google Scholar
  15. Deutsch A, Deutsch D (1963) Attention: some theoretical considerations. Psychol Rev 70(1):80Google Scholar
  16. Fairclough S, Ewing K (2017) The effect of task demand and incentive on neurophysiological and cardiovascular markers of effort. Int J Psychophysiol 119:58–66Google Scholar
  17. Fukanaga K (1990) Statistical pattern recognition, 2nd edn. Academic, New YorkGoogle Scholar
  18. Gaume A (2015) Towards cognitive brain–computer interfaces: real-time monitoring of visual processing and control using electroencephalography. PhD thesis, Université Pierre et Marie CurieGoogle Scholar
  19. Gaume A, Abbasi MA, Dreyfus G, Vialatte FB (2015) Towards cognitive BCI: neural correlates of sustained attention in a continuous performance task. In: 2015 7th international IEEE/EMBS conference, neural engineering (NER), pp 1052–1055Google Scholar
  20. Gaume A, Jaumard-Hakoun A, Mora-Sanchez A, Ramdani C, Vialatte FB (2016) A psychoengineering paradigm for the neurocognitive mechanisms of biofeedback and neurofeedback. Neurosci Biobehav Rev 68:891–910Google Scholar
  21. Geman S, Bienenstock E, Doursat R (1992) Neural networks and the bias/variance dilemma. Neural Comput 4:1–58Google Scholar
  22. Gerjets P, Walter C, Rosenstiel W, Bogdan M, Zander T (2014) Cognitive state monitoring and the design of adaptive instruction in digital environments: lessons learned from cognitive workload assessment using a passive brain–computer interface approach. Front Neurosci 8:385Google Scholar
  23. Gevins A, Smith M (2003) Neurophysiological measures of cognitive workload during human–computer interaction. Theor Issues Ergon Sci 4:113–131Google Scholar
  24. Gevins A, Smith M, Leong H, McEvoy L, Whitfield S, Du R, Rush G (1998) Monitoring working memory load during computer-based tasks with EEG pattern recognition methods. Trends Cogn Sci 40(1):79–91Google Scholar
  25. Grau C, Ginhoux R, Riera A, Nguyen TL, Chauvat H, Berg M, Amengual JL, Pascual-Leone A, Ruffini G (2014) Conscious brain-to-brain communication in humans using non-invasive technologies. PLoS One 9(8):e105,225Google Scholar
  26. Hart S, Staveland L (1988) Development of nasa-tlx (task load index): results of empirical and theoretical research. In: Hancock P, Meshkati N (eds) Human mental workload. Advances in psychology. North Holland, Amsterdam, pp 139–183Google Scholar
  27. Ho Y (1999) The no free lunch theorem and the human–machine interface. IEEE Control Syst 19(3):8–10Google Scholar
  28. Hsieh L, Ranganath C (2014) Frontal midline theta oscillations during working memory maintenance and episodic encoding and retrieval. Neuroimage 85(Pt 2):721–729Google Scholar
  29. Huang M, Jin J, Zhang Y, Hu D, Wang X (2018) Usage of drip drops as stimuli in an auditory P300 BCI paradigm. Cogn Neurodyn 12(1):85–94Google Scholar
  30. James W (1890) The principles of psychology. Henry Holt and Company, New YorkGoogle Scholar
  31. Jensen O, Tesche C (2002) Frontal theta activity in humans increases with memory load in a working memory task. Eur J Neurosci 15(8):1395–1399Google Scholar
  32. Jeunet C, N’Kaoua B, Lotte F (2016) Advances in user-training for mental-imagery-based bci control: psychological and cognitive factors and their neural correlates. Prog Brain Res 228:3–35Google Scholar
  33. Johnston WA, Heinz SP (1978) Flexibility and capacity demands of attention. J Exp Psychol Gen 107(4):420Google Scholar
  34. Jung TP, Makeig S, Humphries C, Lee TW, Mckeown MJ, Iragui V, Sejnowski TJ (2000) Removing electroencephalographic artifacts by blind source separation. Psychophysiology 37(2):163–178Google Scholar
  35. Kahneman D (1973) Attention and effort. CiteSeer, PrincetonGoogle Scholar
  36. Khasnobish A, Datta S, Bose R, Tibarewala D, Konar A (2017) Analyzing text recognition from tactually evoked EEG. Cogn Neurodyn 11(6):501–513Google Scholar
  37. Kim S, Kang M, Kang G, Park S, Shin Y, Jang D (2014) The investigation of the relationship between eye blink and visual attention with video clip. J Biomed Eng Res 35(4):99–104Google Scholar
  38. Kleiner M, Brainard D, Pelli D, Ingling A, Murray R, Broussard C (2007) Whats new in psychtoolbox-3. Perception 36(14):1Google Scholar
  39. Klimesch W (1999) Eeg alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain Res Brain Res Rev 29(2–3):169–195Google Scholar
  40. Knerr S, Personnaz L, Dreyfus G (1990) Single-layer learning revisited: a stepwise procedure for building and training a neural network. In: Fogelman-Soulié F, Hérault J (eds) Neurocomputing algorithms, architectures and applications. NATO ASI Series, vol 68. pp. 41–50Google Scholar
  41. Knudsen EI (2007) Fundamental components of attention. Annu Rev Neurosci 30:57–78Google Scholar
  42. Lachaux JP (2011) Cerveau attentif (Le): Contrôle, maîtrise, lâcher-prise. Odile Jacob, ParisGoogle Scholar
  43. Lotte F, Congedo M, Lécuyer A, Lamarche F, Arnaldi B (2007) A review of classification algorithms for EEG-based brain–computer interfaces. J Neural Eng 4(2):R1–R13Google Scholar
  44. Mackworth N (1948) The breakdown of vigilance durning prolonged visual search. Q J Exp Psychol 1(1):6–21Google Scholar
  45. Makeig S, Inlow M (1993) Lapses in alertness: coherence of fluctuations in performance and EEG spectrum. Electroencephalogr Clin Neurophysiol 86(1):23–35Google Scholar
  46. Marshall D, Coyle D, Wilson S, Callaghan M (2013) Games, gameplay, and BCI: the state of the art. IEEE Trans Comput Intell AI Games 5(2):82–99Google Scholar
  47. Mas-Herrero E, Marco-Pallars J (2016) Theta oscillations integrate functionally segregated sub-regions of the medial prefrontal cortex. Neuroimage 143:166–174Google Scholar
  48. Mazaheri A, van Schouwenburg M, Dimitrijevic A, Denys D, Cools R, Jensen O (2014) Region-specific modulations in oscillatory alpha activity serve to facilitate processing in the visual and auditory modalities. Neuroimage 87:356–362Google Scholar
  49. Mühl C, Jeunet C, Lotte F (2014) EEG-based workload estimation across affective contexts. Front Neurosci 8:114Google Scholar
  50. Nicolelis M (2011) Beyond boundaries: the new neuroscience of connecting brains with machines–and how it will change our lives. Macmillan, New YorkGoogle Scholar
  51. Norman DA (1968) Toward a theory of memory and attention. Psychol Rev 75(6):522Google Scholar
  52. Petersen SE, Posner MI (2012) The attention system of the human brain: 20 years after. Ann Rev Neurosci 35:73Google Scholar
  53. Posner MI, Petersen SE (1989) The attention system of the human brain. Technical report, DTIC DocumentGoogle Scholar
  54. Posner MI, Snyder CRR (1975) Attention and cognitive control. In: Solso R (ed) Information processing and cognition: the Loyola symposium. Lawrence Erlbaum, Hillsdale, pp 55–85Google Scholar
  55. Potes C, Brunner P, Gunduz A, Knight R, Schalk G (2014) Spatial and temporal relationships of electrocorticographic alpha and gamma activity during auditory processing. Neuroimage 97:188–195Google Scholar
  56. Rao RP, Stocco A, Bryan M, Sarma D, Youngquist TM, Wu J, Prat CS (2014) A direct brain-to-brain interface in humans. PLoS ONE 9(11):e111,332Google Scholar
  57. Reinhart R, Mathalon D, Roach B, Ford J (2011) Relationships between pre-stimulus \(\gamma\) power and subsequent p300 and reaction time breakdown in schizophrenia. Int J Psychophysiol 19(1):16–24Google Scholar
  58. Riccio CA, Reynolds CR, Lowe P, Moore JJ (2002) The continuous performance test: a window on the neural substrates for attention? Arch Clin Neuropsychol 17(3):235–272Google Scholar
  59. Richard JF (1980) L’attention. Presses universitaires de France, ParisGoogle Scholar
  60. Roberton I, Garavan H (2004) Vigilant attention. In: Gazzaniga M (ed) The cognitive neurosciences, 3rd edn. MIT Press, Cambridge, pp 631–640Google Scholar
  61. Sauseng P, Klimesch W, Doppelmayr M, Pecherstorfer T, Freunberger R, Hanslmayr S (2005) Eeg alpha synchronization and functional coupling during top-down processing in a working memory task. Hum Brain Mapp 26(2):148–155Google Scholar
  62. Sayan N, Shankha S, Banerjee A, Sengupta R, Ghosh D (2019) Music of brain and music on brain: a novel EEG sonification approach. Cogn Neurodyn 13(1):13–31Google Scholar
  63. Snyder A, Foxe J (2010) Anticipatory attentional suppression of visual features indexed by oscillatory alpha-band power increases: a high-density electrical mapping study. J Neurosci 30(11):4024–4032Google Scholar
  64. Stoppiglia H, Dreyfus G, Dubois R, Oussar Y (2003) Ranking a random feature for variable and feature selection. J Mach Learn Res 3:1399–1414Google Scholar
  65. Stuss D, Shallice T, Alexander M, Picton T (1995) A multidisciplinary approach to anterior attentional functions. Ann NY Acad Sci 769:191–211Google Scholar
  66. Szalma J, Schmidt T, Hancock P (2014) Vigilance on the move: video game-based measurement of sustained attention. Ergonomics 57(9):1315–1336Google Scholar
  67. Tharawadeepimuk K, Wongsawat Y (2017) Quantitative eeg evaluation for performance level analysis of professional female soccer players. Cogn Neurodyn 11(3):233–244Google Scholar
  68. Toscani M, Marzi T, Righi S, Viggiano M, Baldassi S (2010) Alpha waves: a neural signature of visual suppression. Exp Brain Res 207(3–4):213–219Google Scholar
  69. Treisman AM (1964) Monitoring and storage of irrelevant messages in selective attention. J Verbal Learn Verbal Behav 3(6):449–459Google Scholar
  70. Vidal JJ (1973) Toward direct brain–computer communication. Ann Rev Biophys Bioeng 2(1):157–180Google Scholar
  71. Wang C, Rajagovindan R, Han S, Ding M (2016) Top-down control of visual alpha oscillations: sources of control signals and their mechanisms of action. Front Hum Neurosci 10:15Google Scholar
  72. Warm J, Parasuraman R, Matthews G (2008) Vigilance requires hard mental work and is stressful. Hum Factors 50(3):433–441Google Scholar
  73. Wickens C, Alexander A (2009) Attentional tunneling and task management in synthetic vision displays. Int J Aviat Psychol 19(2):182–199Google Scholar
  74. Wokke M, Cleeremans A, Ridderinkhof K (2017) Sure i’m sure: prefrontal oscillations support metacognitive monitoring of decision making. J Neurosci 34(4):781–789Google Scholar
  75. Wolpaw JR, Birbaumer N, Heetderks WJ, McFarland DJ, Peckham PH, Schalk G, Donchin E, Quatrano LA, Robinson CJ, Vaughan TM et al (2000) Brain–computer interface technology: a review of the first international meeting. IEEE Trans Rehabil Eng 8(2):164–173Google Scholar
  76. Wróbel A (2000) Beta activity: a carrier for visual attention. Acta Neurobiol Exp 60(2):247–260Google Scholar
  77. Zander TO, Kothe C (2011) Towards passive brain–computer interfaces: applying brain–computer interface technology to human–machine systems in general. J Neural Eng 8(2):025,005Google Scholar
  78. Zander TO, Kothe C, Welke S, Roetting M (2010) Enhancing human–machine systems with secondary input from passive brain–computer interfaces. In: Tan D, Nijholt A (eds) Brain–computer interfaces applying our minds to human–computer interaction. Springer, London, pp 181–199Google Scholar
  79. Zeng H, Yang C, Dai G, Qin F, Zhang J, Kong W (2018) Eeg classification of driver mental states by deep learning. Cogn Neurodyn 12(6):597–606Google Scholar

Copyright information

© The Author(s) 2019

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  • Antoine Gaume
    • 1
    • 3
  • Gérard Dreyfus
    • 1
  • François-Benoît Vialatte
    • 1
    • 2
    Email author
  1. 1.ESPCI Paris, PSL Université ParisParisFrance
  2. 2.CNRS UMR 8249, Brain Plasticity UnitBrain-Computer Interface TeamParisFrance
  3. 3.EPF École d’ingénieurSceauxFrance

Personalised recommendations