1 Introduction

Motor imagery (MI) is the cognitive process of thinking about an action without motor execution. The characterization of MI tasks by mapping the brain activity has aroused growing interest over the past few years due to its high potential in brain-computer interfaces (BCI) applications, such as physical therapy, rehabilitation, and assistive technologies [2]. Thus, the main cortical brain imaging techniques employed to measure brain activity are electroencephalogram (EEG), magnetoencephalogram (MEG), and functional magnetic resonance imaging (fMRI). Nevertheless, EEG is the most widely used technique in BCI due to its advantages: non-invasive, low cost, and high temporal resolution. Besides, EEG allows observing brain electrical activity changes associated with specific stimuli. However, the development of BCI systems from EEG requires to minimize its poor spatial resolution, which leads to the volume conductor problem and the non-stationary behavior of the brain activity [6]. In particular, such non-stationary yields to performance variability regarding the studied subjects.

Most of the state-of-the-art approaches employ feature extraction techniques based on spectral representation to code significant EEG patterns. Particularity, frequency bands ranging from 8–30 Hz are used in MI tasks due to the relationship of the \(\mu \) and \(\beta \) rhythms in sensorimotor tasks [6, 10]. Also, data projections are employed to support MI discrimination, including Principal Component Analysis (PCA), Independent Component Analysis (ICA), Filter Bank Common Spectral Pattern (FBCSP), Common Spatial Pattern (CSP) [1, 3]. Moreover, connectivity representations like Phase Locking Value (PLV), Coherence, and Granger Causality (GC) [6], are used to discriminate MI classes from brain connectivity maps [5]. Next, given the feature space, classification algorithms have been used for distinguishing between MI tasks, such as Linear Discrimination Analysis (LDA), Neural Network (NN), Bayesian approaches, and Support Vector Machines (SVMs) [8, 11]. Besides, some methods face the non-stationarity in MI by choosing a predefined segment of the EEG signal. Although these approaches obtain acceptable classification results, high variability in performance among subjects persists. Moreover, there is a lack of quantitative relationships between a satisfactory MI classification, the recorded stimuli, and brain patterns with strong activity. That is, there is no clear evidence of the differences between the performance of the subjects to the coupling with the protocol and its stimuli in MI paradigms.

Here, we introduce an MI discrimination system to enhance the identification of relevant brain connectivity patterns and their relationships to the coupling of the subjects with the MI protocol and its stimuli. In particular, our approach comprises two main stages: (i) A CSP-based feature extraction to discriminate from different EEG window segments, and (ii) a PLV-based connectivity analysis to reveal significant dependencies among brain hemispheres concerning the most discriminative window segments. Specifically, our proposal allows finding a relationship between the classification performance of the subject and its coupling with the MI protocol. The obtained results show that subjects exhibiting acceptable classification rates achieve the highest discrimination within the temporal windows associated with the MI stimulus. Besides, in the mentioned segments, these subjects present consistent connectivity patterns in the contralateral hemisphere that seems to be related to the task under study. Otherwise, subjects with low classification performances do not show connectivity patterns according to the specific task; therefore, quantitatively it is demonstrated that their brain connectivity patterns are not in agreement with the MI protocol.

2 Materials and Methods

Let \(\{{\varvec{X}}_{l}{{\,\mathrm{\,\in \,}\,}}\mathbb {R}^{C\times T}\}_{l=1}^{L}\) be an EEG signal set holding L trials with C channels at T time instants. Moreover, each trial can be related to a given MI condition through the label vector \({\varvec{y}}{{\,\mathrm{\,\in \,}\,}}\left\{ -1,+1 \right\} ^{L}\). With the aim to code short-time discriminative patterns, a windowing process is carried out to divide each EEG trial \({\varvec{X}}_{l}\) into V segments of size R. So, at the l-th trial we built the following set: \(\{{\varvec{Z}}_{l,v}{{\,\mathrm{\,\in \,}\,}}\mathbb {R}^{C\times R}\}_{v=1}^{V}\). In turn, to reveal relevant brain patterns related to the studied MI paradigm, our EEG processing approach comprises two main stages: (i) MI discrimination from EEG-based short-time features, and (ii) EEG connectivity analysis in line with the MI acquisition protocol.

MI discrimination from EEG-Based Short-Time Features. Given the v-th segment at the l-trial, we compute a supervised linear mapping based on the well-known Common Spatial Patterns (CSP) algorithm. Namely, the CSP technique rotates the EEG channels through the matrix \({\varvec{W}}_v{{\,\mathrm{\,\in \,}\,}}\mathbb {R}^{C\times C}\) to avoid linear dependencies among EEG channels while enhancing the separability between MI classes. Thereby, the projection matrix at the v-th temporal window can be computed by solving the following eigenvalue problem: \({\varvec{\vartheta }}_v^{+}{\varvec{W}}_v={\varvec{\varDelta }}_v({\varvec{\vartheta }}_v^{-}+{\varvec{\vartheta }}_v^{+}){\varvec{W}}_v,\) where \({\varvec{\varDelta }}_v{{\,\mathrm{\,=\,}\,}}\mathrm{{diag}(\lambda _1,\lambda _2,\dots ,\lambda _C)}\) is a diagonal matrix holding the eigenvalues \((\lambda _1\ge \lambda _2\ge \dots \ge \lambda _C),\) and \({\varvec{\vartheta }}_v^{+},{\varvec{\vartheta }}_v^{-}{{\,\mathrm{\,\in \,}\,}}\mathbb {R}^{C\times C}\) are the covariance matrices of the input space for EEG trials belonging to the class \(+1\) and \(-1\), respectively. Afterward, a feature matrix \({\varvec{H}}_v{{\,\mathrm{\,\in \,}\,}}\mathbb {R}^{L\times 2M}\) is build for each temporal window, holding row vectors \({\varvec{h}}_{l,v}{{\,\mathrm{\,\in \,}\,}}\mathbb {R}^{2M}\) as the log variance of the projected EEG channels:

$$\begin{aligned} {\varvec{h}}_{v,l}=\log \left( \frac{\mathrm{{diag}}\left( {\varvec{\tilde{W}}}_v^\top {\varvec{Z}}_{l,v}{\varvec{Z}}^\top _{l,v}{\varvec{\tilde{W}}}_v \right) }{\mathrm{{Tr}}\left( {\varvec{\tilde{W}}}_v^{\top }{\varvec{Z}}_{l,v}{\varvec{Z}}^\top _{l,v}{\varvec{\tilde{W}}}_v \right) }\right) \end{aligned}$$
(1)

where \(\tilde{{\varvec{W}}}_v{{\,\mathrm{\,\in \,}\,}}\mathbb {R}^{L\times 2M}\) contains the first and the last M eigenvectors of \({\varvec{W}}\), \(\mathrm {diag}(\cdot )\), and \(\mathrm {Tr}(\cdot )\) stand to the diagonal elements and trace functions, respectively [3]. Further, an SVM classifier is trained to estimate the MI condition from the v-th EEG window as follows: \(\hat{y}_{l}=\sum _{{\varvec{h}}_{j,v}{{\,\mathrm{\,\in \,}\,}}\varOmega _v}\alpha _{j}{y}_{j}\kappa _{\sigma }({\varvec{h}}_{l,v},{\varvec{h}}_{j,v})+{b},\) where \(\alpha _{j}{{\,\mathrm{\,\in \,}\,}}\mathbb {R}\) is the weight for the sample \({\varvec{h}}_{j,v}{{\,\mathrm{\,\in \,}\,}}\varOmega _v,\) \(\varOmega _v\) is a set holding the support vectors, \(b{{\,\mathrm{\,\in \,}\,}}\mathbb {R}\) is a bias term, and \(\kappa _\sigma :\mathbb {R}^{2M}\times \mathbb {R}^{2M}\rightarrow \mathbb {R}^+\) is a Gaussian kernel function with bandwidth \(\sigma {{\,\mathrm{\,\in \,}\,}}\mathbb {R}^+.\)

EEG Connectivity Analysis to Code the MI Protocol Coupling. We use a measure based on brain connectivity, knows as Phase Locking Value (PLV), to assess the interaction between different brain regions by means of phase synchronization, that allows a straight comparison between the instantaneous phases of two EEG signals. In particular, this step allows to highlight the relationships between the classification performance of the subject and its coupling with the MI protocol concerning the brain connectivity patterns within each temporal segment. So, given a pair of input channels at the v-th window for the l-th trial \({\varvec{z}},{\varvec{z}}'{{\,\mathrm{\,\in \,}\,}}{\varvec{Z}}_{l,v};\) the instantaneous phase vectors \({\varvec{\phi }},{\varvec{\varphi }}{{\,\mathrm{\,\in \,}\,}}\mathbb {R}^R\) are computed as: \(\phi _t{{\,\mathrm{\,=\,}\,}}\arctan \left( \mathcal {H}\{{z}_t\}/{z}_t\right) ,\) being \(\mathcal {H}\) the Hilbert transform function (analogously for \(\varphi _t\)) [12]. In turn, a PLV-based connectivity matrix \({\varvec{K}}{{\,\mathrm{\,\in \,}\,}}\mathbb {R}^{C\times C}\) can be computed as follows:

$$\begin{aligned} k_{c,c'} = \left| \frac{1}{R}\sum _{t=1}^{R} e^{i\left( \phi _t-\varphi _t\right) }\right| . \end{aligned}$$
(2)

Lastly, to quantify the phase synchronization strength between EEG-channels along the trials associated with an MI condition, the overall brain connectivity matrices \({\varvec{\varPsi }}^+_v,{\varvec{\varPsi }}^-_v{{\,\mathrm{\,\in \,}\,}}\mathbb {R}^{C\times C}\) can be computed as follows: \({\varvec{\varPsi }}^{\xi }_v{{\,\mathrm{\,=\,}\,}}{\mathbb {E}}\left\{ {\varvec{K}}_{l,v} : \forall y_l=\xi \right\} ,\) where \(\xi {{\,\mathrm{\,\in \,}\,}}\{+1,-1\}.\)

3 Experimental Set-Up

For concrete testing, the proposed EEG processing approach is used to highlight relevant spatiotemporal patterns from an MI discrimination task. In particular, the publicly available BCI Competition IV dataset 2a is employed.Footnote 1 This database is provided by the Institute for Knowledge Discovery (Laboratory of Brain-Computer Interfaces) at Graz University of Technology and consists of the recorded EEG data from 9 healthy subjects performing the MI task during one trial. Each trial begins with a fixed cross on the computer screen accompanied by a beep. At second 2, an arrow pointing left, right, down or up (MI of the left hand, right hand, both feet, and tongue, respectively) is shown as a visual stimulus on the screen during 1.25 s. Then, at second 3, the subject is petitioned to perform the MI task until the fixed cross disappeared from the screen at second 6. A short break of 1 s follows, where the screen is blank (see Fig. 1(a)). EEG signals are recorded from 22 Ag/AgCl electrodes positioned according to the international 10/20 placement system. The signals are bandpass-filtered between 0.5–100 Hz and sampled at 250 Hz. Besides, a 50 Hz Notch filter is used. In this work, we select only the first two MI tasks of the experimental paradigm (left and right hands). Afterward, the EEG signals per each trial are down-sampled at 128 Hz and filtered between 8–30 Hz using a 5th order Butterworth bandpass filter to highlight the \(\alpha \) (8–12 Hz) and \(\beta \) (12–30 Hz) frequency bands avoiding the influence of muscle noise [6, 10]. Next, we segment each EEG trial into six-time windows of 2 s with a 50% overlapping through a Hamming window. Namely, the EEG data is divided into six segments of equal length: 0–2 s, 1–3 s, ..., 5–7 s.

Further, the CSP-based representation is carried out on every window. We fix the number of eigenvectors in the CSP algorithm to \(M{{\,\mathrm{\,=\,}\,}}4\) [11]. Then, we train a SVM classifier under a nested 10-folds cross-validation scheme, where 90% of the trials per subject are used as training set and the remaining 10% as testing. The kernel bandwidth value is searched from the set \([0.5\sigma _o,1.0\sigma _o]\), where \(\sigma _o\in \mathbb {R}^+\) holds the median of the Euclidean distances of the input space. Moreover, to evaluate the propagation of neural activity recorded in each trial, the PLV is computed as described in Sect. 2. Besides, we perform a statistical test to determine if there are statistically significant differences between the accumulated electrical activity estimated (regarding the PLV connectivity values) from the left and the right cerebral hemispheres. We test the null hypothesis, that the two data samples are from populations with equal means, assuming a Student’s t distribution with a significance level of 5% (p-value \(<0.05\)).

4 Results and Discussion

Table 1 summarizes the classification results in terms of the accuracy and Cohen’s kappa coefficient. Note that significant state-of-the-art works are shown for comparison purposes [3, 4, 8, 9, 11]. Concerning the introduced approach, we also display the number of the temporal window with the highest classification performance and the p-value measuring the statistical differences of the PLV-based connectivities between the left and the right hemispheres. As seen, our strategy obtains the highest accuracy in average in comparison to the Elasuty et al., Liang et al. and Li et al. methodologies. Moreover, in these approaches, the MI discrimination based on feature extraction is calculated for the specific time segment associated with the stimulus. Now, our MI discrimination system achieves mean classification values along subjects between 62.53\(\%\) to 93.19\(\%\) for accuracy, and 0.29 to 0.86 for the kappa coefficient.

Table 1. MI discrimination results for the BCI Competition IV dataset 2a. The accuracy and kappa coefficients are presented. Also, the window related to the achieved classification is displayed for our approach. The p-value quantifies the statistical difference of the PLV-based connectivity between the left and right hemispheres (the lower p-value the higher the brain connectivity differences).
Full size table
Fig. 1.
figure 1

Connectivity analysis for the best (S08) and the worst (S02) subject based on classification results, in the left hand MI task discrimination. (a) MI protocol. (b) Visualization of variability in classification performance throughout the temporal windows. (c) Probability density function of the electrical activity associated with each cerebral hemisphere from PLV-based connectivity values and (d) Connectivity maps showing interactions between EEG channels for the windows 1, 4 and 6 for the best subject, and analogously for (e) and (f) the worst subject.The dots of the plots (d) and (f) represent the spatial distribution of the EEG electrodes on the sensorimotor area.

Full size image

As a comparative evaluation of the approach proposed in Fig. 1 are showed the statistical and connectivity analysis carried out for the best (S08) and the worst (S02) subject, according to the classification results for the task left hand MI per window. Remarkably, the lowest and highest performances are related to the subjects 02 and 08, respectively. In fact, as seen in Fig. 1(b), the classification performance along windows for the S08 evidence the expected behavior concerning to the MI protocol, having its best result in the window (4) where the stimulus appeared. Otherwise, the performance curve for the S02 shows an indiscriminate behavior, having its worst value in this window. In this sense, these results demonstrate that there is a direct relationship between the subject coupling with MI protocol and the classification performance. Figure 1(c) allow us to see the probability density function of the accumulated connectivity strength linked to each hemisphere in the windows 1, 4, and 6. We noted that the distributions for the S08 are significantly different, while the S02 ones do not reveal discriminative connectivity between brain hemispheres (see Fig. 1(e)). Figure 1(d) and Fig. 1(f) display the connectivity maps for both subjects respectively. The connectivity maps viewed are based on visualization function of the Matlab toolbox HERMES.Footnote 2 Notably, the connectivity distribution for the sensorimotor area shown in Fig. 1(d) and Fig. 1(f) corresponds to the following EEG channels: FC3, C3, and CP3 in the left hemisphere; FCz, Cz, and CPz in the middle, and FC4, C4, and CP4 in the right hemisphere. The observed behavior of the networks generated by PLV for MI task, left hand, per subject varies slightly due to the trials average activity recorded for the segments shown. After visual inspection of the connectivity maps, per window, it is possible to observe the different patterns of dynamic changes in the brain that are generated in subjects 08 and 02 in the states of concentration, execution of the MI task and resting, respectively. The patterns exhibited by the graphs in the middle, for both subjects, represent the average activity measured for the segment between 3–5 s where the brain activity associated with the MI task is mainly recorded. In this sense, the best subject presents more activity in the contralateral hemisphere to the task under study, that is, there is a greater representation of the connection between the nodes of the right hemisphere [7], which does not happen with the worst subject allowing us to interpret with greater clarity the classification results.

5 Conclusions

In this study, we employed an MI discrimination system to enhance the identification of relevant brain connectivity patterns. Although different methodologies based on feature extraction have been used for the distinction between MI tasks, including connectivity analysis, it is not known why those classification systems exhibit differences concerning their accuracy per subject. Our EEG analysis strategy first makes a windows representation of the temporal information of the signal. Then, consistent spatial patterns for MI discrimination are extracted using CSP. Next, the spatiotemporal features are used as inputs to an SVM classifier. Finally, we use a PLV analysis based characterization of the segmented EEG signals to reveal relevant brain connectivity patterns. Our approach focuses on discriminating the spatiotemporal features associated with each MI task for its subsequent visual interpretation employing brain connectivity estimators and a statistical test. According to our results, subjects with higher classification performances exhibited stronger activity in the contralateral hemisphere for the MI task studied. As future work, we plan to test the proposed approach with effective connectivity measures, e.g., Granger Causality, which not only indicate activity between electrodes but also describe the influence that one signal exerts over another. Moreover, to better code brain dynamics we propose employing variants of Canonical Correlation Analysis (CCA) based on kernels [13]. This type of analysis allows improving the robustness of classification systems and provides greater flexibility through the use of non-linear relationships between EEG signals.