Advertisement

Electrocorticography based motor imagery movements classification using long short-term memory (LSTM) based on deep learning approach

  • 190 Accesses

Abstract

Brain–computer interface (BCI) is an important alternative for disabled people that enables the innovative communication pathway among individual thoughts and different assistive appliances. In order to make an efficient BCI system, different physiological signals from the brain have been utilized for instances, steady-state visual evoked potential, motor imagery, P300, movement-related potential and error-related potential. Among these physiological signals, motor imagery is widely used in almost all BCI applications. In this paper, Electrocorticography (ECoG) based motor imagery signal has been classified using long short-term memory (LSTM). ECoG based motor imagery data has been taken from BCI competition III, dataset I. The proposed LSTM approach has achieved the classification accuracy of 99.64%, which is the utmost accuracy in comparison with other state-of-art methods that have employed the same data set.

Introduction

In the present day, the fast growth of machine learning and deep learning approaches enable the investigation of biological signals which happens to be a notable research topic. Professionals have been exploring to figure out and categorize the distinctive biological signals for medical and non-medical application [1, 2]. Invasive and non-invasive strategies are used to capture the biological signals from brain of the human beings. There are two invasive strategies are employed in BCI research, specifically electrocorticography (ECoG) and intracortical neuron recording. In ECoG recording, electrodes are attached on the exterior of the cortex whereas the electrodes are implanted interior the cortex in intracortical neuron recording [3]. The broadly used non-invasive modalities consist of Electroencephalography (EEG), Magnetoencephalography (MEG), Near-Infrared Spectroscopy (NIRS) and Functional Magnetic Resonance Imaging (fMRI).

The trends of mind wave should be decoded in such a manner that persons may able to modulate and interpret their thoughts to deal with a BCI system [3]. These signals are regarded as control signals in BCIs. The widely used physiological control signals are steady state visual evoked potential (SSVEP), slow cortical potentials (SCP), P300 evoked potentials, and motor imagery signal.

In the MI-based BCI framework, the imagination of hand or tongue movement activities is going to be supported by a circumscribed event-related synchronization/desynchronization [4]. In the last few years, a noticeable number of studies have been carried out relating to the MI signal classification. In [5], ECoG features in terms of band powers (BP) have been extracted and the selected features have been classified using probabilistic neural network (PNN). Erdem Erkan and Ismail Kurnaz proposed a new technique to find optimal channel set namely arc detection algorithm (ADM) in [1]. They used discrete wavelet transform (DWT) for the purpose of feature extraction and to classify the extracted features, support vector machine (SVM), K-nearest neighbors (K-NN) and linear discriminant analysis (LDA) were used. They obtained the highest accuracy of 95% in the classification of ECoG data (BCI Competition III, dataset I). Aswinseshadri. K et al. [6] employed wavelet packet tree (WPT) on the same dataset to extract features. Authors applied information gain, mutual information and genetic algorithm (GA) to decide on the best-suited features set and then classified these features using K-NN and Naïve Bayes. Feature selection using GA achieved the highest accuracy. A novel feature extraction technique known as Renyi entropy has been implemented in [7] where they used BLDA to classify the same dataset. Continuous wavelet transform (CWT) based approach has been recommended in [8]. Authors employed PCA to trim down the dimensionality of features set and then implemented LDA, K-NN and SVM to classify the ECoG data. Authors in [9] extracted features using statistical properties of the bispectrum of ECoG data and then classified the features by K-NN. Zheng et al. [10] extracted the time–frequency features by the modified S-transform (MST) algorithm, and then the extracted features are classified using SVM. In reference to [11], authors proposed Hilbert-Huang transform and common spatial subspace decomposition (HCSSD) algorithm to extract time–frequency features and then learning vector quantization neural network (LVQ-NN) was employed to classify the selected features. Chang et al. [12] employed stockwell transform (ST) and GA for feature extraction and selection respectively and finally Bayesian LDA (BLDA) was used to classify the selected features. Another study in [13], continuous wavelet transform (CWT) and K-NN are used as features and classifier respectively to classify the ECoG MI data.

Most of the conventional machine learning approaches described in above paragraph have been extracted different features. The performance of the machine learning approaches is highly affected by the feature extraction techniques. The higher dimensionality of the extracted feature set is another computational complexity for the conventional machine learning approaches. In such cases, a variety of feature reduction algorithms should be employed. To avoid the complexity of feature extraction and feature reduction, we have proposed a LSTM based deep learning approach where no need to employ any feature extraction and reduction framework. In this approach, raw ECoG data have been employed to the LSTM model.

In this article, two classes motor imagery ECoG has been classified using LSTM based deep learning approach. The remaining parts of the article has been organized in the following sections i.e. Sections 2 and 3 discusses issues related to methodology, results and discussion respectively; finally, Sect. 4 deals with the conclusion.

Methodology

Details about ECoG data

The majority of BCI studies have been utilized brain wave based on EEG recording. There are some positive aspects in preference of EEG including low-cost data capturing device, ease of mobility and non-invasive manner of data acquisition. However, the SNR of EEG doesn’t always meet the satisfactory level. Moreover, the algorithm for EEG analysis in certain cases lessens the classification accuracy as well as the data transfer rate. There is a potential alternative utilized in BCIs that is Electrocorticography (ECoG) [14]. Although, ECoG is an invasive approach, the key merits of ECoG are the excellent SNR, considerably greater spatial resolution and it offers better classification accuracy [9]. In this study, the ECoG data has been taken from data set I of the BCI Competition III entitled”motor imagery in ECoG recordings, session-to-session transfer” [15, 16]. Data has been recorded from such subject who was suffered from epilepsy. This ECoG data captured the person’s imagined movements of the tongue or left small finger. An 8 × 8 ECoG platinum electrode grid was positioned on the right motor cortex. The electrode grid size was supposed to meet the right motor cortex entirely. However, it partially covered the motor cortex portions due to its small shape (approx. 8 × 8 cm) shown in Fig. 1.

Fig. 1
figure1

Placement of the 8 × 8 electrode grid

The sampling rate of recorded ECoG data was 1000 Hz. The captured potentials appeared to be stored as microvolt after signal amplification. Each observation is made up of either an imagined finger or imagined tongue movement. The duration of each trial was three seconds. Figure 2 shows the plotting of raw ECoG data for training trial 1 and 2. In this figure, the only first channel of both trials is considered to plot.

Fig. 2
figure2

Motor imagery ECoG raw data plotting

ECoG data has been recorded after 0.5 s from ending of visual cue. This strategy has been followed due to avoid the effect of visually evoked potentials in the ECoG data. The train and test dataset have been captured in two distinct periods when the subjects experienced different mental states. Due to capture data in two distinctive sessions, it is very challenging to classify this dataset. Thus, the classification algorithm should have the ability to classify such dataset which has been captured in distinct sessions.

LSTM network

Deep learning is getting extensive interest in the field of EEG based BCI research. Moreover, the deep learning frameworks are functioning well in comparison with additional conventional approaches. The most commonly employed deep learning techniques are convolution neural network, deep belief network, recurrent neural network and autoencoder. An RNN which consists of LSTM layers is known as LSTM network. Basically, the LSTM network is considerably more effective than the feed-forward neural networks with regard to the sequence prediction and their ability to remember information selectively for a long time. To predict and classify the sequence and time-series data, an LSTM network is effectively employed [17]. In this study, ECoG motor imagery dataset has been classified using LSTM network. Figure 3 shows a single LSTM block where ht and ct denote the hidden or output state and cell state respectively at the time step t. In order to compute the cell state and hidden (or output) at the time state t, the LSTM block uses old hidden (ht−1) and cell state (ct−1).

Fig. 3
figure3

A long short-term memory network architecture

The LSTM networks are capable of adding or erasing information from the memory cell employing gates. These gates are composed of a pointwise multiplication operation and sigmoid neural net layer. The output of the sigmoid layer must be a number whose rage should be in between one and zero. The value belongs to zero denotes “let nothing through,” whereas the value belongs to one refers to “let everything through”. Each LSTM block has three gates, namely forget gate, input gate and output gate. The degree at which the information continues to be in the memory cell of the LSTM layer is handled by the forget gate. The input gate deals with the degree at which the information entering the memory cell of the LSTM layer. The output gate deals with the degree at which the information placed in the memory cell of the LSTM layer is going to be employed to calculate the output [17]. The cell state (ct) and the hidden state (ht) at the time state t could be stated by the Eqs. (1) and (2) respectively [18].

$$c_{t} = f_{t} *c_{t - 1} + i_{t} *g_{t}$$
(1)
$$h_{t} = o_{t} *\sigma_{c} c_{t}$$
(2)

where \(f_{t}\), \(i_{t}\), \(g_{t}\) and \(o_{t}\) denote forget gate, input gate, cell candidate and output gate respectively and mathematically they can be represented by the Eqs. (36) respectively at the time step t as follow [18]:

$$i_{t} = \sigma (W_{i} x_{t} + R_{i} h_{t - 1} + b_{i} )$$
(3)
$$f_{t} = \sigma (W_{f} x_{t} + R_{f} h_{t - 1} + b_{f} )$$
(4)
$$g_{t} = \sigma_{c} (W_{g} x_{t} + R_{g} h_{t - 1} + b_{g} )$$
(5)
$$o_{t} = \sigma (W_{o} x_{t} + R_{o} h_{t - 1} + b_{o} )$$
(6)

The matrices W, R, and b happen to be concatenations of the input weights, the recurrent weights, and the bias of each component, respectively.

Structural outline of proposed approach

The complete workflow of the proposed technique is shown in Fig. 4. In order to build the model, the training data set has been used and this training data consists of the input set and level. A 278 by 1 cell array has been generated from the input set for training purpose and this input set was in 3D data format. Each cell array contains data for 64 electrodes and each electrode has 3000 samples. The LSTM network is initiated with a sequence input layer. In this study, the sequence input layer has been specified by the input size 64. The input as a sequence of vector has been taken from the sequence input layer. The LSTM layer consists of 100 hidden units. In LSTM layer, the output mode “last” has been employed. This mode helps sequence to label classification. In the fully connected layer, the number of the class has been specified. For this dataset, the class number is two. Then the softmax and classification layer have been specified to make the network capable of training. Finally, the training options have been selected as the solver to be ‘sgdm’ and learning rate to be 0.01. Now, the LSTM network is ready for training. Once the model is trained, the model has been tested by test dataset. Finally, the performance of the model has been evaluated using different metrics.

Fig. 4
figure4

Complete workflow of proposed study

Performance evaluation metrics

The performance of the proposed method has been analyzed by confusion matrix, classification accuracy, sensitivity, specificity. The classification accuracy (CA) of the proposed method is calculated by Eq. (7) [17].

$$CA = \frac{TP + TN}{TP + FN + TN + FP} \times 100\%$$
(7)

where TP true positive, FN false negative, TN true negative and FP false positive. Moreover, the formulas for sensitivity, specificity and precision are shown in Eqs. (810) respectively [19].

$${\text{Sensitivity}} = \frac{TP}{TP + FN} \times 100\%$$
(8)
$${\text{Specificity}} = \frac{TN}{TN + FP} \times 100\%$$
(9)
$${\text{Precision}} = \frac{TP}{TP + FP} \times 100\%$$
(10)

Results and discussion

This section presents the performance of the proposed approach based on motor imagery ECoG data. In this study, two-class ECoG data has been classified using LSTM based deep learning approach. However, the LSTM network has been trained using the training data set and multiple layers have been implemented in the proposed LSTM network. During training progress, the epoch and iteration have been set to 150 and 1500 respectively to enhance the performance. Figure 5 shows the accuracy during training progress. From this figure, it is clear that the accuracy stabilizes to 100% from the iteration of 240. Figure 6 presents the loss curve during training accuracy where the loss becomes stable at 0 from the iteration of 400.

Fig. 5
figure5

Training progress in terms of accuracy

Fig. 6
figure6

Training progress in terms of loss

In order to evaluate the performance of the proposed technique in different views, the confusion matrix has been computed. Figure 7a shows the confusion matrix whereas Fig. 7b presents the true and false positive rate for training dataset. From the confusion matrix, it is obvious that all observations for class one have been predicted perfectly whereas 138 observations have been predicted correctly out of 139 observations in class two. The true positive rate for class one and two are 100% and 99.28% respectively whereas the false-negative rate for class one and two are 0 and 0.72% respectively presented in Fig. 7b.

Fig. 7
figure7

a Confusion matrix, b True and false positive rate for training data

Once the model has been constructed, the performance of the model has also been validated using test dataset. The test dataset consist of 100 observations which have been given by the dataset provider separately. Figure 8a shows the confusion matrix whereas Fig. 8b presents the true and false positive rate for testing dataset. During testing, total 97 observations (out of 100) have been recognized correctly by the proposed model. The true positive rate for class one and two are 98% and 96% respectively whereas the false-negative rate for class one and two are 2% and 4% respectively presented in Fig. 8b.

Fig. 8
figure8

a Confusion matrix, b True and false positive rate for testing data

The proposed approach has also been evaluated in terms of classification accuracy, sensitivity, specificity, and precision. These values have been computed using Eqs. (710). In case of training dataset, the classification accuracy, sensitivity, specificity and precision of the proposed LSTM model are 99.64%, 100%, 99.28%, and 99.28% respectively shown in Table 1. Whereas, the classification accuracy, sensitivity, specificity and precision are 97%, 96%, 98%, and 98% respectively for the testing dataset. Results in Table 1 prove that the performance of the proposed LSTM model is excellent.

Table 1 Performance evaluation of proposed approach

The performance of the proposed LSTM model has also been compared with related studies. Table 2 presents the comparison table where the classification accuracy of previous related researches has been listed. All study in Table 2 used the ECoG MI dataset from BCI Competition III dataset I. Table 2, demonstrate that the proposed method outperforms all other previous approaches.

Table 2 Performance and methodology comparison of BCI competition III dataset I

Conclusion

In this study, two classes of motor imagery ECoG data has been classified. The deep learning-based LSTM approach has been employed in the purpose of classification. The performance of the proposed model has been compared with other previous related studies. The proposed method outperforms state-of-art techniques. Due to the non-stationary nature of ECoG data, it is very challenging to process this data for BCI applications. In conventional machine learning techniques, feature extraction, selection and reduction strategies have been employed to boost the performance. However, these supplementary analyses sometimes create algorithm complexity. Hence, deep learning algorithms get started to prove their possibilities to handle the non-stationary nature of ECoG and EEG data and open new opportunities in BCI research.

References

  1. 1.

    Erkan E, Kurnaz I (2017) A study on the effect of psychophysiological signal features on classification methods. Measurement 101:45–52. https://doi.org/10.1016/J.MEASUREMENT.2017.01.019

  2. 2.

    M. Rashid, N. Sulaiman, M. Mustafa, S. Khatun, B.S. Bari, The Classification of EEG Signal Using Different Machine Learning Techniques for BCI Application, in: S.-M.L. Jong-Hwan, KimHyung Myung (Ed.), Robot Intell. Technol. Appl. RiTA 2018. Commun. Comput. Inf. Sci. Vol 1015, Springer, Singapore, 2019: pp. 207–221. https://doi.org/10.1007/978-981-13-7780-8_17

  3. 3.

    Nicolas-Alonso LF, Gomez-Gil J (2012) Brain computer interfaces, a review. Sensors. https://doi.org/10.3390/s120201211

  4. 4.

    Dai M, Zheng D, Na R, Wang S, Zhang S, Dai M, Zheng D, Na R, Wang S, Zhang S (2019) EEG classification of motor imagery using a novel deep learning framework. Sensors 19:551. https://doi.org/10.3390/s19030551

  5. 5.

    Zhao H, Liu C, Wang H, Li C (2010) Classifying ECoG signals using probabilistic neural network. In: 2010 WASE international conference on information engineering. IEEE, 2010, pp 77–80. https://doi.org/10.1109/ICIE.2010.26

  6. 6.

    Aswinseshadri K, Bai VT (2015) Feature selection in brain computer interface using genetics method. In: 2015 IEEE Int Conf Comput Inf Technol Ubiquitous Comput Commun Dependable Auton Secur Comput Pervasive Intell Comput. IEEE, pp 270–275. https://doi.org/10.1109/CIT/IUCC/DASC/PICOM.2015.39

  7. 7.

    Ponnambalam CKSG (2017) Binary and multi-class motor imagery using Renyi entropy for feature extraction. Neural Comput Appl 28:2051–2062. https://doi.org/10.1007/s00521-016-2178-y

  8. 8.

    Islam MR, Fatema U, Bhuiyan MIH, Bashar SK (2016) Classification of electrocorticography based motor imagery movements using continuous wavelet transform. In: 2016 IEEE students’ technology symposium. IEEE, pp 13–17. https://doi.org/10.1109/TechSym.2016.7872647

  9. 9.

    Paul S, Zabir I, Sarker T, Fattah SA, Shahnaz C (2017) Higher order statistics of bispectrum and MRP of ECoG signals for motor imagery tasks classification. In: 2017 IEEE region 10 symposium. IEEE, pp 1–4. https://doi.org/10.1109/TENCONSpring.2017.8070109

  10. 10.

    Zheng W, Xu F, Shu M, Zhang Y, Yuan Q, Lian J, Zheng Y (2019) Classification of motor imagery electrocorticogram signals for brain–computer interface. In: 2019 9th international IEEE/EMBS conference on neural engineering. IEEE, pp 530–533. https://doi.org/10.1109/NER.2019.8716963

  11. 11.

    Li M, Cui Y, Hao D, Yang J (2015) An adaptive feature extraction method in BCI-based rehabilitation. J Intell Fuzzy Syst 28:525–535. https://doi.org/10.3233/IFS-141329

  12. 12.

    Chang H, Yang J (2018) Genetic-based feature selection for efficient motion imaging of a brain–computer interface framework. J Neural Eng 15:56020. https://doi.org/10.1088/1741-2552/aad567

  13. 13.

    Aydemir O, Kayikcioglu T (2011) Wavelet transform based classification of invasive brain computer interface data. Radioengineering 20:31–38

  14. 14.

    Leuthardt EC, Schalk G, Wolpaw JR, Ojemann JG, Moran DW (2004) A brain-computer interface using electrocorticographic signals in humans. J Neural Eng 1:63–71. https://doi.org/10.1088/1741-2560/1/2/001

  15. 15.

    BCI Competition III (2004) http://www.bbci.de/competition/iii/. Accessed 13 Nov 2019

  16. 16.

    Blankertz B, Müller K, Krusienski DJ, Schalk G, Wolpaw JR, Schlögl A, Pfurtscheller G, Millán JR, Schröder M, Birbaumer N (2006) The BCI competition III: validating alternative approaches to actual BCI problems. IEEE Trans Neural Systems Rehabil Eng 14:153–159

  17. 17.

    Kumar S, Sharma A, Tsunoda T (2019) Brain wave classification using long short-term memory network based OPTICAL predictor. Sci Rep 9:9153. https://doi.org/10.1038/s41598-019-45605-1

  18. 18.

    Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9:1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

  19. 19.

    Precision, recall, sensitivity and specificity | Ubershmekel’s Uberpython Pythonlog (2012) https://uberpython.wordpress.com/2012/01/01/precision-recall-sensitivity-and-specificity/

Download references

Acknowledgments

This work is funded by the Universiti Malaysia Pahang, Malaysia through the Research Grant, RDU180396.

Author information

Correspondence to Mamunur Rashid.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Rashid, M., Islam, M., Sulaiman, N. et al. Electrocorticography based motor imagery movements classification using long short-term memory (LSTM) based on deep learning approach. SN Appl. Sci. 2, 211 (2020). https://doi.org/10.1007/s42452-020-2023-x

Download citation

Keywords

  • Electrocorticography (ECoG)
  • Motor imagery (MI)
  • Long short-term memory (LSTM)
  • Brain computer interfaces (BCI)
  • Deep learning