Convolutional Neural Network for Hybrid fNIRS-EEG Mental Workload Classification

  • Marjan SaadatiEmail author
  • Jill Nelson
  • Hasan Ayaz
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 953)


The classification of workload memory tasks based on fNIRS and EEG signals requires solving high-dimensional pattern classification problems with a relatively small number of training patterns. In the use of conventional machine learning algorithms, feature selection is a fundamental difficulty given the large number of possible features and the small amount of available data. In this study, we bypass the challenges of feature selection and investigate the use of Convolutional Neural Networks (CNNs) for classifying workload memory tasks. CNNs are well suited for learning from the raw data without any a priori feature selection. CNNs take as input two-dimensional images, which differ in structure from the neural time series obtained on the scalp surface using EEG and fNIRS. Therefore, both the existing CNN architectures and fNIRS-EEG input must be adapted to allow fNIRS-EEG input to a CNN. In this work, we describe this adaptation, evaluate the performance of CNN classification of mental workload tasks. This study makes use of an open-source meta-dataset collected at the Technische Universität Berlin; including simultaneous EEG and fNIRS recordings of 26 healthy participants during n-back tests. A CNN with three convolution layers and two fully connected layers is adapted to suit the given dataset. ReLU and ELU activation functions are employed to take advantage of their better dampening property in the vanishing gradient problem, fast convergence, and higher accuracy. The results achieved with the two activation functions are compared to select the best performing function. The proposed CNN approach achieves a considerable average improvement relative to conventional methods such as Support Vector Machines. The results across differences in time window length, activation functions, and other hyperparameters are benchmarked for each task. The best result is obtained with a three-second window and the ELU activation function, for which the CNN yields 89% correct classification, while the SVM achieves only 82% correct classification.


EEG fNIRS Convolutional neural networks Deep learning Brain imaging Brain computer interfaces Human machine interfaces 


  1. 1.
    Aghajani, H., Garbey, M., Omurtag, A.: Measuring mental workload with EEG + fNIRS. Front. Hum. Neurosci. 11, 359 (2017)CrossRefGoogle Scholar
  2. 2.
    Jarmasz, J., Herdman, C.M., Johannsdottir, K.R.: Object-based attention and cognitive tunneling. J. Exp. Psychol. Appl. 11(1), 3 (2005)Google Scholar
  3. 3.
    Ayaz, H., Dehais, F.: Neuroergonomics: The Brain at Work and Everyday Life, 1st edn. Elsevier, Academic Press, Cambridge (2019)Google Scholar
  4. 4.
    Wang, L., Curtin, A., Ayaz, H.: Comparison of machine learning approaches for motor imagery based optical brain computer interface. In: Ayaz, H., Mazur, L. (eds.) Advances in Neuroergonomics and Cognitive Engineering, vol. 775, pp. 124–134. Springer, Cham (2019)Google Scholar
  5. 5.
    Liu, Y., Ayaz, H., Shewokis, P.A.: Multisubject “learning” for mental workload classification using concurrent EEG, fNIRS, and physiological measures. Front. Hum. Neurosci. 11(389) (2017).
  6. 6.
    Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Networks 61, 85–117 (2015)CrossRefGoogle Scholar
  7. 7.
    Bashivan, P., Rish, I., Yeasin, M., Codella, N.: Learning representations from EEG with deep recurrent-convolutional neural networks. In: arXiv preprint arXiv:1511.06448 (2015)
  8. 8.
    Hajinoroozi, M., Mao, Z., Jung, T.-P., Lin, C.-T., Huang, Y.: EEG-based prediction of driver’s cognitive performance by deep convolutional neural network. Sig. Process. Image Commun. 47, 549–555 (2016)Google Scholar
  9. 9.
    Thodoro, P., Pineau, J., Lim, A.: Learning robust features using deep learning for automatic seizure detection. In: Machine Learning for Healthcare Conference, pp. 178–190 (2016)Google Scholar
  10. 10.
    Schirrmeister, R.T., Springenberg, J.T., Fiederer, L.D.J., Glasstetter, M., Eggensperger, K., Tangermann, M., Hutter, F., Burgard, W., Ball, T.: Deep learning with convolutional neural networks for EEG decoding and visualization. Hum. Brain Mapp. 38(11), 5391–5420 (2017)CrossRefGoogle Scholar
  11. 11.
    Trakoolwilaiwan, T., Behboodi, B., Lee, J., Kim, K., Choi, J.-W.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain-computer interface: three-class classification of rest, right-, and left-hand motor execution. Neurophotonics 5(1) (2007) CrossRefGoogle Scholar
  12. 12.
    Croce, P., Zappasodi, F., Merla, A., Chiarelli, A.M.: Exploiting neurovascular coupling: a Bayesian sequential Monte Carlo approach applied to simulated EEG fNIRS Data. J. Neural Eng. 14(4) (2017)CrossRefGoogle Scholar
  13. 13.
    Hong, K.-S., Naseer, N., Kim, Y.-H.: Classification of prefrontal and motor cortex signals for three-class fNIRS-BCI. Neurosci. Lett. 587, 87–92 (2015)CrossRefGoogle Scholar
  14. 14.
    Chiarelli, A.M., Zappasodi, F., Di Pompeo, F., Merla, A.: Simultaneous functional near-Infrared spectroscopy and electroencephalography for monitoring of human brain activity and oxygenation: a review. Neurophotonics 4(4) (2017)CrossRefGoogle Scholar
  15. 15.
    Fazli, S., Mehnert, J., Steinbrink, J., Curio, G., Villringer, A., Muller, K.-R., Blankertz, B.: Enhanced performance by a hybrid NIRS-EEG brain computer interface. Neuroimage 59(1), 519–529 (2012)CrossRefGoogle Scholar
  16. 16.
    Saadati, M., Nelson, J.K.: Multiple transmitter localization using clustering by likelihood of transmitter proximity. In: 51st Asilomar Conference on Signals, Systems, and Computers, pp. 1769–1773 (2017)Google Scholar
  17. 17.
    Khan, M.J., Hong, M.J., Hong, K.-S.: Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface. Front. Hum. Neurosci. 8, 244 (2014)Google Scholar
  18. 18.
    Lee, M.-H., Fazli, S., Mehnert, J., Lee, S.-W.: Hybrid brain-computer interface based on EEG and NIRS modalities. In: Brain-Computer Interface (BCI), 2014 International Winter Workshop (2014)Google Scholar
  19. 19.
    Jirayucharoensak, S., Pan-Ngum, S., Israsena, P.: EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation. Sci. World J. (2014) Google Scholar
  20. 20.
    Shin, J., Von Luhmann, A., Kim, D.-W., Mehnert, J., Hwang, H.-J., Muller, K.-R.: Simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset. In: Generic Research Data (2018)Google Scholar
  21. 21.
    Shin, J., Von Luhmann, A., Kim, D.-W., Mehnert, J., Hwang, H.-J., Muller, K.-R.: Simultaneous aquisition of EEG and NIRS during cognitive tasks for an open access dataset. In: Scientific Data, vol. 5 (2018)CrossRefGoogle Scholar
  22. 22.
    Pfurtscheller, G.: Functional brain imaging based on ERD/ERS. Vis. Res. 41(10–11), 1257–1260 (2001)CrossRefGoogle Scholar
  23. 23.
    Neuper, C., Pfurtscheller, G.: Event-related dynamics of cortical rhythms: frequency-specific features and functional correlates. Int. J. Psychophysiol. 43(1), 41–58 (2001)CrossRefGoogle Scholar
  24. 24.
    Pourshafi, A., Saniei, M., Saeedian, A. Saadati, M.: Optimal reactive power compensation in a deregulated distribution network. In: 44th International Universities Power Engineering Conference (UPEC), pp. 1–6 (2009)Google Scholar
  25. 25.
    Ma, L., Zhang, L., Wang, L., Xu, M., Qi, H., Wan, B., Ming, D., Hu, Y.: A hybrid brain-computer interface combining the EEG and NIRS. In: 2012 IEEE International Conference on Virtual Environments Human-Computer Interfaces and Measurement Systems (VECIMS), pp. 159–162 (2012)Google Scholar
  26. 26.
    Saadati, M., Mortazavi, S., Pourshafi, A., Saeedian, A.: Comparing two modified method for harmonic and flicker measurement based on RMS, p. R1 (2009).–27.pdf2009102231223218.pdf
  27. 27.
    Schirrmeister, R.T., Springenberg, J.T., Fiederer, L.D.J., Glasstetter, M., Eggensperger, K., Tangermann, M., Hutter, F., Burgard, W., Ball, T.: Deep learning with convolutional neural networks for EEG decoding and visualization. In: Hum. Brainmapping 38(11), 5391–5420 (2017)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of Electrical and Computer EnginneringGeorge Mason UniversityFairfaxUSA
  2. 2.School of Biomedical Engineering, Science and Health SystemsDrexel UniversityPhiladelphiaUSA

Personalised recommendations