Modeling and classification of voluntary and imagery movements for brain–computer interface from fNIR and EEG signals through convolutional neural network

  • Md. Asadur RahmanEmail author
  • Mohammad Shorif Uddin
  • Mohiuddin Ahmad
Part of the following topical collections:
  1. Special Issue on Artificial Intelligence in Health Informatics


Practical brain–computer interface (BCI) demands the learning-based adaptive model that can handle diverse problems. To implement a BCI, usually functional near-infrared spectroscopy (fNIR) is used for measuring functional changes in brain oxygenation and electroencephalography (EEG) for evaluating the neuronal electric potential regarding the psychophysiological activity. Since the fNIR modality has an issue of temporal resolution, fNIR alone is not enough to achieve satisfactory classification accuracy as multiple neural stimuli are produced by voluntary and imagery movements. This leads us to make a combination of fNIR and EEG with a view to developing a BCI model for the classification of the brain signals of the voluntary and imagery movements. This work proposes a novel approach to prepare functional neuroimages from the fNIR and EEG using eight different movement-related stimuli. The neuroimages are used to train a convolutional neural network (CNN) to formulate a predictive model for classifying the combined fNIR–EEG data. The results reveal that the combined fNIR–EEG modality approach along with a CNN provides improved classification accuracy compared to a single modality and conventional classifiers. So, the outcomes of the proposed research work will be very helpful in the implementation of the finer BCI system.


Voluntary and imagery movements Functional near-infrared spectroscopy (fNIR) Electroencephalography (EEG) Modeling and classification Convolutional neural network (CNN) Brain–computer interface (BCI) 



This work was partially supported by the Higher Education Quality Enhancement Project (HEQEP), UGC, Bangladesh; under Subproject “Postgraduate Research in BME”, CP#3472, KUET, Bangladesh.

Compliance with ethical standards

Conflict of interest

This research work has no conflict of interest to anyone.


  1. 1.
    Fantini S. Dynamic model for the tissue concentration and oxygen saturation of hemoglobin in relation to blood volume, flow velocity, and oxygen consumption: implications for functional neuroimaging and coherent hemodynamics spectroscopy (CHS). NeuroImage. 2014;85:202–21. Scholar
  2. 2.
    Baker JM, Bruno JL, Gundran A, Hadi Hosseini SM, Reiss AL. fNIRS measurement of cortical activation and functional connectivity during a visuospatial working memory task. PLoS ONE. 2018;13(8):1–22. Scholar
  3. 3.
    Shin J, Lühmann AV, Kim DW, Mehnert J, Hwang HJ, Müller KR. Data descriptor: simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset. Sci Data. 2018;5(180003):1–16. Scholar
  4. 4.
    Aghajani H, Garbey M, Omurtag A. Measuring mental workload with EEG + fNIRS. Front Hum Neurosci. 2017;11(359):1–20. Scholar
  5. 5.
    Hong K, Khan MJ, Hong MJ. Feature extraction and classification methods for hybrid fNIRS-EEG brain-computer interfaces. Front Hum Neurosci. 2018;12:246. Scholar
  6. 6.
    Abdulkader SN, Atia A, Mostafa MSM. Brain computer interfacing: applications and challenges. Egypt Inform J. 2015;16:213–30. Scholar
  7. 7.
    Burle B, Spieser L, Roger C, Casini L, Hasbroucq T, Vidal F. Spatial and temporal resolutions of EEG: is it really black and white? A scalp current density view. Int J Psychophysiol. 2015;97(3):210–20. Scholar
  8. 8.
    Basic principles of magnetoencephalography. MIT Class Notes. Accessed 24 Oct 2006.
  9. 9.
    Ariely D, Berns GS. Neuromarketing: the hope and hype of neuroimaging in business. Nat Rev Neurosci. 2010;11:284–92. Scholar
  10. 10.
    Ayaz H, Onaral B, Izzetoglu K, Shewokis PA, McKendrick R, Parasuraman R. Continuous monitoring of brain dynamics with functional near infrared spectroscopy as a tool for neuro ergonomic research: empirical examples and a technological development. Front Hum Neurosci. 2013;7(871):1–13. Scholar
  11. 11.
    Ernst LH, Plichta MM, Lutz E, Zesewitz AK, Tupak SV, Dresler T, Ehlis AC, Fallgatter AJ. Prefrontal activation patterns of automatic and regulated approach avoidance reactions: a functional near-infrared spectroscopy (fNIRS) study. Cortex. 2013;49(1):131–42. Scholar
  12. 12.
    Jöbsis FF. Noninvasive infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. Science. 1977;198(4323):1264–7. Scholar
  13. 13.
    Ferrari M, Quaresima V. A brief review on the history of human functional near-infrared spectroscopy (fNIRS) development and fields of application. NeuroImage. 2012;63(2):921–35. Scholar
  14. 14.
    Ferrari M, Giannini I, Carpi A, Fasella P, Fieschi C, Zanette E. Non-invasive infrared monitoring of tissue oxygenation and circulatory parameters. In: XII world congress of angiology 1980, Athens, September 7–12.Google Scholar
  15. 15.
    Giannini I, Ferrari M, Carpi A, Fasella P. Rat brain monitoring by near-infrared spectroscopy: an assessment of possible clinical significance. Physiol Chem Phys. 1982;14(3):295–305.Google Scholar
  16. 16.
    Cui X, Bray S, Bryant DM, Glover GH, Reiss AL. A quantitative comparison of NIRS and fMRI across multiple cognitive tasks. NeuroImage. 2011;54(4):2808–21. Scholar
  17. 17.
    Noah JA, Ono Y, Nomot Y, Shimada S, Tachibana A, Zhang X, Bronner S, Hirsch J. fMRI validation of fNIRS measurements during a naturalistic task. J Vis Exp. 2015. Scholar
  18. 18.
    Allison BZ, Brunner C, Kaiser V, Muller-Putz GR, Neuper C, Pfurtscheller G. Toward a hybrid brain-computer interface based on imagined movement and visual attention. J Neural Eng. 2010. Scholar
  19. 19.
    Pfurtscheller G, Allison BZ, Brunner C, Bauernfeind G, Solis-Escalante T, Scherer R, Zander TO, Mueller-Putz G, Neuper C, Birbaumer N. The hybrid BCI. Front Neurosci. 2010. Scholar
  20. 20.
    Fazli S, Mehnert J, Steinbrink J, Curio G, Villringer A, Müller KR, Blankertz B. Enhanced performance by a hybrid NIRS–EEG brain-computer interface. NeuroImage. 2012;59(1):519–29. Scholar
  21. 21.
    Lee MH, Fazli S, Mehnert J, Lee SW. Improving the performance of brain-computer interface using multi-modal neuroimaging. In: 2nd IAPR Asian conference on pattern recognition, Naha, 2013, pp. 511–15.
  22. 22.
    Lee MH, Fazli S, Mehnert J, Lee SW. Subject-dependent classification for robust idle state detection using multi-modal neuroimaging and data-fusion techniques in BCI. Pattern Recognit. 2015;48(8):2725–37. Scholar
  23. 23.
    Buccino P, Keles HO, Omurtag A. Hybrid EEG-fNIRS asynchronous brain-computer interface for multiple motor tasks. PLoS ONE. 2016. Scholar
  24. 24.
    World Bank and WHO. World report on disability. World Bank and WHO; 2015.
  25. 25.
    Batula M, Kim YE, Ayaz H. Virtual and actual humanoid robot control with four-class motor-imagery-based optical brain-computer interface. Comput Intell Neurosci. 2017;2017(1463512):1–13. Scholar
  26. 26.
    Batula AM, Ayaz H, Kim YE. Evaluating a four-class motor-imagery-based optical brain-computer interface. In: 36th annual international conference of the IEEE engineering in medicine and biology society, Chicago, IL, 2014, pp. 2000–03.
  27. 27.
    Abbas W, Khan NA. FBCSP-based multi-class motor imagery classification using BP and TDP features. In: 2018 40th annual international conference of the IEEE engineering in medicine and biology society (EMBC), Honolulu, HI, 2018, pp. 215–18.Google Scholar
  28. 28.
    Mahmood A, Zainab R, Ahmad RB, Saeed M, Kamboh AM. Classification of multi-class motor imagery EEG using four band common spatial pattern. In: Annual international conference of the IEEE engineering in medicine and biology society (EMBC), Seogwipo, 2017, pp. 1034–37.
  29. 29.
    Mishra PK, Jagadish B, Kiran MPRS, Rajalakshmi P, Reddy DS. A novel classification for EEG based four class motor imagery using kullback-leibler regularized Riemannian manifold. In: 2018 IEEE 20th international conference on e-Health networking, applications and services (Healthcom), Ostrava, 2018, pp. 1–5.
  30. 30.
    Ge S, Wang R, Yu D. Classification of four-class motor imagery employing single-channel electroencephalography. PLoS ONE. 2014;9(6):1–7. Scholar
  31. 31.
    León CL. Multilabel classification of EEG-based combined motor imageries implemented for the 3D control of a robotic arm. PhD thesis, Université de Lorraine, 2017.Google Scholar
  32. 32.
    Rahman MA. Matlab based graphical protocol. MATLAB Central File Exchange. Accessed 20 Oct 2018.
  33. 33.
    World medical association declaration of Helsinki-ethical principles for medical research involving human subjects. Adopted by 64th WMA General Assembly, Fortaleza, Brazil, Special Communication: Clinical Review & Education, 2013.Google Scholar
  34. 34.
    Rahman MA, Rashid MA, Ahmad M. Selecting the optimal conditions of Savitzky-Golay filter for fNIRS signal. Biocybern Biomed Eng. 2019;39(3):624–37. Scholar
  35. 35.
    Wiriessnegger SC, Kurzmann J, Neuper C. Spatio-temporal differences in brain oxygenation between movement execution and imagery: a multichannel near-infrared spectroscopy study. Int J Psychophysiol. 2008;67(1):54–63. Scholar
  36. 36.
    Batula AM, Mark JA, Kim YE, Ayaz H. Comparison of brain activation during motor imagery and motor movement using fNIRS. Comput Intell Neurosci. 2017. Scholar
  37. 37.
    Nyhof L. Biomedical signal filtering for noisy environments. PhD thesis, Deakin University, Australia, 2014.
  38. 38.
    Chavan MS, Agarwala R, Uplane MD. Digital elliptic filter application for noise reduction in ECG signal. WSEAS Trans Electron. 2006;3(1):210–6.Google Scholar
  39. 39.
    Lutovac MD, Tosic DV, Evans BL. Filter design for signal processing. Upper Saddle River: Prentice Hall; 2001.Google Scholar
  40. 40.
    Vlcek M, Unbehauen R. Degree, ripple, and transition width of elliptic filters. IEEE Trans Circ Syst. 1989;36(3):469–72. Scholar
  41. 41.
    Orfanidis SJ. Introduction to signal processing. Upper Saddle River: Prentice Hall; 1996.Google Scholar
  42. 42.
    Mammone N, Morabito FC. Enhanced automatic wavelet independent component analysis for electroencephalographic artifact removal. Entropy. 2014;16(12):6553–72. Scholar
  43. 43.
    Khanam F, Rahman MA, Ahmad M. Evaluating alpha relative power of EEG signal during psychophysiological activities in Salat. In: International conference on innovations in science, engineering and technology (ICISET), 2018, Bangladesh, pp. 1–6.
  44. 44.
    Ifeachor EC, Jervis BW. Digital signal processing: a practical approach. Boston: Addison Wesley; 1993.Google Scholar
  45. 45.
    Rahman MA, Haque MM, Anjum A, Mollah MN, Ahmad M. Classification of motor imagery events from prefrontal hemodynamics for BCI application. Algorithms for intelligent system. Singapore: Springer; 2018. Scholar
  46. 46.
    Zhijie B, Qiuli L, Lei W, Chengbiao L, Shimin Y, Xiaoli L. Relative power and coherence of EEG series are related to amnestic mild cognitive impairment in diabetes. Front Aging Neurosci. 2014. Scholar
  47. 47.
    Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst. 2012. Scholar
  48. 48.
    Matsugu M, Mori K, Mitari Y, Kaneda Y. Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Netw. 2003. Scholar
  49. 49.
    Rahman MA, Ahmad M. Lie detection from single feature of functional near infrared spectroscopic (fNIRS) signals. In: 2nd international conference on electrical & electronic engineering (ICEEE 2017), 27–29 December, Rajshahi University of Engineering & Technology (RUET), Rajshahi, Bangladesh.
  50. 50.
    Rahman MA, Rashid MMO, Khanam F, Alam MK, Ahmad M. EEG based brain alertness monitoring by statistical and artificial neural network approach. Int J Adv Comput Sci Appl. 2019. Scholar
  51. 51.
    Rahman MA, Khanam F, Ahmad M. Detection of effective temporal window for classification of motor imagery events from prefrontal hemodynamics. In: International conference on electrical, computer and communication engineering (ECCE), Cox’s Bazar, Bangladesh, 2019.
  52. 52.
    Vakkuri A, Yli-Hankala A, Talja P, Mustola S, Tolvanen-Laakso H, Sampson T, Viertiö-Oja H. Time-frequency balanced spectral entropy as a measure of anesthetic drug effect in central nervous system during sevoflurane, propofol, and thiopental anesthesia. Acta Anaesthesiol Scand. 2004;48(2):145–53. Scholar
  53. 53.
    Rahman MA. Topoplot for B-Alert X-10 9-channel EEG signal. MATLAB Central File Exchange. Accessed 2 Apr 2019.
  54. 54.
    Rahman MA, Hossain MK, Khanam F, Alam MK, Ahmad M. Four-class motor imagery EEG signal classification using PCA, wavelet, and two-stage neural network. Int J Adv Comput Sci Appl. 2019. Scholar
  55. 55.
    Shin J, Jeong J. Multiclass classification of hemodynamic responses for performance improvement of functional near-infrared spectroscopy-based brain–computer interface. J Biomed Opt. 2014;19(6):067009-1–9. Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Biomedical EngineeringKhulna University of Engineering & Technology (KUET)KhulnaBangladesh
  2. 2.Department of Computer Science and EngineeringJahangirnagar UniversityDhakaBangladesh
  3. 3.Department of Electrical and Electronic EngineeringKhulna University of Engineering & Technology (KUET)KhulnaBangladesh

Personalised recommendations