Benefits and Limits of Multimodal Neuroimaging for Brain Computer Interfaces

  • Siamac FazliEmail author
  • Min-Ho Lee
  • Seul-Ki Yeom
  • John Williamson
  • Isabella Schlattner
  • Yiyu Chen
  • Seong-Whan LeeEmail author
Part of the Trends in Augmentation of Human Performance book series (TAHP, volume 5)


Recently there has been a surge of interest for combining data from various sources in neuroscience, and also in other scientific domains. In this article we examine some of the benefits as well as limitations that arrise, when various neuroimaging techniques are employed for Brain-Computer Interfacing. In particular we review how setup costs can be reduced for multimodal systems, the NIRS response delay minimized and furthermore show that NIRS can help in the robust detection of the idle state.


BCI EEG NIRS Multi-modal neuroimaging Sensor reduction Idle-state detection 



This work was supported by the Brain Korea 21 Plus Program as well as the SGER Grant 2014055911 through the National Research Foundation of Korea funded by the Ministry of Education. This publication only reflects the authors views. Funding agencies are not liable for any use that may be made of the information contained herein. The authors acknowledge the use of some text from the prior publications [23, 38, 63] and thank their co-authors for allowing them to use materials from prior joint publications.


  1. 1.
    Abibullaev B, An J, Jin S-H, Lee SH, Moon JI (2013) Minimizing inter-subject variability in fNIRS-based brain–computer interfaces via multiple-kernel support vector learning. Med Eng Phys 35(12):1811–1818CrossRefPubMedGoogle Scholar
  2. 2.
    Alamgir M, Grosse-Wentrup M, Altun Y (2010) Multitask learning for brain-computer interfaces. In: AISTATS’10: 13th international conference on articial intelligence and statistics. MIT Press, Cambridge, pp 17–24Google Scholar
  3. 3.
    Babiloni F, Cincotti F, Babiloni C, Carducci F, Mattia D, Astolfi L, Basilisco A, Rossini P, Ding L, Ni Y, Cheng J, Christine K, Sweeney J, He B (2005) Estimation of the cortical functional connectivity with the multimodal integration of high-resolution EEG and fMRI data by directed transfer function. NeuroImage 24(1):118–131CrossRefPubMedGoogle Scholar
  4. 4.
    Baillet S, Garnero L, Marin G, Hugonin J-P (1999) Combined MEG and EEG source imaging by minimization of mutual information. IEEE Trans Biomed Eng 46(5):522–534CrossRefPubMedGoogle Scholar
  5. 5.
    Bauernfeind G, Scherer R, Pfurtscheller G, Neuper C (2011) Single-trial classification of antagonistic oxyhemoglobin responses during mental arithmetic. Med Biol Eng Comput 49(9):979–984CrossRefPubMedGoogle Scholar
  6. 6.
    Berger H (1929) Über das Elektrenkephalogramm des Menschen. Eur Arch Psychiatry Clin Neurosci 87(1):527–570Google Scholar
  7. 7.
    Bießmann F, Plis S, Meinecke FC, Eichele T, Müller K-R (2011) Analysis of multimodal neuroimaging data. IEEE Rev Biomed Eng 4:26–58CrossRefPubMedGoogle Scholar
  8. 8.
    Bogler C, Mehnert J, Steinbrink J, Haynes J-D (2014) Decoding vigilance with NIRS. PLoS ONE 9(7):e101729PubMedCentralCrossRefPubMedGoogle Scholar
  9. 9.
    Coyle S, Ward T, Markham C, McDarby G (2004) On the suitability of near-infrared (NIR) systems for next-generation brain–computer interfaces. Physiol Meas 25(4):815CrossRefPubMedGoogle Scholar
  10. 10.
    Coyle S, Ward T, Markham C (2007) Brain–computer interface using a simplified functional near-infrared spectroscopy system. J Neural Eng 4(3):219CrossRefPubMedGoogle Scholar
  11. 11.
    Cui X, Bray S, Reiss AL (2010) Speeded near infrared spectroscopy (NIRS) response detection. PLoS ONE 5(11):e15474PubMedCentralCrossRefPubMedGoogle Scholar
  12. 12.
    Dähne S, Bießman F, Meinecke FC, Mehnert J, Fazli S, Müller K-R (2013) Integration of multivariate data streams with bandpower signals. IEEE Trans Multimed 15(5):1001–1013CrossRefGoogle Scholar
  13. 13.
    Dähne S, Bießmann F, Samek W, Haufe S, Goltz D, Gundlach C, Villringer A, Fazli S, Müller K-R (2015) Multivariate machine learning methods for fusing functional multimodal neuroimaging data. Proc IEEE 103(9):1507–1530CrossRefGoogle Scholar
  14. 14.
    Dale AM, Sereno MI (1993) Improved localization of cortical activity by combining EEG and MEG with MRI cortical surface reconstruction: a linear approach. J Cogn Neurosci 5(2):162–176CrossRefPubMedGoogle Scholar
  15. 15.
    Dornhege G, del R. Millán J, Hinterberger T, McFarland D, Müller K-R (eds) (2007) Toward brain-computer interfacing. MIT Press, CambridgeGoogle Scholar
  16. 16.
    Ehlis A-C, Ringel T, Plichta M, Richter M, Herrmann M, Fallgatter A (2009) Cortical correlates of auditory sensory gating: a simultaneous near-infrared spectroscopy event-related potential study. Neuroscience 159(3):1032–1043CrossRefPubMedGoogle Scholar
  17. 17.
    Fazli S, Lee S-W (2013) Brain computer interfacing: a multi-modal perspective. J Comput Sci Eng 7(2):132–138CrossRefGoogle Scholar
  18. 18.
    Fazli S, Dähne S, Samek W, Bießman F, Müller K-R (2015) Learning from more than one data source: data fusion techniques for sensorimotor rhythm-based Brain-Computer Interfaces. Proc IEEE 103(6):891–906CrossRefGoogle Scholar
  19. 19.
    Fazli S, Popescu F, Danóczy M, Blankertz B, Müller K-R, Grozea C (2009) Subject-independent mental state classification in single trials. Neural Netw 22(9):1305–1312CrossRefPubMedGoogle Scholar
  20. 20.
    S. Fazli, C. Grozea, M. Danoczy, B. Blankertz, F. Popescu, and K.-R. Müller. Subject independent eeg-based bci decoding. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 513–521. Curran Associates, Inc., 2009.Google Scholar
  21. 21.
    Fazli S, Danóczy M, Schelldorfer J, Müller K-R (2011) L1-penalized Linear Mixed-Effects Models for high dimensional data with application to BCI. NeuroImage 56(4):2100–2108CrossRefPubMedGoogle Scholar
  22. 22.
    S. Fazli, J. Mehnert, J. Steinbrink, and B. Blankertz. Using NIRS as a predictor for EEG-based BCI performance. In Engineering in Medicine and Biology Society (EMBC), 2012 Annual International Conference of the IEEE, pages 4911–4914. IEEE, Aug 2012.Google Scholar
  23. 23.
    Fazli S, Mehnert J, Steinbrink J, Curio G, Villringer A, Müller K-R, Blankertz B (2012) Enhanced performance by a hybrid NIRS–EEG brain computer interface. NeuroImage 59(1):519–529CrossRefPubMedGoogle Scholar
  24. 24.
    Friston KJ (2009) Modalities, modes, and models in functional neuroimaging. Science 326(5951):399–403CrossRefPubMedGoogle Scholar
  25. 25.
    Fukunaga K (1990) Introduction to statistical pattern recognition, 2nd edn. Academic, San DiegoGoogle Scholar
  26. 26.
    Galán F, Nuttin M, Lew E, Ferrez PW, Vanacker G, Philips J, Millán JdR (2008) A brain-actuated wheelchair: asynchronous and non-invasive brain–computer interfaces for continuous control of robots. Clin Neurophysiol 119(9):2159–2169CrossRefPubMedGoogle Scholar
  27. 27.
    Girouard A, Solovey ET, Jacob RJ (2013) Designing a passive brain computer interface using real time classification of functional near–infrared spectroscopy. Int J Auton Adapt Commun Syst 6(1):26–44CrossRefGoogle Scholar
  28. 28.
    Grozea C, Voinescu C, Fazli S (2011) Bristle-sensors – low-cost flexible passive dry EEG electrodes for neurofeedback and BCI applications. J Neural Eng 8:025008CrossRefPubMedGoogle Scholar
  29. 29.
    Hoshi Y (2011) Towards the next generation of near-infrared spectroscopy. Philos Trans R Soc A Math Phys Eng Sci 369(1955):4425–4439CrossRefGoogle Scholar
  30. 30.
    Huppert T, Hoge R, Diamond S, Franceschini MA, Boas DA (2006) A temporal comparison of BOLD, ASL, and NIRS hemodynamic responses to motor stimuli in adult humans. NeuroImage 29(2):368–382PubMedCentralCrossRefPubMedGoogle Scholar
  31. 31.
    Ives J, Warach S, Schmitt F, Edelman R, Schomer D (1993) Monitoring the patient’s EEG during echo planar MRI. Electroencephalogr Clin Neurophysiol 87(6):417–420CrossRefPubMedGoogle Scholar
  32. 32.
    Jobsis FF (1977) Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. Science 198(4323):1264–1267CrossRefPubMedGoogle Scholar
  33. 33.
    Kanoh S, Murayama Y-m, Miyamoto K-i, Yoshinobu T, Kawashima R (2009) A NIRS-based brain-computer interface system during motor imagery: system development and online feedback training. In: Annual international conference of the IEEE engineering in medicine and biology society, Sept 2009 (EMBC 2009). IEEE, Piscataway, pp 594–597Google Scholar
  34. 34.
    Khan B, Wildey C, Francis R, Tian F, Delgado MR, Liu H, MacFarlane D, Alexandrakis G (2012) Improving optical contact for functional near-infrared brain spectroscopy and imaging with brush optodes. Biomed Opt Express 3(5):878–898PubMedCentralCrossRefPubMedGoogle Scholar
  35. 35.
    Khan MJ, Hong MJ, Hong K-S (2014) Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface. Front Hum Neurosci 8:244PubMedCentralPubMedGoogle Scholar
  36. 36.
    Kindermans P-J, Tangermann M, Müller K-R, Schrauwen B (2014) Integrating dynamic stopping, transfer learning and language models in an adaptive zero-training ERP speller. J Neural Eng 11(3):035005CrossRefPubMedGoogle Scholar
  37. 37.
    Kocsis L, Herman P, Eke A (2006) The modified Beer-Lambert law revisited. Phys Med Biol 51:N91–N98CrossRefPubMedGoogle Scholar
  38. 38.
    Lee M-H, Fazli S, Lee S-W (2013) Optimal channel selection based on statistical analysis in high dimensional NIRS data. In: International winter workshop on brain-computer interface (BCI). IEEE, pp 95–97Google Scholar
  39. 39.
    Lee M-H, Fazli S, Mehnert J, Lee S-W (2015) Subject-dependent classification for robust idle state detection using multi-modal neuroimaging and data-fusion techniques in BCI. Pattern Recognit 48(8):2725–2737CrossRefGoogle Scholar
  40. 40.
    Lev J (1949) The point biserial coefficient of correlation. Ann Math Stat 20(1):125–126CrossRefGoogle Scholar
  41. 41.
    MacKay DJC (2002) Information theory, inference & learning algorithms. Cambridge University Press, New YorkGoogle Scholar
  42. 42.
    Matthews F, Pearlmutter BA, Ward TE, Soraghan C, Markham C (2008) Hemodynamics for brain-computer interfaces. IEEE Signal Process Mag 25(1):87–94CrossRefGoogle Scholar
  43. 43.
    Pfurtscheller G, Allison BZ, Bauernfeind G, Brunner C, Solis Escalante T, Scherer R, Zander TO, Müller-Putz G, Neuper C, Birbaumer N (2010) The hybrid BCI. Front Neurosci 4:3Google Scholar
  44. 44.
    Pichler BJ, Wehrl HF, Kolb A, Judenhofer MS (2008) Positron emission tomography/magnetic resonance imaging: the next generation of multimodality imaging? Semin Nucl Med 38(3):199–208PubMedCentralCrossRefPubMedGoogle Scholar
  45. 45.
    Piper SK, Krueger A, Koch SP, Mehnert J, Habermehl C, Steinbrink J, Obrig H, Schmitz CH (2014) A wearable multi-channel fNIRS system for brain imaging in freely moving subjects. NeuroImage 85:64–71CrossRefPubMedGoogle Scholar
  46. 46.
    Popescu F, Fazli S, Badower Y, Blankertz B, Müller K-R (2007) Single trial classification of motor imagination using 6 dry EEG electrodes. PLoS ONE 2(7):e637PubMedCentralCrossRefPubMedGoogle Scholar
  47. 47.
    Ramoser H, Muller-Gerking J, Pfurtscheller G (2000) Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans Rehabil Eng 8(4):441–446CrossRefPubMedGoogle Scholar
  48. 48.
    Saad Z, Ropella K, Cox R, DeYoe E (2001) Analysis and use of fMRI response delays. Hum Brain Mapp 13(2):74–93CrossRefPubMedGoogle Scholar
  49. 49.
    Sannelli C, Dickhaus T, Halder S, Hammer E-M, Müller K-R, Blankertz B (2010) On optimal channel configurations for SMR-based brain–computer interfaces. Brain Topogr 23(2):186–193CrossRefPubMedGoogle Scholar
  50. 50.
    Schröder M, Lal TN, Hinterberger T, Bogdan M, Hill NJ, Birbaumer N, Rosenstiel W, Schölkopf B (2005) Robust EEG channel selection across subjects for brain-computer interfaces. EURASIP J Appl Signal Process 2005:3103–3112CrossRefGoogle Scholar
  51. 51.
    Schroeter ML, Kupka T, Mildner T, Uludağ K, Von Cramon DY (2006) Investigating the post-stimulus undershoot of the BOLD signal–a simultaneous fMRI and fNIRS study. NeuroImage 30(2):349–358CrossRefPubMedGoogle Scholar
  52. 52.
    Schudlo LC, Chau T (2014) Dynamic topographical pattern classification of multichannel prefrontal NIRS signals: II. Online differentiation of mental arithmetic and rest. J Neural Eng 11(1):016003Google Scholar
  53. 53.
    Ge S, Wang R, Yu D (2014) Classification of four-class motor imagery employing single-channel electroencephalography. PLoS ONE 9(6):e98019PubMedCentralCrossRefPubMedGoogle Scholar
  54. 54.
    Sitaram R, Zhang H, Guan C, Thulasidas M, Hoshi Y, Ishikawa A, Shimizu K, Birbaumer N (2007) Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain–computer interface. NeuroImage 34(4):1416–1427CrossRefPubMedGoogle Scholar
  55. 55.
    Solovey ET, Adviser-Jacob RJ (2012) Real-time fNIRS brain input for enhancing interactive systems. Ph.D. thesis, Tufts UniversityGoogle Scholar
  56. 56.
    Tate RF (1954) Correlation between a discrete and a continuous variable. Point-biserial correlation. Ann Math Stat 25(3):603–607CrossRefGoogle Scholar
  57. 57.
    Van De Ville D, Lee S-W (2012) Brain decoding: opportunities and challenges for pattern recognition. Pattern Recognit 45(6):2033–2034CrossRefGoogle Scholar
  58. 58.
    Villringer A, Chance B (1997) Non-invasive optical spectroscopy and imaging of human brain function. Trends Neurosci 20(10):435–442CrossRefPubMedGoogle Scholar
  59. 59.
    Villringer A, Planck J, Hock C, Schleinkofer L, Dirnagl U (1993) Near infrared spectroscopy (NIRS): a new tool to study hemodynamic changes during activation of brain function in human adults. Neurosci Lett 154(1):101–104CrossRefPubMedGoogle Scholar
  60. 60.
    Wallois F, Mahmoudzadeh M, Patil A, Grebe R (2012) Usefulness of simultaneous EEG–NIRS recording in language studies. Brain Lang 121(2):110–123CrossRefPubMedGoogle Scholar
  61. 61.
    Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM (2002) Brain–computer interfaces for communication and control. Clin Neurophysiol 113(6):767–791CrossRefPubMedGoogle Scholar
  62. 62.
    Wriessnegger S, Kurzmann J, Neuper C (2008) Spatio-temporal differences in brain oxygenation between movement execution and imagery: a multichannel near-infrared spectroscopy study. Int J Psychophysiol 67(1):54–63CrossRefPubMedGoogle Scholar
  63. 63.
    Yeom S-K, Fazli S, Mehnert J, Blankertz B, Steinbrink J, Müller K-R, Lee S-W (2013) Multimodal imaging technique for rapid response brain-computer interface feedback. In: International winter workshop on Brain-Computer Interface (BCI). IEEE, pp 92–94Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  • Siamac Fazli
    • 1
    Email author
  • Min-Ho Lee
    • 1
  • Seul-Ki Yeom
    • 1
  • John Williamson
    • 1
  • Isabella Schlattner
    • 1
  • Yiyu Chen
    • 1
  • Seong-Whan Lee
    • 1
    Email author
  1. 1.Department of Brain and Cognitive EngineeringKorea UniversitySeoulRepublic of Korea

Personalised recommendations