Towards a Hybrid P300-Based BCI Using Simultaneous fNIR and EEG

  • Yichuan Liu
  • Hasan Ayaz
  • Adrian Curtin
  • Banu Onaral
  • Patricia A. Shewokis
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8027)


Next generation brain computer interfaces (BCI) are expected to provide robust and continuous control mechanism. In this study, we assessed integration of optical brain imaging (fNIR: functional near infrared spectroscopy) to a P300-BCI for improving BCI usability by monitoring cognitive workload and performance. fNIR is a safe and wearable neuroimaging modality that tracks cortical hemodynamics in response to sensory, motor, or cognitive activation. Eight volunteers participated in the study where simultaneous EEG and 16 optode fNIR from anterior prefrontal cortex were recorded while participants engaged with the P300-BCI for spatial navigation. The results showed a significant response in fNIR signals during high, medium and low performance indicating a positive correlation between prefrontal oxygenation changes and BCI performance. This preliminary study provided evidence that the performance of P300-BCI can be monitored by fNIR which in turn can help improve the robustness of the BCI classification.


BCI P300 fNIR Performance Optical brain imaging EEG 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M.: Brain-computer interfaces for communication and control. Clinical Neurophysiology 113, 767–791 (2002)CrossRefGoogle Scholar
  2. 2.
    Farwell, L.A., Donchin, E.: Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalography and clinical Neurophysiology 70, 510–523 (1988)CrossRefGoogle Scholar
  3. 3.
    Pfurtscheller, G., Neuper, C.: Motor imagery and direct brain-computer communication. Proceedings of the IEEE 89, 1123–1134 (2001)CrossRefGoogle Scholar
  4. 4.
    Mellinger, J., Schalk, G., Braun, C., Preissl, H., Rosenstiel, W., Birbaumer, N., Kübler, A.: An MEG-based brain-computer interface (BCI). Neuroimage 36, 581 (2007)CrossRefGoogle Scholar
  5. 5.
    Coyle, S., Ward, T., Markham, C., McDarby, G.: On the suitability of near-infrared (NIR) systems for next-generation brain–computer interfaces. Physiological Measurement 25, 815 (2004)CrossRefGoogle Scholar
  6. 6.
    Sitaram, R., Zhang, H., Guan, C., Thulasidas, M., Hoshi, Y., Ishikawa, A., Shimizu, K., Birbaumer, N.: Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain–computer interface. NeuroImage 34, 1416–1427 (2007)CrossRefGoogle Scholar
  7. 7.
    Ayaz, H., Izzetoglu, M., Bunce, S., Heiman-Patterson, T., Onaral, B.: Detecting cognitive activity related hemodynamic signal for brain computer interface using functional near infrared spectroscopy, pp. 342–345. IEEE (2007)Google Scholar
  8. 8.
    Limongi, T., Di Sante, G., Ferrari, M., Quaresima, V.: Detecting mental calculation related frontal cortex oxygenation changes for brain computer interface using multi-channel functional near infrared topography. International Journal of Bioelectromagnetism 11, 86–90 (2009)Google Scholar
  9. 9.
    Ayaz, H., Shewokis, P., Bunce, S., Schultheis, M., Onaral, B.: Assessment of cognitive neural correlates for a functional near infrared-based brain computer interface system. Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience, 699–708 (2009) Google Scholar
  10. 10.
    Weiskopf, N., Mathiak, K., Bock, S.W., Scharnowski, F., Veit, R., Grodd, W., Goebel, R., Birbaumer, N.: Principles of a brain-computer interface (BCI) based on real-time functional magnetic resonance imaging (fMRI). IEEE Transactions on Biomedical Engineering 51, 966–970 (2004)CrossRefGoogle Scholar
  11. 11.
    Yoo, S.S., Fairneny, T., Chen, N.K., Choo, S.E., Panych, L.P., Park, H.W., Lee, S.Y., Jolesz, F.A.: Brain-computer interface using fMRI: spatial navigation by thoughts. Neuroreport 15, 1591–1595 (2004)CrossRefGoogle Scholar
  12. 12.
    Ferrez, P.W., Millán, J.R.: Simultaneous real-time detection of motor imagery and error-related potentials for improved BCI accuracy. In: Proceedings of the 4th International Brain-Computer Interface Workshop, pp. 197–202 (2008)Google Scholar
  13. 13.
    Allison, B., Brunner, C., Kaiser, V., Müller-Putz, G., Neuper, C., Pfurtscheller, G.: Toward a hybrid brain–computer interface based on imagined movement and visual attention. Journal of Neural Engineering 7, 026007 (2010)Google Scholar
  14. 14.
    Pfurtscheller, G., Allison, B.Z., Bauernfeind, G.N., Brunner, C., Solis Escalante, T., Scherer, R., Zander, T.O., Mueller-Putz, G., Neuper, C., Birbaumer, N.: The hybrid BCI. Frontiers in Neuroscience 4 (2010)Google Scholar
  15. 15.
    Rebsamen, B., Burdet, E., Zeng, Q., Zhang, H., Ang, M., Teo, C.L., Guan, C., Laugier, C.: Hybrid P300 and mu-beta brain computer interface to operate a brain controlled wheelchair. In: Proceedings of the 2nd International Convention on Rehabilitation Engineering & Assistive Technology, pp. 51-55. Singapore Therapeutic, Assistive & Rehabilitative Technologies (START) Centre, Bangkok, Thailand (2008)Google Scholar
  16. 16.
    Müller-Putz, G.: Hybrid brain-computer interfaces: current state and future directions. PPT 55, 923–929 (2011)Google Scholar
  17. 17.
    Mak, J., Arbel, Y., Minett, J., McCane, L., Yuksel, B., Ryan, D., Thompson, D., Bianchi, L., Erdogmus, D.: Optimizing the P300-based brain–computer interface: current status, limitations and future directions. Journal of Neural Engineering 8, 025003 (2011)CrossRefGoogle Scholar
  18. 18.
    Bunce, S.C., Izzetoglu, M., Izzetoglu, K., Onaral, B., Pourrezaei, K.: Functional near-infrared spectroscopy. IEEE Engineering in Medicine and Biology Magazine 25, 54–62 (2006)CrossRefGoogle Scholar
  19. 19.
    Fazli, S., Mehnert, J., Steinbrink, J., Curio, G., Villringer, A., Müller, K.-R., Blankertz, B.: Enhanced performance by a hybrid NIRS–EEG brain computer interface. Neuroimage 59, 519–529 (2012)CrossRefGoogle Scholar
  20. 20.
    Fazli, S., Mehnert, J., Steinbrink, J., Blankertz, B.: Using NIRS as a predictor for EEG-based BCI performance. In: Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 4911–4914 (2012)Google Scholar
  21. 21.
    Mak, J.N., McFarland, D.J., Vaughan, T.M., McCane, L.M., Tsui, P.Z., Zeitlin, D.J., Sellers, E.W., Wolpaw, J.R.: EEG correlates of P300-based brain–computer interface (BCI) performance in people with amyotrophic lateral sclerosis. Journal of Neural Engineering 9, 026014 (2012)CrossRefGoogle Scholar
  22. 22.
    Halder, S., Hammer, E.M., Kleih, S.C., Bogdan, M., Rosenstiel, W., Birbaumer, N., Kübler, A.: Prediction of Auditory and Visual P300 Brain-Computer Interface Aptitude. PLOS One 8, e53513 (2013)Google Scholar
  23. 23.
    Bledowski, C., Prvulovic, D., Goebel, R., Zanella, F.E., Linden, D.E.J.: Attentional systems in target and distractor processing: a combined ERP and fMRI study. Neuroimage 22, 530–540 (2004)CrossRefGoogle Scholar
  24. 24.
    Mccarthy, G., Luby, M., Gore, J., Goldman-Rakic, P.: Infrequent events transiently activate human prefrontal and parietal cortex as measured by functional MRI. Journal of Neurophysiology 77, 1630–1634 (1997)Google Scholar
  25. 25.
    Horovitz, S.G., Skudlarski, P., Gore, J.C.: Correlations and dissociations between BOLD signal and P300 amplitude in an auditory oddball task: a parametric approach to combining fMRI and ERP. Magnetic Resonance Imaging 20, 319–325 (2002)CrossRefGoogle Scholar
  26. 26.
    Thomas, M., Sing, H., Belenky, G., Holcomb, H., Mayberg, H., Dannals, R., Wagner, J., Thorne, D., Popp, K., Rowland, L.: Neural basis of alertness and cognitive performance impairments during sleepiness. I. Effects of 24 h of sleep deprivation on waking human regional brain activity. Journal of Sleep Research 9, 335–352 (2008)CrossRefGoogle Scholar
  27. 27.
    Moller, H.J., Rizzo, A.A., Mikulis, D.J.: Prefrontal cortex activation mediates cognitive reserve alertness and attention in the Virtual Classroom: preliminary fMRI findings and clinical implications. In: Virtual Rehabilitation, pp. 146–150 (2007)Google Scholar
  28. 28.
    Fuster, J.M.: The prefrontal cortex. Academic Press (2008)Google Scholar
  29. 29.
    Ayaz, H., Bunce, S., Shewokis, P., Izzetoglu, K., Willems, B., Onaral, B.: Using Brain Activity to Predict Task Performance and Operator Efficiency. In: Zhang, H., Hussain, A., Liu, D., Wang, Z. (eds.) BICS 2012. LNCS, vol. 7366, pp. 147–155. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  30. 30.
    Liu, Y., Ayaz, H., Curtin, A., Shewokis, P.A., Onaral, B.: Detection of attention shift for asynchronous P300-based BCI. In: Proc. IEEE Eng. Med. Biol. Soc., pp. 4724–4727 (2012)Google Scholar
  31. 31.
    Curtin, A., Ayaz, H., Liu, Y., Shewokis, P.A., Onaral, B.: A P300-based EEG-BCI for Spatial Navigation Control. In: Conf. Proc. IEEE Eng.Med. Biol. Soc., pp. 3841–3844 (2012)Google Scholar
  32. 32.
    Ayaz, H., Shewokis, P.A., Curtin, A., Izzetoglu, M., Izzetoglu, K., Onaral, B.: Using MazeSuite and functional near infrared spectroscopy to study learning in spatial navigation. J. Vis. Exp., e3443 (2011)Google Scholar
  33. 33.
    Ayaz, H., Allen, S.L., Platek, S.M., Onaral, B.: Maze Suite 1.0: a complete set of tools to prepare, present, and analyze navigational and spatial cognitive neuroscience experiments. Behavior Research Methods 40, 353–359 (2008)CrossRefGoogle Scholar
  34. 34.
    Schalk, G., McFarland, D.J., Hinterberger, T., Birbaumer, N., Wolpaw, J.R.: BCI2000: a general-purpose brain-computer interface (BCI) system. IEEE Transactions on Biomedical Engineering 51, 1034–1043 (2004)CrossRefGoogle Scholar
  35. 35.
    Ayaz, H., Shewokis, P.A., Bunce, S., Izzetoglu, K., Willems, B., Onaral, B.: Optical brain monitoring for operator training and mental workload assessment. Neuroimage 59, 36–47 (2012)CrossRefGoogle Scholar
  36. 36.
    Ayaz, H., Izzetoglu, M., Shewokis, P.A., Onaral, B.: Sliding-window motion artifact rejection for functional near-infrared spectroscopy. In: Annual International Conf. on Engineering in Medicine and Biology Society (EMBC), pp. 6567–6570. IEEE (2010)Google Scholar
  37. 37.
    Halder, S., Agorastos, D., Veit, R., Hammer, E., Lee, S., Varkuti, B., Bogdan, M., Rosenstiel, W., Birbaumer, N., Kübler, A.: Neural mechanisms of brain–computer interface control. NeuroImage 55, 1779–1790 (2011)CrossRefGoogle Scholar
  38. 38.
    Miller, E.K., Cohen, J.D.: An integrative theory of prefrontal cortex function. Annual Review of Neuroscience 24, 167–202 (2001)CrossRefGoogle Scholar
  39. 39.
    Posner, M.I., Rothbart, M.K.: Research on attention networks as a model for the integration of psychological science. Annu. Rev. Psychol. 58, 1–23 (2007)CrossRefGoogle Scholar
  40. 40.
    Ayaz, H., Shewokis, P.A., İzzetoğlu, M., Çakır, M.P., Onaral, B.: Tangram solved? Prefrontal cortex activation analysis during geometric problem solving. In: 34th Annual International IEEE EMBS Conference, pp. 4724–4727. IEEE (2012)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Yichuan Liu
    • 1
    • 2
  • Hasan Ayaz
    • 1
    • 2
  • Adrian Curtin
    • 1
    • 2
  • Banu Onaral
    • 1
    • 2
  • Patricia A. Shewokis
    • 1
    • 2
    • 3
  1. 1.School of Biomedical Engineering, Science & Health SystemsDrexel UniversityPhiladelphiaUSA
  2. 2.Cognitive Neuroengineering and Quantitative Experimental Research (CONQUER) CollaborativeDrexel UniversityPhiladelphiaUSA
  3. 3.Nutrition Sciences Department, College of Nursing and Health ProfessionsDrexel UniversityPhiladelphiaUSA

Personalised recommendations