Skip to main content
SpringerLink
Log in

Semi-autonomous Hybrid Brain-Machine Interface

  • Chapter
  • First Online:
Brain-Computer Interface Research

Part of the book series: SpringerBriefs in Electrical and Computer Engineering ((BRIEFSELECTRIC))

Abstract

Although advanced prosthetic limbs, such as the modular prosthetic limb (MPL), are now capable of mimicking the dexterity of human limbs, brain-machine interfaces (BMIs) are not yet able to take full advantage of their capabilities. To improve BMI control of the MPL, we are developing a semi-autonomous system, the Hybrid Augmented Reality Multimodal Operation Neural Integration Environment (HARMONIE). This system is designed to utilize novel control strategies including hybrid input (adding eye tracking to neural control), supervisory control (decoding high-level patient goals), and intelligent robotics (incorporating computer vision and route planning algorithms). Patients use eye gaze to indicate a desired object that has been recognized by computer vision. They then perform a desired action, such as reaching and grasping, which is decoded and carried out by the MPL via route planning algorithms. Here we present two patients, implanted with electrocorticography (ECoG) and depth electrodes, who controlled the HARMONIE system to perform reach and grasping tasks; in addition, one patient also used the HARMONIE system to simulate self-feeding. This work builds upon prior research to demonstrate the feasibility of using novel control strategies to enable patients to perform a wider variety of activities of daily living (ADLs).

D.P. McMullen and M.S. Fifer are co-first authors.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • B.Z. Allison, C. Brunner, V. Kaiser, G.R. Müller-Putz, C. Neuper, G. Pfurtscheller, Toward a hybrid brain–computer interface based on imagined movement and visual attention. J. Neural Eng. 7(2), 026007 (2010)

    Article  Google Scholar 

  • B.Z. Allison, R. Leeb, C. Brunner, G.R. Müller-Putz, G. Bauernfeind, J.W. Kelly, C. Neuper, Toward smarter BCIs: extending BCIs through hybridization and intelligent control. J. Neural Eng. 9(1), 013001 (2012)

    Article  Google Scholar 

  • R.A. Andersen, J.W. Burdick, S. Musallam, B. Pesaran, J.G. Cham, Cognitive neural prosthetics. Trends Cogn. Sci. 8(11), 486–493 (2004)

    Article  Google Scholar 

  • Y. Benjamini, Y. Hochberg, Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. Ser. B Methodol. 57(1), 289–300 (1995)

    MathSciNet  MATH  Google Scholar 

  • J.L. Collinger, B. Wodlinger, J.E. Downey, W. Wang, E.C. Tyler-Kabara, D.J. Weber, A.J.C. McMorland, M. Velliste, M.L. Boninger, A.B. Schwartz, High-performance neuroprosthetic control by an individual with tetraplegia. Lancet 381(9866), 557–564 (2013)

    Article  Google Scholar 

  • N.E. Crone, D.L. Miglioretti, B. Gordon, R.P. Lesser, Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. II. Event-related synchronization in the gamma band. Brain 121(Pt 12), 2301–2315 (1998)

    Article  Google Scholar 

  • N.E. Crone, D. Boatman, B. Gordon, L. Hao, Induced electrocorticographic gamma activity during auditory perception. Brazier Award-winning article, 2001. Clin. Neurophysiol. 112(4), 565–582 (2001a)

    Article  Google Scholar 

  • N.E. Crone, L. Hao, J. Hart Jr, D. Boatman, R.P. Lesser, R. Irizarry, B. Gordon, Electrocorticographic gamma activity during word production in spoken and sign language. Neurology 57(11), 2045–2053 (2001b)

    Article  Google Scholar 

  • N.E. Crone, A. Korzeniewska, P.J. Franaszczuk, Cortical gamma responses: searching high and low. Int. J. Psychophysiol. 79(1), 9–15 (2011)

    Article  Google Scholar 

  • C. Escolano, J.M. Antelis, J. Minguez, A telepresence mobile robot controlled with a noninvasive brain-computer interface. IEEE Trans. Syst. Man Cybern. Part B Cybern. 42(3), 793–804 (2012)

    Article  Google Scholar 

  • M.S. Fifer, G. Hotson, B. Wester, D.P. McMullen, Y. Wang, M.S. Johannes, K.D. Katyal, J.B. Helder, M.P. Para, R.J. Vogelstein, W.S. Anderson, N.V. Thakor, N.E. Crone, Simultaneous neural control of simple reaching and grasping with the modular prosthetic limb using intracranial EEG. IEEE Trans. Neural Syst. Rehabil. Eng. 22(3), 695–705 (2014)

    Article  Google Scholar 

  • F.O. Flemisch, C.A. Adams, S.R. Conway, K.H. Goodrich, M.T. Palmer, P.C. Schutte, The H-metaphor as a guideline for vehicle automation and interaction. NASA Technical Report NASA/TM—2003-212672 (2003)

    Google Scholar 

  • A. Frisoli, C. Loconsole, D. Leonardis, F. Banno, M. Barsotti, C. Chisari, M. Bergamasco, A new gaze-BCI-driven control of an upper limb exoskeleton for rehabilitation in real-world tasks. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 42(6), 1169–1179 (2012)

    Article  Google Scholar 

  • K.H. Goodrich, P.C. Schutte, F.O. Flemisch, R.A. Williams, Application of the H-mode, a design and interaction concept for highly automated vehicles, to aircraft, in 25th Digital Avionics Systems Conference, 2006 IEEE/AIAA, 2006, pp. 1–13

    Google Scholar 

  • S. Hayati, S.T. Venkataraman, Design and implementation of a robot control system with traded and shared control capability, in Proceedings IEEE International Conference on Robotics and Automation, 1989, vol. 3, pp. 1310–1315

    Google Scholar 

  • I. Iturrate, J.M. Antelis, A. Kubler, J. Minguez, A noninvasive brain-actuated wheelchair based on a P300 neurophysiological protocol and automated navigation. IEEE Trans. Robot. 25(3), 614–627 (2009)

    Article  Google Scholar 

  • K.D. Katyal, M.S. Johannes, T.G. McGee, A.J. Harris, R.S. Armiger, A.H. Firpi, D. McMullen, G. Hotson, M.S. Fifer, N.E. Crone, R.J. Vogelstein, B.A. Wester, HARMONIE: A multimodal control framework for human assistive robotics, in 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER), 2013, pp. 1274–1278

    Google Scholar 

  • B.H. Kim, M. Kim, S. Jo, Quadcopter flight control using a low-cost hybrid interface with EEG-based classification and eye tracking. Comput. Biol. Med. 51, 82–92 (2014)

    Article  Google Scholar 

  • J.P. Lachaux, N. George, C. Tallon-Baudry, J. Martinerie, L. Hugueville, L. Minotti, P. Kahane, B. Renault, The many faces of the gamma band response to complex visual stimuli. Neuroimage 25(2), 491–501 (2005)

    Article  Google Scholar 

  • E.C. Lee, J.C. Woo, J.H. Kim, M. Whang, K.R. Park, A brain–computer interface method combined with eye tracking for 3D interaction. J. Neurosci. Methods 190(2), 289–298 (2010)

    Article  Google Scholar 

  • E.C. Leuthardt, G. Schalk, J.R. Wolpaw, J.G. Ojemann, D.W. Moran, A brain-computer interface using electrocorticographic signals in humans. J. Neural Eng. 1(2), 63–71 (2004)

    Article  Google Scholar 

  • D.P. McMullen, G. Hotson, K.D. Katyal, B.A. Wester, M.S. Fifer, T.G. McGee, A. Harris, M.S. Johannes, R.J. Vogelstein, A.D. Ravitz, W.S. Anderson, N.V. Thakor, N.E. Crone, Demonstration of a semi-autonomous hybrid brain-machine interface using human intracranial eeg, eye tracking, and computer vision to control a robotic upper limb prosthetic. IEEE Trans. Neural Syst. Rehabil. Eng. 22(4), 784–796 (2014)

    Article  Google Scholar 

  • T. Milekovic, T. Ball, A. Schulze-Bonhage, A. Aertsen, C. Mehring, Detection of error related neuronal responses recorded by electrocorticography in humans during continuous movements. PLoS One 8(2), e55235 (2013)

    Article  Google Scholar 

  • J.D.R. Millán, R. Rupp, G.R. Müller-Putz, R. Murray-Smith, C. Giugliemma, M. Tangermann, C. Vidaurre, F. Cincotti, A. Kübler, R. Leeb, C. Neuper, K.-R. Müller, D. Mattia, Combining brain–computer interfaces and assistive technologies: state-of-the-art and challenges. Front. Neuroprosthetics 4, 161 (2010)

    Google Scholar 

  • S. Musallam, B.D. Corneil, B. Greger, H. Scherberger, R.A. Andersen, cognitive control signals for neural prosthetics. Science 305(5681), 258–262 (2004)

    Article  Google Scholar 

  • G. Pfurtscheller, B.Z. Allison, C. Brunner, G. Bauernfeind, T. Solis-Escalante, R. Scherer, T.O. Zander, G. Mueller-Putz, C. Neuper, N. Birbaumer, The hybrid BCI. Front. Neurosci. 4, 30 (2010a)

    Google Scholar 

  • G. Pfurtscheller, T. Solis-Escalante, R. Ortner, P. Linortner, G.R. Muller-Putz, Self-paced operation of an SSVEP-based orthosis with and without an imagery-based ‘brain switch:’ a feasibility study towards a hybrid BCI. IEEE Trans. Neural Syst. Rehabil. Eng. 18(4), 409–414 (2010b)

    Article  Google Scholar 

  • S. Ray, E. Niebur, S.S. Hsiao, A. Sinai, N.E. Crone, High-frequency gamma activity (80-150 Hz) is increased in human cortex during selective attention. Clin. Neurophysiol. 119(1), 116–133 (2008)

    Article  Google Scholar 

  • A.S. Royer, B. He, Goal selection versus process control in a brain–computer interface based on sensorimotor rhythms. J. Neural Eng. 6(1), 016005 (2009)

    Article  Google Scholar 

  • A.S. Royer, M.L. Rose, B. He, Goal selection versus process control while learning to use a brain–computer interface. J. Neural Eng. 8(3), 036012 (2011)

    Article  Google Scholar 

  • G. Schalk, K.J. Miller, N.R. Anderson, J.A. Wilson, M.D. Smyth, J.G. Ojemann, D.W. Moran, J.R. Wolpaw, E.C. Leuthardt, Two-dimensional movement control using electrocorticographic signals in humans. J. Neural Eng. 5(1), 75–84 (2008)

    Article  Google Scholar 

  • R. Vinjamuri, D.J. Weber, Z.-H. Mao, J.L. Collinger, A.D. Degenhart, J.W. Kelly, M.L. Boninger, E.C. Tyler-Kabara, W. Wang, Toward synergy-based brain-machine interfaces. IEEE Trans. Inf. Technol. Biomed. Publ. IEEE Eng. Med. Biol. Soc. 15(5), 726–736 (2011)

    Article  Google Scholar 

  • W. Wang, J.L. Collinger, A.D. Degenhart, E.C. Tyler-Kabara, A.B. Schwartz, D.W. Moran, D.J. Weber, B. Wodlinger, R.K. Vinjamuri, R.C. Ashmore, J.W. Kelly, M.L. Boninger, An electrocorticographic brain interface in an individual with tetraplegia. PLoS One 8(2), e55344 (2013)

    Article  Google Scholar 

  • J.R. Wolpaw, Brain–computer interfaces as new brain output pathways. J. Physiol. 579(3), 613–619 (2007)

    Article  Google Scholar 

  • T. Yanagisawa, M. Hirata, Y. Saitoh, H. Kishima, K. Matsushita, T. Goto, R. Fukuma, H. Yokoi, Y. Kamitani, T. Yoshimine, Electrocorticographic control of a prosthetic arm in paralyzed patients. Ann. Neurol. 71(3), 353–361 (2012)

    Article  Google Scholar 

  • T.O. Zander, M. Gaertner, C. Kothe, R. Vilimek, Combining eye gaze input with a brain-computer interface for touchless human-computer interaction. Int. J. Hum. Comput. Interact. 27(1), 38–51 (2010)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Neurosurgery, Johns Hopkins University, 600 N Wolfe Street/Meyer 8-181, 21287, Baltimore, MD, USA

    David P. McMullen & William S. Anderson

  2. Department of Biomedical Engineering, Johns Hopkins University, Baltimore, USA

    Matthew S. Fifer

  3. Applied Physics Laboratory, Johns Hopkins University, Baltimore, USA

    Brock A. Wester, Kapil D. Katyal, Matthew S. Johannes, Timothy G. McGee, Andrew Harris, Alan D. Ravitz & Michael P. McLoughlin

  4. Department of Electrical Engineering, Johns Hopkins University, Baltimore, USA

    Guy Hotson & Nitish V. Thakor

  5. Department of Neurology, Johns Hopkins University, Baltimore, USA

    Nathan E. Crone

Authors
  1. David P. McMullen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matthew S. Fifer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Brock A. Wester
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guy Hotson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kapil D. Katyal
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Matthew S. Johannes
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Timothy G. McGee
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Andrew Harris
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Alan D. Ravitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Michael P. McLoughlin
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. William S. Anderson
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Nitish V. Thakor
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Nathan E. Crone
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David P. McMullen .

Editor information

Editors and Affiliations

  1. g.tec Guger Technologies OG, Schiedlberg, Austria

    Christoph Guger

  2. Graz, Austria

    Gernot Müller-Putz

  3. Cognitive Science Dept., UC San Diego, La Jolla, California, USA

    Brendan Allison

Rights and permissions

Reprints and permissions

Copyright information

© 2015 The Author(s)

About this chapter

Cite this chapter

McMullen, D.P. et al. (2015). Semi-autonomous Hybrid Brain-Machine Interface. In: Guger, C., Müller-Putz, G., Allison, B. (eds) Brain-Computer Interface Research. SpringerBriefs in Electrical and Computer Engineering. Springer, Cham. https://doi.org/10.1007/978-3-319-25190-5_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-25190-5_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-25188-2

  • Online ISBN: 978-3-319-25190-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation