Springer Nature is making Coronavirus research free. View research | View latest news | Sign up for updates

Effects of congruent and incongruent visual cues on speech perception and brain activity in cochlear implant users


While deafness-induced plasticity has been investigated in the visual and auditory domains, not much is known about language processing in audiovisual multimodal environments for patients with restored hearing via cochlear implant (CI) devices. Here, we examined the effect of agreeing or conflicting visual inputs on auditory processing in deaf patients equipped with degraded artificial hearing. Ten post-lingually deafened CI users with good performance, along with matched control subjects, underwent H 2 15 O-positron emission tomography scans while carrying out a behavioral task requiring the extraction of speech information from unimodal auditory stimuli, bimodal audiovisual congruent stimuli, and incongruent stimuli. Regardless of congruency, the control subjects demonstrated activation of the auditory and visual sensory cortices, as well as the superior temporal sulcus, the classical multisensory integration area, indicating a bottom-up multisensory processing strategy. Compared to CI users, the control subjects exhibited activation of the right ventral premotor-supramarginal pathway. In contrast, CI users activated primarily the visual cortices more in the congruent audiovisual condition than in the null condition. In addition, compared to controls, CI users displayed an activation focus in the right amygdala for congruent audiovisual stimuli. The most notable difference between the two groups was an activation focus in the left inferior frontal gyrus in CI users confronted with incongruent audiovisual stimuli, suggesting top-down cognitive modulation for audiovisual conflict. Correlation analysis revealed that good speech performance was positively correlated with right amygdala activity for the congruent condition, but negatively correlated with bilateral visual cortices regardless of congruency. Taken together these results suggest that for multimodal inputs, cochlear implant users are more vision-reliant when processing congruent stimuli and are disturbed more by visual distractors when confronted with incongruent audiovisual stimuli. To cope with this multimodal conflict, CI users activate the left inferior frontal gyrus to adopt a top-down cognitive modulation pathway, whereas normal hearing individuals primarily adopt a bottom-up strategy.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6



Normal hearing




Superior temporal sulcus


Inferior frontal gyrus


Cochlear implant


Positron emission tomography


Magnetic resonance imaging


Categories of auditory performance


Middle temporal gyrus


Inferior temporal gyrus


Regional cerebral blood flow


Ventral premotor cortex


Supramarginal gyrus


Superior frontal gyrus


Medial frontal gyrus


  1. Aboitiz F, Garcia R (2009) Merging of phonological and gestural circuits in early language evolution. Rev Neurosci 20:71–84

  2. Adam R, Noppeney U (2010) Prior auditory information shapes visual category-selectivity in ventral occipito-temporal cortex. Neuroimage 52:1592–1602

  3. Archbold S, Lutman ME, Marshall DH (1995) Categories of auditory performance. Ann Otol Rhinol Laryngol Suppl 166:312–314

  4. Aron AR, Fletcher PC, Bullmore ET, Sahakian BJ, Robbins TW (2003) Stop-signal inhibition disrupted by damage to right inferior frontal gyrus in humans. Nat Neurosci 6:115–116

  5. Beauchamp MS (2005) Statistical criteria in FMRI studies of multisensory integration. Neuroinformatics 3:93–113

  6. Beauchamp MS, Argall BD, Bodurka J, Duyn JH, Martin A (2004a) Unraveling multisensory integration: patchy organization within human STS multisensory cortex. Nat Neurosci 7:1190–1192

  7. Beauchamp MS, Lee KE, Argall BD, Martin A (2004b) Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41:809–823

  8. Beauchamp MS, Nath AR, Pasalar S (2010) fMRI-guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect. J Neurosci 30:2414–2417

  9. Bernstein LE, Auer ET Jr, Wagner M, Ponton CW (2008) Spatiotemporal dynamics of audiovisual speech processing. Neuroimage 39:423–435

  10. Bitan T, Lifshitz A, Breznitz Z, Booth JR (2010) Bidirectional connectivity between hemispheres occurs at multiple levels in language processing but depends on sex. J Neurosci 30:11576–11585

  11. Bunge SA, Dudukovic NM, Thomason ME, Vaidya CJ, Gabrieli JD (2002) Immature frontal lobe contributions to cognitive control in children: evidence from fMRI. Neuron 33:301–311

  12. Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Curr Biol 10:649–657

  13. Champoux F, Lepore F, Gagne JP, Theoret H (2009) Visual stimuli can impair auditory processing in cochlear implant users. Neuropsychologia 47:17–22

  14. Corbetta M, Shulman GL (2002) Control of goal-directed and stimulus-driven attention in the brain. Nat Rev Neurosci 3:201–215

  15. Doucet ME, Bergeron F, Lassonde M, Ferron P, Lepore F (2006) Cross-modal reorganization and speech perception in cochlear implant users. Brain 129:3376–3383

  16. Eickhoff SB, Stephan KE, Mohlberg H, Grefkes C, Fink GR, Amunts K, Zilles K (2005) A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. Neuroimage 25:1325–1335

  17. Finney EM, Fine I, Dobkins KR (2001) Visual stimuli activate auditory cortex in the deaf. Nat Neurosci 4:1171–1173

  18. Fletcher PC, Henson RN (2001) Frontal lobes and human memory: insights from functional neuroimaging. Brain 124:849–881

  19. Garavan H, Ross TJ, Stein EA (1999) Right hemispheric dominance of inhibitory control: an event-related functional MRI study. Proc Natl Acad Sci USA 96:8301–8306

  20. Genovese CR, Lazar NA, Nichols T (2002) Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage 15:870–878

  21. Giraud AL, Lee HJ (2007) Predicting cochlear implant outcome from brain organisation in the deaf. Restor Neurol Neurosci 25:381–390

  22. Giraud AL, Truy E (2002) The contribution of visual areas to speech comprehension: a PET study in cochlear implants patients and normal-hearing subjects. Neuropsychologia 40:1562–1569

  23. Giraud AL, Price CJ, Graham JM, Truy E, Frackowiak RS (2001) Cross-modal plasticity underpins language recovery after cochlear implantation. Neuron 30:657–663

  24. Goldberg RF, Perfetti CA, Fiez JA, Schneider W (2007) Selective retrieval of abstract semantic knowledge in left prefrontal cortex. J Neurosci 27:3790–3798

  25. Guenther FH, Ghosh SS, Tourville JA (2006) Neural modeling and imaging of the cortical interactions underlying syllable production. Brain Lang 96:280–301

  26. Haddon JE, Killcross S (2006) Prefrontal cortex lesions disrupt the contextual control of response conflict. J Neurosci 26:2933–2940

  27. Hein G, Doehrmann O, Muller NG, Kaiser J, Muckli L, Naumer MJ (2007) Object familiarity and semantic congruency modulate responses in cortical audiovisual integration areas. J Neurosci 27:7881–7887

  28. Jones JA, Callan DE (2003) Brain activity during audiovisual speech perception: an fMRI study of the McGurk effect. Neuroreport 14:1129–1133

  29. Kempe V, Thoresen JC, Kirk NW, Schaeffler F, Brooks PJ (2012) Individual differences in the discrimination of novel speech sounds: effects of sex, temporal processing, musical and cognitive abilities. PLoS One 7:e48623

  30. Koechlin E, Hyafil A (2007) Anterior prefrontal function and the limits of human decision-making. Science 318:594–598

  31. Koechlin E, Ody C, Kouneiher F (2003) The architecture of cognitive control in the human prefrontal cortex. Science 302:1181–1185

  32. Koles ZJ, Lind JC, Flor-Henry P (2010) Gender differences in brain functional organization during verbal and spatial cognitive challenges. Brain Topogr 23:199–204

  33. Kral A, O’Donoghue GM (2010) Profound deafness in childhood. N Engl J Med 363:1438–1450

  34. Kreifelts B, Ethofer T, Huberle E, Grodd W, Wildgruber D (2010) Association of trait emotional intelligence and individual fMRI-activation patterns during the perception of social signals from voice and face. Hum Brain Mapp 31:979–991

  35. Lazard DS, Lee HJ, Gaebler M, Kell CA, Truy E, Giraud AL (2010) Phonological processing in post-lingual deafness and cochlear implant outcome. Neuroimage 49:3443–3451

  36. Lazard DS, Giraud AL, Truy E, Lee HJ (2011) Evolution of non-speech sound memory in postlingual deafness: implications for cochlear implant rehabilitation. Neuropsychologia 49:2475–2482

  37. Lazard DS, Giraud AL, Gnansia D, Meyer B, Sterkers O (2012) Understanding the deafened brain: implications for cochlear implant rehabilitation. Eur Ann Otorhinolaryngol Head Neck Dis 129:98–103

  38. Lazard DS, Lee HJ, Truy E, Giraud AL (2013) Bilateral reorganization of posterior temporal cortices in post-lingual deafness and its relation to cochlear implant outcome. Hum Brain Mapp 34:1208–1219

  39. Lee DS, Lee JS, Oh SH, Kim SK, Kim JW, Chung JK, Lee MC, Kim CS (2001) Cross-modal plasticity and cochlear implants. Nature 409:149–150

  40. Lee JS, Lee DS, Oh SH, Kim CS, Kim JW, Hwang CH, Koo J, Kang E, Chung JK, Lee MC (2003) PET evidence of neuroplasticity in adult auditory cortex of postlingual deafness. J Nucl Med 44:1435–1439

  41. Lee HJ, Giraud AL, Kang E, Oh SH, Kang H, Kim CS, Lee DS (2007a) Cortical activity at rest predicts cochlear implantation outcome. Cereb Cortex 17:909–917

  42. Lee HJ, Truy E, Mamou G, Sappey-Marinier D, Giraud AL (2007b) Visual speech circuits in profound acquired deafness: a possible role for latent multimodal connectivity. Brain 130:2929–2941

  43. Marquis JP, Killcross S, Haddon JE (2007) Inactivation of the prelimbic, but not infralimbic, prefrontal cortex impairs the contextual control of response conflict in rats. Eur J Neurosci 25:559–566

  44. McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748

  45. Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221:389–391

  46. Mesulam MM (1998) From sensation to cognition. Brain 121(Pt 6):1013–1052

  47. Miller LM, D’Esposito M (2005) Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. J Neurosci 25:5884–5893

  48. Molenberghs P, Brander C, Mattingley JB, Cunnington R (2010) The role of the superior temporal sulcus and the mirror neuron system in imitation. Hum Brain Mapp 31:1316–1326

  49. Nath AR, Beauchamp MS (2011) Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech. J Neurosci 31:1704–1714

  50. Nath AR, Beauchamp MS (2012) A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion. Neuroimage 59:781–787

  51. Nishitani N, Hari R (2002) Viewing lip forms: cortical dynamics. Neuron 36:1211–1220

  52. Noppeney U, Josephs O, Hocking J, Price CJ, Friston KJ (2008) The effect of prior visual information on recognition of speech and sounds. Cereb Cortex 18:598–609

  53. Park H, Kang E, Kang H, Kim JS, Jensen O, Chung CK, Lee DS (2011) Cross-frequency power correlations reveal the right superior temporal gyrus as a hub region during working memory maintenance. Brain Connect 1:460–472

  54. Rauschecker JP (1999) Auditory cortical plasticity: a comparison with other sensory systems. Trends Neurosci 22:74–80

  55. Ross LA, Saint-Amour D, Leavitt VM, Javitt DC, Foxe JJ (2007) Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cereb Cortex 17:1147–1153

  56. Rouger J, Lagleyre S, Fraysse B, Deneve S, Deguine O, Barone P (2007) Evidence that cochlear-implanted deaf patients are better multisensory integrators. Proc Natl Acad Sci USA 104:7295–7300

  57. Rouger J, Fraysse B, Deguine O, Barone P (2008) McGurk effects in cochlear-implanted deaf subjects. Brain Res 1188:87–99

  58. Rouger J, Lagleyre S, Demonet JF, Fraysse B, Deguine O, Barone P (2012) Evolution of crossmodal reorganization of the voice area in cochlear-implanted deaf patients. Hum Brain Mapp 33:1929–1940

  59. Ruff DA, Marrett S, Heekeren HR, Bandettini PA, Ungerleider LG (2010) Complementary roles of systems representing sensory evidence and systems detecting task difficulty during perceptual decision making. Front Neurosci 4:190

  60. Schaefer A, Braver TS, Reynolds JR, Burgess GC, Yarkoni T, Gray JR (2006) Individual differences in amygdala activity predict response speed during working memory. J Neurosci 26:10120–10128

  61. Skipper JI, Nusbaum HC, Small SL (2005) Listening to talking faces: motor cortical activation during speech perception. Neuroimage 25:76–89

  62. Skipper JI, Goldin-Meadow S, Nusbaum HC, Small SL (2007) Speech-associated gestures, Broca’s area, and the human mirror system. Brain Lang 101:260–277

  63. Song JJ, Choi HG, Oh SH, Chang SO, Kim CS, Lee JH (2009) Unilateral sensorineural hearing loss in children: the importance of temporal bone computed tomography and audiometric follow-up. Otol Neurotol 30:604–608

  64. Song JJ, De Ridder D, Van de Heyning P, Vanneste S (2012a) Mapping tinnitus-related brain activation: an activation-likelihood estimation metaanalysis of PET studies. J Nucl Med 53:1550–1557

  65. Song JJ, Yoo YT, An YH, Yoo JC, Kim JS, Koo JW (2012b) Comorbid benign paroxysmal positional vertigo in idiopathic sudden sensorineural hearing loss: an ominous sign for hearing recovery. Otol Neurotol 33:137–141

  66. Song JJ, Punte AK, De Ridder D, Vanneste S, Van de Heyning P (2013a) Neural substrates predicting improvement of tinnitus after cochlear implantation in patients with single-sided deafness. Hear Res 299:1–9

  67. Song JJ, Vanneste S, Schlee W, Van de Heyning P, De Ridder D (2013b) Onset-related differences in neural substrates of tinnitus-related distress: the anterior cingulate cortex in late-onset tinnitus, and the frontal cortex in early-onset tinnitus. Brain Struct Funct [Epub ahead of print]

  68. Strelnikov K, Rosito M, Barone P (2011) Effect of audiovisual training on monaural spatial hearing in horizontal plane. PLoS One 6:e18344

  69. Strelnikov K, Rouger J, Demonet JF, Lagleyre S, Fraysse B, Deguine O, Barone P (2013) Visual activity predicts auditory recovery from deafness after adult cochlear implantation. Brain 136:3682–3695

  70. Sumby WH, Pollack I (1954) Visual contribution to speech intelligibility in noise. J Acoust Soc Am 26:212–215

  71. Szycik GR, Jansma H, Munte TF (2009) Audiovisual integration during speech comprehension: an fMRI study comparing ROI-based and whole brain analyses. Hum Brain Mapp 30:1990–1999

  72. Talairach J, Tornoux P (1988) Co-planar stereotaxic atlas of the human brain: 3-dimensional proportional system: An approach to cerebral imaging. Georg Thieme, Stuttgart

  73. Talsma D, Doty TJ, Woldorff MG (2007) Selective attention and audiovisual integration: is attending to both modalities a prerequisite for early integration? Cereb Cortex 17:679–690

  74. Tremblay C, Champoux F, Lepore F, Theoret H (2010) Audiovisual fusion and cochlear implant proficiency. Restor Neurol Neurosci 28:283–291

  75. van Atteveldt N, Formisano E, Goebel R, Blomert L (2004) Integration of letters and speech sounds in the human brain. Neuron 43:271–282

  76. van Atteveldt NM, Formisano E, Goebel R, Blomert L (2007) Top-down task effects overrule automatic multisensory responses to letter-sound pairs in auditory association cortex. Neuroimage 36:1345–1360

  77. van Wassenhove V, Grant KW, Poeppel D (2005) Visual speech speeds up the neural processing of auditory speech. Proc Natl Acad Sci USA 102:1181–1186

  78. Vanneste S, Plazier M, der Loo E, de Heyning PV, Congedo M, De Ridder D (2010) The neural correlates of tinnitus-related distress. Neuroimage 52:470–480

  79. Vigneau M, Beaucousin V, Herve PY, Duffau H, Crivello F, Houde O, Mazoyer B, Tzourio-Mazoyer N (2006) Meta-analyzing left hemisphere language areas: phonology, semantics, and sentence processing. Neuroimage 30:1414–1432

  80. Werner S, Noppeney U (2010a) Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization. J Neurosci 30:2662–2675

  81. Werner S, Noppeney U (2010b) Superadditive responses in superior temporal sulcus predict audiovisual benefits in object categorization. Cereb Cortex 20:1829–1842

  82. Yun RJ, Krystal JH, Mathalon DH (2010) Working memory overload: fronto-limbic interactions and effects on subsequent working memory function. Brain Imaging Behav 4:96–108

Download references


The authors thank Dr. Yong-Hwi Ahn for his support on the manuscript. Also, the first author thanks Dr. DY Yoon for giving precious support to the study. This work was supported by Korean government (MOST) [Korea Science and Engineering Foundation (KOSEF) (no. 2012-0030102)].

Author information

Correspondence to Seung Ha Oh.

Additional information

Jae-Jin Song and Hyo-Jeong Lee contributed equally to this work.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Song, J., Lee, H., Kang, H. et al. Effects of congruent and incongruent visual cues on speech perception and brain activity in cochlear implant users. Brain Struct Funct 220, 1109–1125 (2015).

Download citation


  • Cochlear implant
  • Deafness
  • Positron emission tomography
  • Audiovisual
  • Plasticity