Cross-Modal Integration of Identity and Gender Information Through Faces and Voices Involves a Similar Cortical Network

  • Salvatore Campanella
  • Frédéric Joassin


We investigate the cerebral cross-modal interactions between human faces and voices involved during gender and identity categorization in two separate functional magnetic resonance imaging (fMRI) studies. In each of these experiments, participants were scanned in four runs that contained three conditions consisting in the presentation of faces, voices, or congruent face–voice pairs. The task consisted in categorizing each trial (visual, auditory, or associations) according to its gender or identity. The subtraction between the bimodal condition and the sum of the unimodal ones, as well as psychophysiological interaction analyses (PPI), were performed. Main results suggest that the cross-modal auditory–visual categorization of human gender and identity is sustained by a network of highly similar cerebral regions. This network included several regions such as the unimodal visual and auditory regions processing the perceived faces and voices and inter-connected via a subcortical relay located in the striatum, the left superior parietal gyrus, part of a larger parieto-motor network dispatching the attentional resources to the visual and auditory modalities, and the right inferior frontal gyrus sustaining the integration of the semantically congruent information into a coherent multimodal representation. Therefore, we suggest that cross-modal processing of human stimuli requires the activation of a network of cortical regions, including both unimodal visual and auditory regions and supramodal parietal and frontal regions involved in the integration of both faces and voices and in the cross-modal attentional processes.


Inferior Frontal Gyrus Fusiform Face Area Bimodal Condition Calcarine Sulcus Auditory Region 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Beauchamp MS (2005) Statistical criteria in fMRI studies of multisensory integration. Neuroinformatics 3:93–113PubMedCrossRefGoogle Scholar
  2. Beauchemin M et al (2006) Electrophysiological markers of voice familiarity. European Journal of Neuroscience 23:3081–3086PubMedCrossRefGoogle Scholar
  3. Belin P, Zatorre RJ, Lafaille P, Ahad P, Pike B (2000) Voice-selective areas in human auditory cortex. Nature 403:309–312PubMedCrossRefGoogle Scholar
  4. Bernstein LE, Auer ET Jr, Wagner M, Ponton CW (2008) Spatiotemporal dynamics of audiovisual speech processing. Neuroimage 39:423–435PubMedCrossRefGoogle Scholar
  5. Bodamer J (1947) Die Prosop-Agnosia (Die Agnosie des Physionomeerkennens). Archives fur Psychiatrie und Nervenkrankenheiten 179:6–33CrossRefGoogle Scholar
  6. Bruce V, Young A (1986) Understanding face recognition. British Journal of Psychology 77(3):305–327PubMedCrossRefGoogle Scholar
  7. Burton AM, Bruce V, Johnston RA (1990) Understanding face recognition with an interactive model. British Journal of Psychology 81:361–380PubMedCrossRefGoogle Scholar
  8. Bushara KO, Hanakawa T, Immish I, Toma K, Kansaku K, Hallett M (2003) Neural correlates of cross-modal binding. Nature Neuroscience 6(2):190–195PubMedCrossRefGoogle Scholar
  9. Bushara KO, Weeks RA, Ishii K, Catalan MJ, Tian B, Rauschecker JP et al (1999) Modality-specific frontal and parietal areas for auditory and visual spatial localization in humans. Nature Neuroscience 2:759–766PubMedCrossRefGoogle Scholar
  10. Calvert GA (2001) Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex 11:1110–1123PubMedCrossRefGoogle Scholar
  11. Calvert GA, Campbell R, Brammer MJ (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in human heteromodal cortex. Current Biology 10:649–657PubMedCrossRefGoogle Scholar
  12. Campanella S, Belin P (2007) Integrating face and voice in person perception. Trends in Cognitive Sciences 11(12):535–543PubMedCrossRefGoogle Scholar
  13. Campanella S, Hanoteau C, Depy D, Rossion B, Bruyer R, Crommelinck M (2000) Right N170 modulation in a face discrimination task: An account for categorical perception of familiar faces. Psychophysiology 37:796–806PubMedCrossRefGoogle Scholar
  14. Campanella S, Joassin F, Rossion B, De Volder AG, Bruyer R, Crommelinck M (2001) Associations of the distinct visual representations of faces and names: A PET activation study. Neuroimage 14:873–882PubMedCrossRefGoogle Scholar
  15. Driver J, Spence C (2000) Multisensory perception: Beyond modularity and convergence. Current Biology 10:731–735CrossRefGoogle Scholar
  16. Ganel T, Goshen-Gottstein Y (2002) Perceptual integrity of sex and identity of faces: Further evidence for the single-route hypothesis. Journal of Experimental Psychology Human Perception and Performance 28:854–867PubMedCrossRefGoogle Scholar
  17. Garrido L, Eisner F, McGettigan C, Stewart L, Sauter D, Hanley JR, Schweinberger SR, Warren JD, Duchaine B (2009) Developmental phonagnosia: A selective deficit of vocal identity recognition. Neuropsychologia 47(1):123–131PubMedCrossRefGoogle Scholar
  18. Gauthier I, Skudlarski P, Gore JC, Anderson AW (2000) Expertise for cars and birds recruits brain areas involved in face recognition. Nature Neuroscience 3:191–197PubMedCrossRefGoogle Scholar
  19. Gonzalo D, Shallice T, Dolan R (2000) Time-dependent changes in learning audiovisual associations: A single-trial fMRI study. Neuroimage 11:243–255PubMedCrossRefGoogle Scholar
  20. Haruno M, Kawato M (2006) Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus–action–reward association learning. Neural Networks 19:1242–1254PubMedCrossRefGoogle Scholar
  21. Haxby JV et al (2000) The distributed human neural system for face perception. Trends in Cognitive Sciences 4:223–233PubMedCrossRefGoogle Scholar
  22. Hesling I, Clément S, Bordessoules M, Allard M (2005) Cerebral mechanisms of prosodic integration: Evidence from connected speech. Neuroimage 24:937–947PubMedCrossRefGoogle Scholar
  23. Joassin F, Campanella S, Debatisse D, Guérit JM, Bruyer R, Crommelinck M (2004a) The electrophysiological correlates sustaining the retrieval of face–name associations: An ERP study. Psychophysiology 41:625–635PubMedCrossRefGoogle Scholar
  24. Joassin F, Maurage P, Bruyer R, Crommelinck M, Campanella S (2004b) When audition alters vision: An event-related potential study of the cross-modal interactions between faces and voices. Neuroscience Letters 369:132–137PubMedCrossRefGoogle Scholar
  25. Joassin F, Maurage P, Campanella S (2011a) The neural network sustaining the crossmodal processing of human gender from faces and voices: An fMRI study. Neuroimage 54(2):1654–1661PubMedCrossRefGoogle Scholar
  26. Joassin F, Meert G, Campanella S, Bruyer R (2007) The associative processes involved in faces–proper names vs. objects–common names binding: A comparative ERP study. Biological Psychology 75(3):286–299PubMedCrossRefGoogle Scholar
  27. Joassin F, Pesenti M, Maurage P, Verreckt E, Bruyer R, Campanella S (2011b) Cross-modal interactions between human faces and voices involved in person recognition. Cortex 47:367–376PubMedCrossRefGoogle Scholar
  28. Kanwisher N, McDermott J, Chun MM (1997) The fusiform face area: A module in human extrastriate cortex specialized for face perception. The Journal of Neuroscience 9:462–475Google Scholar
  29. Kerlin JR, Shahin AJ, Miller LM (2010) Attentional grain control of ongoing cortical speech representations in a “cocktail party”. The Journal of Neuroscience 30(2):620–628PubMedCrossRefGoogle Scholar
  30. Laurienti PJ, Perrault TJ, Stanford TR, Wallace MT, Stein BE (2005) On the use of superadditivity as a metric for characterizing multisensory integration in functional neuroimaging studies. Experimental Brain Research 166:289–297CrossRefGoogle Scholar
  31. Leube DT, Erb M, Grodd W, Bartels M, Kircher TTJ (2001) Differential activation in parahippocampal and prefrontal cortex during word and face encoding tasks. Neuroreport 12(12):2773–2777PubMedCrossRefGoogle Scholar
  32. Magnée M, de Gelder B, van Engeland H, Kemner C (2008) Atypical processing of fearful face–voice pairs in Pervasive Developmental Disorder: An ERP study. Clinical Neurophysiology 119:2004–2010CrossRefGoogle Scholar
  33. Maurage P, Campanella S, Philippot P, Pham T, Joassin F (2007) The crossmodal facilitation effect is disrupted in alcoholism: A study with emotional stimuli. Alcohol and Alcoholism 42:552–559PubMedCrossRefGoogle Scholar
  34. Maurage P, Philippot P, Joassin F, Alonso Prieto E, Palmero Soler E, Zanow F, Campanella S (2008) The auditory–visual integration of anger is disrupted in alcoholism: An ERP study. Journal of Psychiatry and Neuroscience 33(2):111–122PubMedGoogle Scholar
  35. McNamara A, Buccino G, Menz MM, Gläsher J, Wolbers T, Baumgärtner A, Binkofski F (2008) Neural dynamics of learning sound–action associations. PLoS One 3(12):1–10CrossRefGoogle Scholar
  36. Melillo R, Leisman G (2009) Autistic spectrum disorders as functional disconnection syndrome. Reviews in the Neurosciences 20(2):111–131PubMedCrossRefGoogle Scholar
  37. Monk C, Weng SJ, Wiggins J, Kurapati N, Louro H, Carrasco M, Maslowsky J, Risi S, Lord C (2010) Neural circuitry of emotional face processing in autism spectrum disorders. Journal of Psychiatry and Neuroscience 35(2):105–114PubMedCrossRefGoogle Scholar
  38. Puce A, Allison T, Gore JC, McCarthy G (1995) Face-sensitive regions in human extrastriate cortex studied by functional MRI. Journal of Neurophysiology 74(3):1192–1199PubMedGoogle Scholar
  39. Rama P, Courtney SM (2005) Functional topography of working memory for face or voice identity. NeuroImage 24:224–234PubMedCrossRefGoogle Scholar
  40. Rolls ET (2000) The orbitofrontal cortex and reward. Cerebral Cortex 10:284–294PubMedCrossRefGoogle Scholar
  41. Schweinberger SR, Robertson D, Kaufmann JM (2007) Hearing facial identities. The Quarterly Journal of Experimental Psychology 60(10):1446–1456PubMedCrossRefGoogle Scholar
  42. Seiferth N, Pauly K, Kellermann T, Shah N, Ott G, Herpertz-Dahlmann B, Kircher T, Schneider F, Habel U (2009) Neuronal correlates of facial emotion discrimination in early onset schizophrenia. Neuropsychopharmacology 34:477–487PubMedCrossRefGoogle Scholar
  43. Senkowski D, Schneider TR, Foxe JJ, Engel AK (2008) Crossmodal binding through neural coherence: Implications for multisensory processing. Trends in Cognitive Sciences 31(8):401–409Google Scholar
  44. Sheffert SM, Olson E (2004) Audiovisual speech facilitates voice learning. Perception & Psychophysics 66(2):352–362CrossRefGoogle Scholar
  45. Shomstein S, Yantis S (2004) Control of attention shifts between vision and audition in human cortex. The Journal of Neuroscience 24(47):10702–10706PubMedCrossRefGoogle Scholar
  46. Smith EL, Grabowecky M, Suzuki S (2007) Auditory–visual crossmodal integration in perception of face gender. Current Biology 17:1680–1685PubMedCrossRefGoogle Scholar
  47. Steeves J, Dricot L, Goltz HC, Sorger B, Peters J, Milner AD, Goodale MA, Goebel R, Rossion B (2009) Abnormal face identity coding in the middle fusiform gyrus of two brain-damaged prosopagnosic patients. Neuropsychologia 47(12):2584–2592PubMedCrossRefGoogle Scholar
  48. Van Lancker DR, Canter GJ (1982) Impairment of voice and face recognition in patients with hemispheric damage. Brain and Cognition 1(2):185–195PubMedCrossRefGoogle Scholar
  49. Von Kriegstein K, Giraud AL (2006) Implicit multisensory associations influence voice recognition. PLoS Biology 4:e326CrossRefGoogle Scholar
  50. Von Kriegstein K, Kleinschmidt A, Sterzer P, Giraud AL (2005) Interaction of face and voice areas during speaker recognition. Journal of Cognitive Neuroscience 17(3):367–376CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Laboratory of Psychological MedicineFree University of BrusselsBrusselsBelgium
  2. 2.CHU Brugmann, Psychiatry Department (EEG)The Belgian Fund for Scientific Research (FNRS)BrusselsBelgium
  3. 3.Clinique de la mémoireCHU Ambroise ParéMonsBelgium

Personalised recommendations