Skip to main content
SpringerLink
Log in
Book cover

Integrating Face and Voice in Person Perception pp 163–178Cite as

Neurophysiological Correlates of Face and Voice Integration

  • Chapter
  • First Online:

  • 1962 Accesses

Abstract

On meeting another individual for the first time during the course of a conversation we learn a lot unique and idiosyncratic facts about that person. If that meeting is socially or professionally significant, in the future, on crossing paths with that individual again, we can easily remember their details and the circumstances of the meeting. The person’s age, gender, ethnic or racial background, place of birth, current place of residence, professional and personal interests are all important pieces of data for building up a mental picture of that individual which we notice without going to too much effort. We might also notice the tone of their voice, the prosody with which they speak, and a foreign accent, if present. Interestingly, all of those details can be readily recalled on either seeing their face or just hearing their voice. Many of the chapters in this book are devoted to the importance of the face and voice in the formed percept we have of another individual—assigning an individual their own unique identity. Other chapters focus on how animals decode these important conspecific details. The questions asked in this chapter pertain less to brain mechanisms active in identifying specific individuals and their characteristics, but focus on issues relating to how non-verbal face and voice cues are integrated by the human brain. Early behavioral studies have noted how important non-verbal behaviors are for the interpretation of the actions of others, in terms of presenting important information relating to the social interaction (Campbell & Rushton, 1978; Mehrabian & Ferris, 1967). Yet, this remains a poorly studied area in social neuroscience and is a major focus for our laboratory.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  • Beauchamp, M. S., Nath, A. R., & Pasalar, S. (2010). fMRI-Guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect. Journal of Neuroscience, 30(7), 2414–2417.

    Article  PubMed  CAS  Google Scholar 

  • Bentin, S., Allison, T., Puce, A., Perez, A., & McCarthy, G. (1996). Electrophysiological studies of face perception in humans. Journal of Cognitive Neuroscience, 8, 551–565.

    Article  PubMed  Google Scholar 

  • Brefczynski-Lewis, J., Lowitszch, S., Parsons, M., Lemieux, S., & Puce, A. (2009). Audiovisual non-verbal dynamic faces elicit converging fMRI and ERP responses. Brain Topography, 21(3–4), 193–206.

    Article  PubMed  Google Scholar 

  • Busch, N. A., Dubois, J., & VanRullen, R. (2009). The phase of ongoing EEG oscillations predicts visual perception. Journal of Neuroscience, 29(24), 7869–7876.

    Article  PubMed  CAS  Google Scholar 

  • Calvert, G. A. (2001). Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex, 11(12), 1110–1123.

    Article  PubMed  CAS  Google Scholar 

  • Campbell, A., & Rushton, J. P. (1978). Bodily communication and personality. The British Journal of Social and Clinical Psychology, 17(1), 31–36.

    Article  PubMed  CAS  Google Scholar 

  • Carrick, O. K., Thompson, J. C., Epling, J. A., & Puce, A. (2007). It’s all in the eyes: Neural responses to socially significant gaze shifts. Neuroreport, 18(8), 763–766.

    Article  PubMed  Google Scholar 

  • Eggermont, J. J., & Ponton, C. W. (2002). The neurophysiology of auditory perception: From single units to evoked potentials. Audiology & Neuro-Otology, 7(2), 71–99.

    Article  Google Scholar 

  • Engel, A. K., & Fries, P. (2010). Beta-band oscillations–signalling the status quo? Current Opinion in Neurobiology, 20(2), 156–165.

    Article  PubMed  CAS  Google Scholar 

  • Engel, A. K., Fries, P., & Singer, W. (2001). Dynamic predictions: Oscillations and synchrony in top-down processing. Nature Reviews. Neuroscience, 2(10), 704–716.

    Article  PubMed  CAS  Google Scholar 

  • Foxe, J. J., Morocz, I. A., Murray, M. M., Higgins, B. A., Javitt, D. C., & Schroeder, C. E. (2000). Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping. Brain Research. Cognitive Brain Research, 10(1–2), 77–83.

    Article  PubMed  CAS  Google Scholar 

  • Ghazanfar, A. A., & Schroeder, C. E. (2006). Is neocortex essentially multisensory? Trends in Cognitive Sciences, 10(6), 278–285.

    Article  PubMed  Google Scholar 

  • Giard, M. H., & Peronnet, F. (1999). Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. Journal of Cognitive Neuroscience, 11(5), 473–490.

    Article  PubMed  CAS  Google Scholar 

  • Hagan, C. C., Woods, W., Johnson, S., Calder, A. J., Green, G. G., & Young, A. W. (2009). MEG demonstrates a supra-additive response to facial and vocal emotion in the right superior temporal sulcus. Proceedings of the National Academy of Sciences of the United States of America, 106(47), 20010–20015.

    PubMed  CAS  Google Scholar 

  • Herrmann, C. S., & Debener, S. (2008). Simultaneous recording of EEG and BOLD responses: A historical perspective. International Journal of Psychophysiology, 67(3), 161–168.

    Article  PubMed  Google Scholar 

  • Herrmann, C. S., Frund, I., & Lenz, D. (2010). Human gamma-band activity: A review on cognitive and behavioral correlates and network models. Neuroscience and Biobehavioral Reviews, 34(7), 981–992.

    Article  PubMed  Google Scholar 

  • Herrmann, C. S., Munk, M. H., & Engel, A. K. (2004). Cognitive functions of gamma-band activity: Memory match and utilization. Trends in Cognitive Sciences, 8(8), 347–355.

    Article  PubMed  Google Scholar 

  • Itier, R. J., & Batty, M. (2009). Neural bases of eye and gaze processing: The core of social cognition. Neuroscience and Biobehavioral Reviews, 33(6), 843–863.

    Article  PubMed  Google Scholar 

  • Kayser, C., Petkov, C. I., Augath, M., & Logothetis, N. K. (2007). Functional imaging reveals visual modulation of specific fields in auditory cortex. Journal of Neuroscience, 27(8), 1824–1835.

    Article  PubMed  CAS  Google Scholar 

  • Kim, S., & James, T. W. (2010). Enhanced effectiveness in visuo-haptic object-selective brain regions with increasing stimulus salience. Human Brain Mapping, 31(5), 678–693.

    Article  PubMed  CAS  Google Scholar 

  • Kutas, M., & Hillyard, S. A. (1980). Reading senseless sentences: Brain potentials reflect semantic incongruity. Science, 207(4427), 203–205.

    Article  PubMed  CAS  Google Scholar 

  • Lewis, J. W., Talkington, W. J., Puce, A., Engel, L. R., & Frum, C. (2011). Cortical networks representing object categories and high-level attributes of familiar real-world action sounds. Journal of Cognitive Neuroscience, 23(8), 2079–2101.

    Article  PubMed  Google Scholar 

  • Mazaheri, A., & Jensen, O. (2010). Rhythmic pulsing: Linking ongoing brain activity with evoked responses. Frontiers in Human Neuroscience, 4, 177.

    Article  PubMed  Google Scholar 

  • McCallum, W. C., Farmer, S. F., & Pocock, P. V. (1984). The effects of physical and semantic incongruities on auditory event-related potentials. Electroencephalography and Clinical Neurophysiology, 59(6), 477–488.

    Article  PubMed  CAS  Google Scholar 

  • McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746–748.

    Article  PubMed  CAS  Google Scholar 

  • Mehrabian, A., & Ferris, S. R. (1967). Inference of attitudes from nonverbal communication in two channels. Journal of Consulting Psychology, 31(3), 248–252.

    Article  PubMed  CAS  Google Scholar 

  • Meredith, M. A., & Stein, B. E. (1986). Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. Journal of Neurophysiology, 56(3), 640–662.

    PubMed  CAS  Google Scholar 

  • Munhall, K. G., Gribble, P., Sacco, L., & Ward, M. (1996). Temporal constraints on the McGurk effect. Perception & Psychophysics, 58(3), 351–362.

    Article  CAS  Google Scholar 

  • Nath, A. R., & Beauchamp, M. S. (2011a). Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech. Journal of Neuroscience, 31(5), 1704–1714.

    Article  PubMed  CAS  Google Scholar 

  • Nath, A. R., & Beauchamp, M. S. (2011b). A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion. NeuroImage, 59(1), 781–787.

    Article  PubMed  Google Scholar 

  • Naue, N., Rach, S., Struber, D., Huster, R. J., Zaehle, T., Korner, U., et al. (2011). Auditory event-related response in visual cortex modulates subsequent visual responses in humans. Journal of Neuroscience, 31(21), 7729–7736.

    Article  PubMed  CAS  Google Scholar 

  • Puce, A., Epling, J. A., Thompson, J. C., & Carrick, O. K. (2007). Neural responses elicited to face motion and vocalization pairings. Neuropsychologia, 45(1), 93–106.

    Article  PubMed  Google Scholar 

  • Puce, A., & Perrett, D. (2003). Electrophysiology and brain imaging of biological motion. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 358(1431), 435–445.

    Article  PubMed  Google Scholar 

  • Puce, A., Smith, A., & Allison, T. (2000). ERPs evoked by viewing facial movements. Cognitive neuropsychology, 17, 221–239.

    Article  PubMed  CAS  Google Scholar 

  • Reale, R. A., Calvert, G. A., Thesen, T., Jenison, R. L., Kawasaki, H., Oya, H., et al. (2007). Auditory-visual processing represented in the human superior temporal gyrus. Neuroscience, 145(1), 162–184.

    Article  PubMed  CAS  Google Scholar 

  • Rossion, B., & Jacques, C. (2008). Does physical interstimulus variance account for early electrophysiological face sensitive responses in the human brain? Ten lessons on the N170. NeuroImage, 39(4), 1959–1979.

    Article  PubMed  Google Scholar 

  • Sabbagh, M. A., Moulson, M. C., & Harkness, K. L. (2004). Neural correlates of mental state decoding in human adults: An event-related potential study. Journal of Cognitive Neuroscience, 16(3), 415–426.

    Article  PubMed  Google Scholar 

  • Sams, M., Aulanko, R., Hamalainen, M., Hari, R., Lounasmaa, O. V., Lu, S. T., et al. (1991). Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters, 127(1), 141–145.

    Article  PubMed  CAS  Google Scholar 

  • Stein, B. E., Stanford, T. R., Ramachandran, R., Perrault, T. J., Jr., & Rowland, B. A. (2009). Challenges in quantifying multisensory integration: Alternative criteria, models, and inverse effectiveness. Experimental Brain Research, 198(2–3), 113–126.

    Article  Google Scholar 

  • Stevenson, R. A., Geoghegan, M. L., & James, T. W. (2007). Superadditive BOLD activation in superior temporal sulcus with threshold non-speech objects. Experimental Brain Research, 179(1), 85–95.

    Article  Google Scholar 

  • Stevenson, R. A., & James, T. W. (2009). Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition. NeuroImage, 44(3), 1210–1223.

    Article  PubMed  Google Scholar 

  • Stevenson, R. A., Kim, S., & James, T. W. (2009). An additive-factors design to disambiguate neuronal and areal convergence: Measuring multisensory interactions between audio, visual, and haptic sensory streams using fMRI. Experimental Brain Research, 198(2–3), 183–194.

    Article  Google Scholar 

  • Wheaton, K. J., Pipingas, A., Silberstein, R. B., & Puce, A. (2001). Human neural responses elicited to observing the actions of others. Visual Neuroscience, 18(3), 401–406.

    Article  PubMed  CAS  Google Scholar 

  • Wright, T. M., Pelphrey, K. A., Allison, T., McKeown, M. J., & McCarthy, G. (2003). Polysensory interactions along lateral temporal regions evoked by audiovisual speech. Cerebral Cortex, 13(10), 1034–1043.

    Article  PubMed  Google Scholar 

  • Young, C. K., & Eggermont, J. J. (2009). Coupling of mesoscopic brain oscillations: Recent advances in analytical and theoretical perspectives. Progress in Neurobiology, 89(1), 61–78.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychological and Brain Sciences, Indiana University, 1101 E 10th St., Bloomington, IN, 47401, USA

    Aina Puce Ph.D.

Authors
  1. Aina Puce Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aina Puce Ph.D. .

Editor information

Editors and Affiliations

  1. Dept. Psychology, Voice Neurocognition Lab., University of Glasgow, Glasgow, Glasgow, G12 8QB, United Kingdom

    Pascal Belin

  2. Brugmann University Hospital, Brussels, B-1020, Belgium

    Salvatore Campanella

  3. University of Tubingen, Tubingen, 72074, Germany

    Thomas Ethofer

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer Science+Business Media New York

About this chapter

Cite this chapter

Puce, A. (2013). Neurophysiological Correlates of Face and Voice Integration. In: Belin, P., Campanella, S., Ethofer, T. (eds) Integrating Face and Voice in Person Perception. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-3585-3_9

Download citation

Publish with us

Policies and ethics

Search

Navigation