Skip to main content
SpringerLink
Log in
Book cover

Multisensory Processes pp 135–159Cite as

Audiovisual Integration in the Primate Prefrontal Cortex

  • Chapter
  • First Online:

Part of the book series: Springer Handbook of Auditory Research ((SHAR,volume 68))

Abstract

Language and communication rely on the combination of visual and auditory information. The frontal lobes have long been known to support communication processing and receive a wide array of sensory inputs from many brain regions. The ventral frontal lobe, specifically the ventrolateral prefrontal cortex (VLPFC), receives afferents from auditory and visual association cortices. Recordings in nonhuman primates indicate that single neurons in the VLPFC integrate face and vocal stimuli. These multisensory neurons show enhanced and suppressed responses to face and vocalization combinations. Furthermore, studies indicate that ventral prefrontal neurons are affected by the semantic congruence of face-vocalization pairs and by the temporal synchrony of dynamic face-vocalization stimuli. Recordings of VLPFC neurons in macaques performing working memory tasks demonstrate that neurons are context dependent and respond to specific combinations of face and vocal stimuli during memory and decision tasks. Finally, transient inactivation of the prefrontal cortex impairs working memory for face-vocalization stimuli. Thus, results from several studies indicate that the primate prefrontal cortex plays an important role in the processing and integration of face and vocalization information that is essential during communication and social cognition.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD   159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  • Adachi, I., & Hampton, R. (2011). Rhesus monkeys see who they hear: Spontaneous cross-modal memory for familiar conspecifics. PLoS One, 6(8), e23345.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Asaad, W. F., Rainer, G., & Miller, E. K. (2000). Task-specific neural activity in the primate prefrontal cortex. Journal of Neurophysiology, 84, 451–459.

    CAS  PubMed  Google Scholar 

  • Averbeck, B. B., & Romanski, L. M. (2006). Probabilistic encoding of vocalizations in macaque ventral lateral prefrontal cortex. The Journal of Neuroscience, 26, 11023–11033.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Barbas, H. (1992). Architecture and cortical connections of the prefrontal cortex in the rhesus monkey. Advances in Neurology, 57, 91–115.

    CAS  PubMed  Google Scholar 

  • Barraclough, N. E., Xiao, D., Baker, C. I., Oram, M. W., & Perrett, D. I. (2005). Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. Journal of Cognitive Neuroscience, 17(3), 377–391.

    PubMed  Google Scholar 

  • Baum, S. H., Stevenson, R. A., & Wallace, M. T. (2015). Behavioral, perceptual, and neural alterations in sensory and multisensory function in autism spectrum disorder. Progress in Neurobiology, 134, 140–160.

    PubMed  PubMed Central  Google Scholar 

  • Baxter, M. G., Gaffan, D., Kyriazis, D. A., & Mitchell, A. S. (2009). Ventrolateral prefrontal cortex is required for performance of a strategy implementation task but not reinforcer devaluation effects in rhesus monkeys. European Journal of Neuroscience, 29(10), 2049–2059.

    PubMed  Google Scholar 

  • Bodner, M., Kroger, J., & Fuster, J. M. (1996). Auditory memory cells in dorsolateral prefrontal cortex. Neuroreport, 7, 1905–1908.

    CAS  PubMed  Google Scholar 

  • Brody, C. D., Hernandez, A., Zainos, A., & Romo, R. (2003). Timing and neural encoding of somatosensory parametric working memory in macaque prefrontal cortex. Cerebral Cortex, 13(11), 1196–1207.

    PubMed  Google Scholar 

  • Bussey, T. J., Wise, S. P., & Murray, E. A. (2002). Interaction of ventral and orbital prefrontal cortex with inferotemporal cortex in conditional visuomotor learning. Behavioral Neuroscience, 116(4), 703–715.

    PubMed  Google Scholar 

  • Cohen, Y. E., Russ, B. E., Davis, S. J., Baker, A. E., Ackelson, A. L., & Nitecki, R. (2009). A functional role for the ventrolateral prefrontal cortex in non-spatial auditory cognition. Proceedings of the National Academy of Sciences of the United States of America, 106(47), 20045–20050.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Crottaz-Herbette, S., Anagnoson, R. T., & Menon, V. (2004). Modality effects in verbal working memory: Differential prefrontal and parietal responses to auditory and visual stimuli. NeuroImage, 21(1), 340–351.

    CAS  PubMed  Google Scholar 

  • Diehl, M. M., & Romanski, L. M. (2014). Responses of prefrontal multisensory neurons to mismatching faces and vocalizations. The Journal of Neuroscience, 34(34), 11233–11243.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Eliades, S. J., & Wang, X. (2012). Neural correlates of the Lombard effect in primate auditory cortex. The Journal of Neuroscience, 32(31), 10737–10748.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Fuster, J. M., Bodner, M., & Kroger, J. K. (2000). Cross-modal and cross-temporal association in neurons of frontal cortex. Nature, 405(6784), 347–351.

    CAS  PubMed  Google Scholar 

  • Ghazanfar, A. A., Maier, J. X., Hoffman, K. L., & Logothetis, N. K. (2005). Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. The Journal of Neuroscience, 25(20), 5004–5012.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Ghazanfar, A. A., Morrill, R. J., & Kayser, C. (2013). Monkeys are perceptually tuned to facial expressions that exhibit a theta-like speech rhythm. Proceedings of the National Academy of Sciences of the United States of America, 110(5), 1959–1963.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Gifford, G. W., III, Maclean, K. A., Hauser, M. D., & Cohen, Y. E. (2005). The neurophysiology of functionally meaningful categories: Macaque ventrolateral prefrontal cortex plays a critical role in spontaneous categorization of species-specific vocalizations. Journal of Cognitive Neuroscience, 17, 1471–1482.

    PubMed  Google Scholar 

  • Goldman-Rakic, P. S. (1987). Circuitry of primate prefrontal cortex and regulation of behavior by representational memory. In F. Plum (Ed.), Handbook of physiology, Vol. V: Higher functions of the brain: The nervous system, Sect. 1 (pp. 373–418). Bethesda: American Physiological Society.

    Google Scholar 

  • Gross, C. G., & Weiskrantz, L. (1962). Evidence for dissociation of impairment on auditory discrimination and delayed response following lateral frontal lesions in monkeys. Experimental Neurology, 5, 453–476.

    CAS  PubMed  Google Scholar 

  • Habbershon, H. M., Ahmed, S. Z., & Cohen, Y. E. (2013). Rhesus macaques recognize unique multimodal face-voice relations of familiar individuals and not of unfamiliar ones. Brain, Behavior and Evolution, 81(4), 219–225.

    PubMed  PubMed Central  Google Scholar 

  • Hackett, T. A., Stepniewska, I., & Kaas, J. H. (1998). Subdivisions of auditory cortex and ipsilateral cortical connections of the parabelt auditory cortex in macaque monkeys. Journal of Comparative Neurology, 394, 475–495.

    CAS  PubMed  Google Scholar 

  • Hackett, T. A., Stepniewska, I., & Kaas, J. H. (1999). Prefrontal connections of the parabelt auditory cortex in macaque monkeys. Brain Research, 817, 45–58.

    CAS  PubMed  Google Scholar 

  • Hackett, T. A., de la Mothe, L. A., Camalier, C. R., Falchier, A., Lakatos, P., Kajikawa, Y., & Schroeder, C. E. (2014). Feedforward and feedback projections of caudal belt and parabelt areas of auditory cortex: Refining the hierarchical model. Frontiers in Neuroscience, 8, 72.

    PubMed  PubMed Central  Google Scholar 

  • Hage, S. R., & Nieder, A. (2013). Single neurons in monkey prefrontal cortex encode volitional initiation of vocalizations. Nature Communications, 4, 2409.

    PubMed  Google Scholar 

  • Hage, S. R., & Nieder, A. (2015). Audio-vocal interaction in single neurons of the monkey ventrolateral prefrontal cortex. The Journal of Neuroscience, 35, 7030–7040.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Hershenson, M. (1962). Reaction time as a measure of intersensory facilitation. Journal of Experimental Psychology, 63, 289–293.

    CAS  PubMed  Google Scholar 

  • Hwang, J., & Romanski, L. M. (2015). Prefrontal neuronal responses during audiovisual nmemonic processing. The Journal of Neuroscience, 35, 960–971.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Iversen, S. D., & Mishkin, M. (1973). Comparison of superior temporal and inferior prefrontal lesions on auditory and non-auditory tasks in rhesus monkeys. Brain Research, 55(2), 355–367.

    CAS  PubMed  Google Scholar 

  • Jones, E. G., Dell’Anna, M. E., Molinari, M., Rausell, E., & Hashikawa, T. (1995). Subdivisions of macaque monkey auditory cortex revealed by calcium-binding protein immunoreactivity. Journal of Comparative Neurology, 362, 153–170.

    CAS  PubMed  Google Scholar 

  • Jurgens, U. (2009). The neural control of vocalization in mammals: A review. Journal of Voice, 23(1), 1–10.

    CAS  PubMed  Google Scholar 

  • Kosaki, H., Hashikawa, T., He, J., & Jones, E. G. (1997). Tonotopic organization of auditory cortical fields delineated by parvalbumin immunoreactivity in macaque monkeys. Journal of Comparative Neurology, 386(2), 304–316.

    CAS  PubMed  Google Scholar 

  • Laurienti, P. J., Wallace, M. T., Maldjian, J. A., Susi, C. M., Stein, B. E., & Burdette, J. H. (2003). Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Human Brain Mapping, 19(4), 213–223.

    PubMed  Google Scholar 

  • McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748.

    CAS  PubMed  Google Scholar 

  • Medalla, M., & Barbas, H. (2014). Specialized prefrontal “auditory fields”: Organization of primate prefrontal-temporal pathways. Frontiers in Neuroscience, 8, 77.

    PubMed  PubMed Central  Google Scholar 

  • Meredith, M. A., & Stein, B. E. (1986). Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. Journal of Neurophysiology, 56(3), 640–662.

    CAS  PubMed  Google Scholar 

  • Miller, L. M., & D’Esposito, M. (2005). Perceptual fusion and stimulus coincidence in the cross-modal integration of speech. The Journal of Neuroscience, 25, 5884–5893.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Miller, E. K., Li, L., & Desimone, R. (1991). A neural mechanism for working and recognition memory in inferior temporal cortex. Science, 254(5036), 1377–1379.

    CAS  PubMed  Google Scholar 

  • Miller, E. K., Erickson, C. A., & Desimone, R. (1996). Neural mechanisms of visual working memory in prefrontal cortex of the macaque. The Journal of Neuroscience, 16, 5154–5167.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Mishkin, M., & Manning, F. J. (1978). Non-spatial memory after selective prefrontal lesions in monkeys. Brain Research, 143, 313–323.

    CAS  PubMed  Google Scholar 

  • Molholm, S., Ritter, W., Javitt, D. C., & Foxe, J. J. (2004). Multisensory visual-auditory object recognition in humans: A high-density electrical mapping study. Cerebral Cortex, 14(4), 452–465.

    PubMed  Google Scholar 

  • O’Scalaidhe, S. P., Wilson, F. A., & Goldman-Rakic, P. S. (1997). Areal segregation of face-processing neurons in prefrontal cortex. Science, 278, 1135–1138.

    Google Scholar 

  • O’Scalaidhe, S. P. O., Wilson, F. A. W., & Goldman-Rakic, P. G. R. (1999). Face-selective neurons during passive viewing and working memory performance of rhesus monkeys: Evidence for intrinsic specialization of neuronal coding. Cerebral Cortex, 9, 459–475.

    Google Scholar 

  • Passingham, R. (1975). Delayed matching after selective prefrontal lesions in monkeys (Macaca mulatta). Brain Research, 92, 89–102.

    CAS  PubMed  Google Scholar 

  • Passingham, R. E., Toni, I., & Rushworth, M. F. (2000). Specialisation within the prefrontal cortex: The ventral prefrontal cortex and associative learning. Experimental Brain Research, 133(1), 103–113.

    CAS  PubMed  Google Scholar 

  • Petrides, M., & Pandya, D. N. (2002). Comparative cytoarchitectonic analysis of the human and the macaque ventrolateral prefrontal cortex and corticocortical connection patterns in the monkey. European Journal of Neuroscience, 16(2), 291–310.

    CAS  PubMed  Google Scholar 

  • Plakke, B., & Romanski, L. M. (2014). Auditory connections and functions of prefrontal cortex. Frontiers in Neuroscience, 8, 199.

    PubMed  PubMed Central  Google Scholar 

  • Plakke, B., Diltz, M. D., & Romanski, L. M. (2013a). Coding of vocalizations by single neurons in ventrolateral prefrontal cortex. Hearing Research, 305, 135–143.

    PubMed  PubMed Central  Google Scholar 

  • Plakke, B., Ng, C. W., & Poremba, A. (2013b). Neural correlates of auditory recognition memory in primate lateral prefrontal cortex. Neuroscience, 244, 62–76.

    CAS  PubMed  Google Scholar 

  • Plakke, B., Hwang, J., & Romanski, L. M. (2015). Inactivation of primate prefrontal cortex impairs auditory and audiovisual working memory. The Journal of Neuroscience, 35, 9666–9675.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Posner, M. I., Nissen, M. J., & Klein, R. M. (1976). Visual dominance: An information-processing account of its origins and significance. Psychological Review, 83(2), 157–171.

    CAS  PubMed  Google Scholar 

  • Price, C. J. (2012). A review and synthesis of the first 20 years of PET and fMRI studies of heard speech, spoken language and reading. NeuroImage, 62(2), 816–847.

    PubMed  PubMed Central  Google Scholar 

  • Rama, P., & Courtney, S. M. (2005). Functional topography of working memory for face or voice identity. NeuroImage, 24(1), 224–234.

    PubMed  Google Scholar 

  • Rama, P., Poremba, A., Sala, J. B., Yee, L., Malloy, M., Mishkin, M., & Courtney, S. M. (2004). Dissociable functional cortical topographies for working memory maintenance of voice identity and location. Cerebral Cortex, 14, 768–780.

    PubMed  Google Scholar 

  • Rauschecker, J. P., Tian, B., & Hauser, M. (1995). Processing of complex sounds in the macaque nonprimary auditory cortex. Science, 268(5207), 111–114.

    CAS  PubMed  Google Scholar 

  • Romanski, L. M. (2012). Integration of faces and vocalizations in ventral prefrontal cortex: Implications for the evolution of audiovisual speech. Proceedings of the National Academy of Sciences of the United States of America, 109(Suppl 1), 10717–10724.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Romanski, L. M., & Averbeck, B. B. (2009). The primate cortical auditory system and neural representation of conspecific vocalizations. Annual Review of Neuroscience, 32, 315–346.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Romanski, L. M., & Diehl, M. M. (2011). Neurons responsive to face-view in the primate ventrolateral prefrontal cortex. Neuroscience, 189, 223–235.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Romanski, L. M., & Goldman-Rakic, P. S. (2002). An auditory domain in primate prefrontal cortex. Nature Neuroscience, 5, 15–16.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Romanski, L. M., & Hwang, J. (2012). Timing of audiovisual inputs to the prefrontal cortex and multisensory integration. Neuroscience, 214, 36–48.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Romanski, L. M., Bates, J. F., & Goldman-Rakic, P. S. (1999a). Auditory belt and parabelt projections to the prefrontal cortex in the rhesus monkey. Journal of Comparative Neurology, 403, 141–157.

    CAS  PubMed  Google Scholar 

  • Romanski, L. M., Tian, B., Fritz, J., Mishkin, M., Goldman-Rakic, P. S., & Rauschecker, J. P. (1999b). Dual streams of auditory afferents target multiple domains in the primate prefrontal cortex. Nature Neuroscience, 2(12), 1131–1136.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Romanski, L. M., Averbeck, B. B., & Diltz, M. (2005). Neural representation of vocalizations in the primate ventrolateral prefrontal cortex. Journal of Neurophysiology, 93(2), 734–747.

    PubMed  Google Scholar 

  • Rushworth, M. F., Nixon, P. D., Eacott, M. J., & Passingham, R. E. (1997). Ventral prefrontal cortex is not essential for working memory. The Journal of Neuroscience, 17(12), 4829–4838.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Russ, B. E., Orr, L. E., & Cohen, Y. E. (2008). Prefrontal neurons predict choices during an auditory same-different task. Current Biology, 18(19), 1483–1488.

    CAS  PubMed  Google Scholar 

  • Rygula, R., Walker, S. C., Clarke, H. F., Robbins, T. W., & Roberts, A. C. (2010). Differential contributions of the primate ventrolateral prefrontal and orbitofrontal cortex to serial reversal learning. The Journal of Neuroscience, 30(43), 14552–14559.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Schumacher, E. H., Lauber, E., Awh, E., Jonides, J., Smith, E. E., & Koeppe, R. A. (1996). PET evidence for an amodal verbal working memory system. NeuroImage, 3(2), 79–88.

    CAS  PubMed  Google Scholar 

  • Strand, F., Forssberg, H., Klingberg, T., & Norrelgen, F. (2008). Phonological working memory with auditory presentation of pseudo-words—An event related fMRI study. Brain Research, 1212, 48–54.

    CAS  PubMed  Google Scholar 

  • Sugihara, T., Diltz, M. D., Averbeck, B. B., & Romanski, L. M. (2006). Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex. The Journal of Neuroscience, 26, 11138–11147.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Tanila, H., Carlson, S., Linnankoski, I., Lindroos, F., & Kahila, H. (1992). Functional properties of dorsolateral prefrontal cortical neurons in awake monkey. Behavioral Brain Research, 47, 169–180.

    CAS  Google Scholar 

  • Tanila, H., Carlson, S., Linnankoski, I., & Kahila, H. (1993). Regional distribution of functions in dorsolateral prefrontal cortex of the monkey. Behavioral Brain Research, 53, 63–71.

    CAS  Google Scholar 

  • Vaadia, E., Benson, D. A., Hienz, R. D., & Goldstein, M. H., Jr. (1986). Unit study of monkey frontal cortex: Active localization of auditory and of visual stimuli. Journal of Neurophysiology, 56, 934–952.

    CAS  PubMed  Google Scholar 

  • Watson, R., Latinus, M., Noguchi, T., Garrod, O., Crabbe, F., & Belin, P. (2013). Dissociating task difficulty from incongruence in face-voice emotion integration. Frontiers in Human Neuroscience, 7, 744.

    PubMed  PubMed Central  Google Scholar 

  • West, R. A., & Larson, C. R. (1995). Neurons of the anterior mesial cortex related to faciovocal activity in the awake monkey. Journal of Neurophysiology, 74(5), 1856–1869.

    CAS  PubMed  Google Scholar 

  • Wilson, F. A., O’Scalaidhe, S. P., & Goldman-Rakic, P. S. (1993). Dissociation of object and spatial processing domains in primate prefrontal cortex. Science, 260, 1955–1958.

    CAS  PubMed  Google Scholar 

  • Yu, J. C., Chang, T. Y., & Yang, C. T. (2014). Individual differences in working memory capacity and workload capacity. Frontiers in Psychology, 5, 1465.

    PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgments

The authors acknowledge the hard work and assistance of Theodore Lincoln and Skye Kevelson in writing, editing, and figure creation. We also thank Adrian K. C. Lee, Mark Wallace, Allison Coffin, and Arthur N. Popper for all their helpful comments and suggestions in editing this chapter.

Compliance with Ethics Requirements

Lizabeth M. Romanski declares that she has no conflict of interest.

Bethany Plakke declares that she has no conflict of interest.

Author information

Authors and Affiliations

  1. Department of Psychological Sciences, Kansas State University, Manhattan, KS, USA

    Bethany Plakke

  2. Department of Neuroscience, University of Rochester School of Medicine, Rochester, NY, USA

    Lizabeth M. Romanski

Authors
  1. Bethany Plakke
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lizabeth M. Romanski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lizabeth M. Romanski .

Editor information

Editors and Affiliations

  1. Department of Speech and Hearing Sciences, Institute for Learning and Brain Sciences, University of Washington, Seattle, WA, USA

    Adrian K. C. Lee

  2. Departments of Hearing and Speech Sciences, Psychiatry, Psychology and Vanderbilt Brain Institute, Vanderbilt University, Nashville, TN, USA

    Mark T. Wallace

  3. Department of Integrative Physiology and Neuroscience, Washington State University Vancouver, Vancouver, WA, USA

    Allison B. Coffin

  4. Department of Biology, University of Maryland, College Park, MD, USA

    Arthur N. Popper

  5. Department of Psychology, Loyola University Chicago, Chicago, IL, USA

    Richard R. Fay

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Plakke, B., Romanski, L.M. (2019). Audiovisual Integration in the Primate Prefrontal Cortex. In: Lee, A., Wallace, M., Coffin, A., Popper, A., Fay, R. (eds) Multisensory Processes. Springer Handbook of Auditory Research, vol 68. Springer, Cham. https://doi.org/10.1007/978-3-030-10461-0_7

Download citation

Publish with us

Policies and ethics

Search

Navigation