Abstract
Unisensory auditory representations are strongly shaped by multisensory experience, and, likewise, audition contributes to cross-modal learning in other sensory systems. This applies to lower-level sensory features like spatial and temporal processing as well as to higher-level features like speech identification. Cross-modal learning has particularly profound influences during development, but its effects on unisensory processing are ubiquitous throughout life. Moreover, influences of cross-modal learning on unisensory processing have been observed at various timescales, ranging from long-term structural changes over months to short-term plasticity of auditory representations after minutes or only seconds of cross-modal exposure. This chapter focuses particularly on cross-modal learning and its underlying neural mechanisms in the healthy adult auditory system. Recent findings suggest that cross-modal learning operates in parallel on different neural representations and at different timescales. With an increasing amount of exposure to new cross-modal associations, cross-modal learning seems to progress from higher level multisensory representations to lower level modality-specific representations, possibly even in primary auditory cortex. In addition to cortically mediated learning mechanisms, auditory representations are shaped via subcortical multisensory pathways including the superior colliculi in the midbrain. The emerging view from these findings is that auditory-guided behavior is jointly shaped by cross-modal learning in distinct neural systems. To fully understand the dynamic nature of the auditory system, it will be important to identify how short-term and long-term learning processes interact in the mature brain.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14, 257–262.
Bergan, J. F., Ro, P., Ro, D., & Knudsen, E. I. (2005). Hunting increases adaptive auditory map plasticity in adult barn owls. The Journal of Neuroscience, 25, 9816–9820.
Bertelson, P., & Aschersleben, G. (1998). Automatic visual bias of perceived auditory location. Psychonomic Bulletin & Review, 5, 482–489.
Bertelson, P., Vroomen, J., de Gelder, B., & Driver, J. (2000). The ventriloquist effect does not depend on the direction of deliberate visual attention. Perception & Psychophysics, 62, 321–332.
Bertelson, P., Vroomen, J., & de Gelder, B. (2003). Visual recalibration of auditory speech identification: A McGurk aftereffect. Psychological Science, 14, 592–597.
Bertelson, P., Frissen, I., Vroomen, J., & de Gelder, B. (2006). The aftereffects of ventriloquism: Patterns of spatial generalization. Perception & Psychophysics, 68, 428–436.
Bertini, C., Leo, F., Avenanti, A., & Làdavas, E. (2010). Independent mechanisms for ventriloquism and multisensory integration as revealed by theta-burst stimulation. European Journal of Neuroscience, 31, 1791–1799.
Bonath, B., Noesselt, T., Martinez, A., Mishra, J., Schwiecker, K., Heinze, H. J., & Hillyard, S. A. (2007). Neural basis of the ventriloquist illusion. Current Biology, 17, 1697–1703.
Bruns, P., & Röder, B. (2010). Tactile capture of auditory localization: An event-related potential study. European Journal of Neuroscience, 31, 1844–1857.
Bruns, P., & Röder, B. (2015). Sensory recalibration integrates information from the immediate and the cumulative past. Scientific Reports, 5, 12739.
Bruns, P., Liebnau, R., & Röder, B. (2011a). Cross-modal training induces changes in spatial representations early in the auditory processing pathway. Psychological Science, 22, 1120–1126.
Bruns, P., Spence, C., & Röder, B. (2011b). Tactile recalibration of auditory spatial representations. Experimental Brain Research, 209, 333–344.
Bruns, P., Maiworm, M., & Röder, B. (2014). Reward expectation influences audiovisual spatial integration. Attention, Perception, & Psychophysics, 76, 1815–1827.
Caclin, A., Soto-Faraco, S., Kingstone, A., & Spence, C. (2002). Tactile “capture” of audition. Perception & Psychophysics, 64, 616–630.
Chen, L., & Vroomen, J. (2013). Intersensory binding across space and time: A tutorial review. Attention, Perception, & Psychophysics, 75, 790–811.
Dahmen, J. C., & King, A. J. (2007). Learning to hear: Plasticity of auditory cortical processing. Current Opinion in Neurobiology, 17, 456–464.
Dahmen, J. C., Keating, P., Nodal, F. R., Schulz, A. L., & King, A. J. (2010). Adaptation to stimulus statistics in the perception and neural representation of auditory space. Neuron, 66, 937–948.
Dean, I., Harper, N. S., & McAlpine, D. (2005). Neural population coding of sound level adapts to stimulus statistics. Nature Neuroscience, 8, 1684–1689.
Eramudugolla, R., Kamke, M. R., Soto-Faraco, S., & Mattingley, J. B. (2011). Perceptual load influences auditory space perception in the ventriloquist aftereffect. Cognition, 118, 62–74.
Ernst, M. O., & Bülthoff, H. H. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8, 162–169.
Frissen, I., Vroomen, J., de Gelder, B., & Bertelson, P. (2003). The aftereffects of ventriloquism: Are they sound-frequency specific? Acta Psychologica, 113, 315–327.
Frissen, I., Vroomen, J., de Gelder, B., & Bertelson, P. (2005). The aftereffects of ventriloquism: Generalization across sound-frequencies. Acta Psychologica, 118, 93–100.
Frissen, I., Vroomen, J., & de Gelder, B. (2012). The aftereffects of ventriloquism: The time course of the visual recalibration of auditory localization. Seeing and Perceiving, 25, 1–14.
Fujisaki, W., Shimojo, S., Kashino, M., & Nishida, S. (2004). Recalibration of audiovisual simultaneity. Nature Neuroscience, 7, 773–778.
Gutfreund, Y., & King, A. J. (2012). What is the role of vision in the development of the auditory space map? In B. E. Stein (Ed.), The New Handbook of Multisensory Processing (pp. 573–587). Cambridge, MA: The MIT Press.
Heron, J., Roach, N. W., Whitaker, D., & Hanson, J. V. M. (2010). Attention regulates the plasticity of multisensory timing. European Journal of Neuroscience, 31, 1755–1762.
Heron, J., Roach, N. W., Hanson, J. V. M., McGraw, P. V., & Whitaker, D. (2012). Audiovisual time perception is spatially specific. Experimental Brain Research, 218, 477–485.
Ikumi, N., & Soto-Faraco, S. (2014). Selective attention modulates the direction of audio-visual temporal recalibration. PLoS One, 9, e99311.
Kim, R. S., Seitz, A. R., & Shams, L. (2008). Benefits of stimulus congruency for multisensory facilitation of visual learning. PLoS One, 3, e1532.
Kim, R., Peters, M. A. K., & Shams, L. (2012). 0 + 1 > 1: How adding noninformative sound improves performance on a visual task. Psychological Science, 23, 6–12.
King, A. J. (2009). Visual influences on auditory spatial learning. Philosophical Transactions of the Royal Society, B: Biological Sciences, 364, 331–339.
Knudsen, E. I., & Brainard, M. S. (1991). Visual instruction of the neural map of auditory space in the developing optic tectum. Science, 253, 85–87.
Knudsen, E. I., & Knudsen, P. F. (1990). Sensitive and critical periods for visual calibration of sound localization by barn owls. The Journal of Neuroscience, 10, 222–232.
Kopčo, N., Lin, I.-F., Shinn-Cunningham, B. G., & Groh, J. M. (2009). Reference frame of the ventriloquism aftereffect. The Journal of Neuroscience, 29, 13809–13814.
Leo, F., Bolognini, N., Passamonti, C., Stein, B. E., & Làdavas, E. (2008). Cross-modal localization in hemianopia: New insights on multisensory integration. Brain, 131, 855–865.
Lewald, J. (2002). Rapid adaptation to auditory-visual spatial disparity. Learning & Memory, 9, 268–278.
Linkenhoker, B. A., & Knudsen, E. I. (2002). Incremental training increases the plasticity of the auditory space map in adult barn owls. Nature, 419, 293–296.
Lüttke, C. S., Ekman, M., van Gerven, M. A. J., & de Lange, F. P. (2016). McGurk illusion recalibrates subsequent auditory perception. Scientific Reports, 6, 32891.
Maiworm, M., Bellantoni, M., Spence, C., & Röder, B. (2012). When emotional valence modulates audiovisual integration. Attention, Perception, & Psychophysics, 74, 1302–1311.
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748.
Morein-Zamir, S., Soto-Faraco, S., & Kingstone, A. (2003). Auditory capture of vision: Examining temporal ventriloquism. Cognitive Brain Research, 17, 154–163.
Murray, M. M., Lewkowicz, D. J., Amedi, A., & Wallace, M. T. (2016). Multisensory processes: A balancing act across the lifespan. Trends in Neurosciences, 39, 567–579.
Navarra, J., Soto-Faraco, S., & Spence, C. (2007). Adaptation to audiotactile asynchrony. Neuroscience Letters, 413, 72–76.
Navarra, J., Hartcher-O’Brien, J., Piazza, E., & Spence, C. (2009). Adaptation to audiovisual asynchrony modulates the speeded detection of sound. Proceedings of the National Academy of Sciences of the United States of America, 106, 9169–9173.
Noel, J.-P., De Niear, M., Van der Burg, E., & Wallace, M. T. (2016). Audiovisual simultaneity judgment and rapid recalibration throughout the lifespan. PLoS One, 11, e0161698.
Pages, D. S., & Groh, J. M. (2013). Looking at the ventriloquist: Visual outcome of eye movements calibrates sound localization. PLoS One, 8, e72562.
Passamonti, C., Frissen, I., & Làdavas, E. (2009). Visual recalibration of auditory spatial perception: Two separate neural circuits for perceptual learning. European Journal of Neuroscience, 30, 1141–1150.
Polley, D. B., Steinberg, E. E., & Merzenich, M. M. (2006). Perceptual learning directs auditory cortical map reorganization through top-down influences. The Journal of Neuroscience, 26, 4970–4982.
Powers, A. R., III, Hillock, A. R., & Wallace, M. T. (2009). Perceptual training narrows the temporal window of multisensory binding. The Journal of Neuroscience, 29, 12265–12274.
Powers, A. R., III, Hevey, M. A., & Wallace, M. T. (2012). Neural correlates of multisensory perceptual learning. The Journal of Neuroscience, 32(18), 6263–6274.
Radeau, M., & Bertelson, P. (1974). The after-effects of ventriloquism. Quarterly Journal of Experimental Psychology, 26, 63–71.
Radeau, M., & Bertelson, P. (1977). Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations. Perception & Psychophysics, 22, 137–146.
Radeau, M., & Bertelson, P. (1978). Cognitive factors and adaptation to auditory-visual discordance. Perception & Psychophysics, 23, 341–343.
Recanzone, G. H. (1998). Rapidly induced auditory plasticity: The ventriloquism aftereffect. Proceedings of the National Academy of Sciences of the United States of America, 95, 869–875.
Recanzone, G. H. (2009). Interactions of auditory and visual stimuli in space and time. Hearing Research, 258, 89–99.
Recanzone, G. H., & Sutter, M. L. (2008). The biological basis of audition. Annual Review of Psychology, 59, 119–142.
Renzi, C., Bruns, P., Heise, K.-F., Zimerman, M., Feldheim, J. F., Hummel, F. C., & Röder, B. (2013). Spatial remapping in the audio-tactile ventriloquism effect: A TMS investigation on the role of the ventral intraparietal area. Journal of Cognitive Neuroscience, 25, 790–801.
Rohe, T., & Noppeney, U. (2015). Cortical hierarchies perform Bayesian causal inference in multisensory perception. PLoS Biology, 13, e1002073.
Roseboom, W., & Arnold, D. H. (2011). Twice upon a time: Multiple concurrent temporal recalibrations of audiovisual speech. Psychological Science, 22, 872–877.
Seitz, A. R., Kim, R., & Shams, L. (2006). Sound facilitates visual learning. Current Biology, 16, 1422–1427.
Shams, L., & Seitz, A. R. (2008). Benefits of multisensory learning. Trends in Cognitive Sciences, 12, 411–417.
Shams, L., Wozny, D. R., Kim, R., & Seitz, A. (2011). Influences of multisensory experience on subsequent unisensory processing. Frontiers in Psychology, 2, 264.
Slutsky, D. A., & Recanzone, G. H. (2001). Temporal and spatial dependency of the ventriloquism effect. NeuroReport, 12, 7–10.
Stein, B. E., & Stanford, T. R. (2008). Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews Neuroscience, 9, 255–266.
Stein, B. E., Huneycutt, W. S., & Meredith, M. A. (1988). Neurons and behavior: The same rules of multisensory integration apply. Brain Research, 448, 355–358.
Strelnikov, K., Rosito, M., & Barone, P. (2011). Effect of audiovisual training on monaural spatial hearing in horizontal plane. PLoS One, 6, e18344.
Talsma, D., Senkowski, D., Soto-Faraco, S., & Woldorff, M. G. (2010). The multifaceted interplay between attention and multisensory integration. Trends in Cognitive Sciences, 14, 400–410.
Tzounopoulos, T., & Leão, R. M. (2012). Mechanisms of memory and learning in the auditory system. In L. O. Trussel, A. N. Popper, & R. R. Fay (Eds.), Synaptic Mechanisms in the Auditory System (pp. 203–226). New York: Springer-Verlag.
Van der Burg, E., Alais, D., & Cass, J. (2013). Rapid recalibration to audiovisual asynchrony. The Journal of Neuroscience, 33, 14633–14637.
Van der Burg, E., Alais, D., & Cass, J. (2015). Audiovisual temporal recalibration occurs independently at two different time scales. Scientific Reports, 5, 14526.
Vatakis, A., Navarra, J., Soto-Faraco, S., & Spence, C. (2007). Temporal recalibration during asynchronous audiovisual speech perception. Experimental Brain Research, 181, 173–181.
Von Kriegstein, K., & Giraud, A.-L. (2006). Implicit multisensory associations influence voice recognition. PLoS Biology, 4, e326.
Vroomen, J., & de Gelder, B. (2004). Temporal ventriloquism: Sound modulates the flash-lag effect. Journal of Experimental Psychology: Human Perception and Performance, 30, 513–518.
Vroomen, J., & Keetels, M. (2010). Perception of intersensory synchrony: A tutorial review. Attention, Perception, & Psychophysics, 72, 871–884.
Vroomen, J., Bertelson, P., & de Gelder, B. (2001). The ventriloquist effect does not depend on the direction of automatic visual attention. Perception & Psychophysics, 63, 651–659.
Vroomen, J., Keetels, M., de Gelder, B., & Bertelson, P. (2004). Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cognitive Brain Research, 22, 32–35.
Wallace, M. T., & Stein, B. E. (1997). Development of multisensory neurons and multisensory integration in cat superior colliculus. The Journal of Neuroscience, 17, 2429–2444.
Wallace, M. T., & Stein, B. E. (2007). Early experience determines how the senses will interact. Journal of Neurophysiology, 97, 921–926.
Woods, T. M., & Recanzone, G. H. (2004). Visually induced plasticity of auditory spatial perception in macaques. Current Biology, 14, 1559–1564.
Wozny, D. R., & Shams, L. (2011). Recalibration of auditory space following milliseconds of cross-modal discrepancy. The Journal of Neuroscience, 31, 4607–4612.
Yu, L., Rowland, B. A., & Stein, B. E. (2010). Initiating the development of multisensory integration by manipulating sensory experience. The Journal of Neuroscience, 30, 4904–4913.
Zaidel, A., Turner, A. H., & Angelaki, D. E. (2011). Multisensory calibration is independent of cue reliability. The Journal of Neuroscience, 31, 13949–13962.
Zwiers, M. P., van Opstal, A. J., & Paige, G. D. (2003). Plasticity in human sound localization induced by compressed spatial vision. Nature Neuroscience, 6, 175–181.
Acknowledgments
The authors’ work was supported by German Research Foundation (DFG) Grants BR 4913/2-1 and TRR 169 Subproject A1 and the City of Hamburg Grant “Crossmodal Learning.”
Compliance with Ethics Requirements
Patrick Bruns declares that he has no conflict of interest.
Brigitte Röder declares that she has no conflict of interest.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Bruns, P., Röder, B. (2019). Cross-Modal Learning in the Auditory System. In: Lee, A., Wallace, M., Coffin, A., Popper, A., Fay, R. (eds) Multisensory Processes. Springer Handbook of Auditory Research, vol 68. Springer, Cham. https://doi.org/10.1007/978-3-030-10461-0_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-10461-0_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-10459-7
Online ISBN: 978-3-030-10461-0
eBook Packages: Biomedical and Life SciencesBiomedical and Life Sciences (R0)