Abstract
Why does seeing a speaker’s lip movements improve understanding speech in noisy environments? Why does simultaneous ringing and vibrating quicken answering a phone? These are questions of interest for researchers in the field of multisensory information processing. Electrophysiological approaches are suited to map the neural network dynamics underlying multisensory perception. Combining findings from behavioral, functional neuroimaging, and electrophysiological studies allows a comprehensive understanding of how information is integrated across the different senses. This chapter first provides an introduction on the relationships between neural network dynamics, as reflected in neural oscillations, and unisensory perception. Then, the relevance of neural network dynamics for multisensory perception is described, with a special focus on the auditory system. Moreover, the chapter provides an overview on how visual and auditory information can mutually influence each other and highlights the crucial role of ongoing neural network dynamics for upcoming multisensory perception. Finally, general principles of audiovisual integration are established, and open questions and future direction in the field of multisensory perception are discussed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adrian, E. D. (1942). Olfactory reactions in the brain of the hedgehog. The Journal of Physiology, 100(4), 459–473.
Arnal, L. H., & Giraud, A.-L. (2012). Cortical oscillations and sensory predictions. Trends in Cognitive Sciences, 16(7), 390–398. https://doi.org/10.1016/j.tics.2012.05.003.
Arnal, L. H., Morillon, B., Kell, C. A., & Giraud, A.-L. (2009). Dual neural routing of visual facilitation in speech processing. The Journal of Neuroscience, 29(43), 13445–13453. https://doi.org/10.1523/JNEUROSCI.3194-09.2009.
Arnal, L. H., Wyart, V., & Giraud, A.-L. (2011). Transitions in neural oscillations reflect prediction errors generated in audiovisual speech. Nature Neuroscience, 14(6), 797–801. https://doi.org/10.1038/nn.2810.
Balz, J., Keil, J., Roa Romero, Y., Mekle, R., Schubert, F., Aydin, S., Ittermann, B., Gallinat, J., & Senkowski, D. (2016). GABA concentration in superior temporal sulcus predicts gamma power and perception in the sound-induced flash illusion. NeuroImage, 125, 724–730. https://doi.org/10.1016/j.neuroimage.2015.10.087.
Berger, H. (1929). Über das elektroenkephalogramm des menschen. Archiv für Psychiatrie und Nervenkrankheiten, 87, 527–570.
Bhattacharya, J., Shams, L., & Shimojo, S. (2002). Sound-induced illusory flash perception: Role of gamma band responses. NeuroReport, 13(14), 1727–1730.
Biau, E., Torralba, M., Fuentemilla, L., de Diego Balaguer, R., & Soto-Faraco, S. (2015). Speaker’s hand gestures modulate speech perception through phase resetting of ongoing neural oscillations. Cortex, 68, 76–85. https://doi.org/10.1016/j.cortex.2014.11.018.
Birbaumer, N., Elbert, T., Canavan, A. G., & Rockstroh, B. (1990). Slow potentials of the cerebral cortex and behavior. Physiological Reviews, 70(1), 1–41.
Bizley, J. K., Maddox, R. K., & Lee, A. K. C. (2016). Defining auditory-visual objects: Behavioral tests and physiological mechanisms. Trends in Neurosciences, 39(2), 74–85. https://doi.org/10.1016/j.tins.2015.12.007.
Bremer, F. (1958). Cerebral and cerebellar potentials. Physiological Reviews, 38(3), 357–388.
Bressler, S. L. (1990). The gamma wave: A cortical information carrier? Trends in Neurosciences, 13(5), 161–162.
Busch, N. A., Dubois, J., & Vanrullen, R. (2009). The phase of ongoing EEG oscillations predicts visual perception. The Journal of Neuroscience, 29(24), 7869–7876. https://doi.org/10.1523/JNEUROSCI.0113-09.2009.
Buzsáki, G., & Draguhn, A. (2004). Neuronal oscillations in cortical networks. Science, 304(5679), 1926–1929. https://doi.org/10.1126/science.1099745.
Cavanagh, J. F., & Frank, M. J. (2014). Frontal theta as a mechanism for cognitive control. Trends in Cognitive Sciences, 18(8), 414–421. https://doi.org/10.1016/j.tics.2014.04.012.
Cecere, R., Rees, G., & Romei, V. (2015). Individual differences in alpha frequency drive crossmodal illusory perception. Current Biology, 25(2), 231–235. https://doi.org/10.1016/j.cub.2014.11.034.
Chatrian, G. E., Bickford, R. G., & Uihlein, A. (1960). Depth electrographic study of a fast rhythm evoked from the human calcarine region by steady illumination. Electroencephalography and Clinical Neurophysiology, 12, 167–176.
Cheron, G., Cebolla, A., De Saedeleer, C., Bengoetxea, A., Leurs, F., Leroy, A., & Dan, B. (2007). Pure phase-locking of beta/gamma oscillation contributes to the N30 frontal component of somatosensory evoked potentials. BMC Neuroscience, 8(1), 75. https://doi.org/10.1186/1471-2202-8-75.
Davis, H., & Davis, P. A. (1936). Action potentials of the brain: In normal persons and in normal states of cerebral activity. Archives of Neurology and Psychiatry, 36(6), 1214–1224.
Diekelmann, S., & Born, J. (2010). The memory function of sleep. Nature Reviews Neuroscience, 11(2), 114–126. https://doi.org/10.1038/nrn2762.
Driver, J., & Noesselt, T. (2008). Multisensory interplay reveals crossmodal influences on “sensory-specific” brain regions, neural responses, and judgments. Neuron, 57(1), 11–23. https://doi.org/10.1016/j.neuron.2007.12.013.
Engel, A. K., & Fries, P. (2010). Beta-band oscillations—signalling the status quo? Current Opinion in Neurobiology, 20(2), 156–165. https://doi.org/10.1016/j.conb.2010.02.015.
Engel, A. K., Fries, P., & Singer, W. (2001). Dynamic predictions: Oscillations and synchrony in top-down processing. Nature Reviews Neuroscience, 2(10), 704–716. https://doi.org/10.1038/35094565.
Ernst, M. O., & Bülthoff, H. H. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8(4), 162–169. https://doi.org/10.1016/j.tics.2004.02.002.
Fries, P. (2005). A mechanism for cognitive dynamics: Neuronal communication through neuronal coherence. Trends in Cognitive Sciences, 9(10), 474–480. https://doi.org/10.1016/j.tics.2005.08.011.
Fries, P. (2015). Rhythms for cognition: Communication through coherence. Neuron, 88(1), 220–235. https://doi.org/10.1016/j.neuron.2015.09.034.
Ghazanfar, A., & Schroeder, C. (2006). Is neocortex essentially multisensory? Trends in Cognitive Sciences, 10(6), 278–285. https://doi.org/10.1016/j.tics.2006.04.008.
He, Y., Gebhardt, H., Steines, M., Sammer, G., Kircher, T., Nagels, A., & Straube, B. (2015). The EEG and fMRI signatures of neural integration: An investigation of meaningful gestures and corresponding speech. Neuropsychologia, 72, 27–42. https://doi.org/10.1016/j.neuropsychologia.2015.04.018.
Herrmann, C. S., Mecklinger, A., & Pfeifer, E. (1999). Gamma responses and ERPs in a visual classification task. Clinical Neurophysiology, 110(4), 636–642.
Jensen, O., & Mazaheri, A. (2010). Shaping functional architecture by oscillatory alpha activity: Gating by inhibition. Frontiers in Human Neuroscience, 4, 186. https://doi.org/10.3389/fnhum.2010.00186.
Kaiser, J., Hertrich, I., Ackermann, H., Mathiak, K., & Lutzenberger, W. (2005). Hearing lips: Gamma-band activity during audiovisual speech perception. Cerebral Cortex, 15(5), 646–653. https://doi.org/10.1093/cercor/bhh166.
Kayser, C., Petkov, C. I., & Logothetis, N. K. (2008). Visual modulation of neurons in auditory cortex. Cerebral Cortex, 18(7), 1560–1574. https://doi.org/10.1093/cercor/bhm187.
Keil, J., & Senkowski, D. (2017). Individual alpha frequency relates to the sound-induced flash illusion. Multisensory Research, 30(6), 565–578. https://doi.org/10.1163/22134808-00002572.
Keil, J. & Senkowski, D. (2018). Neural oscillations orchestrate multisensory processing. Neuroscientist, 24(6), 609–626. https://journals.sagepub.com/doi/10.1177/10738584187553522.
Keil, J., Müller, N., Ihssen, N., & Weisz, N. (2012). On the variability of the McGurk effect: Audiovisual integration depends on prestimulus brain states. Cerebral Cortex, 22(1), 221–231. https://doi.org/10.1093/cercor/bhr125.
Keil, J., Müller, N., Hartmann, T., & Weisz, N. (2014a). Prestimulus beta power and phase synchrony influence the sound-induced flash illusion. Cerebral Cortex, 24(5), 1278–1288. https://doi.org/10.1093/cercor/bhs409.
Keil, J., Timm, J., SanMiguel, I., Schulz, H., Obleser, J., & Schönwiesner, M. (2014b). Cortical brain states and corticospinal synchronization influence TMS-evoked motor potentials. Journal of Neurophysiology, 111(3), 513–519. https://doi.org/10.1152/jn.00387.2013.
Keil, J., Pomper, U., & Senkowski, D. (2016). Distinct patterns of local oscillatory activity and functional connectivity underlie intersensory attention and temporal prediction. Cortex, 74, 277–288. https://doi.org/10.1016/j.cortex.2015.10.023.
Klimesch, W. (1999). EEG alpha and theta oscillations reflect cognitive and memory performance: A review and analysis. Brain Research Reviews, 29, 169–195.
Klimesch, W. (2012). Alpha-band oscillations, attention, and controlled access to stored information. Trends in Cognitive Sciences, 16(12), 606–617. https://doi.org/10.1016/j.tics.2012.10.007.
Lakatos, P., Chen, C.-M., O’Connell, M. N., Mills, A., & Schroeder, C. E. (2007). Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron, 53(2), 279–292. https://doi.org/10.1016/j.neuron.2006.12.011.
Lange, J., Oostenveld, R., & Fries, P. (2011). Perception of the touch-induced visual double-flash illusion correlates with changes of rhythmic neuronal activity in human visual and somatosensory areas. NeuroImage, 54(2), 1395–1405. https://doi.org/10.1016/j.neuroimage.2010.09.031.
Lange, J., Christian, N., & Schnitzler, A. (2013). Audio-visual congruency alters power and coherence of oscillatory activity within and between cortical areas. NeuroImage, 79, 111–120. https://doi.org/10.1016/j.neuroimage.2013.04.064.
Lange, J., Keil, J., Schnitzler, A., Van Dijk, H., & Weisz, N. (2014). The role of alpha oscillations for illusory perception. Behavioural Brain Research, 271, 294–301. https://doi.org/10.1016/j.bbr.2014.06.015.
Leonardelli, E., Braun, C., Weisz, N., Lithari, C., Occelli, V., & Zampini, M. (2015). Prestimulus oscillatory alpha power and connectivity patterns predispose perceptual integration of an audio and a tactile stimulus. Human Brain Mapping, 36(9), 3486–3498. https://doi.org/10.1002/hbm.22857.
Leske, S., Ruhnau, P., Frey, J., Lithari, C., Müller, N., Hartmann, T., & Weisz, N. (2015). Prestimulus network integration of auditory cortex predisposes near-threshold perception independently of local excitability. Cerebral Cortex, 25, 4898–4907. https://doi.org/10.1093/cercor/bhv212.
Lindsley, D. B. (1952). Psychological phenomena and the electroencephalogram. Electroencephalography and Clinical Neurophysiology, 4(4), 443–456.
Lopes da Silva, F. (1991). Neural mechanisms underlying brain waves: From neural membranes to networks. Electroencephalography and Clinical Neurophysiology, 79(2), 81–93.
Luck, S. J. (2014). An introduction to the event-related potential technique. Cambridge, MA: The MIT Press.
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746–748.
Mercier, M. R., Foxe, J. J., Fiebelkorn, I. C., Butler, J. S., Schwartz, T. H., & Molholm, S. (2013). Auditory-driven phase reset in visual cortex: Human electrocorticography reveals mechanisms of early multisensory integration. NeuroImage, 79, 19–29. https://doi.org/10.1016/j.neuroimage.2013.04.060.
Mercier, M. R., Molholm, S., Fiebelkorn, I. C., Butler, J. S., Schwartz, T. H., & Foxe, J. J. (2015). Neuro-oscillatory phase alignment drives speeded multisensory response times: An electro-corticographic investigation. The Journal of Neuroscience, 35(22), 8546–8557. https://doi.org/10.1523/JNEUROSCI.4527-14.2015.
Michalareas, G., Vezoli, J., van Pelt, S., Schoffelen, J.-M., Kennedy, H., & Fries, P. (2016). Alpha-beta and gamma rhythms subserve feedback and feedforward influences among human visual cortical areas. Neuron, 89(2), 384–397. https://doi.org/10.1016/j.neuron.2015.12.018.
Mishra, J., Martinez, A., Sejnowski, T. J., & Hillyard, S. A. (2007). Early cross-modal interactions in auditory and visual cortex underlie a sound-induced visual illusion. The Journal of Neuroscience, 27(15), 4120–4131. https://doi.org/10.1523/JNEUROSCI.4912-06.2007.
Mitra, P. P., & Pesaran, B. (1999). Analysis of dynamic brain imaging data. Biophysical Journal, 76(2), 691–708. https://doi.org/10.1016/S0006-3495(99)77236-X.
Pantev, C., Makeig, S., Hoke, M., Galambos, R., Hampson, S., & Gallen, C. (1991). Human auditory evoked gamma-band magnetic fields. Proceedings of the National Academy of Sciences of the United States of America, 88(20), 8996–9000.
Peelle, J. E., & Sommers, M. S. (2015). Prediction and constraint in audiovisual speech perception. Cortex, 68, 169–181. https://doi.org/10.1016/j.cortex.2015.03.006.
Penttonen, M., & Buzsáki, G. (2003). Natural logarithmic relationship between brain oscillators. Thalamus & Related Systems, 2(2), 145–152. https://doi.org/10.1016/S1472-9288(03)00007-4.
Pfurtscheller, G. (1992). Event-related synchronization (ERS): An electrophysiological correlate of cortical areas at rest. Electroencephalography and Clinical Neurophysiology, 83(1), 62–69.
Pomper, U., Brincker, J., Harwood, J., Prikhodko, I., & Senkowski, D. (2014). Taking a call is facilitated by the multisensory processing of smartphone vibrations, sounds, and flashes. PLoS One, 9(8), e103238. https://doi.org/10.1371/journal.pone.0103238.s002.
Pomper, U., Keil, J., Foxe, J. J., & Senkowski, D. (2015). Intersensory selective attention and temporal orienting operate in parallel and are instantiated in spatially distinct sensory and motor cortices. Human Brain Mapping, 36(8), 3246–3259. https://doi.org/10.1002/hbm.22845.
Rao, R. P., & Ballard, D. H. (1999). Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1), 79–87. https://doi.org/10.1038/4580.
Ray, W. J., & Cole, H. W. (1985). EEG alpha activity reflects attentional demands, and beta activity reflects emotional and cognitive processes. Science, 228(4700), 750–752.
Roa Romero, Y., Senkowski, D., & Keil, J. (2015). Early and late beta-band power reflect audiovisual perception in the McGurk illusion. Journal of Neurophysiology, 113(7), 2342–2350. https://doi.org/10.1152/jn.00783.2014.
Roa Romero, Y., Keil, J., Balz, J., Gallinat, J., & Senkowski, D. (2016). Reduced frontal theta oscillations indicate altered crossmodal prediction error processing in schizophrenia. Journal of Neurophysiology, 116(3), 1396–1407. https://doi.org/10.1152/jn.00096.2016.
Rockstroh, B., Müller, M., Cohen, R., & Elbert, T. (1992). Probing the functional brain state during P300-evocation. Journal of Psychophysiology, 6, 175–184.
Romei, V., Gross, J., & Thut, G. (2010). On the role of prestimulus alpha rhythms over occipito-parietal areas in visual input regulation: Correlation or causation? The Journal of Neuroscience, 30(25), 8692–8697. https://doi.org/10.1523/JNEUROSCI.0160-10.2010.
Sakowitz, O. W., Quiroga, R. Q., Schürmann, M., & Başar, E. (2001). Bisensory stimulation increases gamma-responses over multiple cortical regions. Cognitive Brain Research, 11(2), 267–279. https://doi.org/10.1016/S0926-6410(00)00081-1.
Sams, M., Aulanko, R., Hämäläinen, M., Hari, R., Lounasmaa, O. V., Lu, S. T., & Simola, J. (1991). Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neuroscience Letters, 127(1), 141–145.
Sauseng, P., Klimesch, W., Schabus, M., & Doppelmayr, M. (2005). Fronto-parietal EEG coherence in theta and upper alpha reflect central executive functions of working memory. International Journal of Psychophysiology, 57(2), 97–103. https://doi.org/10.1016/j.ijpsycho.2005.03.018.
Schepers, I. M., Schneider, T. R., Hipp, J. F., Engel, A. K., & Senkowski, D. (2013). Noise alters beta-band activity in superior temporal cortex during audiovisual speech processing. NeuroImage, 70, 101–112. https://doi.org/10.1016/j.neuroimage.2012.11.066.
Schroeder, C. E., & Foxe, J. (2005). Multisensory contributions to low-level, “unisensory” processing. Current Opinion in Neurobiology, 15(4), 454–458. https://doi.org/10.1016/j.conb.2005.06.008.
Schroeder, C. E., Lakatos, P., Kajikawa, Y., Partan, S., & Puce, A. (2008). Neuronal oscillations and visual amplification of speech. Trends in Cognitive Sciences, 12(3), 106–113. https://doi.org/10.1016/j.tics.2008.01.002.
Senkowski, D., Talsma, D., Herrmann, C. S., & Woldorff, M. G. (2005). Multisensory processing and oscillatory gamma responses: Effects of spatial selective attention. Experimental Brain Research, 166(3-4), 411–426. https://doi.org/10.1007/s00221-005-2381-z.
Senkowski, D., Talsma, D., Grigutsch, M., Herrmann, C. S., & Woldorff, M. G. (2007). Good times for multisensory integration: Effects of the precision of temporal synchrony as revealed by gamma-band oscillations. Neuropsychologia, 45(3), 561–571. https://doi.org/10.1016/j.neuropsychologia.2006.01.013.
Senkowski, D., Schneider, T., Foxe, J., & Engel, A. (2008). Crossmodal binding through neural coherence: Implications for multisensory processing. Trends in Neurosciences, 31(8), 401–409. https://doi.org/10.1016/j.tins.2008.05.002.
Senkowski, D., Saint-Amour, D., Höfle, M., & Foxe, J. J. (2011). Multisensory interactions in early evoked brain activity follow the principle of inverse effectiveness. NeuroImage, 56(4), 2200–2208. https://doi.org/10.1016/j.neuroimage.2011.03.075.
Shams, L., Kamitani, Y., & Shimojo, S. (2000). Illusions. What you see is what you hear. Nature, 408(6814), 788. https://doi.org/10.1038/35048669.
Shams, L., Kamitani, Y., Thompson, S., & Shimojo, S. (2001). Sound alters visual evoked potentials in humans. NeuroReport, 12(17), 3849–3852.
Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Cambridge, MA: The MIT Press.
Stein, B. E., Stanford, T. R., & Rowland, B. A. (2014). Development of multisensory integration from the perspective of the individual neuron. Nature Reviews Neuroscience, 15(8), 520–535. https://doi.org/10.1038/nrn3742.
Steriade, M. (2001). Impact of network activities on neuronal properties in corticothalamic systems. Journal of Neurophysiology, 86(1), 1–39.
Sumby, W. H., & Polack, I. (1954). Perceptual amplification of speech sounds by visual cues. The Journal of the Acoustical Society of America, 26, 212–215.
Tallon-Baudry, C., & Bertrand, O. (1999). Oscillatory gamma activity in humans and its role in object representation. Trends in Cognitive Sciences, 3(4), 151–162.
van Atteveldt, N., Murray, M. M., Thut, G., & Schroeder, C. E. (2014). Multisensory integration: Flexible use of general operations. Neuron, 81(6), 1240–1253. https://doi.org/10.1016/j.neuron.2014.02.044.
Van Dijk, H., Schoffelen, J.-M., Oostenveld, R., & Jensen, O. (2008). Prestimulus oscillatory activity in the alpha band predicts visual discrimination ability. The Journal of Neuroscience, 28(8), 1816–1823. https://doi.org/10.1523/JNEUROSCI.1853-07.2008.
van Wassenhove, V., & Grzeczkowski, L. (2015). Visual-induced expectations modulate auditory cortical responses. Frontiers in Neuroscience, 9, 11. https://doi.org/10.3389/fnins.2015.00011.
Walter, W. G. (1936). The location of cerebral tumours by electro-encephalography. The Lancet, 228(5893), 305–308.
Walter, D. O., Rhodes, J. M., Brown, D., & Adey, W. R. (1966). Comprehensive spectral analysis of human EEG generators in posterior cerebral regions. Electroencephalography and Clinical Neurophysiology, 20(3), 224–237.
Wang, X.-J. (2010). Neurophysiological and computational principles of cortical rhythms in cognition. Physiological Reviews, 90, 1195–1268. https://doi.org/10.1152/physrev.00035.2008.
Weisz, N., Wühle, A., Monittola, G., Demarchi, G., Frey, J., Popov, T., & Braun, C. (2014). Prestimulus oscillatory power and connectivity patterns predispose conscious somatosensory perception. Proceedings of the National Academy of Sciences of the United States of America, 111(4), E417–E425. https://doi.org/10.1073/pnas.1317267111.
Welch, R. B., & Warren, D. H. (1980). Immediate perceptual responses to intersensory discrepancy. Psychological Bulletin, 88(3), 638–667.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Keil, J., Senkowski, D. (2019). Neural Network Dynamics and Audiovisual Integration. In: Lee, A., Wallace, M., Coffin, A., Popper, A., Fay, R. (eds) Multisensory Processes. Springer Handbook of Auditory Research, vol 68. Springer, Cham. https://doi.org/10.1007/978-3-030-10461-0_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-10461-0_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-10459-7
Online ISBN: 978-3-030-10461-0
eBook Packages: Biomedical and Life SciencesBiomedical and Life Sciences (R0)