Journal of Computational Neuroscience

, Volume 41, Issue 2, pp 143–155 | Cite as

Parallel linear dynamic models can mimic the McGurk effect in clinical populations

  • Nicholas Altieri
  • Cheng-Ta Yang


One of the most common examples of audiovisual speech integration is the McGurk effect. As an example, an auditory syllable /ba/ recorded over incongruent lip movements that produce “ga” typically causes listeners to hear “da”. This report hypothesizes reasons why certain clinical and listeners who are hard of hearing might be more susceptible to visual influence. Conversely, we also examine why other listeners appear less susceptible to the McGurk effect (i.e., they report hearing just the auditory stimulus without being influenced by the visual). Such explanations are accompanied by a mechanistic explanation of integration phenomena including visual inhibition of auditory information, or slower rate of accumulation of inputs. First, simulations of a linear dynamic parallel interactive model were instantiated using inhibition and facilitation to examine potential mechanisms underlying integration. In a second set of simulations, we systematically manipulated the inhibition parameter values to model data obtained from listeners with autism spectrum disorder. In summary, we argue that cross-modal inhibition parameter values explain individual variability in McGurk perceptibility. Nonetheless, different mechanisms should continue to be explored in an effort to better understand current data patterns in the audiovisual integration literature.


Audiovisual integration Parallel interactive linear dynamic model McGurk effect 



The project described was supported by Grant No. (NIGMS) 5U54GM104944-03. Portions of this report, including the basic model set-up, appeared in the author’s Doctoral Dissertation and in Altieri (2016). Finally, we thank Ryan A. Stevenson for his data set.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. Altieri, N. (2016). Why we hear what we see: The temporal dynamics of audiovisual speech integration. In Houpt, J.W. and Blaha, L.M. (Eds.). Mathematical Models of Perception and Cognition (Vols. 1–2)Google Scholar
  2. Altieri, N., & Hudock, D. (2014). Variability in audiovisual speech integration skills assessed by combined capacity and accuracy measures. International Journal of Audiology, 53, 710–718.CrossRefPubMedGoogle Scholar
  3. Altieri, N., & Hudock, D. (2016). Normative data on audiovisual speech integration using sentence recognition and capacity measures. International Journal of Audiology, 55, 206–214.CrossRefPubMedGoogle Scholar
  4. Altieri, N., Pisoni, D. B., & Townsend, J. T. (2011). Some behavioral and neurobiological constraints on theories of audiovisual speech integration: a review and suggestions for new directions. Seeing and Perceiving, 24, 513–539.Google Scholar
  5. Altieri, N., Lentz, J., Townsend, J.T., & Wenger, M.J. (2016). The McGurk effect: An investigation of attentional capacity employing response times. Attention, Perception, & Psychophysics. (Online First)Google Scholar
  6. Bergeson, T. R., & Pisoni, D. B. (2004). Audiovisual speech perception in deaf adults and children following cochlear implantation. In G. A. Calvert, C. Spence, & B. E. Stein (Eds.), The handbook of multisensory processes (pp. 153–176). Cambridge, MA: The MIT Press.Google Scholar
  7. Bishop, C. W., & Miller, L. M. (2011). Speech cues contribute to audiovisual spatial integration. PLoS ONE, 6(8), e24016.CrossRefPubMedPubMedCentralGoogle Scholar
  8. Cienkowski, K. M., & Carney, A. E. (2002). Auditory–visual speech perception and aging. Ear and Hearing, 23, 439–449.CrossRefPubMedGoogle Scholar
  9. Dodd, B., McIntosh, B., Erdener, D., & Burnham, D. (2008). Perception of the auditory–visual illusion in speech perception by children with phonological disorders. Clinical Linguistics & Phonetics, 22(1), 69–82.CrossRefGoogle Scholar
  10. Dupont, S., Aubin, J., & Menard, L. (2005). A study of the McGurk effect in 4 and 5-year-old French Canadian children. ZAS Papers in Linguistics, 40, 1–17.Google Scholar
  11. Eidels, A., Houpt, J., Altieri, N., Pei, L., & Townsend, J. T. (2011). Nice guys finish fast and bad guys finish last: a theory of interactive parallel processing. Journal of Mathematical Psychology, 55(2), 176–190.CrossRefPubMedPubMedCentralGoogle Scholar
  12. French-St. George, M., & Stoker, R. G. (1988). Speechreading: an historical perspective. The Volta Review, 90(5), 17–31.Google Scholar
  13. Johnson, S. A., Blaha, L. M., Houpt, J. W., & Townsend, J. T. (2010). Systems factorial technology provides new insights on global–local information processing in autism spectrum disorders. Journal of Mathematical Psychology, 54, 53–72.CrossRefPubMedGoogle Scholar
  14. Magnotti, J. F., & Beauchamp, M. S. (2015). The noisy encoding of disparity model of the McGurk effect. Psychonomic Bulletin & Review, 22, 701–709.CrossRefGoogle Scholar
  15. Mallick, D. B., Magnotti, J. F., & Beauchamp, M. S. (2015). Variability and stability in the McGurk effect: contributions of participants, stimuli, time, and response type. Psychonomic Bulletin & Review, 22, 1299–1307.CrossRefGoogle Scholar
  16. Massaro, D. W. (1987). Speech perception by ear and eye. In B. Dodd & R. Campbell (Eds.), Hearing by eye: the psychology of lip-reading (pp. 53–83). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  17. Massaro, D. W. (2004). From multisensory integration to talking heads and language learning. In G. A. Calvert, C. Spence, & B. E. Stein (Eds.), The handbook of multisensory processes (pp. 153–176). Cambridge, MA: The MIT Press.Google Scholar
  18. McGurk, H., & MacDonald, J. W. (1976). Hearing lips and seeing voices. Nature, 264, 746–748.CrossRefPubMedGoogle Scholar
  19. Miller, J. (1982). Divided attention: evidence for coactivation with redundant signals. Cognitive Psychology, 14, 247–279.Google Scholar
  20. Norrix, L. W., Plante, E., & Vance, R. (2006). Auditory–visual speech integration by adults with and without language-learning disabilities. Journal of Communication Disorders, 39(1), 22–36.CrossRefPubMedGoogle Scholar
  21. Rosenblum, L. D., Schmuckler, M. A., & Johnson, J. A. (1997). The McGurk effect in infants. Perception & Psychophysics, 59(3), 347–357.CrossRefGoogle Scholar
  22. Sekiyama, K., Sochi, T., & Sakamoto, S. (2013). Enhanced audiovisual integration with aging in speech perception: a heightened McGurk effect in older adults. Frontiers in Psychology, 5(323).Google Scholar
  23. Setti, A., Burke, K. E., Kenny, R., & Newell, F. N. (2013). Susceptibility to a multi- sensory speech illusion in older persons is driven by perceptual processes. Frontiers in Psychology, 5(323), 1–11.Google Scholar
  24. Soto-Faraco, S., Navarra, J., & Alsius, A. (2004). Assessing automaticity in audiovisual speech integration: evidence from the speeded classification task. Cognition, 92, B13–B23.CrossRefPubMedGoogle Scholar
  25. Stevenson, R. A., Zemtsov, R. K., & Wallace, M. T. (2012). Individual differences in the multisensory temporal binding window predict susceptibility to audiovisual illusions. Journal of Experimental Psychology: Human Perception and Performance, 38, 1517–1529.PubMedPubMedCentralGoogle Scholar
  26. Stevenson, R. A., Siemann, J. K., Woynaroski, T. G., Schneider, B. C., Camarata, S. M., & Wallace, M. T. (2014). Arrested development of audiovisual speech perception in autism Spectrum disorders. Journal of Autism and Developmental Disorders, 4(6), 1470–1477.CrossRefGoogle Scholar
  27. Strelnikov, K., Rouger, J., Lagleyre, S., Fraysse, J.-F., Demonet, O., & Barone, P. (2015). Increased audiovisual integration in cochlear-implanted deaf patients: independent components analysis of longitudinal positron emission tomography data. European Journal of Neuroscience, 41, 677–685.CrossRefPubMedGoogle Scholar
  28. Sumby, W., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212–215.CrossRefGoogle Scholar
  29. Summerfield, Q. (1987). Some preliminaries to a comprehensive account of audio-visual speech perception. In B. Dodd & R. Campbell (Eds.), The psychology of lip-reading (pp. 3–50). Hillsdale, NJ: LEA.Google Scholar
  30. Tiippana, K., Andersen, T. S., & Sams, M. (2004). Visual attention modulates audiovisual speech perception. European Journal of Cognitive Psychology, 16(3), 457–472.CrossRefGoogle Scholar
  31. Tiippana, K., Mottonen, R., & Schwartz, J. L (2015). Multisensory and sensorimotor interactions in speech perception. Frontiers in Psychology, 6, 1–3.Google Scholar
  32. Townsend, J. T. & Nozawa, G. (1995). Spatio-temporal properties of elementary perception: An investigation of parallel, serial and coactive theories. Journal of Mathematical Psychology, 39, 321–360.Google Scholar
  33. Townsend, J. T., & Wenger, M. J. (2004). A theory of interactive parallel processing: new capacity measures and predictions for a response time inequality series. Psychological Review, 111(4), 1003–1035.CrossRefPubMedGoogle Scholar
  34. van Wassenhove, V., Grant, K., & Poeppel, D. (2005). Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Science, U.S.A., 102, 1181–1186.Google Scholar
  35. Wallace, M. T., Carriere, B. N., Perrault, T. J., Vaughan, J. W., & Stein, B. E. (2006). The development of cortical multisensory neurons. The Journal of Neuroscience, 15, 11844–11849.CrossRefGoogle Scholar
  36. Werker, J. F., & Hensch, T. K. (2015). Critical periods in speech perception: new directions. Annual Review of Psychology, 66, 173–196.CrossRefPubMedGoogle Scholar
  37. White, T. P., Wigton, R. L., Joyce, D. W., Bobin, T., Ferragamo, C., Wasim, N., et al. (2014). Eluding the illusion? Schizophrenia, dopamine and the McGurk effect. Frontiers in Human Neuroscience, 9(565), 1–12.Google Scholar
  38. Woynaroski, T. G., Kwakye, L. D., Foss-Feig, J. H., Stevenson, R. A., Stone, W. L., & Wallace, M. T. (2013). Multisensory speech perception in children with autism Spectrum disorders. Journal of Autism and Developmental Disorders, 43, 2891–2902.CrossRefPubMedPubMedCentralGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Department of Communication Sciences and DisordersIdaho State UniversityPocatelloUSA
  2. 2.Department of PsychologyNational Cheng Kung UniversityTainan CityTaiwan

Personalised recommendations