Cue Combination Within a Bayesian Framework

  • David AlaisEmail author
  • David Burr
Part of the Springer Handbook of Auditory Research book series (SHAR, volume 68)


To interact effectively with the world, the brain must optimize its perception of the objects and events in the environment, many of which are signaled by more than one sense. Optimal perception requires the brain to integrate redundant cues from the different senses as efficiently as possible. One effective model of cue combination is maximum likelihood estimation (MLE), a Bayesian model that deals with the fundamental uncertainty and noise associated with sensory signals and provides a statistically optimal way to integrate them. MLE achieves this through a weighted linear sum of two or more cues in which each cue is weighted inversely to its variance or “uncertainty.” This produces an integrated sensory estimate with minimal uncertainty and thus maximized perceptual precision. Many studies show that adults integrate redundant sensory information consistent with MLE predictions. When the MLE model is tested in school-aged children, it is found that predictions for multisensory integration are confirmed in older children (>10 years) but not in younger children. Younger children show unisensory dominance and do not exploit the statistical benefits of multisensory integration, even when their dominant sense is far less precise than the other. This curious finding may result from each sensory system having an inherent specialization, with each specialist sense tuning the other senses, such as vision calibrating audition for space (or audition calibrating vision for time). This cross-sensory tuning would preclude useful combination of two senses until calibration is complete, after which MLE integration provides an excellent model of multisensory cue combination.


Bayesian Calibration Cross-modal Cue combination Development Maximum likelihood model Optimal integration Sensory deficit Sensory integration Sensory noise Space perception Time perception Uncertainty reduction Weighted sum 


  1. Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14(3), 257–262.PubMedPubMedCentralCrossRefGoogle Scholar
  2. Alais, D., Newell, F. N., & Mamassian, P. (2010). Multisensory processing in review: From physiology to behaviour. Seeing and Perceiving, 23(1), 3–38.PubMedCrossRefPubMedCentralGoogle Scholar
  3. Angelaki, D. E., Gu, Y., & DeAngelis, G. C. (2009). Multisensory integration: Psychophysics, neurophysiology, and computation. Current Opinion in Neurobiology, 19(4), 452–458.PubMedPubMedCentralCrossRefGoogle Scholar
  4. Bahrick, L. E., & Lickliter, R. (2004). Infants’ perception of rhythm and tempo in unimodal and multimodal stimulation: A developmental test of the intersensory redundancy hypothesis. Cognitive, Affective, & Behavioral Neuroscience, 4(2), 137–147.CrossRefGoogle Scholar
  5. Barutchu, A., Crewther, D. P., & Crewther, S. G. (2009). The race that precedes coactivation: Development of multisensory facilitation in children. Developmental Science, 12(3), 464–473.PubMedCrossRefPubMedCentralGoogle Scholar
  6. Barutchu, A., Danaher, J., Crewther, S. G., Innes-Brown, H., Shivdasani, M. N., & Paolini, A. G. (2010). Audiovisual integration in noise by children and adults. Journal of Experimental Child Psychology, 105(1–2), 38–50.PubMedCrossRefPubMedCentralGoogle Scholar
  7. Benevento, L. A., Fallon, J., Davis, B. J., & Rezak, M. (1977). Auditory-visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey. Experimental Neurology, 57(3), 849–872.PubMedCrossRefPubMedCentralGoogle Scholar
  8. Berkeley, G. (1963). An essay towards a new theory of vision. Indianapolis: Bobbs-Merrill. (Original work published 1709).Google Scholar
  9. Bertelson, P., & Radeau, M. (1981). Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Perception & Psychophysics, 29(6), 578–584.CrossRefGoogle Scholar
  10. Bresciani, J. P., Dammeier, F., & Ernst, M. O. (2006). Vision and touch are automatically integrated for the perception of sequences of events. Journal of Vision, 6(5), 554–564.PubMedCrossRefPubMedCentralGoogle Scholar
  11. Brown, A. M., Dobson, V., & Maier, J. (1987). Visual acuity of human infants at scotopic, mesopic and photopic luminances. Vision Research, 27(10), 1845–1858.PubMedCrossRefPubMedCentralGoogle Scholar
  12. Burr, D., Banks, M. S., & Morrone, M. C. (2009). Auditory dominance over vision in the perception of interval duration. Experimental Brain Research, 198(1), 49–57.PubMedCrossRefPubMedCentralGoogle Scholar
  13. Caclin, A., Soto-Faraco, S., Kingstone, A., & Spence, C. (2002). Tactile “capture” of audition. Perception & Psychophysics, 64(4), 616–630.CrossRefGoogle Scholar
  14. Cicchini, G. M., Arrighi, R., Cecchetti, L., Giusti, M., & Burr, D. C. (2012). Optimal encoding of interval timing in expert percussionists. Journal of Neuroscience, 32(3), 1056–1060.PubMedCrossRefPubMedCentralGoogle Scholar
  15. Clarke, J. J., & Yuille, A. L. (1990). Data fusion for sensory information processing. Boston: Kluwer Academic.CrossRefGoogle Scholar
  16. Dodd, B. (1979). Lip reading in infants: Attention to speech presented in- and out-of-synchrony. Cognitive Psychology, 11(4), 478–484.PubMedCrossRefPubMedCentralGoogle Scholar
  17. Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415(6870), 429–433.PubMedCrossRefPubMedCentralGoogle Scholar
  18. Ernst, M. O., & Bulthoff, H. H. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8(4), 162–169.PubMedPubMedCentralCrossRefGoogle Scholar
  19. Gebhard, J. W., & Mowbray, G. H. (1959). On discriminating the rate of visual flicker and auditory flutter. American Journal of Psychology, 72, 521–529.PubMedCrossRefPubMedCentralGoogle Scholar
  20. Ghahramani, Z., & Wolpert, D. M. (1997). Modular decomposition in visuomotor learning. Nature, 386(6623), 392–395.PubMedCrossRefPubMedCentralGoogle Scholar
  21. Gori, M., Del Viva, M., Sandini, G., & Burr, D. C. (2008). Young children do not integrate visual and haptic form information. Current Biology, 18(9), 694–698.PubMedCrossRefPubMedCentralGoogle Scholar
  22. Gori, M., Sandini, G., Martinoli, C., & Burr, D. (2010). Poor haptic orientation discrimination in nonsighted children may reflect disruption of cross-sensory calibration. Current Biology, 20(3), 223–225.PubMedCrossRefPubMedCentralGoogle Scholar
  23. Gori, M., Sandini, G., & Burr, D. (2012). Development of visuo-auditory integration in space and time. Frontiers in Integrative Neuroscience, 6, 77.PubMedPubMedCentralCrossRefGoogle Scholar
  24. Gori, M., Sandini, G., Martinoli, C., & Burr, D. (2014). Impairment of auditory spatial localization in congenitally blind human subjects. Brain, 20, 288–293.CrossRefGoogle Scholar
  25. Gottlieb, G. (1990). Development of species identification in birds: An inquiry into the prenatal determinants of perception. Chicago: University of Chicago Press.Google Scholar
  26. Gu, Y., Angelaki, D. E., & Deangelis, G. C. (2008). Neural correlates of multisensory cue integration in macaque MSTd. Nature Neuroscience, 11(10), 1201–1210.PubMedPubMedCentralCrossRefGoogle Scholar
  27. Guest, S., Catmur, C., Lloyd, D., & Spence, C. (2002). Audiotactile interactions in roughness perception. Experimental Brain Research, 146(2), 161–171.PubMedCrossRefPubMedCentralGoogle Scholar
  28. Hartcher-O’Brien, J., & Alais, D. (2011). Temporal ventriloquism in a purely temporal context. Journal of Experimental Psychology: Human Perception and Performance, 37(5), 1383–1395.PubMedPubMedCentralGoogle Scholar
  29. Hartcher-O’Brien, J., Di Luca, M., & Ernst, M. O. (2014). The duration of uncertain times: Audiovisual information about intervals is integrated in a statistically optimal fashion. PLoS One, 9(3), e89339.PubMedPubMedCentralCrossRefGoogle Scholar
  30. Hatwell, Y. (1987). Motor and cognitive functions of the hand in infancy and childhood. International Journal of Behavioural Development, 10, 509–526.CrossRefGoogle Scholar
  31. Helbig, H. B., & Ernst, M. O. (2008). Visual-haptic cue weighting is independent of modality-specific attention. Journal of Vision, 8(1), 21.1–21.16.Google Scholar
  32. Hillis, J. M., Ernst, M. O., Banks, M. S., & Landy, M. S. (2002). Combining sensory information: Mandatory fusion within, but not between, senses. Science, 298(5598), 1627–1630.PubMedCrossRefPubMedCentralGoogle Scholar
  33. Hillis, J. M., Watt, S. J., Landy, M. S., & Banks, M. S. (2004). Slant from texture and disparity cues: Optimal cue combination. Journal of Vision, 4(12), 967–992.PubMedCrossRefPubMedCentralGoogle Scholar
  34. Howard, I. P., & Templeton, W. B. (1966). Human spatial orientation. New York: Wiley.Google Scholar
  35. Hubel, D. H., & Wiesel, T. N. (1968). Receptive fields and functional architecture of monkey striate cortex. Journal of Physiology, 195(1), 215–243.PubMedCrossRefGoogle Scholar
  36. Jazayeri, M., & Shadlen, M. N. (2010). Temporal context calibrates interval timing. Nature Neuroscience, 13(8), 1020–1026.PubMedPubMedCentralCrossRefGoogle Scholar
  37. Jones, E. G., & Powell, T. P. (1970). An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. Brain, 93(4), 793–820.PubMedCrossRefGoogle Scholar
  38. Jusczyk, P., Houston, D., & Goodman, M. (1998). Speech perception during the first year. In A. Slater (Ed.), Perceptual development: Visual, auditory, and speech perception in infancy (pp. 357–388). Hove: Psychology Press.Google Scholar
  39. Kayser, C., Petkov, C. I., & Logothetis, N. K. (2008). Visual modulation of neurons in auditory cortex. Cerebral Cortex, 18(7), 1560–1574.PubMedCrossRefPubMedCentralGoogle Scholar
  40. Kersten, D., Mamassian, P., & Yuille, A. (2004). Object perception as Bayesian inference. Annual Review of Psychology, 55, 271–304.PubMedCrossRefPubMedCentralGoogle Scholar
  41. King, A. J. (2009). Visual influences on auditory spatial learning. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 364(1515), 331–339.PubMedCrossRefPubMedCentralGoogle Scholar
  42. Knill, D. C. (2007). Learning Bayesian priors for depth perception. Journal of Vision, 7(8), 13.PubMedCrossRefPubMedCentralGoogle Scholar
  43. Knill, D. C., & Pouget, A. (2004). The Bayesian brain: The role of uncertainty in neural coding and computation. Trends in Neurosciences, 27(12), 712–719.PubMedCrossRefPubMedCentralGoogle Scholar
  44. Knill, D. C., & Saunders, J. A. (2003). Do humans optimally integrate stereo and texture information for judgments of surface slant? Vision Research, 43(24), 2539–2558.PubMedCrossRefPubMedCentralGoogle Scholar
  45. Kording, K. P., & Wolpert, D. M. (2004). Bayesian integration in sensorimotor learning. Nature, 427(6971), 244–247.PubMedCrossRefPubMedCentralGoogle Scholar
  46. Kwon, O. S., & Knill, D. C. (2013). The brain uses adaptive internal models of scene statistics for sensorimotor estimation and planning. Proceedings of the National Academy of Sciences of the United States of America, 110(11), E1064–E1073.PubMedPubMedCentralCrossRefGoogle Scholar
  47. Landy, M. S., Banks, M. S., & Knill, D. C. (2011). Ideal-observer models of cue integration. In K. Tromershauser, K. Körding, & M. S. Landy (Eds.), Book of sensory cue integration (pp. 5–30). New York: Oxford University Press.CrossRefGoogle Scholar
  48. Lewkowicz, D. J. (1992). Infants’ responsiveness to the auditory and visual attributes of a sounding/moving stimulus. Perception & Psychophysics, 52(5), 519–528.CrossRefGoogle Scholar
  49. Lewkowicz, D. J. (2000). The development of intersensory temporal perception: An epigenetic systems/limitations view. Psychological Bulletin, 126(2), 281–308.PubMedCrossRefPubMedCentralGoogle Scholar
  50. Lewkowicz, D. J., & Turkewitz, G. (1981). Intersensory interaction in newborns: Modification of visual preferences following exposure to sound. Child Development, 52(3), 827–832.PubMedCrossRefPubMedCentralGoogle Scholar
  51. McGurk, H., & Power, R. P. (1980). Intermodal coordination in young children: Vision and touch. Developmental Psychology, 16, 679–680.CrossRefGoogle Scholar
  52. Misceo, G. F., Hershberger, W. A., & Mancini, R. L. (1999). Haptic estimates of discordant visual-haptic size vary developmentally. Perception & Psychophysics, 61(4), 608–614.CrossRefGoogle Scholar
  53. Murray, M. M., Thelen, A., Thut, G., Romei, V., Martuzzi, R., & Matusz, P. J. (2015). The multisensory function of the human primary visual cortex. Neuropsychologia, 83, 161–169.PubMedCrossRefPubMedCentralGoogle Scholar
  54. Nardini, M., Jones, P., Bedford, R., & Braddick, O. (2008). Development of cue integration in human navigation. Current Biology, 18(9), 689–693.PubMedCrossRefPubMedCentralGoogle Scholar
  55. Nardini, M., Bedford, R., & Mareschal, D. (2010). Fusion of visual cues is not mandatory in children. Proceedings of the National Academy of Sciences of the United States of America, 107(39), 17041–17046.PubMedPubMedCentralCrossRefGoogle Scholar
  56. Olsho, L. W. (1984). Infant frequency discrimination as a function of frequency. Infant Behavior and Development, 7, 27–35.CrossRefGoogle Scholar
  57. Olsho, L. W., Koch, E. G., Carter, E. A., Halpin, C. F., & Spetner, N. B. (1988). Pure-tone sensitivity of human infants. The Journal of the Acoustical Society of America, 84(4), 1316–1324.PubMedCrossRefPubMedCentralGoogle Scholar
  58. Patterson, M. L., & Werker, J. F. (2002). Infants’ ability to match dynamic phonetic and gender information in the face and voice. Journal of Experimental Child Psychology, 81(1), 93–115.PubMedCrossRefPubMedCentralGoogle Scholar
  59. Paus, T. (2005). Mapping brain development and aggression. Journal of the Canadian Academy of Child and Adolescent Psychiatry, 14(1), 10–15.Google Scholar
  60. Pick, H. L., Warren, D. H., & Hay, J. (1969). Sensory conflict in judgements of spatial direction. Perception & Psychophysics, 6, 203–205.CrossRefGoogle Scholar
  61. Pouget, A., Beck, J. M., Ma, W. J., & Latham, P. E. (2013). Probabilistic brains: Known and unknowns. Nature Neuroscience, 16, 1170–1178.PubMedPubMedCentralCrossRefGoogle Scholar
  62. Recanzone, G. H. (2003). Auditory influences on visual temporal rate perception. Journal of Neurophysiology, 89(2), 1078–1093.PubMedCrossRefPubMedCentralGoogle Scholar
  63. Rock, I., & Victor, J. (1964). Vision and touch: An experimentally created conflict between the two senses. Science, 143, 594–596.PubMedCrossRefPubMedCentralGoogle Scholar
  64. Rose, S. A. (1981). Developmental changes in infants’ retention of visual stimuli. Child Development, 52(1), 227–233.PubMedCrossRefPubMedCentralGoogle Scholar
  65. Rowland, B., Stanford, T., & Stein, B. (2007). A Bayesian model unifies multisensory spatial localization with the physiological properties of the superior colliculus. Experimental Brain Research, 180(1), 153–161.PubMedCrossRefPubMedCentralGoogle Scholar
  66. Schorr, E. A., Fox, N. A., van Wassenhove, V., & Knudsen, E. I. (2005). Auditory-visual fusion in speech perception in children with cochlear implants. Proceedings of the National Academy of Sciences of the United States of America, 102, 18748–18750.PubMedPubMedCentralCrossRefGoogle Scholar
  67. Shams, L., Kamitani, Y., & Shimojo, S. (2000). Illusions. What you see is what you hear. Nature, 408(6814), 788.CrossRefGoogle Scholar
  68. Shi, Z., & Burr, D. (2015). Predictive coding of multisensory timing. Current Opinion in Behavioral Sciences, 8, 200–206.CrossRefGoogle Scholar
  69. Shipley, T. (1964). Auditory flutter-driving of visual flicker. Science, 145, 1328–1330.PubMedCrossRefPubMedCentralGoogle Scholar
  70. Slutsky, D. A., & Recanzone, G. H. (2001). Temporal and spatial dependency of the ventriloquism effect. Neuroreport, 12(1), 7–10.PubMedCrossRefPubMedCentralGoogle Scholar
  71. Streri, A. (2003). Cross-modal recognition of shape from hand to eyes in human newborns. Somatosensory and Motor Research, 20(1), 13–18.PubMedCrossRefPubMedCentralGoogle Scholar
  72. Tomassini, A., Gori, M., Burr, D., Sandini, G., & Morrone, M. C. (2011). Perceived duration of visual and tactile stimuli depends on perceived speed. Frontiers in Integrative Neuroscience, 5, 51.PubMedPubMedCentralCrossRefGoogle Scholar
  73. Trehub, S. E., Schneider, B. A., & Henderson, J. L. (1995). Gap detection in infants, children, and adults. The Journal of the Acoustical Society of America, 98(5), 2532–2541.PubMedCrossRefPubMedCentralGoogle Scholar
  74. Ulrich, R., Nitschke, J., & Rammsayer, T. (2006). Crossmodal temporal discrimination: Assessing the predictions of a general pacemaker-counter model. Perception & Psychophysics, 68(7), 1140–1152.CrossRefGoogle Scholar
  75. van Beers, R. J., Sittig, A. C., & Gon, J. J. (1999). Integration of proprioceptive and visual position-information: An experimentally supported model. Journal of Neurophysiology, 81(3), 1355–1364.PubMedCrossRefPubMedCentralGoogle Scholar
  76. von Helmholtz, H. (1925). Treatise on physiological optics (Vol. 3). New York: Dover.Google Scholar
  77. Vroomen, J., Bertelson, P., & de Gelder, B. (2001). The ventriloquist effect does not depend on the direction of automatic visual attention. Perception & Psychophysics, 63(4), 651–659.CrossRefGoogle Scholar
  78. Warren, D. H., Welch, R. B., & McCarthy, T. J. (1981). The role of visual-auditory “compellingness” in the ventriloquism effect: Implications for transitivity among the spatial senses. Perception & Psychophysics, 30, 557–564.CrossRefGoogle Scholar
  79. Welch, R. B., & Warren, D. H. (1980). Immediate perceptual response to intersensory discrepancy. Psychological Bulletin, 88(3), 638–667.PubMedCrossRefPubMedCentralGoogle Scholar
  80. Wozny, D. R., Beierholm, U. R., & Shams, L. (2008). Human trimodal perception follows optimal statistical inference. Journal of Vision, 8(3), 24.1–24.11.Google Scholar
  81. Xiang, N., & Fackler, C. (2015). Objective Bayesian analysis in acoustics. Acoustics Today, 11(2), 54–61.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.School of PsychologyThe University of SydneySydneyAustralia
  2. 2.Neuroscience Institute, National Research CouncilPisaItaly
  3. 3.Department of NeuroscienceUniversity of FlorenceFlorenceItaly

Personalised recommendations