Hearing in a “Moving” Visual World: Coordinate Transformations Along the Auditory Pathway

  • Shawn M. WillettEmail author
  • Jennifer M. Groh
  • Ross K. Maddox
Part of the Springer Handbook of Auditory Research book series (SHAR, volume 68)


This chapter reviews the literature on how auditory signals are transformed into a coordinate system that facilitates interactions with the visual system. Sound location is deduced from cues that depend on the position of the sound with respect to the head, but visual location is deduced from the pattern of light illuminating the retina, yielding an eye-centered code. Connecting sights and sounds originating from the same position in the physical world requires the brain to incorporate information about the position of the eyes with respect to the head. Eye position has been found to interact with auditory signals at all levels of the auditory pathway that have been tested but usually yields a code that is in a hybrid reference frame: neither head nor eye centered. Computing a coordinate transformation, in principle, may be easy, which could suggest that the looseness of the computational constraints may permit hybrid coding. A review of the behavioral literature addressing the effects of eye gaze on auditory spatial perception and a discussion of its consistency with physiological observations concludes the chapter.


Eye centered Frontal eye field Head centered Hybrid Inferior colliculus Intraparietal cortex Multimodal Multisensory integration Reference frame Sound localization Superior colliculus 


  1. Andersen, R. A., & Buneo, C. A. (2002). Intentional maps in posterior parietal cortex. Annual Review of Neuroscience, 25(1), 189–220.PubMedGoogle Scholar
  2. Andersen, R. A., & Mountcastle, V. B. (1983). The influence of the angle of gaze upon the excitability of the light-sensitive neurons of the posterior parietal cortex. The Journal of Neuroscience, 3(3), 532–548.PubMedPubMedCentralGoogle Scholar
  3. Andersen, R. A., Essick, G. K., & Siegel, R. M. (1985). Encoding of spatial location by posterior parietal neurons. Science, 230(4724), 456–458.PubMedGoogle Scholar
  4. Best, V., Ozmeral, E. J., & Shinn-Cunningham, B. G. (2007). Visually-guided attention enhances target identification in a complex auditory scene. Journal of the Association for Research in Otolaryngology, 8(2), 294–304.PubMedPubMedCentralGoogle Scholar
  5. Bohlander, R. W. (1984). Eye position and visual attention influence perceived auditory direction. Perceptual and Motor Skills, 59(2), 483–510.PubMedGoogle Scholar
  6. Caruso, V., Pages, D. S., Sommer, M., & Groh, J. M. (2017). Beyond the labeled line: Variation in visual reference frames from intraparietal cortex to frontal eye fields and the superior colliculus. Journal of Neurophysiology, 119(4), 1411–1421.PubMedPubMedCentralGoogle Scholar
  7. Chalupa, L. M., & Rhoades, R. W. (1977). Responses of visual, somatosensory, and auditory neurones in the golden hamster’s superior colliculus. The Journal of Physiology, 270(3), 595–626.PubMedPubMedCentralGoogle Scholar
  8. Cui, Q. N., O’Neill, W. E., & Paige, G. D. (2010a). Advancing age alters the influence of eye position on sound localization. Experimental Brain Research, 206(4), 371–379.PubMedPubMedCentralGoogle Scholar
  9. Cui, Q. N., Razavi, B., O’Neill, W. E., & Paige, G. D. (2010b). Perception of auditory, visual, and egocentric spatial alignment adapts differently to changes in eye position. Journal of Neurophysiology, 103(2), 1020–1035.PubMedGoogle Scholar
  10. Deneve, S., Latham, P. E., & Pouget, A. (2001). Efficient computation and cue integration with noisy population codes. Nature Neuroscience, 4(8), 826–831.PubMedGoogle Scholar
  11. Drager, U. C., & Hubel, D. H. (1975). Responses to visual stimulation and relationship between visual, auditory, and somatosensory inputs in mouse superior colliculus. Journal of Neurophysiology, 38(3), 690–713.PubMedGoogle Scholar
  12. Fu, K.-M. G., Shah, A. S., O’Connell, M. N., McGinnis, T., Eckholdt, H., Lakatos, P., Smiley, J., & Schroeder, C. E. (2004). Timing and laminar profile of eye-position effects on auditory responses in primate auditory cortex. Journal of Neurophysiology, 92(6), 3522–3531.PubMedGoogle Scholar
  13. Fuchs, A. F., & Luschei, E. S. (1970). Firing patterns of abducens neurons of alert monkeys in relationship to horizontal eye movement. Journal of Neurophysiology, 33(3), 382–392.PubMedGoogle Scholar
  14. Getzmann, S. (2002). The effect of eye position and background noise on vertical sound localization. Hearing Research, 169(1–2), 130–139.PubMedGoogle Scholar
  15. Groh, J. M., & Sparks, D. L. (1992). Two models for transforming auditory signals from head-centered to eye-centered coordinates. Biological Cybernetics, 67(4), 291–302.PubMedGoogle Scholar
  16. Groh, J. M., & Sparks, D. L. (1996). Saccades to somatosensory targets. I. Behavioral characteristics. Journal of Neurophysiology, 75(1), 412–427.PubMedGoogle Scholar
  17. Groh, J. M., Trause, A. S., Underhill, A. M., Clark, K. R., & Inati, S. (2001). Eye position influences auditory responses in primate inferior colliculus. Neuron, 29(2), 509–518.PubMedGoogle Scholar
  18. Groh, J. M., Kelly, K. A., & Underhill, A. M. (2003). A monotonic code for sound azimuth in primate inferior colliculus. Journal of Cognitive Neuroscience, 15(8), 1217–1231.PubMedGoogle Scholar
  19. Grothe, B., & Pecka, M. (2014). The natural history of sound localization in mammals—A story of neuronal inhibition. Frontiers in Neural Circuits, 8(116), 1–19.Google Scholar
  20. Guthrie, B. L., Porter, J. D., & Sparks, D. L. (1983). Corollary discharge provides accurate eye position information to the oculomotor system. Science, 221(4616), 1193–1195.PubMedGoogle Scholar
  21. Hafter, E. R., & Maio, J. D. (1975). Difference thresholds for interaural delay. The Journal of the Acoustical Society of America, 57(1), 181–187.PubMedGoogle Scholar
  22. Jay, M. F., & Sparks, D. L. (1984). Auditory receptive fields in primate superior colliculus shift with changes in eye position. Nature, 309(5966), 345–347.PubMedGoogle Scholar
  23. Jay, M. F., & Sparks, D. L. (1987a). Sensorimotor integration in the primate superior colliculus. I. Motor convergence. Journal of Neurophysiology, 57(1), 22–34.PubMedGoogle Scholar
  24. Jay, M. F., & Sparks, D. L. (1987b). Sensorimotor integration in the primate superior colliculus. II. Coordinates of auditory signals. Journal of Neurophysiology, 57(1), 35–55.PubMedGoogle Scholar
  25. Lee, J., & Groh, J. M. (2012). Auditory signals evolve from hybrid- to eye-centered coordinates in the primate superior colliculus. Journal of Neurophysiology, 108(1), 227–242.PubMedPubMedCentralGoogle Scholar
  26. Lee, J., & Groh, J. M. (2014). Different stimuli, different spatial codes: A visual map and an auditory rate code for oculomotor space in the primate superior colliculus. PLoS One, 9(1), e85017.PubMedPubMedCentralGoogle Scholar
  27. Lewald, J. (1997). Eye-position effects in directional hearing. Behavioural Brain Research, 87(1), 35–48.PubMedGoogle Scholar
  28. Lewald, J. (1998). The effect of gaze eccentricity on perceived sound direction and its relation to visual localization. Hearing Research, 115(1–2), 206–216.PubMedGoogle Scholar
  29. Lewald, J., & Ehrenstein, W. H. (1996). The effect of eye position on auditory lateralization. Experimental Brain Research, 108(3), 473–485.PubMedGoogle Scholar
  30. Lewald, J., & Ehrenstein, W. H. (2001). Effect of gaze direction on sound localization in rear space. Neuroscience Research, 39(2), 253–257.PubMedGoogle Scholar
  31. Lewald, J., & Getzmann, S. (2006). Horizontal and vertical effects of eye-position on sound localization. Hearing Research, 213(1–2), 99–106.PubMedGoogle Scholar
  32. Linden, J. F., Grunewald, A., & Andersen, R. A. (1999). Responses to auditory stimuli in macaque lateral intraparietal area II. Behavioral modulation. Journal of Neurophysiology, 82(1), 343–358.PubMedGoogle Scholar
  33. Luschei, E. S., & Fuchs, A. F. (1972). Activity of brain stem neurons during eye movements of alert monkeys. Journal of Neurophysiology, 35(4), 445–461.PubMedGoogle Scholar
  34. Maddox, R. K., Pospisil, D. A., Stecker, G. C., & Lee, A. K. C. (2014). Directing eye gaze enhances auditory spatial cue discrimination. Current Biology, 24(7), 748–752.PubMedGoogle Scholar
  35. Maier, J. X., & Groh, J. M. (2009). Multisensory guidance of orienting behavior. Hearing Research, 258(1–2), 106–112.PubMedPubMedCentralGoogle Scholar
  36. Maier, J. X., & Groh, J. M. (2010). Comparison of gain-like properties of eye position signals in inferior colliculus versus auditory cortex of primates. Frontiers in Integrative Neuroscience, 4, 121.PubMedPubMedCentralGoogle Scholar
  37. Marrone, N., Mason, C. R., & Kidd, G. (2008). Tuning in the spatial dimension: Evidence from a masked speech identification task. The Journal of the Acoustical Society of America, 124(2), 1146–1158.PubMedPubMedCentralGoogle Scholar
  38. Mays, L. E., & Sparks, D. L. (1980). Dissociation of visual and saccade-related responses in superior colliculus neurons. Journal of Neurophysiology, 43(1), 207–232.PubMedGoogle Scholar
  39. McAlpine, D., & Grothe, B. (2003). Sound localization and delay lines—Do mammals fit the model? Trends in Neurosciences, 26(7), 347–350.PubMedGoogle Scholar
  40. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746–748.PubMedGoogle Scholar
  41. Meredith, A. M., & Stein, B. E. (1986a). Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Research, 365(2), 350–354.PubMedGoogle Scholar
  42. Meredith, M. A., & Stein, B. E. (1986b). Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. Journal of Neurophysiology, 56(3), 640–662.PubMedGoogle Scholar
  43. Metzger, R. R., Mullette-Gillman, O. D. A., Underhill, A. M., Cohen, Y. E., & Groh, J. M. (2004). Auditory saccades from different eye positions in the monkey: Implications for coordinate transformations. Journal of Neurophysiology, 92(4), 2622–2627.PubMedGoogle Scholar
  44. Middlebrooks, J. C., & Onsan, Z. A. (2012). Stream segregation with high spatial acuity. The Journal of the Acoustical Society of America, 132(6), 3896–3911.PubMedPubMedCentralGoogle Scholar
  45. Middlebrooks, J. C., Clock, A. E., Xu, L., & Green, D. M. (1994). A panoramic code for sound location by cortical neurons. Science, 264(5160), 842–843.PubMedGoogle Scholar
  46. Middlebrooks, J. C., Xu, L., Eddins, A. C., & Green, D. M. (1998). Codes for sound-source location in nontonotopic auditory cortex. Journal of Neurophysiology, 80(2), 863–881.PubMedGoogle Scholar
  47. Mills, A. W. (1958). On the minimum audible angle. The Journal of the Acoustical Society of America, 30(4), 237–246.Google Scholar
  48. Mohler, C. W., Goldberg, M. E., & Wurtz, R. H. (1973). Visual receptive fields of frontal eye field neurons. Brain Research, 61, 385–389.PubMedGoogle Scholar
  49. Mullette-Gillman, O. D. A., Cohen, Y. E., & Groh, J. M. (2005). Eye-centered, head-centered, and complex coding of visual and auditory targets in the intraparietal sulcus. Journal of Neurophysiology, 94(4), 2331–2352.PubMedGoogle Scholar
  50. Mullette-Gillman, O. D. A., Cohen, Y. E., & Groh, J. M. (2009). Motor-related signals in the intraparietal cortex encode locations in a hybrid, rather than eye-centered reference frame. Cerebral Cortex, 19(8), 1761–1775.PubMedGoogle Scholar
  51. Populin, L. C., & Yin, T. C. T. (1998). Sensitivity of auditory cells in the superior colliculus to eye position in the behaving cat. In A. R. Palmer, A. Q. Summerfield, & R. Meddis (Eds.), Psychophysical and physiological advances in hearing (pp. 441–448). London: Whurr.Google Scholar
  52. Populin, L. C., Tollin, D. J., & Yin, T. C. T. (2004). Effect of eye position on saccades and neuronal responses to acoustic stimuli in the superior colliculus of the behaving cat. Journal of Neurophysiology, 92(4), 2151–2167.PubMedGoogle Scholar
  53. Porter, K. K., Metzger, R. R., & Groh, J. M. (2006). Representation of eye position in primate inferior colliculus. Journal of Neurophysiology, 95(3), 1826–1842.PubMedGoogle Scholar
  54. Porter, K. K., Metzger, R. R., & Groh, J. M. (2007). Visual- and saccade-related signals in the primate inferior colliculus. Proceedings of the National Academy of Sciences of the United States of America, 104(45), 17855–17860.PubMedPubMedCentralGoogle Scholar
  55. Pouget, A., & Sejnowski, T. J. (1997). Spatial transformations in the parietal cortex using basis functions. Journal of Cognitive Neuroscience, 9(2), 222–237.PubMedGoogle Scholar
  56. Razavi, B., O’Neill, W. E., & Paige, G. D. (2007). Auditory Spatial perception dynamically realigns with changing eye position. The Journal of Neuroscience, 27(38), 10249–10258.PubMedPubMedCentralGoogle Scholar
  57. Robinson, D. A. (1972). Eye movements evoked by collicular stimulation in the alert monkey. Vision Research, 12(11), 1795–1808.PubMedGoogle Scholar
  58. Robinson, D. A., & Fuchs, A. F. (1969). Eye movements evoked by stimulation of frontal eye fields. Journal of Neurophysiology, 32(5), 637–648.PubMedGoogle Scholar
  59. Russo, G. S., & Bruce, C. J. (1994). Frontal eye field activity preceding aurally guided saccades. Journal of Neurophysiology, 71(3), 1250–1253.PubMedGoogle Scholar
  60. Sajad, A., Sadeh, M., Keith, G. P., Yan, X., Wang, H., & Crawford, J. D. (2015). Visual-motor transformations within frontal eye fields during head-unrestrained gaze shifts in the monkey. Cerebral Cortex, 25(10), 3932–3952.PubMedGoogle Scholar
  61. Schiller, P. H., True, S. D., & Conway, J. L. (1979). Effects of frontal eye field and superior colliculus ablations on eye movements. Science, 206(4418), 590–592.PubMedGoogle Scholar
  62. Schiller, P. H., True, S. D., & Conway, J. L. (1980). Deficits in eye movements following frontal eye-field and superior colliculus ablations. Journal of Neurophysiology, 44(6), 1175–1189.PubMedGoogle Scholar
  63. Sommer, M. A., & Wurtz, R. H. (2008). Brain circuits for the internal monitoring of movements. Annual Review of Neuroscience, 31, 317–338.PubMedPubMedCentralGoogle Scholar
  64. Sparks, D. L. (1975). Response properties of eye movement-related neurons in the monkey superior colliculus. Brain Research, 90(1), 147–152.PubMedGoogle Scholar
  65. Sparks, D. L. (1978). Functional properties of neurons in the monkey superior colliculus: Coupling of neuronal activity and saccade onset. Brain Research, 156(1), 1–16.PubMedGoogle Scholar
  66. Sparks, D. L., & Hartwich-Young, R. (1989). The deep layers of the superior colliculus. Reviews of Oculomotor Research, 3, 213–255.PubMedGoogle Scholar
  67. Stricanne, B., Andersen, R. A., & Mazzoni, P. (1996). Eye-centered, head-centered, and intermediate coding of remembered sound locations in area LIP. Journal of Neurophysiology, 76(3), 2071–2076.PubMedGoogle Scholar
  68. Wann, J. P., & Ibrahim, S. F. (1992). Does limb proprioception drift? Experimental Brain Research, 91(1), 162–166.PubMedGoogle Scholar
  69. Weerts, T. C., & Thurlow, W. R. (1971). The effects of eye position and expectation on sound localization. Perception & Psychophysics, 9(1), 35–39.Google Scholar
  70. Werner-Reiss, U., & Groh, J. M. (2008). A rate code for sound azimuth in monkey auditory cortex: Implications for human neuroimaging studies. The Journal of Neuroscience, 28(14), 3747–3758.PubMedPubMedCentralGoogle Scholar
  71. Werner-Reiss, U., Kelly, K. A., Trause, A. S., Underhill, A. M., & Groh, J. M. (2003). Eye position affects activity in primary auditory cortex of primates. Current Biology, 13(7), 554–562.PubMedGoogle Scholar
  72. Wood, K. C., & Bizley, J. K. (2015). Relative sound localisation abilities in human listeners. The Journal of the Acoustical Society of America, 138(2), 674–686.PubMedPubMedCentralGoogle Scholar
  73. Woods, T. M., Lopez, S. E., Long, J. H., Rahman, J. E., & Recanzone, G. H. (2006). Effects of stimulus azimuth and intensity on the single-neuron activity in the auditory cortex of the alert macaque monkey. Journal of Neurophysiology, 96(6), 3323–3337.PubMedGoogle Scholar
  74. Wurtz, R. H., & Goldberg, M. E. (1972). Activity of superior colliculus in behaving monkey. III. Cells discharging before eye movements. Journal of Neurophysiology, 35(4), 575–586.PubMedGoogle Scholar
  75. Zahn, J. R., Abel, L. A., Dell’Osso, L. F., & Daroff, R. B. (1979). The audioocular response: Intersensory delay. Sensory Processes, 3(1), 60.PubMedGoogle Scholar
  76. Zambarbieri, D., Schmid, R., Magenes, G., & Prablanc, C. (1982). Saccadic responses evoked by presentation of visual and auditory targets. Experimental Brain Research, 47(3), 417–427.PubMedGoogle Scholar
  77. Zwiers, M. P., Versnel, H., & Van Opstal, A. J. (2004). Involvement of monkey inferior colliculus in spatial hearing. The Journal of Neuroscience, 24(17), 4145–4156.PubMedPubMedCentralGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Shawn M. Willett
    • 1
    Email author
  • Jennifer M. Groh
    • 1
    • 2
  • Ross K. Maddox
    • 3
    • 4
    • 5
    • 6
  1. 1.Department of NeurobiologyCenter for Cognitive Neuroscience, Duke UniversityDurhamUSA
  2. 2.Department of Psychology and NeuroscienceCenter for Cognitive Neuroscience, Duke UniversityDurhamUSA
  3. 3.Department of Biomedical EngineeringUniversity of RochesterRochesterUSA
  4. 4.Department of NeuroscienceUniversity of RochesterRochesterUSA
  5. 5.Del Monte Institute for NeuroscienceUniversity of RochesterRochesterUSA
  6. 6.Center for Visual ScienceUniversity of RochesterRochesterUSA

Personalised recommendations