Cross-modal representations in early visual and auditory cortices revealed by multi-voxel pattern analysis

  • Jin Gu
  • Baolin LiuEmail author
  • Xianglin Li
  • Peiyuan Wang
  • Bin Wang


Primary sensory cortices can respond not only to their defined sensory modality but also to cross-modal information. In addition to the observed cross-modal phenomenon, it is valuable to research further whether cross-modal information can be valuable for categorizing stimuli and what effect other factors, such as experience and imagination, may have on cross-modal processing. In this study, we researched cross-modal information processing in the early visual cortex (EVC, including the visual area 1, 2, and 3 (V1, V2, and V3)) and auditory cortex (primary (A1) and secondary (A2) auditory cortex). Images and sound clips were presented to participants separately in two experiments in which participants’ imagination and expectations were restricted by an orthogonal fixation task and the data were collected by functional magnetic resonance imaging (fMRI). We successfully decoded categories of the cross-modal stimuli in the ROIs except for V1 by multi-voxel pattern analysis (MVPA). It was further shown that familiar sounds had the advantage of classification accuracies in V2 and V3 when compared with unfamiliar sounds. The results of the cross-classification analysis showed that there was no significant similarity between the activity patterns induced by different stimulus modalities. Even though the cross-modal representation is robust when considering the restriction of top-down expectations and mental imagery in our experiments, the sound experience showed effects on cross-modal representation in V2 and V3. In addition, primary sensory cortices may receive information from different modalities in different ways, so the activity patterns between two modalities were not similar enough to complete the cross-classification successfully.


Cross-modal Auditory cortex Early visual cortex MVPA fMRI 



This work was supported by the National Natural Science Foundation of China (No. U1736219 and No.61571327).

Compliance with ethical standards

Conflict of interest

Jin Gu, Baolin Liu, Xianglin Li, Peiyuan Wang, Bin Wang declare that they have no actual or potential conflict of interest including any financial, personal or other relationships with other people or organizations that can inappropriately influence our work. All of the authors declare that the work described in the manuscript was original research that has not been published previously, and was not under consideration for publication elsewhere, in whole or in part.

Ethical approval

This study was approved by the Research Ethics Committee of Tianjin University. All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, and the applicable revisions at the time of the investigation.

Informed consent

Informed consent was obtained from all subjects for being included in the study.


  1. Albers, A. M., Kok, P., Toni, I., Dijkerman, H. C., & de Lange, F. P. (2013). Shared representations for working memory and mental imagery in early visual cortex. Current Biology, 23(15), 1427–1431. Scholar
  2. Axelrod, V., & Yovel, G. (2012). Hierarchical processing of face viewpoint in human visual cortex. Journal of Neuroscience, 32(7), 2442–2452. Scholar
  3. Beer, A. L., Plank, T., & Greenlee, M. W. (2011). Diffusion tensor imaging shows white matter tracts between human auditory and visual cortex. Experimental Brain Research, 213(2–3), 299–308. Scholar
  4. Beer, A. L., Plank, T., Meyer, G., & Greenlee, M. W. (2013). Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal-occipital network involved in auditory-visual object processing. Frontiers in Integrative Neuroscience, 7, 5. Scholar
  5. Benetti, S., van Ackeren, M. J., Rabini, G., Zonca, J., Foa, V., Baruffaldi, F., et al. (2017). Functional selectivity for face processing in the temporal voice area of early deaf individuals. Proceedings of the National Academy of Sciences of the United States of America, 114(31), E6437–E6446. Scholar
  6. Bieler, M., Sieben, K., Cichon, N., Schildt, S., Roder, B., & Hanganu-Opatz, I. L. (2017a). Rate and temporal coding convey multisensory information in primary sensory cortices. eNeuro, 4(2).
  7. Bieler, M., Sieben, K., Schildt, S., Roder, B., & Hanganu-Opatz, I. L. (2017b). Visual-tactile processing in primary somatosensory cortex emerges before cross-modal experience. Synapse, 71(6).
  8. Braga, R. M., Hellyer, P. J., Wise, R. J., & Leech, R. (2017). Auditory and visual connectivity gradients in frontoparietal cortex. Human Brain Mapping, 38(1), 255–270. Scholar
  9. Chang, C.-C., & Lin, C.-J. (2011). LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(27), 21–27:27.Google Scholar
  10. Collignon, O., Dormal, G., Albouy, G., Vandewalle, G., Voss, P., Phillips, C., et al. (2013). Impact of blindness onset on the functional organization and the connectivity of the occipital cortex. Brain, 136(Pt9), 2769–2783. Scholar
  11. Damasio, A. R. (1989). Time-locked multiregional retroactivation: A systems-level proposal for the neural substrates of recall and recognition. Cognition, 33(1–2), 25–62.CrossRefGoogle Scholar
  12. de Haas, B., Schwarzkopf, D. S., Urner, M., & Rees, G. (2013). Auditory modulation of visual stimulus encoding in human retinotopic cortex. NeuroImage, 70, 258–267. Scholar
  13. Driver, J., & Noesselt, T. (2008). Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron, 57(1), 11–23. Scholar
  14. Eckert, M. A., Kamdar, N. V., Chang, C. E., Beckmann, C. F., Greicius, M. D., & Menon, V. (2008). A cross-modal system linking primary auditory and visual cortices: evidence from intrinsic fMRI connectivity analysis. Human Brain Mapping, 29(7), 848–857. Scholar
  15. Eickhoff, S. B., Stephan, K. E., Mohlberg, H., Grefkes, C., Fink, G. R., Amunts, K., et al. (2005). A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. NeuroImage, 25(4), 1325–1335. Scholar
  16. Geng, X., Xu, J., Liu, B., & Shi, Y. (2018). Multivariate classification of major depressive disorder using the effective connectivity and functional connectivity. Frontiers in Neuroscience, 12, 38. Scholar
  17. Ghazanfar, A. A., & Schroeder, C. E. (2006). Is neocortex essentially multisensory? Trends in Cognitive Sciences, 10(6), 278–285. Scholar
  18. Harrison, S. A., & Tong, F. (2009). Decoding reveals the contents of visual working memory in early visual areas. Nature, 458(7238), 632–635. Scholar
  19. Hsieh, P. J., Colas, J. T., & Kanwisher, N. (2012). Spatial pattern of BOLD fMRI activation reveals cross-modal information in auditory cortex. Journal of Neurophysiology, 107(12), 3428–3432. Scholar
  20. Iordan, M. C., Greene, M. R., Beck, D. M., & Fei-Fei, L. (2015). Basic level category structure emerges gradually across human ventral visual cortex. Journal of Cognitive Neuroscience, 27(7), 1427–1446. Scholar
  21. Iordan, M. C., Greene, M. R., Beck, D. M., & Fei-Fei, L. (2016). Typicality sharpens category representations in object-selective cortex. NeuroImage, 134, 170–179. Scholar
  22. Iurilli, G., Ghezzi, D., Olcese, U., Lassi, G., Nazzaro, C., Tonini, R., et al. (2012). Sound-driven synaptic inhibition in primary visual cortex. Neuron, 73(4), 814–828. Scholar
  23. Kayser, C., Petkov, C. I., & Logothetis, N. K. (2008). Visual modulation of neurons in auditory cortex. Cerebral Cortex, 18(7), 1560–1574. Scholar
  24. Klemen, J., & Chambers, C. D. (2012). Current perspectives and methods in studying neural mechanisms of multisensory interactions. Neuroscience and Biobehavioral Reviews, 36(1), 111–133. Scholar
  25. Klinge, C., Eippert, F., Roder, B., & Buchel, C. (2010). Corticocortical connections mediate primary visual cortex responses to auditory stimulation in the blind. Journal of Neuroscience, 30(38), 12798–12805. Scholar
  26. Kok, P., Failing, M. F., & de Lange, F. P. (2014). Prior expectations evoke stimulus templates in the primary visual cortex. Journal of Cognitive Neuroscience, 26(7), 1546–1554. Scholar
  27. Lee, H. K., & Whitt, J. L. (2015). Cross-modal synaptic plasticity in adult primary sensory cortices. Current Opinion in Neurobiology, 35, 119–126. Scholar
  28. Liang, M., Mouraux, A., Hu, L., & Iannetti, G. D. (2013a). Primary sensory cortices contain distinguishable spatial patterns of activity for each sense. Nature Communications, 4, 1979. Scholar
  29. Liang, M., Mouraux, A., & Iannetti, G. D. (2013b). Bypassing primary sensory cortices--a direct thalamocortical pathway for transmitting salient sensory information. Cerebral Cortex, 23(1), 1–11. Scholar
  30. Liang, Y., Liu, B., Xu, J., Zhang, G., Li, X., Wang, P., et al. (2017). Decoding facial expressions based on face-selective and motion-sensitive areas. Human Brain Mapping, 38(6), 3113–3125. Scholar
  31. Liang, Y., Liu, B., Li, X., & Wang, P. (2018). Multivariate pattern classification of facial expressions based on large-scale functional connectivity. Frontiers in Human Neuroscience, 12.
  32. Lo Verde, L., Morrone, M. C., & Lunghi, C. (2017). Early cross-modal plasticity in adults. Journal of Cognitive Neuroscience, 29(3), 520–529. Scholar
  33. Man, K., Kaplan, J. T., Damasio, A., & Meyer, K. (2012). Sight and sound converge to form modality-invariant representations in temporoparietal cortex. Journal of Neuroscience, 32(47), 16629–16636. Scholar
  34. Man, K., Damasio, A., Meyer, K., & Kaplan, J. T. (2015). Convergent and invariant object representations for sight, sound, and touch. Human Brain Mapping, 36(9), 3629–3640. Scholar
  35. Mesulam, M. M. (1998). From sensation to cognition. Brain, 121(Pt 6), 1013–1052. CrossRefGoogle Scholar
  36. Meyer, K., & Damasio, A. (2009). Convergence and divergence in a neural architecture for recognition and memory. Trends in Neurosciences, 32(7), 376–382. Scholar
  37. Meyer, K., Kaplan, J. T., Essex, R., Webber, C., Damasio, H., & Damasio, A. (2010). Predicting visual stimuli on the basis of activity in auditory cortices. Nature Neuroscience, 13(6), 667–668. Scholar
  38. Meyer, K., Kaplan, J. T., Essex, R., Damasio, H., & Damasio, A. (2011). Seeing touch is correlated with content-specific activity in primary somatosensory cortex. Cerebral Cortex, 21(9), 2113–2121. Scholar
  39. Petro, L. S., Paton, A. T., & Muckli, L. (2017). Contextual modulation of primary visual cortex by auditory signals. In Philosophical transactions of the Royal Society of London. Series B, Biological sciences. Scholar
  40. Renier, L., De Volder, A. G., & Rauschecker, J. P. (2014). Cortical plasticity and preserved function in early blindness. Neuroscience and Biobehavioral Reviews, 41, 53–63. Scholar
  41. Ricciardi, E., Bonino, D., Pellegrini, S., & Pietrini, P. (2014). Mind the blind brain to understand the sighted one! Is there a supramodal cortical functional architecture? Neuroscience and Biobehavioral Reviews, 41, 64–77. Scholar
  42. Rohe, T., & Noppeney, U. (2016). Distinct computational principles govern multisensory integration in primary sensory and association cortices. Current Biology, 26(4), 509–514. Scholar
  43. Sieben, K., Roder, B., & Hanganu-Opatz, I. L. (2013). Oscillatory entrainment of primary somatosensory cortex encodes visual control of tactile processing. Journal of Neuroscience, 33(13), 5736–5749. Scholar
  44. Sieben, K., Bieler, M., Roder, B., & Hanganu-Opatz, I. L. (2015). Neonatal restriction of tactile inputs leads to long-lasting impairments of cross-modal processing. PLoS Biology, 13(11), e1002304. Scholar
  45. Smith, F. W., & Goodale, M. A. (2015). Decoding visual object categories in early somatosensory cortex. Cerebral Cortex, 25(4), 1020–1031. Scholar
  46. Vetter, P., Smith, F. W., & Muckli, L. (2014). Decoding sound and imagery content in early visual cortex. Current Biology, 24(11), 1256–1262. Scholar
  47. Werner, S., & Noppeney, U. (2010). Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization. Journal of Neuroscience, 30(7), 2662–2675. Scholar
  48. Wolbers, T., Zahorik, P., & Giudice, N. A. (2011). Decoding the direction of auditory motion in blind humans. NeuroImage, 56(2), 681–687. Scholar
  49. Xu, J., Yin, X., Ge, H., Han, Y., Pang, Z., Liu, B., et al. (2016). Heritability of the effective connectivity in the resting-state default mode network. Cerebral Cortex.
  50. Yang, X., Xu, J., Cao, L., Li, X., Wang, P., Wang, B., et al. (2017). Linear representation of emotions in whole persons by combining facial and bodily expressions in the Extrastriate body area. Frontiers in Human Neuroscience, 11, 653. Scholar
  51. Zeidman, P., Mullally, S. L., & Maguire, E. A. (2015). Constructing, perceiving, and maintaining scenes: Hippocampal activity and connectivity. Cerebral Cortex, 25(10), 3836–3855. Scholar
  52. Zhang, G., Cheng, Y., & Liu, B. (2017). Abnormalities of voxel-based whole-brain functional connectivity patterns predict the progression of hepatic encephalopathy. Brain Imaging and Behavior, 11(3), 784–796. Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.College of Intelligence and Computing, Tianjin Key Laboratory of Cognitive Computing and ApplicationTianjin UniversityTianjinPeople’s Republic of China
  2. 2.School of Computer and Communication EngineeringUniversity of Science and Technology BeijingBeijingPeople’s Republic of China
  3. 3.Medical Imaging Research InstituteBinzhou Medical UniversityYantaiPeople’s Republic of China
  4. 4.Department of RadiologyYantai Affiliated Hospital of Binzhou Medical UniversityYantaiPeople’s Republic of China

Personalised recommendations