Cross-modal representations in early visual and auditory cortices revealed by multi-voxel pattern analysis
Primary sensory cortices can respond not only to their defined sensory modality but also to cross-modal information. In addition to the observed cross-modal phenomenon, it is valuable to research further whether cross-modal information can be valuable for categorizing stimuli and what effect other factors, such as experience and imagination, may have on cross-modal processing. In this study, we researched cross-modal information processing in the early visual cortex (EVC, including the visual area 1, 2, and 3 (V1, V2, and V3)) and auditory cortex (primary (A1) and secondary (A2) auditory cortex). Images and sound clips were presented to participants separately in two experiments in which participants’ imagination and expectations were restricted by an orthogonal fixation task and the data were collected by functional magnetic resonance imaging (fMRI). We successfully decoded categories of the cross-modal stimuli in the ROIs except for V1 by multi-voxel pattern analysis (MVPA). It was further shown that familiar sounds had the advantage of classification accuracies in V2 and V3 when compared with unfamiliar sounds. The results of the cross-classification analysis showed that there was no significant similarity between the activity patterns induced by different stimulus modalities. Even though the cross-modal representation is robust when considering the restriction of top-down expectations and mental imagery in our experiments, the sound experience showed effects on cross-modal representation in V2 and V3. In addition, primary sensory cortices may receive information from different modalities in different ways, so the activity patterns between two modalities were not similar enough to complete the cross-classification successfully.
KeywordsCross-modal Auditory cortex Early visual cortex MVPA fMRI
This work was supported by the National Natural Science Foundation of China (No. U1736219 and No.61571327).
Compliance with ethical standards
Conflict of interest
Jin Gu, Baolin Liu, Xianglin Li, Peiyuan Wang, Bin Wang declare that they have no actual or potential conflict of interest including any financial, personal or other relationships with other people or organizations that can inappropriately influence our work. All of the authors declare that the work described in the manuscript was original research that has not been published previously, and was not under consideration for publication elsewhere, in whole or in part.
This study was approved by the Research Ethics Committee of Tianjin University. All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, and the applicable revisions at the time of the investigation.
Informed consent was obtained from all subjects for being included in the study.
- Beer, A. L., Plank, T., Meyer, G., & Greenlee, M. W. (2013). Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal-occipital network involved in auditory-visual object processing. Frontiers in Integrative Neuroscience, 7, 5. https://doi.org/10.3389/fnint.2013.00005.CrossRefGoogle Scholar
- Benetti, S., van Ackeren, M. J., Rabini, G., Zonca, J., Foa, V., Baruffaldi, F., et al. (2017). Functional selectivity for face processing in the temporal voice area of early deaf individuals. Proceedings of the National Academy of Sciences of the United States of America, 114(31), E6437–E6446. https://doi.org/10.1073/pnas.1618287114.CrossRefGoogle Scholar
- Bieler, M., Sieben, K., Cichon, N., Schildt, S., Roder, B., & Hanganu-Opatz, I. L. (2017a). Rate and temporal coding convey multisensory information in primary sensory cortices. eNeuro, 4(2). https://doi.org/10.1523/ENEURO.0037-17.2017.
- Bieler, M., Sieben, K., Schildt, S., Roder, B., & Hanganu-Opatz, I. L. (2017b). Visual-tactile processing in primary somatosensory cortex emerges before cross-modal experience. Synapse, 71(6). https://doi.org/10.1002/syn.21958.
- Chang, C.-C., & Lin, C.-J. (2011). LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(27), 21–27:27.Google Scholar
- Eckert, M. A., Kamdar, N. V., Chang, C. E., Beckmann, C. F., Greicius, M. D., & Menon, V. (2008). A cross-modal system linking primary auditory and visual cortices: evidence from intrinsic fMRI connectivity analysis. Human Brain Mapping, 29(7), 848–857. https://doi.org/10.1002/hbm.20560.CrossRefGoogle Scholar
- Liang, Y., Liu, B., Li, X., & Wang, P. (2018). Multivariate pattern classification of facial expressions based on large-scale functional connectivity. Frontiers in Human Neuroscience, 12. https://doi.org/10.3389/fnhum.2018.00094.
- Xu, J., Yin, X., Ge, H., Han, Y., Pang, Z., Liu, B., et al. (2016). Heritability of the effective connectivity in the resting-state default mode network. Cerebral Cortex. https://doi.org/10.1093/cercor/bhw332.
- Yang, X., Xu, J., Cao, L., Li, X., Wang, P., Wang, B., et al. (2017). Linear representation of emotions in whole persons by combining facial and bodily expressions in the Extrastriate body area. Frontiers in Human Neuroscience, 11, 653. https://doi.org/10.3389/fnhum.2017.00653.CrossRefGoogle Scholar