When repeatedly exposed to simultaneously presented stimuli, associations between these stimuli are nearly always established, both within as well as between sensory modalities. Such associations guide our subsequent actions and may also play a role in multisensory selection. Thus, crossmodal associations (i.e., associations between stimuli from different modalities) learned in a multisensory interference task might affect subsequent information processing. The aim of this study was to investigate the processing level of multisensory stimuli in multisensory selection by means of crossmodal aftereffects. Either feature or response associations were induced in a multisensory flanker task while the amount of interference in a subsequent crossmodal flanker task was measured. The results of Experiment 1 revealed the existence of crossmodal interference after multisensory selection. Experiments 2 and 3 then went on to demonstrate the dependence of this effect on the perceptual associations between features themselves, rather than on the associations between feature and response. Establishing response associations did not lead to a subsequent crossmodal interference effect (Experiment 2), while stimulus feature associations without response associations (obtained by changing the response effectors) did (Experiment 3). Taken together, this pattern of results suggests that associations in multisensory selection, and the interference of (crossmodal) distractors, predominantly work at the perceptual, rather than at the response, level.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
The onset of the distractor occurred 100 ms before the onset of the target in order to maximize any congruency effects that were observed. Note that congruency effects, unisensory as well as crossmodal, have been shown to increase when the distractor precedes the target, often peaking at an SOA of about 100 ms (e.g., Chen & Spence, 2013, 2018; Flowers & Wilcox, 1982; Shore, Barnes, & Spence, 2006).
An additional analysis with target modality as the between-participants variable was conducted in order to check for modality differences. In this 2 (target modality: visual vs. auditory) × 3 (compatibility: compatible vs. neutral vs. incompatible) MANOVA with Pillai’s trace as the criterion, an effect of compatibility was still obtained, F(2, 24) = 3.74, p = .039, ηp² = .24, but no other effects were significant (all ps > .107). Thus, the compatibility effect was independent of modality. In the following experiments, modality was left out of all analyses, as there was never any effect (all ps > .197), and the size of the compatibility effects was not influenced by modality.
Note that the number of participants was reduced in this sample. However, with the effect size of the (non-significant) crossmodal compatibility effect in Experiment 2 of ηp² = .08 in mind, and given an alpha level of .05 and a desired power of at least 1 − ß > .80 in a repeated-measures MANOVA with three measures, we calculated a new minimum of 23 participants, which is fulfilled (power analyses were run with G*Power Version 18.104.22.168; Erdfelder et al., 1996; Faul, Erdfelder, Lang, & Buchner, 2007).
Barenholtz, E., Lewkowicz, D. J., Davidson, M., & Mavica, L. (2014). Categorical congruence facilitates multisensory associative learning. Psychonomic Bulletin & Review, 21, 1346–1352.
Chen, Y.-C., & Spence, C. (2010). When hearing the bark helps to identify the dog: Semantically-congruent sounds modulate the identification of masked pictures. Cognition, 114, 389–404.
Chen, Y.-C., & Spence, C. (2011). Crossmodal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity. Journal of Experimental Psychology: Human Perception and Performance, 37, 1554–1568.
Chen, Y.-C., & Spence, C. (2013). The time-course of the cross-modal semantic modulation of visual picture processing by naturalistic sounds and spoken words. Multisensory Research, 26, 371–386.
Chen, Y.-C., & Spence, C. (2018). Dissociating the time courses of the cross-modal semantic priming effects elicited by naturalistic sounds and spoken words. Psychonomic Bulletin & Review, 25, 1138–1146.
Erdfelder, E., Faul, F., & Buchner, A. (1996). GPOWER: A general power analysis program. Behavior Research Methods, Instruments, & Computers, 28, 1–11.
Eriksen, B. A., & Eriksen, C. W. (1974). Effects of noise letters upon the identification of a target letter in a nonsearch task. Perception & Psychophysics, 16, 143–149.
Ernst, M. O. (2007). Learning to integrate arbitrary signals from vision and touch. Journal of Vision, 7(7), 1–14.
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.
Fiebelkorn, I. C., Foxe, J. J., & Molholm, S. (2010). Dual mechanisms for the cross-sensory spread of attention: How much do learned associations matter? Cerebral Cortex, 20, 109–120.
Flowers, J. H., & Wilcox, N. (1982). The effect of flanking context on visual classification: The joint contribution of interactions at different processing levels. Perception & Psychophysics, 32, 581–591.
Frings, C., Koch, I., Rothermund, K., Dignath, D., Giesen, C., Hommel, B., . . . Philipp, A. (in press). Merkmalsintegration und Abruf als wichtige Prozesse der Handlungssteuerung – eine Paradigmen-übergreifende Perspektive [Feature integration and retrieval as core processes of action control – an across paradigm perspective]. Psychologische Rundschau.
Frings, C., & Spence, C. (2010). Crossmodal congruency effects based on stimulus identity. Brain Research, 1354, 113–122.
Guo, J., & Guo, A. (2005). Crossmodal interactions between olfactory and visual learning in Drosophila. Science, 309, 307.
Hall, G. (1991). Oxford psychology series, No. 18: Perceptual and associative learning. New York, NY: Oxford University Press.
Hall, G., & Honey, R. (2014). Perceptual and associative learning. In S. B. Klein & R. R. Mowrer (Eds.), Contemporary learning theories: Volume II: Instrumental conditioning theory and the impact of biological constraints on learning (pp. 117–148). New York, NY: Psychology Press.
Hommel, B. (2004). Event files: Feature binding in and across perception and action. Trends in Cognitive Sciences, 8, 494–500.
Hommel, B. (2009). Action control according to TEC (theory of event coding). Psychological Research, 73, 512–526.
Hommel, B. (2019). Theory of event coding (TEC) V2. 0: Representing and controlling perception and action. Attention, Perception, & Psychophysics, 1–16. doi:https://doi.org/10.3758/s13414-019-01779-4
Hommel, B., Müsseler, J., Aschersleben, G., & Prinz, W. (2001). The theory of event coding (TEC): A framework for perception and action planning. Behavioral and Brain Sciences, 24(5), 849–878.
Jensen, A., Merz, S., Spence, C., & Frings, C. (2019). Overt spatial attention modulates multisensory selection. Journal of Experimental Psychology: Human Perception and Performance, 45, 174–188.
Jonides, J., & Mack, R. (1984). On the cost and benefit of cost and benefit. Psychological Bulletin, 96, 29–44.
Knoeferle, K., Knoeferle, P., Velasco, C., & Spence, C. (2016). Multisensory brand search: How the meaning of sounds guides consumers' visual attention. Journal of Experimental Psychology: Applied, 22, 196–210.
Lehmann, S., & Murray, M. M. (2005). The role of multisensory memories in unisensory object discrimination. Cognitive Brain Research, 24, 326–334.
Mackintosh, N. J. (1965). Selective attention in animal discrimination learning. Psychological Bulletin, 64, 124–150.
McLaren, I. P. L., & Mackintosh, N. J. (2000). An elemental model of associative learning: I. Latent inhibition and perceptual learning. Animal Learning & Behavior, 28, 211–246.
Merz, S., Jensen, A., Spence, C., & Frings, C. (2019). Multisensory distractor processing is modulated by spatial attention. Journal of Experimental Psychology: Human Perception and Performance. Advance online publication. doi:https://doi.org/10.1037/xhp0000678
Molholm, S., Martinez, A., Shpaner, M., & Foxe, J. J. (2007). Object-based attention is multisensory: Co-activation of an object’s representations in ignored sensory modalities. European Journal of Neuroscience, 26, 449–509.
Molholm, S., Ritter, W., Javitt, D. C., & Foxe, J. J. (2004). Multisensory visual-auditory object recognition in humans: A high-density electrical mapping study. Cerebral Cortex, 14, 452–465.
Murray, M. M., Foxe, J. J., & Wylie, G. R. (2005). The brain uses single-trial multisensory memories to discriminate without awareness. NeuroImage, 27, 473–478.
Murray, M. M., Lewkowicz, D. J., Amedi, A., & Wallace, M. T. (2016). Multisensory processes: A balancing act across the lifespan. Trends in Neurosciences, 39, 567–579.
Murray, M. M., Michel, C., M., Grave de Peralta, R., Ortigue, S., Brunet, D., Gonzalez Andino, S., & Schnider, A. (2004). Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. NeuroImage, 21, 125–135.
Pavlov, I. P. (1927). Conditioned reflexes. London, UK: Oxford University Press.
Prinz, W. (1990). A common coding approach to perception and action. In O. Neumann & W. Prinz (Eds.), Relationships between perception and action (pp. 167–201). Berlin, Germany: Springer.
Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current research and theory (pp. 64–99). New York, NY: Appleton-Century-Crofts.
Seitz, A. R., Kim, R., & Shams, L. (2006). Sound facilitates visual learning. Current Biology, 16, 1422–1427.
Shams, L., & Seitz, A. R. (2008). Benefits of multisensory learning. Trends in Cognitive Sciences, 12, 411–417.
Shore, D. I., Barnes, M. E., & Spence, C. (2006). Temporal aspects of the visuotactile congruency effect. Neuroscience Letters, 392, 96–100.
Spence, C. (2011). Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics, 73, 971–995.
Spence, C., & Frings, C. (in press). Multisensory feature integration in (and out) of the focus of spatial attention. Attention, Perception & Psychophysics.
Spence, C., Pavani, F., & Driver, J. (2004). Spatial constraints on visual-tactile cross-modal distractor congruency effects. Cognitive, Affective, & Behavioral Neuroscience, 4, 148–169.
Treisman, A. (1988). Features and objects: The fourteenth Bartlett memorial lecture. The Quarterly Journal of Experimental Psychology, 40, 201–237.
Tukey, J. W. (1977). Exploratory data analysis. Reading, MA: Addison Wesley.
Von Kriegstein, K., & Giraud, A.-L. (2006). Implicit multisensory associations influence voice recognition. PLOS Biology, 4, 1809–1820.
Wagenmakers, E. J., Love, J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Selker, R., . . . Morey, R. D. (2018b). Bayesian inference for psychology. Part II: Example applications with JASP. Psychonomic Bulletin & Review, 25, 58–76.
Wagenmakers, E. J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., . . . Morey, R. D. (2018a). Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications. Psychonomic Bulletin & Review, 25, 35–57.
Wagner, A. R. (1969a). Stimulus selection and a “modified continuity theory”. In G. H. Bower & J. T. Spence (Eds.), The psychology of learning and motivation (Vol. 3, pp. 1–41). New York, NY: Academic Press.
Wagner, A. R. (1969b). Stimulus validity and stimulus selection. In W. K. Honig & N. J. Mackintosh (Eds.), Fundamental issues in associative learning (pp. 90–122). Halifax, Nova Scotia: Dalhousie University Press.
Walker-Andrews, A. (1994). Taxonomy for intermodal relations. In D. J. Lewkowicz & R. Lickliter (Eds.), The development of intersensory perception: Comparative perspectives (pp. 39–56). Hillsdale, NJ: Erlbaum.
Wesslein, A. K., Spence, C., & Frings, C. (2014). When vision influences the invisible distractor: Tactile response compatibility effects require vision. Journal of Experimental Psychology: Human Perception and Performance, 40, 763–774.
Zangenehpour, S., & Zatorre, R. J. (2010). Crossmodal recruitment of primary visual cortex following brief exposure to bimodal audiovisual stimuli. Neuropsychologia, 48, 591–600.
Zhang, H., Zhang, J., & Kornblum, S. (1999). A parallel distributed processing model of stimulus–stimulus and stimulus–response compatibility. Cognitive Psychology, 38, 386–432.
Anne Jensen, Simon Merz, and Christian Frings, University of Trier, Department of Psychology, D-54286, Germany, and Charles Spence, Crossmodal Research Laboratory, Department of Experimental Psychology, Anna Watts Building, University of Oxford, Radcliffe Observatory Quarter, Woodstock Road, Oxford, OX2 6GG, UK.
The research reported in this article was supported by a grant from the Deutsche Forschungsgemeinschaft to Christian Frings and Charles Spence (FR 2133/5-3).
Open practices statement
The data and codes for all experiments are available at PsychArchives (https://doi.org/10.23668/psycharchives.2465).
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Statement of significance
This study demonstrates that crossmodal distractor interference induced by the learning of multisensory associations predominantly operates on a perceptual, rather than on a response, level. Crossmodal interference was elicited when features in a previous multisensory task were frequently presented together. Interference was still found when the response changed across tasks, but not when associations between stimulus features were counterbalanced. These findings provide novel insight concerning the processing of multisensory stimuli. Additionally, such insights may have significant implications for the design of future multisensory alerts and machine interfaces.
About this article
Cite this article
Jensen, A., Merz, S., Spence, C. et al. Perception it is: Processing level in multisensory selection. Atten Percept Psychophys 82, 1391–1406 (2020). https://doi.org/10.3758/s13414-019-01830-4
- Multisensory perception
- Multisensory selection
- Distractor processing
- Multisensory associations
- Processing level