Perception it is: Processing level in multisensory selection

Abstract

When repeatedly exposed to simultaneously presented stimuli, associations between these stimuli are nearly always established, both within as well as between sensory modalities. Such associations guide our subsequent actions and may also play a role in multisensory selection. Thus, crossmodal associations (i.e., associations between stimuli from different modalities) learned in a multisensory interference task might affect subsequent information processing. The aim of this study was to investigate the processing level of multisensory stimuli in multisensory selection by means of crossmodal aftereffects. Either feature or response associations were induced in a multisensory flanker task while the amount of interference in a subsequent crossmodal flanker task was measured. The results of Experiment 1 revealed the existence of crossmodal interference after multisensory selection. Experiments 2 and 3 then went on to demonstrate the dependence of this effect on the perceptual associations between features themselves, rather than on the associations between feature and response. Establishing response associations did not lead to a subsequent crossmodal interference effect (Experiment 2), while stimulus feature associations without response associations (obtained by changing the response effectors) did (Experiment 3). Taken together, this pattern of results suggests that associations in multisensory selection, and the interference of (crossmodal) distractors, predominantly work at the perceptual, rather than at the response, level.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Notes

  1. 1.

    The onset of the distractor occurred 100 ms before the onset of the target in order to maximize any congruency effects that were observed. Note that congruency effects, unisensory as well as crossmodal, have been shown to increase when the distractor precedes the target, often peaking at an SOA of about 100 ms (e.g., Chen & Spence, 2013, 2018; Flowers & Wilcox, 1982; Shore, Barnes, & Spence, 2006).

  2. 2.

    An additional analysis with target modality as the between-participants variable was conducted in order to check for modality differences. In this 2 (target modality: visual vs. auditory) × 3 (compatibility: compatible vs. neutral vs. incompatible) MANOVA with Pillai’s trace as the criterion, an effect of compatibility was still obtained, F(2, 24) = 3.74, p = .039, ηp² = .24, but no other effects were significant (all ps > .107). Thus, the compatibility effect was independent of modality. In the following experiments, modality was left out of all analyses, as there was never any effect (all ps > .197), and the size of the compatibility effects was not influenced by modality.

  3. 3.

    Note that the number of participants was reduced in this sample. However, with the effect size of the (non-significant) crossmodal compatibility effect in Experiment 2 of ηp² = .08 in mind, and given an alpha level of .05 and a desired power of at least 1 − ß > .80 in a repeated-measures MANOVA with three measures, we calculated a new minimum of 23 participants, which is fulfilled (power analyses were run with G*Power Version 3.1.9.2; Erdfelder et al., 1996; Faul, Erdfelder, Lang, & Buchner, 2007).

References

  1. Barenholtz, E., Lewkowicz, D. J., Davidson, M., & Mavica, L. (2014). Categorical congruence facilitates multisensory associative learning. Psychonomic Bulletin & Review, 21, 1346–1352.

    Google Scholar 

  2. Chen, Y.-C., & Spence, C. (2010). When hearing the bark helps to identify the dog: Semantically-congruent sounds modulate the identification of masked pictures. Cognition, 114, 389–404.

    PubMed  Google Scholar 

  3. Chen, Y.-C., & Spence, C. (2011). Crossmodal semantic priming by naturalistic sounds and spoken words enhances visual sensitivity. Journal of Experimental Psychology: Human Perception and Performance, 37, 1554–1568.

    PubMed  Google Scholar 

  4. Chen, Y.-C., & Spence, C. (2013). The time-course of the cross-modal semantic modulation of visual picture processing by naturalistic sounds and spoken words. Multisensory Research, 26, 371–386.

    PubMed  Google Scholar 

  5. Chen, Y.-C., & Spence, C. (2018). Dissociating the time courses of the cross-modal semantic priming effects elicited by naturalistic sounds and spoken words. Psychonomic Bulletin & Review, 25, 1138–1146.

    Google Scholar 

  6. Erdfelder, E., Faul, F., & Buchner, A. (1996). GPOWER: A general power analysis program. Behavior Research Methods, Instruments, & Computers, 28, 1–11.

    Google Scholar 

  7. Eriksen, B. A., & Eriksen, C. W. (1974). Effects of noise letters upon the identification of a target letter in a nonsearch task. Perception & Psychophysics, 16, 143–149.

    Google Scholar 

  8. Ernst, M. O. (2007). Learning to integrate arbitrary signals from vision and touch. Journal of Vision, 7(7), 1–14.

    PubMed  Google Scholar 

  9. Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.

    PubMed  PubMed Central  Google Scholar 

  10. Fiebelkorn, I. C., Foxe, J. J., & Molholm, S. (2010). Dual mechanisms for the cross-sensory spread of attention: How much do learned associations matter? Cerebral Cortex, 20, 109–120.

    PubMed  Google Scholar 

  11. Flowers, J. H., & Wilcox, N. (1982). The effect of flanking context on visual classification: The joint contribution of interactions at different processing levels. Perception & Psychophysics, 32, 581–591.

    Google Scholar 

  12. Frings, C., Koch, I., Rothermund, K., Dignath, D., Giesen, C., Hommel, B., . . . Philipp, A. (in press). Merkmalsintegration und Abruf als wichtige Prozesse der Handlungssteuerung – eine Paradigmen-übergreifende Perspektive [Feature integration and retrieval as core processes of action control – an across paradigm perspective]. Psychologische Rundschau.

  13. Frings, C., & Spence, C. (2010). Crossmodal congruency effects based on stimulus identity. Brain Research, 1354, 113–122.

    PubMed  Google Scholar 

  14. Guo, J., & Guo, A. (2005). Crossmodal interactions between olfactory and visual learning in Drosophila. Science, 309, 307.

    PubMed  Google Scholar 

  15. Hall, G. (1991). Oxford psychology series, No. 18: Perceptual and associative learning. New York, NY: Oxford University Press.

    Google Scholar 

  16. Hall, G., & Honey, R. (2014). Perceptual and associative learning. In S. B. Klein & R. R. Mowrer (Eds.), Contemporary learning theories: Volume II: Instrumental conditioning theory and the impact of biological constraints on learning (pp. 117–148). New York, NY: Psychology Press.

    Google Scholar 

  17. Hommel, B. (2004). Event files: Feature binding in and across perception and action. Trends in Cognitive Sciences, 8, 494–500.

    PubMed  Google Scholar 

  18. Hommel, B. (2009). Action control according to TEC (theory of event coding). Psychological Research, 73, 512–526.

    PubMed  PubMed Central  Google Scholar 

  19. Hommel, B. (2019). Theory of event coding (TEC) V2. 0: Representing and controlling perception and action. Attention, Perception, & Psychophysics, 1–16. doi:https://doi.org/10.3758/s13414-019-01779-4

  20. Hommel, B., Müsseler, J., Aschersleben, G., & Prinz, W. (2001). The theory of event coding (TEC): A framework for perception and action planning. Behavioral and Brain Sciences, 24(5), 849–878.

    PubMed  Google Scholar 

  21. Jensen, A., Merz, S., Spence, C., & Frings, C. (2019). Overt spatial attention modulates multisensory selection. Journal of Experimental Psychology: Human Perception and Performance, 45, 174–188.

    PubMed  Google Scholar 

  22. Jonides, J., & Mack, R. (1984). On the cost and benefit of cost and benefit. Psychological Bulletin, 96, 29–44.

    Google Scholar 

  23. Knoeferle, K., Knoeferle, P., Velasco, C., & Spence, C. (2016). Multisensory brand search: How the meaning of sounds guides consumers' visual attention. Journal of Experimental Psychology: Applied, 22, 196–210.

    PubMed  Google Scholar 

  24. Lehmann, S., & Murray, M. M. (2005). The role of multisensory memories in unisensory object discrimination. Cognitive Brain Research, 24, 326–334.

    PubMed  Google Scholar 

  25. Mackintosh, N. J. (1965). Selective attention in animal discrimination learning. Psychological Bulletin, 64, 124–150.

    PubMed  Google Scholar 

  26. McLaren, I. P. L., & Mackintosh, N. J. (2000). An elemental model of associative learning: I. Latent inhibition and perceptual learning. Animal Learning & Behavior, 28, 211–246.

  27. Merz, S., Jensen, A., Spence, C., & Frings, C. (2019). Multisensory distractor processing is modulated by spatial attention. Journal of Experimental Psychology: Human Perception and Performance. Advance online publication. doi:https://doi.org/10.1037/xhp0000678

  28. Molholm, S., Martinez, A., Shpaner, M., & Foxe, J. J. (2007). Object-based attention is multisensory: Co-activation of an object’s representations in ignored sensory modalities. European Journal of Neuroscience, 26, 449–509.

    Google Scholar 

  29. Molholm, S., Ritter, W., Javitt, D. C., & Foxe, J. J. (2004). Multisensory visual-auditory object recognition in humans: A high-density electrical mapping study. Cerebral Cortex, 14, 452–465.

    PubMed  Google Scholar 

  30. Murray, M. M., Foxe, J. J., & Wylie, G. R. (2005). The brain uses single-trial multisensory memories to discriminate without awareness. NeuroImage, 27, 473–478.

    PubMed  Google Scholar 

  31. Murray, M. M., Lewkowicz, D. J., Amedi, A., & Wallace, M. T. (2016). Multisensory processes: A balancing act across the lifespan. Trends in Neurosciences, 39, 567–579.

    PubMed  PubMed Central  Google Scholar 

  32. Murray, M. M., Michel, C., M., Grave de Peralta, R., Ortigue, S., Brunet, D., Gonzalez Andino, S., & Schnider, A. (2004). Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. NeuroImage, 21, 125–135.

  33. Pavlov, I. P. (1927). Conditioned reflexes. London, UK: Oxford University Press.

    Google Scholar 

  34. Prinz, W. (1990). A common coding approach to perception and action. In O. Neumann & W. Prinz (Eds.), Relationships between perception and action (pp. 167–201). Berlin, Germany: Springer.

    Google Scholar 

  35. Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current research and theory (pp. 64–99). New York, NY: Appleton-Century-Crofts.

    Google Scholar 

  36. Seitz, A. R., Kim, R., & Shams, L. (2006). Sound facilitates visual learning. Current Biology, 16, 1422–1427.

    PubMed  Google Scholar 

  37. Shams, L., & Seitz, A. R. (2008). Benefits of multisensory learning. Trends in Cognitive Sciences, 12, 411–417.

    PubMed  Google Scholar 

  38. Shore, D. I., Barnes, M. E., & Spence, C. (2006). Temporal aspects of the visuotactile congruency effect. Neuroscience Letters, 392, 96–100.

    PubMed  Google Scholar 

  39. Spence, C. (2011). Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics, 73, 971–995.

    Google Scholar 

  40. Spence, C., & Frings, C. (in press). Multisensory feature integration in (and out) of the focus of spatial attention. Attention, Perception & Psychophysics.

  41. Spence, C., Pavani, F., & Driver, J. (2004). Spatial constraints on visual-tactile cross-modal distractor congruency effects. Cognitive, Affective, & Behavioral Neuroscience, 4, 148–169.

    Google Scholar 

  42. Treisman, A. (1988). Features and objects: The fourteenth Bartlett memorial lecture. The Quarterly Journal of Experimental Psychology, 40, 201–237.

    PubMed  Google Scholar 

  43. Tukey, J. W. (1977). Exploratory data analysis. Reading, MA: Addison Wesley.

    Google Scholar 

  44. Von Kriegstein, K., & Giraud, A.-L. (2006). Implicit multisensory associations influence voice recognition. PLOS Biology, 4, 1809–1820.

    Google Scholar 

  45. Wagenmakers, E. J., Love, J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Selker, R., . . . Morey, R. D. (2018b). Bayesian inference for psychology. Part II: Example applications with JASP. Psychonomic Bulletin & Review, 25, 58–76.

  46. Wagenmakers, E. J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., Selker, R., . . . Morey, R. D. (2018a). Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications. Psychonomic Bulletin & Review, 25, 35–57.

  47. Wagner, A. R. (1969a). Stimulus selection and a “modified continuity theory”. In G. H. Bower & J. T. Spence (Eds.), The psychology of learning and motivation (Vol. 3, pp. 1–41). New York, NY: Academic Press.

    Google Scholar 

  48. Wagner, A. R. (1969b). Stimulus validity and stimulus selection. In W. K. Honig & N. J. Mackintosh (Eds.), Fundamental issues in associative learning (pp. 90–122). Halifax, Nova Scotia: Dalhousie University Press.

    Google Scholar 

  49. Walker-Andrews, A. (1994). Taxonomy for intermodal relations. In D. J. Lewkowicz & R. Lickliter (Eds.), The development of intersensory perception: Comparative perspectives (pp. 39–56). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  50. Wesslein, A. K., Spence, C., & Frings, C. (2014). When vision influences the invisible distractor: Tactile response compatibility effects require vision. Journal of Experimental Psychology: Human Perception and Performance, 40, 763–774.

    PubMed  Google Scholar 

  51. Zangenehpour, S., & Zatorre, R. J. (2010). Crossmodal recruitment of primary visual cortex following brief exposure to bimodal audiovisual stimuli. Neuropsychologia, 48, 591–600.

    PubMed  Google Scholar 

  52. Zhang, H., Zhang, J., & Kornblum, S. (1999). A parallel distributed processing model of stimulus–stimulus and stimulus–response compatibility. Cognitive Psychology, 38, 386–432.

    PubMed  Google Scholar 

Download references

Author note

Anne Jensen, Simon Merz, and Christian Frings, University of Trier, Department of Psychology, D-54286, Germany, and Charles Spence, Crossmodal Research Laboratory, Department of Experimental Psychology, Anna Watts Building, University of Oxford, Radcliffe Observatory Quarter, Woodstock Road, Oxford, OX2 6GG, UK.

The research reported in this article was supported by a grant from the Deutsche Forschungsgemeinschaft to Christian Frings and Charles Spence (FR 2133/5-3).

Open practices statement

The data and codes for all experiments are available at PsychArchives (https://doi.org/10.23668/psycharchives.2465).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Anne Jensen.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Statement of significance

This study demonstrates that crossmodal distractor interference induced by the learning of multisensory associations predominantly operates on a perceptual, rather than on a response, level. Crossmodal interference was elicited when features in a previous multisensory task were frequently presented together. Interference was still found when the response changed across tasks, but not when associations between stimulus features were counterbalanced. These findings provide novel insight concerning the processing of multisensory stimuli. Additionally, such insights may have significant implications for the design of future multisensory alerts and machine interfaces.

Appendix

Appendix

Table 2 Mean RT (in milliseconds; error rates in %, in parentheses) as a function of visual and auditory distractor feature congruency in Experiments 1–3

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Jensen, A., Merz, S., Spence, C. et al. Perception it is: Processing level in multisensory selection. Atten Percept Psychophys 82, 1391–1406 (2020). https://doi.org/10.3758/s13414-019-01830-4

Download citation

Keywords

  • Multisensory perception
  • Multisensory selection
  • Distractor processing
  • Multisensory associations
  • Processing level