Audiovisual integration capacity modulates as a function of illusory visual contours, visual display circumference, and sound type

Abstract

Research into the capacity of audiovisual integration has previously assessed whether capacity is strictly limited to a single item, or whether it can exceed one item under certain environmental conditions. More recently, investigations have turned to examining the effects of various stimulus factors on capacity. Across two experiments, we looked at a number of factors that were expected to play a modulatory role on capacity. Experiment 1 deployed a manipulation of illusory polygons, revealing an increase in audiovisual capacity, even in an absence of visual connections. This demonstrates that exceeding the capacity of 1 does not only represent a functional increase in the binding of a singular, complex visual object, but that it can also represent binding of multiple simpler objects. Findings also support the hypothesis that capacity modulates quantitatively, but not qualitatively, with respect to speed of presentation. Experiment 2 examined the effects of different sound types (sine tones or white noise) and of different spatial visual field sizes on the capacity of audiovisual integration. The results indicate that capacity is maximized when stimuli are presented in a smaller circle (7.5°) if alongside a sine tone, and when presented in a larger circle (18.5°) alongside white noise. These results suggest that audiovisual integration capacity is dependent on the combination of sound type and visual spatial field size. The combination of these results reveal additional phenomenological features of the capacity of audiovisual integration, and provides impetus for further research into applications of the findings.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Notes

  1. 1.

    The pattern of results using the full data set (N = 42) was similar to that of the trimmed data set (N = 26), with a significant main effect of SOA (p < .001, ηp2 = .617), along with a significant polygon × SOA interaction (p = .001, ηp2 = .098). One difference between the full data and trimmed data set is that the main effect of polygon crossed the standard threshold of significance (p = .064, ηp2 = .087), although the pattern of results did not change.

  2. 2.

    The data as a function of SOA only, along with the linear trend analysis, were reported in Wilbiks and Dyson (2018) in order to satisfy a reviewer’s query. However, the remainder of the data analysis in Experiment 1, and the entirety of Experiment 2 have not been reported elsewhere.

  3. 3.

    We thank a reviewer for this recommendation for further study.

  4. 4.

    The pattern of results using the full data set (N = 35) was similar to that of the trimmed data set (N = 26), with a significant main effect of SOA (p < .001, ηp2 = .622), along with a significant sound type × circle diameter interaction (p = .049, ηp2 = .131).

References

  1. Alvarez, G. A., Cavanagh, P. (2004). The capacity of visual short-term memory is set both by visual information load and by number of objects. Psychological Science, 15, 2. doi:https://doi.org/10.1111/j.0963-7214.2004.01502006.x

    Article  Google Scholar 

  2. Awh, E., Barton, B., & Vogel, E. K. (2007). Visual working memory represents a fixed number of items regardless of complexity. Psychological Science, 18, 7. doi:https://doi.org/10.1111/j.1467-9280.2007.01949.x

    Article  Google Scholar 

  3. Baron-Cohen, S., Wheelwright, S., Skinner, R., Martin, J., & Clubley, E. (2001). The autism-spectrum quotient (AQ) : Evidence from asperger syndrome/high-functioning autism, males and females, scientists and mathematicians. Journal of Autism and Developmental Disorders, 31(1), 5-17.

    Article  Google Scholar 

  4. Baum, S. H., Stevenson, R. A., & Wallace, M. T. (2015). Behavioral, perceptual, and neural alterations in sensory and multisensory function in autism spectrum disorder. Progress in Neurobiology, 134, 140-160.

    Article  Google Scholar 

  5. Bertelson, P., Pavani, F., Ladavas, E., Vroomen, J., & de Gelder, B. (2000). Ventriloquism in patients with unilateral visual neglect. Neuropsychologia, 38(12), 1634-1642. doi:https://doi.org/10.1016/s0028-3932(00)00067-1

    Article  PubMed  Google Scholar 

  6. Botta, F., Lupianez, J., & Sanabria, D. (2013). Visual unimodal grouping mediates auditory attentional bias in visuo-spatial working memory. Acta Psychologica, 144, 104-111. doi: https://doi.org/10.1016/j.actapsy.2013.05.010

    Article  PubMed  Google Scholar 

  7. Botta, F., Santangelo, V., Raffone, A., Lupianez, J., & Belardinelli, M. O. (2010). Exogenous and endogenous spatial attention effects on visuospatial working memory. The Quarterly Journal of Experimental Psychology, 63(8), 1590-1602. doi: https://doi.org/10.1080/17470210903443836

    Article  PubMed  Google Scholar 

  8. Botta, F., Santangelo, V., Raffone, A., Sanabria, D., Lupianez, J., & Belardinelli, M. O. (2011). Multisensory integration affects visuo-spatial working memory. Journal of Experimental Psychology: Human Perception and Performance, 37(4), 1099-1109. doi: https://doi.org/10.1037/a0023513

    Article  PubMed  Google Scholar 

  9. Castiello, U., & Umilta, C. (1989). Size of the attentional focus and efficiency of processing. Acta Psychologica, 73(3), 195-209. doi:https://doi.org/10.1016/00016918(90)90022-8

    Article  Google Scholar 

  10. Cowan, N. (2001). The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behavioral Brain Science, 24, 87-114. doi:https://doi.org/10.1017/S0140525X01003922

    Article  Google Scholar 

  11. Cowan, N. (2010). The magical mystery four: How is working memory capacity limited, and why? Current Directions in Psychological Science, 19(1), 51-57.

    Article  Google Scholar 

  12. Deruelle, C., Rondan, C., Gepner, B., & Fagot, J. (2006). Processing of compound stimuli by children with autism and Asperger syndrome. International Journal of Psychology, 41(2), 97-106.

    Article  Google Scholar 

  13. Eng, H. Y., Chen, D., & Jiang, Y. (2005). Visual working memory for simple and complex visual stimuli. Psychonomic Bulletin & Review, 12, 1127-1133. doi:https://doi.org/10.3758/BF03206454

    Article  Google Scholar 

  14. Fernell, E., Eriksson, M. A., & Gillberg, C. (2013). Early diagnosis of autism and impact on prognosis: a narrative review. Clinical Epidemiology, 5, 33-43.

    Article  Google Scholar 

  15. Fujisaki, W., Koene, A., Arnold, D., Johnston, A., & Nishida, S. (2006). Visual search for a target changing in synchrony with an auditory signal. Proceedings of the Royal Society B: Biological Sciences, 273(1588), 865-874. doi:https://doi.org/10.1098/rspb.2005.3327

    Article  PubMed  Google Scholar 

  16. Gleiss, S., & Kayser, C. (2014). Acoustic noise improves visual perception and modulates occipital oscillatory states. Journal of Cognitive Neuroscience, 26(4), 699-711. doi: https://doi.org/10.1162/jocn_a_00524

    Article  PubMed  Google Scholar 

  17. Gobet, F., Lane, P. C. R., Croker, S., Cheng, P. C.-H., Jones, G., Oliver, I., & Pine, J. M. (2001) Chunking mechanisms in human learning. Trends in Cognitive Sciences, 5236–243. doi:https://doi.org/10.1016/S1364-6613(00)01662-4

  18. Golomb, J. D., McDavitt, J. R., Ruf, B. M., Chen, J. I., Saricicek, A., Maloney, K. H., ... & Bhagwagar, Z. (2009). Enhanced visual motion perception in major depressive disorder. Journal of Neuroscience, 29(28), 9072-9077.

  19. Hommel, B. (1998). Event files: Evidence for automatic integration of stimulus-response episodes. Visual Cognition, 5, 183-216.

    Article  Google Scholar 

  20. Hommel, B. (2004). Event files: Feature binding in and across perception and action. Trends in Cognitive Sciences, 8, 494-500.

    Article  Google Scholar 

  21. Hommel, B. & Colzato, L. (2004). Visual attention and the temporal dynamics of feature integration. Visual Cognition, 11, 483-521.

    Article  Google Scholar 

  22. Kanizsa, G. (1976). Subjective contours. Scientific American, 23(4), 48-52. doi:https://doi.org/10.1038/scientificamerican0476-48

    Article  Google Scholar 

  23. Kawachi, Y., Grove, P. M., & Sakurai, K. (2014). A single auditory tone alters the perception of multiple visual events. Journal of Vision, 14(8), 16-16. doi:https://doi.org/10.1167/14.8.16

    Article  PubMed  Google Scholar 

  24. Kösem, A., & van Wassenhove, V. (2012). Temporal Structure in Audiovisual Sensory Selection. PLoS ONE, 7(7), 1-11. doi:https://doi.org/10.1371/journal.pone.0040936

    Article  Google Scholar 

  25. Laberge, D., & Brown, V. (1986). Variations in size of the visual field in which targets are presented: An attentional range effect. Perception & Psychophysics, 40(3), 188-200. doi:https://doi.org/10.3758/bf03203016

    Article  Google Scholar 

  26. Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions. Nature, 390, 279-81. doi:https://doi.org/10.1038/36846

    Article  PubMed  PubMed Central  Google Scholar 

  27. Manjarrez, E., Mendez, I., Martinez, L., Flores, A., & Mirasso, C. R. (2007). Effects of auditory noise on the psychophysical detection of visual signals: Cross-modal stochastic resonance. Neuroscience Letters, 415, 231-236. doi: https://doi.org/10.1016/j/neulet.2007.01.030

    Article  PubMed  Google Scholar 

  28. Miller, G. A. (1956). The magical number seven, plus or minus two some limits on our capacity for processing information. Psychological Review, 101, 343-352. doi:https://doi.org/10.1037/h0043158

    Article  Google Scholar 

  29. Mottron, L., Dawson, M., Soulieres, I., Hubert, B., & Burack, J. (2006). Enhanced perceptual functioning in autism: an update, and eight principles of autistic perception. Journal of Autism and Developmental Disorders, 36(1), 27-43.

    Article  Google Scholar 

  30. Nicholls, M. E. R., Loveless, K. M., Thomas, N. A., Loetscher, T., & Churches, T. (2015). Some participants may be better than others: Sustained attention and motivation are higher early in the semester. The Quarterly Journal of Experimental Psychology, 68(1), 10-18.

    Article  Google Scholar 

  31. Olivers, C. N. L., Awh, E., & Van der Burg, E. (2016). The capacity to detect synchronous audiovisual events is severely limited: Evidence from mixture modelling. Journal of Experimental Psychology: Human Perception and Performance, 42, 2115-2124.

    PubMed  Google Scholar 

  32. Pylyshyn, Z. W. (2001). Visual indexes, preconceptual objects, and situated vision. Cognition, 80, 127-58. doi:https://doi.org/10.1016/S0010-0277(00)00156-6

    Article  PubMed  Google Scholar 

  33. Sargent, J., Dopkins, S., Philbeck, J., & Chickha, D. (2010). Chunking in spatial memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36, 576-589. doi: https://doi.org/10.1037/a0017528

    Article  PubMed  Google Scholar 

  34. Schutz, M., & Gillard, J. (2018). Generalizing audio-visual integration: what kinds of stimuli have we been using? Paper presented at the meeting of the International Multisensory Research Forum, Toronto, ON, Canada.

  35. Schutz, M., & Vaisberg, J. M. (2014). Surveying the temporal structure of sounds used in Music Perception. Music Perception, 31(3), 288-296.

  36. Seibold, V. C. (2018). Do alerting signals increase the size of the attentional focus? Attention, Perception, & Psychophysics, 80(2), 402-425. doi:https://doi.org/10.3758/s13414-017-1451-1

    Article  Google Scholar 

  37. Seitz, A. R., Kim, R., & Shams, L. (2006). Sound facilitates visual learning. Current Biology, 16(14), 1422-1427.

    Article  Google Scholar 

  38. Stevenson, R. A., Zemtsov, R. K., & Wallace, M. T. (2012). Individual differences in the multisensory temporal binding window predict susceptibility to audiovisual illusions. Journal of Experimental Psychology: Human Perception and Performance, 38(6), 1517-1529.

    PubMed  Google Scholar 

  39. Usher, M., & Feingold, M. (2000). Stochastic resonance in the speed of memory retrieval. Biological Cybernetics, 83, 11-16.

    Article  Google Scholar 

  40. Van der Burg, E., Awh, E., & Olivers, C. N. (2013). The capacity of audiovisual integration is limited to one item. Psychological Science, 24(3), 345-351. doi:https://doi.org/10.1177/0956797612452865

    Article  PubMed  PubMed Central  Google Scholar 

  41. Van der Burg, E., Cass, J., Olivers, C., Theeuwes, J., & Alais, D. (2010). Efficient visual search from synchronized auditory signals requires transient audiovisual events. Journal of Vision, 10(7), 855-855. doi:https://doi.org/10.1167/10.7.855

    Article  Google Scholar 

  42. Van der Burg, E., Olivers, C. N., Bronkhorst, A. W., & Theeuwes, J. (2008). Pip and pop: Nonspatial auditory signals improve spatial visual search. Journal of Experimental Psychology: Human Perception and Performance, 34(5), 1053-1065. doi:https://doi.org/10.1037/0096-1523.34.5.1053

    Article  PubMed  Google Scholar 

  43. Van der Stoep, N., Van der Stigchel, S., Nijboer, T. C. W., & Van der Smagt, M. J. (2016). Audiovisual integration in near and far space: effects of changes in distance and stimulus effectiveness. Experimental Brain Research, 234, 1175-1188. doi: https://doi.org/10.1007/s00221-015-4248-2

    Article  PubMed  Google Scholar 

  44. Van Wanrooij, M. M., Bremen, P., & Opstal, J. A. (2010) Acquired prior knowledge modulates audiovisual integration. European Journal of Neuroscience, 31, 1763-71. doi:https://doi.org/10.1111/j.1460-9568.2010.07198.x.

    Article  PubMed  Google Scholar 

  45. Vogel, E. K., Woodman, G. F., & Luck, S. J. (2001). Storage of features, conjunctions, and objects in visual working memory. Psychonomic Bulletin & Review, 27, 92-114. doi:https://doi.org/10.1037//0096-1523.27.1.92

    Article  Google Scholar 

  46. Vroomen, J., & de Gelder, B. (2000). Sound enhances visual perception: Cross-modal effects of auditory organization on vision. Journal of Experimental Psychology: Human Perception and Performance, 26(5), 1583-1590. doi:https://doi.org/10.1037//0096-1523.26.5.1583

    Article  PubMed  Google Scholar 

  47. Weinbach, N., & Henik, A. (2012). The relationship between alertness and executive control. Journal of Experimental Psychology: Human Perception and Performance, 38(6), 1530-1540. doi:https://doi.org/10.1037/a0027875

  48. Weinbach, N., & Henik, A. (2014). Alerting enhances attentional bias for salient stimuli; evidence from a global/local processing task. Cognition, 133, 414-419.

    Article  Google Scholar 

  49. Wilbiks, J. M. P., & Dyson, B. J. (2013). Effects of temporal asynchrony and stimulus magnitude on competitive audio-visual binding. Attention, Perception, & Psychophysics, 75(8), 1883-1891.

    Article  Google Scholar 

  50. Wilbiks, J. M. P., & Dyson, B. J. (2016). The dynamics and neural correlates of audio-visual integration capacity as determined by temporal unpredictability, proactive interference, and SOA. PLOS One, 11(12). doi:https://doi.org/10.1371/journal.pone.0168304

  51. Wilbiks, J. M. P., & Dyson, B. J. (2018). The contribution of perceptual factors and training on varying audiovisual integration capacity. Journal of Experimental Psychology: Human Perception and Performance, 44(6), 871-884.

    PubMed  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jonathan M. P. Wilbiks.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wilbiks, J.M.P., Pavilanis, A.D.S. & Rioux, D.M. Audiovisual integration capacity modulates as a function of illusory visual contours, visual display circumference, and sound type. Atten Percept Psychophys 82, 1971–1986 (2020). https://doi.org/10.3758/s13414-019-01882-6

Download citation

Keywords

  • Audition
  • Multisensory processing
  • Visual perception