Skip to main content
SpringerLink
Log in

Spoken Word Mediated Interference in a Visual Task: Eye Tracking Evidence from Bilinguals

  • Chapter
  • First Online:
Attention and Vision in Language Processing

Abstract

Listening and speaking constitute two most fundamental aspects of human communication. While speaking is largely intentional and calls for voluntary control and planning, listening could be largely unintentional. One listens and understands speech even when one is not being deliberate. Not much is known about control, attentional mechanisms and intention during listening. Most psycholinguistic studies ask participants to comprehend speech under different task requirements and measure comprehension. However, recently it has been observed that listening without any intention to comprehend can lead to attention capture and change in action plan. This nexus between speech and action control can be very useful in understanding goal planning and cognition in humans. The abundance of speech around us and its evolutionary importance makes it a very important biological stimulus in understanding different facets of control. In this chapter, we will review studies where unintentional comprehension of speech has been shown to cause changes in goal-directed action. We particularly look at studies where eye movements and other such motor actions have been shown to deviate during such multi-modal processing. It appears that speech just cannot be ignored in many situations and this penetrates into processing modules that require no language. We look at visual world studies with bilinguals where listening to words causes slowness in a visual search task.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 139.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Allopenna, P. D., Magnuson, J. S., & Tanenhaus, M. K. (1998). Tracking the time course ofspoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory and Language, 38(4), 419–439.

    Google Scholar 

  • Altmann, G. T. M., & Kamide, Y. (2004). Now you see it, now you don’t: Mediating the mapping between language and the visual world. In J. Henderson & F. Ferreira (Eds.), The integration of language, vision and action (pp. 347–386). Psychology Press.

    Google Scholar 

  • Bargh, J. A. (1994). The four horsemen of automaticity. In R. S. Wyer & T. K. Srull (Eds.), Handbook of social cognition (pp. 1–40). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Bertels, J., Kolinsky, R., Bernaerts, A., & Morais, J. (2011). Effects of emotional spoken words on exogenous attentional orienting. Journal of Cognitive Psychology, 23(4), 435–452.

    Article  Google Scholar 

  • Blumenfeld, H. K., & Marian, V. (2007). Constraints on parallel activation in bilingual spoken language processing: Examining proficiency and lexical status using eye-tracking. Language and Cognitive Processes, 22, 633–660.

    Article  Google Scholar 

  • Blumenfeld, H. K., & Marian, V. (2013). Parallel language activation and cognitive control during spoken word recognition in bilinguals. Journal of Cognitive Psychology, 25(5), 547–567.

    Google Scholar 

  • Buchner, A., & Erdfelder, E. (2005). Word frequency of irrelevant speech distractors affects serial recall. Memory and Cognition, 33, 86–97.

    Article  Google Scholar 

  • Buchner, A., Rothermund, K., Wentura, D., & Mehl, B. (2004). Valence of distractor words increases the effects of irrelevant speech on serial recall. Memory and Cognition, 32, 722–731.

    Article  Google Scholar 

  • Buchner, A. (1996). On the irrelevance of semantic information for the llIrrelevant speech effect. The Quarterly Journal of Experimental Psychology: Section A, 49(3), 765–779.

    Article  Google Scholar 

  • Chua, F. K. (2013). Attentional capture by onsets and offsets. Visual Cognition, 21(5), 569–598.

    Article  Google Scholar 

  • Cleland, A. A., Tamminen, J., Quinlan, P. T., & Gaskell, M. G. (2012). Spoken word processing creates a lexical bottleneck. Language and Cognitive Processes, 27(4), 572–593.

    Article  Google Scholar 

  • Cooper, R. M. (1974). The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6(1), 84–107.

    Article  Google Scholar 

  • Dahan, D., Magnuson, J. S., & Tanenhaus, M. K. (2001). Time course of frequency effects in spoken-word recognition: evidence from eye movements. Cognitive Psychology, 42, 317–367.

    Article  Google Scholar 

  • Dijkstra, T., Miwa, K., Brummelhuis, B., Sappelli, M., & Baayen, H. (2010). How crosslanguage similarity and task demands affect cognate recognition. Journal of Memory and Language, 62, 284–301.

    Article  Google Scholar 

  • Dijkstra, A. F. J., & Van Heuven, W. J. B. (2002). The architecture of the bilingual word recognition system: From identification to decision. Bilingualism: Language and Cognition, 5(3), 175–197.

    Google Scholar 

  • Downing, P. E. (2000). Interactions between visual working memory and selective attention. Psychological Science, 11, 467–473.

    Article  Google Scholar 

  • Driver, J., & Spence, C. (1998). Cross-modal links in spatial attention. Philosophical Transactions of the Royal Society B: Biological Sciences, 353(1373), 1319.

    Article  Google Scholar 

  • Elliott, E. M., & Briganti, A. M. (2012). Investigating the role of attentional processes in the irrelevant speech effect. Acta Psychologica, 140, 64–74.

    Article  Google Scholar 

  • Guo, T., Misra, M., Tam, J. W., & Kroll, J. F. (2012). On the time course of accessing meaning in a second language: An electrophysiological investigation of translation recognition. Journal of Experimental Psychology. Learning, Memory, and Cognition, 38, 1165–1186. doi:10.1037/a0028076.

    Article  Google Scholar 

  • Huettig, F., & Altmann, G. T. M. (2004). Language-mediated eye movements and the resolution of lexical ambiguity. In M. Carreiras & C. Clifton (Eds.), The on-line study of sentence comprehension: Eye-tracking, ERP, and beyond (pp. 187–207). New York, NY: Psychology Press.

    Google Scholar 

  • Huettig, F., & Altmann, G. T. M. (2005). Word meaning and the control of eye fixation: semantic competitor effects and the visual world paradigm. Cognition, 96(1), 23–32.

    Article  Google Scholar 

  • Huettig, F., & Altmann, G. T. M. (2007). Visual-shape competition and the control of eye fixation during the processing of unambiguous and ambiguous words. Visual Cognition, 15(8), 985–1018.

    Google Scholar 

  • Huettig, F., & Altmann, G. T. M. (2011). Looking at anything that is green when hearing ‘frog’—How object surface colour and stored object colour knowledge influence language-mediated overt attention. Quarterly Journal of Experimental Psychology, 64(1), 122–145.

    Article  Google Scholar 

  • Huettig, F., & Hartsuiker, R. J. (2008). When you name the pizza you look at the coin and the bread: Eye movements reveal semantic activation during word production. Memory and Cognition, 36(2), 341–360. doi:10.3758/MC.36.2.341.

    Article  Google Scholar 

  • Huettig, F., Mishra, R. K., & Olivers, C. N. (2012). On the mechanisms and representations of language-mediated visual attention. Frontiers in Cognition, 2, 394.

    Google Scholar 

  • Huettig, F., & McQueen, J. M. (2007). The tug of war between phonological, semantic and shape information in language-mediated visual search. Journal of Memory and Language, 57(4), 460–482. doi:10.1016/j.jml.2007.02.001.

    Article  Google Scholar 

  • Irwin, D. E., Colcombe, A. M., Kramer, A. F., & Hahn, S. (2000). Attention and oculomotor capture by onset luminance and color singletons. Vision Research, 40, 1443–1458.

    Article  Google Scholar 

  • Jones, D. M., Miles, C., & Page, J. (1990). Disruption of proofreading by irrelevant speech: Effects of attention, arousal or memory? Applied Cognitive Psychology, 4(2), 89–108.

    Article  Google Scholar 

  • Jones, D. M., Madden, C., & Miles, C. (1992). Privileged access by irrelevant speech to short-term memory: The role of changing state. Quarterly Journal of Experimental Psychology: Human Experimental Psychology, 44, 645–669.

    Article  Google Scholar 

  • Jonides, J., Naveh-Benjamin, M., & Palmer, J. (1985). Assessing automaticity. Acta Psychologica, 60(2), 157–171.

    Article  Google Scholar 

  • Ju, M., & Luce, P. A. (2004). Falling on sensitive ears. Psychological Science, 15, 314–318.

    Article  Google Scholar 

  • Kamide, Y., & Altmann, G. T. M. (2004). The time-course of constraint-application during sentence processing in visual contexts: Anticipatory eye-movements in English and Japanese. In M. Tanenhaus & J. Trueswell (Eds.), World situated language use: Psycholinguistic, linguistic and computational perspectives on bridging the product and action traditions. Cambridge: MIT Press.

    Google Scholar 

  • Kamide, Y., Altmann, G. T. M., & Haywood, S. (2003). The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye-movements. Journal of Memory and Language, 49, 133–159.

    Article  Google Scholar 

  • Klatte, M., Lachmann, T., Schlittmeier, S., & Hellbrück, J. (2010). The irrelevant sound effect in short-term memory: Is there developmental change?European. Journal of Cognitive Psychology, 22(8), 1168–1191.

    Article  Google Scholar 

  • Lavie, N. (2005). Distracted and confused? Selective attention under load. Trends in Cognitive Sciences, 9, 75–82.

    Article  Google Scholar 

  • Lavie, N., & Tsal, Y. (1994). Perceptual load as a major determinant of the locus of selection in visual attention. Perception and Psychophysics, 56, 183–197.

    Article  Google Scholar 

  • LeCompte, D. C., Neely, C. B., & Wilson, J. R. (1997). Irrelevant speech and irrelevant tones: The relative importance of speech to the irrelevant speech effect. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23, 472–483.

    Google Scholar 

  • Logan, G. D. (1998). What is learned during automatization? Obligatory encoding of location information. Journal of Experimental Psychology: Human Perception and Performance, 24, 1720–1736.

    Google Scholar 

  • Lupyan, G., & Swingley, D. (2011). Self-directed speech affects visual processing. Quarterly Journal of Experimental Psychology, 65(6), 1068–1085.

    Article  Google Scholar 

  • Ljungberg, J. K., Parmentier, F. B., Hughes, R. W., Macken, W. J., & Jones, D. M. (2012). Listen out! Behavioural and subjective responses to verbal warnings. Applied Cognitive Psychology, 26(3), 451–461.

    Article  Google Scholar 

  • Mani, N., & Huettig, F. (2012). Prediction is a piece of cake—but only for skilled producers. Journal of Experimental Psychology: Human Perception and Performance, 38, 843–847.

    Google Scholar 

  • Marsh, J. E., Hughes, R. W., & Jones, D. M. (2008). Auditory distraction in semantic memory: A process-based approach. Journal of Memory and Language, 58(3), 682–700.

    Article  Google Scholar 

  • Marian, V., & Spivey, M. (2003a). Bilingual and monolingual processing of competing lexical items. Applied Psycholinguistics, 24, 173–193.

    Article  Google Scholar 

  • Marian, V., & Spivey, M. (2003b). Competing activation in bilingual language processing: Within and between-language competition. Bilingualism: Language and Cognition, 6, 97–115.

    Article  Google Scholar 

  • Martin, R. C., Wogalter, M. S., & Forlano, J. G. (1988). Reading comprehension in the presence of unattended speech and music. Journal of Memory and Language, 27, 382–398.

    Article  Google Scholar 

  • Mishra, R. (2009). Interface of language and visual attention: Evidence from production and comprehension. Progress in Brain Research, 176, 277–292.

    Article  Google Scholar 

  • Mishra, R. K., Huettig, F., & Olivers, C. N. (2012). Automaticity and conscious decisions during language-mediated eye gaze in the visual world. In N. Srinivasan & V. S. C. Pammi (Eds.), Progress in Brain Research: Decision Making: Neural and Behavioral Approaches. Amsterdam: Elsevier.

    Google Scholar 

  • Moors, A.., & De Houwer, J. (2006). Automaticity: a theoretical and conceptual analysis. Psychological bulletin, 132(2), 297.

    Google Scholar 

  • Moores, E., Laiti, L., & Chelazzi, L. (2003). Associative knowledge controls deployment of visual selective attention. Nature Neuroscience, 6, 182–189.

    Article  Google Scholar 

  • Nation, K., Marshall, C., & Altmann, G. T. M. (2003). Investigating individual differences in children’s real-time sentence comprehension using language-mediated eye movements. Journal of Experimental Child Psychology, 86, 314–329.

    Article  Google Scholar 

  • Neely, C. B., & LeCompte, D. C. (1999). The importance of semantic similarity to the irrelevant speech effect. Memory and Cognition, 27, 37–44.

    Article  Google Scholar 

  • Oswald, C. J. P., Tremblay, S., & Jones, D. M. (2000). Disruption of comprehension by the meaning of irrelevant sound. Memory, 8, 345–350. doi:10.1080/09658210050117762.

    Article  Google Scholar 

  • Parmentier, F. B. R., Turner, J., & Perez, L. (2013). A dual contribution to the involuntary semantic processing of unexpected spoken words. Journal of Experimental Psychology: General. doi:10.1037/a0031550.

  • Pickering, M. J., & Garrod, S. (2004). Toward a mechanistic psychology of dialogue. Behavioral and Brain Sciences, 27(2), 169–189.

    Google Scholar 

  • Pulvermüller, F., & Shtyrov, Y. (2006). Language outside the focus of attention: The mismatch negativity as a tool for studying higher cognitive processes. Progress in Neurobiology, 79(1), 49.

    Article  Google Scholar 

  • Salamé, P., & Baddeley, A. (1982). Disruption of short term memory by unattended speech: Implications for thestructure of working memory. Journal of Verbal Learning and Verbal Behavior, 21, 150–164.

    Article  Google Scholar 

  • Salverda, A. P., & Altmann, G. T. M. (2011). Attentional capture of objects referred to by spoken language. Journal of Experimental Psychology: Human Perception and Performance, 37, 1122–1133.

    Google Scholar 

  • Shtyrov. Y., Kimppa, L., Pulvermüller, F., Kujala, T. (2010). Event-related potentials reflecting the frequency of unattended spoken words: A neuronal index of connection strength in lexical memory circuits? Neuroimage, 15;55(2), 658–668.

    Google Scholar 

  • Singh, N., & Mishra, R. K. (2013). Second language proficiency modulates conflict-monitoring in an oculomotor stroop task: Evidence from Hindi-English Bilinguals. Frontiers in Psychology,. doi:10.3389/fpsyg.2013.00322.

    Google Scholar 

  • Singh, N., & Mishra, R. K. (2012). Does language proficiency modulate oculomotor control? Evidence from Hindi-English bilinguals. Bilingualism: Language and Cognition, 15, 771–781.

    Google Scholar 

  • Soto, D., & Humphreys, G. W. (2007). Automatic guidance of visual attention from verbal working memory. The Journal of Experimental Psychology: Human Perception and Performance, 33(3), 730–737.

    Google Scholar 

  • Spivey, M. J., & Marian, V. (1999). Cross talk between native and second languages: Partial activation of an irrelevant lexicon. Psychological Science, 10(3), 281–284.

    Article  Google Scholar 

  • Sunderman, G., & Priya, K. (2012). Translation recognition in highly proficient Hindi-Englis bilinguals the influence of different scripts but connectable phonologies. Language and Cognitive Processes, 27(9), 1265–1285.

    Article  Google Scholar 

  • Telling, A. L., Kumar, S., Meyer, A. S., & Humphreys, G. W. (2010). Electrophysiological evidence of semantic interference in visual search. Journal of Cognitive Neuroscience, 22(10), 2212–2225.

    Article  Google Scholar 

  • Tellinghuisen, D. J., & Nowak, E. J. (2003). The inability to ignore auditory distractors as a function of visual task perceptual load. Perception and Psychophysics, 65, 817–828.

    Article  Google Scholar 

  • Thierry, G., & Wu, Y. J. (2010). Chinese–English bilinguals reading english hear Chinese. The Journal of Neuroscience, 30, 7646–7651.

    Article  Google Scholar 

  • Tun, P. A., O’Kane, G., & Wingfield, A. (2002). Distraction by competing speech in younger and older listeners. Psychology and Aging, 17, 453–467.

    Article  Google Scholar 

  • Weaver, M. D., Lauwereyns, J., & Theeuwes, J. (2011). The effect of semantic information on saccade trajectory deviations. Vision Research, 51, 1124–1128.

    Article  Google Scholar 

  • Weber, A., & Cutler, A. (2004). Lexical competition in non-native spoken-word recognition. Journal of Memory and Language, 50, 1–25.

    Article  Google Scholar 

  • Yantis, S., & Jonides, J. (1984). Abrupt visual onsets and selective attention: Evidence from visual search. Journal of Experimental Psychology: Human Perception and Performance, 5, 625–638.

    Google Scholar 

  • Yee, E., Overton, E., & Thompson-Schill, S. L. (2009). Looking for meaning: Eye movements are sensitive to overlapping semantic features, not association. Psychonomic Bulletin and Review, 16(5), 869–874.

    Article  Google Scholar 

  • Yee, E., & Sedivy, J. C. (2006). Eye movements to pictures reveal transient semantic activation during spoken word recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32, 1–14.

    Google Scholar 

  • Zeelenberg, R., & Bocanegra, B. R. (2010). Auditory emotional cues enhance visual perception. Cognition, 115, 202–206.

    Article  Google Scholar 

Download references

Acknowledgments

Support for this chapter came from a DST grant awarded to RKM under the Cognitive Science Initiative Scheme.

Author information

Authors and Affiliations

  1. Center for Neural and Cognitive Sciences, University of Hyderabad, Hyderabad, India

    Ramesh Kumar Mishra

  2. Center for Behavioural and Cognitive Sciences, University of Allahabad, Allahabad, India

    Ramesh Kumar Mishra & Niharika Singh

Authors
  1. Ramesh Kumar Mishra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Niharika Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ramesh Kumar Mishra .

Editor information

Editors and Affiliations

  1. Center for Neural and Cognitive Sciences, University of Hyderabad, Hyderabad, Andhra Pradesh, India

    Ramesh Kumar Mishra

  2. Centre of Behavioural and Cognitive Sciences, University of Allahabad, Allahabad, Uttar Pradesh, India

    Narayanan Srinivasan

  3. Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands

    Falk Huettig

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer India

About this chapter

Cite this chapter

Mishra, R.K., Singh, N. (2015). Spoken Word Mediated Interference in a Visual Task: Eye Tracking Evidence from Bilinguals. In: Mishra, R., Srinivasan, N., Huettig, F. (eds) Attention and Vision in Language Processing. Springer, New Delhi. https://doi.org/10.1007/978-81-322-2443-3_10

Download citation

Publish with us

Policies and ethics

Search

Navigation