Skip to main content

Attention, Language and Vision

  • Chapter
  • First Online:
Interaction Between Attention and Language Systems in Humans
  • 840 Accesses

Abstract

So far we have looked at scenarios where information only from one modality was used during language processing. Recently, several researchers have explored how spoken language influences attentional selections in simple looking and listening tasks using eye tracking as methodology. Attentional mechanisms influence how linguistic and non-linguistic information are processed together. Eye movements during language comprehension in the presence of visual images have revealed how conceptual activation leads to shifts in attention and cognition. The aim of this chapter is to examine how linguistic and visual information interact dynamically during cognitive processing in different situations. Data from both monolingual and bilingual studies are considered to examine issues of lexical access and visual search. First, the chapter covers some theoretical issues related to language and vision interaction and later, findings from the visual world studies are examined.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Altmann, G. (2004). Language-mediated eye movements in the absence of a visual world: The blank screen paradigm. Cognition, 93(2), B79–B87.

    Article  PubMed  Google Scholar 

  • Altmann, G. T. M. (2011). Language can mediate eye movement control within 100 milliseconds, regardless of whether there is anything to move the eyes to. Acta Psychol. (Amst.), 137, 190–200.

    Article  Google Scholar 

  • Altmann, G. T., & Kamide, Y. (2007). The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing. Journal of Memory and Language, 57(4), 502–518.

    Article  Google Scholar 

  • Altmann, G., & Kamide, Y. (2009). Discourse-mediation of the mapping between language and the visual world: Eye movements and mental representation. Cognition, 111(1), 55–71.

    Article  PubMed Central  PubMed  Google Scholar 

  • Allopenna, P. D., Magnuson, J. S., & Tanenhaus, M. K. (1998). Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory and Language, 38(4), 419–439.

    Article  Google Scholar 

  • Bertels, J., Kolinsky, R., Bernaerts, A., & Morais, J. (2011). Effects of emotional spoken words on exogenous attentional orienting. Journal of Cognitive Psychology, 23(4), 435–452.

    Article  Google Scholar 

  • Buswell, G. T. (1935). How people look at pictures: a study of the psychology and perception in art. Oxford, England: University of Chicago Press.

    Google Scholar 

  • Chambers, C. G., Tanenhaus, M. K., Eberhard, K. M., Filip, H., & Carlson, G. N. (2002). Circumscribing referential domains during real-time language comprehension. Journal of Memory and Language, 47(1), 30–49.

    Google Scholar 

  • Chomsky, N. (2007). Approaching UG from below. Interfaces + recursion = language, 1–29.

    Google Scholar 

  • Cleland, A. A., Tamminen, J., Quinlan, P. T., & Gaskell, M. G. (2012). Spoken word processing creates a lexical bottleneck. Language and Cognitive Processes, 27(4), 572–593.

    Article  Google Scholar 

  • Cooper, R. M. (1974). The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6(1), 84–107.

    Article  Google Scholar 

  • Dahan, D., & Tanenhaus, M. K. (2005). Looking at the rope when looking for the snake: Conceptually mediated eye movements during spoken-word recognition. Psychological Bulletin & Review, 12, 455–459.

    Article  Google Scholar 

  • De Houwer, J., Smith, C. (in press). Go with your gut! Effects of Affect Misattribution Procedures become stronger when participants are encouraged to rely on their gut feelings. Social Psychology.

    Google Scholar 

  • Desimone, R., & Duncan, J. (1995). Neural mechanisms of selective visual attention. Annual Review of Neuroscience, 18, 193–222.

    Article  PubMed  Google Scholar 

  • Downing, P. E. (2000). Interactions between visual working memory and selective attention. Psychological Science, 11(6), 467–473.

    Article  PubMed  Google Scholar 

  • Ellermeier, W., & Zimmer, K. (1997). Individual differences in susceptibility to the irrelevant speech effect. The Journal of the Acoustical Society of America, 102(4), 2191–2199.

    Article  PubMed  Google Scholar 

  • Elliott, E. M., & Briganti, A. M. (2012). Investigating the role of attentional processes in the irrelevant speech effect. Acta Psychologica, 140, 64–74.

    Article  PubMed  Google Scholar 

  • Ferreira, F., Apel, J., & Henderson, J. M. (2008). Taking a new look at looking at nothing. Trends in cognitive sciences, 12(11), 405–410.

    Article  PubMed  Google Scholar 

  • Findlay, J. M., Brown, V., & Gilchrist, I. D. (2001). Saccade target selection in visual search: the effect of information from the previous fixation. Vision Research, 41(1), 87–95.

    Article  PubMed  Google Scholar 

  • Fodor, J. A. (1987). Psychosemantics: The problem of meaning in the philosophy of mind. Cambridge: The MIT Press.

    Google Scholar 

  • Foulsham, T., & Underwood, G. (2011). If visual saliency predicts search, then why? evidence from normal and gaze-contingent search tasks in natural scenes. Cognitive Computation, 3(1), 48–63.

    Article  Google Scholar 

  • Guo, T., Misra, M., Tam, J. W., & Kroll, J. F. (2012). On the time course of accessing meaning in a second language: An electrophysiological investigation of translation recognition. Journal of Experimental Psychology. Learning, Memory, and Cognition, 38, 1165–1186.

    Article  PubMed Central  PubMed  Google Scholar 

  • Henderson, J. M. (2003). Human gaze control during real-world scene perception. Trends in cognitive sciences, 7(11), 498–504.

    Article  PubMed  Google Scholar 

  • Henderson, J. M., & Ferreira, F. (2004). Scene perception for psycholinguists. In J. M. Henderson & F. Ferreira (Eds.), The interface of language, vision, and action: Eye movements and the visual world (pp. 1–58). New York: Psychology Press.

    Google Scholar 

  • Huang, Y. T., & Snedeker, J. (2011). Logic and conversation revisited: Evidence for a division between semantic and pragmatic content in real-time language comprehension. Language and Cognitive Processes, 26(8), 1161–1172.

    Article  Google Scholar 

  • Huettig, F., & Altmann, G. T. (2005). Word meaning and the control of eye fixation: Semantic competitor effects and the visual world paradigm. Cognition, 96(1), B23–B32.

    Article  PubMed  Google Scholar 

  • Huettig, F., & Altmann, G. T. M. (2007). Visual-shape competition during language-mediated attention is based on lexical input and not modulated by contextual appropriateness. Visual Cognition, 15(8), 985–1018. doi:10.1080/13506280601130875.

    Article  Google Scholar 

  • Huettig, F., & McQueen, J. M. (2007). The tug of war between phonological, semantic and shape information in language-mediated visual search. Journal of Memory and Language, 57(4), 460–482. doi:10.1016/j.jml.2007.02.001.

    Article  Google Scholar 

  • Huettig, F., Mishra, R. K., & Olivers, C. N. (2012). On the mechanisms and representations of language-mediated visual attention. Frontiers in Cognition, 2, 394.

    Google Scholar 

  • Huettig, F., Olivers, C. N. L., & Hartsuiker, R. J. (2011a). Looking, language, and memory: Bridging research from the visual world and visual search paradigms. Acta Psychologica, 137, 138–150. doi:10.1016/j.actpsy.2010.07.013.

    Article  PubMed  Google Scholar 

  • Huettig, F., Rommers, J., & Meyer, A. S. (2011b). Using the visual world paradigm to study language processing: A review and critical evaluation. Acta Psychologica, 137(2), 151–171.

    Article  PubMed  Google Scholar 

  • Huettig, F., Singh, N., & Mishra, R. K. (2011c). Language-mediated visual orienting behavior in low and high literates. Frontiers in Psychology, 2, 285. doi:10.3389/fpsyg.2011.00285.

    Article  PubMed Central  PubMed  Google Scholar 

  • Jackendoff, R. (1997). The architecture of the language faculty (Vol. 28). Cambridge: The MIT Press.

    Google Scholar 

  • Jackendoff, R. (2011). What is the human language faculty?: Two views. Language, 87(3), 586–624.

    Article  Google Scholar 

  • Johansson, R., Holsanova, J., & Holmqvist, K. (2006). Pictures and spoken descriptions elicit similar eye movements during mental imagery, both in light and in complete darkness. Cognitive Science, 30(6), 1053–1079.

    Article  PubMed  Google Scholar 

  • Jones, D. M., & Macken, W. J. (1993). Irrelevant tones produce an irrelevant speech effect: implications for phonological coding in working memory. Journal of Experimental Psychology. Learning, Memory, and Cognition, 19(2), 369–381.

    Article  Google Scholar 

  • Jesperson, O. (1924). The Philosophy of Grammar. University of Chicago Press.

    Google Scholar 

  • Kamide, Y., Altmann, G. T., & Haywood, S. L. (2003). The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye movements. Journal of Memory and Language, 49(1), 133–156.

    Article  Google Scholar 

  • Klatte, M., Lachmann, T., Schlittmeier, S., & Hellbrück, J. (2010). The irrelevant sound effect in short-term memory: Is there developmental change? European Journal of Cognitive Psychology, 22(8), 1168–1191.

    Article  Google Scholar 

  • Klein, R. M. (2000). Inhibition of return. Trends in cognitive sciences, 4(4), 138–147.

    Article  PubMed  Google Scholar 

  • Knoeferle, P., Crocker, M. W., Scheepers, C., & Pickering, M. J. (2005). The influence of the immediate visual context on incremental thematic role-assignment: Evidence from eye-movements in depicted events. Cognition, 95(1), 95–127.

    Article  PubMed  Google Scholar 

  • Knoeferle, P., & Crocker, M. W. (2006). The coordinated interplay of scene, utterance, and world knowledge: evidence from eye tracking. Cognitive Science, 30(3), 481–529.

    Article  PubMed  Google Scholar 

  • Kukona, A., Tabor, W. (2011). Multi-word bottom-up effects in the visual world paradigm. Poster presented at the 17th Annual Architectures and Mechanisms for Language Processing Conference, Paris, France.

    Google Scholar 

  • Lavie, N. (2005). Distracted and confused?: selective attention under load. Trends in Cognitive Sciences, 9, 75–82.

    Article  PubMed  Google Scholar 

  • Land, M. F., & Hayhoe, M. (2001). In what ways do eye movements contribute to everyday activities? Vision Research, 41(25), 3559–3565.

    Article  PubMed  Google Scholar 

  • Landau, B., Jackendoff, R. (1993). What and where in spatial language and spatial cognition. Behavioral and brain sciences, 16, 217–217.

    Google Scholar 

  • Logan, G. D. (1978). Attention in character classification: Evidence for the automaticity of component stages. Journal of Experimental Psychology: General, 107, 32–63.

    Article  Google Scholar 

  • Lupyan, G., & Swingley, D. (2012). Self-directed speech affects visual search performance. The Quarterly Journal of Experimental Psychology, 65(6), 1068–1085.

    Article  PubMed  Google Scholar 

  • Lupyan, G., Thompson-Schill, S. L., & Swingley, D. (2010). Conceptual penetration of visual processing. Psychological Science, 21, 682–691.

    Article  PubMed Central  PubMed  Google Scholar 

  • Lupyan, G., & Ward, E. J. (2013). Language can boost otherwise unseen objects into visual awareness. Proceedings of the National Academy of Sciences, 110(35), 14196–14201.

    Article  Google Scholar 

  • Mani, N., & Huettig, F. (2012). Prediction during language processing is a piece of cake—But only for skilled producers. Journal of Experimental Psychology: Human Perception and Performance, 38(4), 843–847.

    PubMed  Google Scholar 

  • Mani, N., Durrant, S., & Floccia, C. (2012). Activation of phonological and semantic codes in toddlers. Journal of Memory and Language, 66(4), 612–622.

    Article  Google Scholar 

  • Marian, V., & Spivey, M. (2003a). Bilingual and monolingual processing of competing lexical items. Applied Psycholinguistics, 24, 173–193.

    Article  Google Scholar 

  • Marian, V., Spivey, M. (2003b). Competing activation in bilingual language processing: Within and between-language competition. Bilingualism: Language and Cognition, 6, 97–115.

    Google Scholar 

  • Marslen-Wilson, W., & Zwitserlood, P. (1989). Accessing spoken words: The importance of word Knoeferle, P., Crocker, M. W., Scheepers, C., Pickering, M. J. (2005). The influence of the immediate visual context on incremental thematic role-assignment: Evidence from eye-movements in depicted events. Cognition, 95(1), 95-127.onsets. Journal of Experimental Psychology: Human Perception and Performance, 15(3), 576–585.

    Google Scholar 

  • Marr, D. (1982). Vision. New York: W. H. Freeman and Co.

    Google Scholar 

  • Mayberry, M. R., Crocker, M. W., & Knoeferle, P. (2009). Learning to attend: A connectionist model of situated language comprehension. Cognitive science, 33(3), 449–496.

    Article  PubMed  Google Scholar 

  • McQueen, J. M., & Viebahn, M. C. (2007). Tracking recognition of spoken words by tracking looks to printed words. Quarterly Journal of Experimental Psychology, 60(5), 661–671. doi:10.1080/17470210601183890.

    Article  Google Scholar 

  • Meier, B. P., Schnall, S., Schwarz, N., & Bargh, J. A. (2012). Embodiment in social psychology. Topics in cognitive science, 4(4), 705–716.

    Article  PubMed  Google Scholar 

  • Meyer, A. S., Sleiderink, A. M., & Levelt, W. J. (1998). Viewing and naming objects: Eye movements during noun phrase production. Cognition, 66(2), B25–B33.

    Article  PubMed  Google Scholar 

  • Mishra, R. K. (2009). Interface of language and visual attention: Evidence from production and comprehension. Progress in Brain Research, 176, 277–292.

    Article  PubMed  Google Scholar 

  • Mishra, R. K., & Marmolejo-Ramos, F. (2010). On the mental representations originating during the interaction between language and vision. Cognitive Processing, 11(4), 295–305.

    Article  PubMed  Google Scholar 

  • Mishra, R. K., & Singh, N. (2010). Online fictive motion understanding: An eye-movement study with Hindi. Metaphor & Symbol, 25(3), 144–161.

    Article  Google Scholar 

  • Mishra, R. K., Singh, N., Pandey, A., & Huettig, F. (2012a). Spoken language-mediated anticipatory eye movements are modulated by reading ability: Evidence from Indian low and high literates. Journal of Eye Movement Research, 5(1), 1–10.

    Google Scholar 

  • Mishra, R. K., Olivers, C. N., Huettig, F. (2012). Spoken language and the decision to move the eyes: To what extent are language-mediated eye movements automatic?. In Progress in Brain Research, Vol. 202, Decision Making: Neural and Behavioural Approaches.

    Google Scholar 

  • Mishra, R. K., Huettig, F., & Olivers, C. N. (2013). Automaticity and conscious decisions during language-mediated eye gaze in the visual world. In N. Srinivasan & V. S. C. Pammi (Eds.), Progress in Brain Research: Decision Making: Neural and Behavioral Approaches. Amsterdam: Elsevier.

    Google Scholar 

  • Mishra, R. K., & Singh, N. (2014a). Language non-selective activation of orthography during spoken word processing in Hindi-English sequential bilinguals: an eye tracking visual world study. Reading and Writing, 27(1), 129–151. doi:10.1007/s11145-013-9436-5.

    Article  Google Scholar 

  • Mishra, R. K., & Singh, S. (2014b). Activation of shape and semantic information during ambiguous homophone processing: eye tracking evidence from Hindi. Cognitive Processing, 15(4), 451–465.

    Article  PubMed  Google Scholar 

  • Moores, E., Laiti, L., & Chelazzi, L. (2003). Associative knowledge controls deployment of visual selective attention. Nature Neuroscience, 6, 182–189.

    Article  PubMed  Google Scholar 

  • Moors, A., & De Houwer, J. (2006). Automaticity: a theoretical and conceptual analysis. Psychological Bulletin, 132(2), 297–326.

    Article  PubMed  Google Scholar 

  • Noudoost, B., Chang, M. H., Steinmetz, N. A., & Moore, T. (2010). Top-down control of visual attention. Current Opinion in Neurobiology, 20(2), 183–190.

    Article  PubMed Central  PubMed  Google Scholar 

  • O’Regan, J. K., & Noë, A. (2001). A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences, 24(5), 939–972.

    Article  PubMed  Google Scholar 

  • Posner, M. I. (2012). Attention in a social world. New York: Oxford University Press.

    Book  Google Scholar 

  • Pylyshyn, Z. W., & Storm, R. W. (1988). Tracking multiple independent targets: Evidence for a parallel tracking mechanism*. Spatial Vision, 3(3), 179–197.

    Article  PubMed  Google Scholar 

  • Pylyshyn, Z. (1999). Is vision continuous with cognition? The case for cognitive impenetrability of visual perception. Behavioural and brain sciences, 22(3), 341–365.

    Google Scholar 

  • Pylyshyn, Z. (2007). Things and places: How the Mind connects with the World. Cambridge: MIT Press.

    Google Scholar 

  • Pylyshyn, Z. (2009). The empirical case for bare demonstratives in vision. Compositionality, Context and Semantic Values. Studies in Linguistics and Philosophy Volume 85, pp 254–274.

    Google Scholar 

  • Rayner, K., & Duffy, S. A. (1986). Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity. Memory & Cognition, 14(3), 191–201.

    Article  Google Scholar 

  • Richardson, D., & Matlock, T. (2007). The integration of figurative language and static depictions: An eye movement study of fictive motion. Cognition, 102(1), 129–138.

    Article  PubMed  Google Scholar 

  • Richardson, D. C., Altmann, G. T., Spivey, M. J., Hoover, M. A. (2009). Much ado about eye movements to nothing: a response to Ferreira et al.: Taking a new look at looking at nothing. Trends in Cognitive Sciences, 13(6), 235–236.

    Google Scholar 

  • Rolfs, M., Jonikaitis, D., Deubel, H., & Cavanagh, P. (2011). Predictive remapping of attention across eye movements. Nature Neuroscience, 14(2), 252–256.

    Article  PubMed  Google Scholar 

  • Rommers, J., Meyer, A. S., Praamstra, P., & Huettig, F. (2013). The contents of predictions in sentence comprehension: Activation of the shape of objects before they are referred to. Neuropsychologia, 51(3), 437–447.

    Article  PubMed  Google Scholar 

  • Salverda, A. P., & Altmann, G. (2011). Attentional capture of objects referred to by spoken language. Journal of Experimental Psychology: Human Perception and Performance, 37(4), 1122–1133.

    PubMed Central  PubMed  Google Scholar 

  • Salverda, A. P., Brown, M., & Tanenhaus, M. K. (2011). A goal-based perspective on eye movements in visual world studies. Acta Psychologica, 137(2), 172–180.

    Article  PubMed Central  PubMed  Google Scholar 

  • Saslow, M. G. (1967). Effects of components of displacement-step stimuli upon latency for saccadic eye movement. JOSA, 57(8), 1024–1029.

    Article  Google Scholar 

  • Shtyrov, Y., Kimppa, L., Pulvermüller, F., Kujala, T.(2010). Event-related potentials reflecting the frequency of unattended spoken words: a neuronal index of connection strength in lexical memory circuits?. Neuroimage, 15:55(2), 658–68.

    Google Scholar 

  • Soto, D., & Humphreys, G. W. (2007). Automatic guidance of visual attention from verbal working memory. Journal of Experimental Psychology: Human Perception and Performance, 33(3), 730–737.

    PubMed  Google Scholar 

  • Spivey, M. J., & Marian, V. (1999). Cross talk between native and second languages: Partial Activation of an Irrelevant Lexicon. Psychological Science, 10(3), 281–284.

    Article  Google Scholar 

  • Spivey, M. J., & Geng, J. J. (2001). Oculomotor mechanisms activated by imagery and memory: Eye movements to absent objects. Psychological Research, 65(4), 235–241.

    Article  PubMed  Google Scholar 

  • Spivey, M. J., Tyler, M. J., Eberhard, K. M., & Tanenhaus, M. K. (2001). Linguistically mediated visual search. Psychological Science, 12(4), 282–286.

    Article  PubMed  Google Scholar 

  • Spivey, M. J., & Gonzalez-Marquez, M. (2003). Rescuing generative linguistics: Too little, too late? Behavioral and Brain Sciences, 26(06), 690–691.

    Article  Google Scholar 

  • Spivey, M. J., & Dale, R. (2006). Continuous dynamics in real-time cognition. Current Directions in Psychological Science, 15(5), 207–211.

    Article  Google Scholar 

  • Spivey, M. J. (2007). The continuity of mind. Oxford, England: Oxford University Press.

    Google Scholar 

  • Sunderman, G. L., & Priya, K. (2012). Translation recognition in highly proficient Hindi-English bilinguals: The influence of different scripts but connectable phonologies. Language and Cognitive Processes, 27(9), 1265–1285.

    Article  Google Scholar 

  • Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., & Sedivy, J. E. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 1632–1634.

    Article  PubMed  Google Scholar 

  • Telling, A. L., Kumar, S., Meyer, A. S., & Humphreys, G. W. (2010). Electrophysiological evidence of semantic interference in visual search. Journal of Cognitive Neuroscience, 22(10), 2212–2225. doi:10.1162/jocn.2009.21348.

    Article  PubMed  Google Scholar 

  • Trueswell, J. C., & Tanenhaus, M. K. (2005). Approaches to studying world-situated language use: Bridging the language-as-product and language-as-action traditions. Cambridge: The MIT Press.

    Google Scholar 

  • Vecera, S. P., & Farah, M. J. (1994). Does visual attention select objects or locations? Journal of Experimental Psychology: General, 123(2), 146–160.

    Article  Google Scholar 

  • Vu, H., Kellas, G., & Paul, S. T. (1998). Sources of sentence constraint on lexical ambiguity resolution. Memory & cognition, 26(5), 979–1001.

    Article  Google Scholar 

  • Weber, A., & Cutler, A. (2004). Lexical competition in non-native spoken- word recognition. Journal of Memory and Language, 50, 1–25.

    Article  Google Scholar 

  • Wolfe, J. M. (1994). Guided Search 2.0: A Revised Model of Visual Search. Psychonomic Bulletin & Review, 1(2), 202–238.

    Article  Google Scholar 

  • Wolfe, J. M., Horowitz, T. S., Kenner, N., Hyle, M., & Vasan, N. (2004). How fast can you change your mind? The speed of top-down guidance in visual search. Vision Research, 44(12), 1411–1426.

    Article  PubMed  Google Scholar 

  • Yee, E., & Sedivy, J. C. (2006). Eye movements to pictures reveal transient semantic activation during spoken word recognition. Journal of Experimental Psychology. Learning, Memory, and Cognition, 32(1), 1–14.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ramesh Kumar Mishra .

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer India

About this chapter

Cite this chapter

Mishra, R.K. (2015). Attention, Language and Vision. In: Interaction Between Attention and Language Systems in Humans. Springer, New Delhi. https://doi.org/10.1007/978-81-322-2592-8_7

Download citation

Publish with us

Policies and ethics