Abstract
So far we have looked at scenarios where information only from one modality was used during language processing. Recently, several researchers have explored how spoken language influences attentional selections in simple looking and listening tasks using eye tracking as methodology. Attentional mechanisms influence how linguistic and non-linguistic information are processed together. Eye movements during language comprehension in the presence of visual images have revealed how conceptual activation leads to shifts in attention and cognition. The aim of this chapter is to examine how linguistic and visual information interact dynamically during cognitive processing in different situations. Data from both monolingual and bilingual studies are considered to examine issues of lexical access and visual search. First, the chapter covers some theoretical issues related to language and vision interaction and later, findings from the visual world studies are examined.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Altmann, G. (2004). Language-mediated eye movements in the absence of a visual world: The blank screen paradigm. Cognition, 93(2), B79–B87.
Altmann, G. T. M. (2011). Language can mediate eye movement control within 100 milliseconds, regardless of whether there is anything to move the eyes to. Acta Psychol. (Amst.), 137, 190–200.
Altmann, G. T., & Kamide, Y. (2007). The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing. Journal of Memory and Language, 57(4), 502–518.
Altmann, G., & Kamide, Y. (2009). Discourse-mediation of the mapping between language and the visual world: Eye movements and mental representation. Cognition, 111(1), 55–71.
Allopenna, P. D., Magnuson, J. S., & Tanenhaus, M. K. (1998). Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory and Language, 38(4), 419–439.
Bertels, J., Kolinsky, R., Bernaerts, A., & Morais, J. (2011). Effects of emotional spoken words on exogenous attentional orienting. Journal of Cognitive Psychology, 23(4), 435–452.
Buswell, G. T. (1935). How people look at pictures: a study of the psychology and perception in art. Oxford, England: University of Chicago Press.
Chambers, C. G., Tanenhaus, M. K., Eberhard, K. M., Filip, H., & Carlson, G. N. (2002). Circumscribing referential domains during real-time language comprehension. Journal of Memory and Language, 47(1), 30–49.
Chomsky, N. (2007). Approaching UG from below. Interfaces + recursion = language, 1–29.
Cleland, A. A., Tamminen, J., Quinlan, P. T., & Gaskell, M. G. (2012). Spoken word processing creates a lexical bottleneck. Language and Cognitive Processes, 27(4), 572–593.
Cooper, R. M. (1974). The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6(1), 84–107.
Dahan, D., & Tanenhaus, M. K. (2005). Looking at the rope when looking for the snake: Conceptually mediated eye movements during spoken-word recognition. Psychological Bulletin & Review, 12, 455–459.
De Houwer, J., Smith, C. (in press). Go with your gut! Effects of Affect Misattribution Procedures become stronger when participants are encouraged to rely on their gut feelings. Social Psychology.
Desimone, R., & Duncan, J. (1995). Neural mechanisms of selective visual attention. Annual Review of Neuroscience, 18, 193–222.
Downing, P. E. (2000). Interactions between visual working memory and selective attention. Psychological Science, 11(6), 467–473.
Ellermeier, W., & Zimmer, K. (1997). Individual differences in susceptibility to the irrelevant speech effect. The Journal of the Acoustical Society of America, 102(4), 2191–2199.
Elliott, E. M., & Briganti, A. M. (2012). Investigating the role of attentional processes in the irrelevant speech effect. Acta Psychologica, 140, 64–74.
Ferreira, F., Apel, J., & Henderson, J. M. (2008). Taking a new look at looking at nothing. Trends in cognitive sciences, 12(11), 405–410.
Findlay, J. M., Brown, V., & Gilchrist, I. D. (2001). Saccade target selection in visual search: the effect of information from the previous fixation. Vision Research, 41(1), 87–95.
Fodor, J. A. (1987). Psychosemantics: The problem of meaning in the philosophy of mind. Cambridge: The MIT Press.
Foulsham, T., & Underwood, G. (2011). If visual saliency predicts search, then why? evidence from normal and gaze-contingent search tasks in natural scenes. Cognitive Computation, 3(1), 48–63.
Guo, T., Misra, M., Tam, J. W., & Kroll, J. F. (2012). On the time course of accessing meaning in a second language: An electrophysiological investigation of translation recognition. Journal of Experimental Psychology. Learning, Memory, and Cognition, 38, 1165–1186.
Henderson, J. M. (2003). Human gaze control during real-world scene perception. Trends in cognitive sciences, 7(11), 498–504.
Henderson, J. M., & Ferreira, F. (2004). Scene perception for psycholinguists. In J. M. Henderson & F. Ferreira (Eds.), The interface of language, vision, and action: Eye movements and the visual world (pp. 1–58). New York: Psychology Press.
Huang, Y. T., & Snedeker, J. (2011). Logic and conversation revisited: Evidence for a division between semantic and pragmatic content in real-time language comprehension. Language and Cognitive Processes, 26(8), 1161–1172.
Huettig, F., & Altmann, G. T. (2005). Word meaning and the control of eye fixation: Semantic competitor effects and the visual world paradigm. Cognition, 96(1), B23–B32.
Huettig, F., & Altmann, G. T. M. (2007). Visual-shape competition during language-mediated attention is based on lexical input and not modulated by contextual appropriateness. Visual Cognition, 15(8), 985–1018. doi:10.1080/13506280601130875.
Huettig, F., & McQueen, J. M. (2007). The tug of war between phonological, semantic and shape information in language-mediated visual search. Journal of Memory and Language, 57(4), 460–482. doi:10.1016/j.jml.2007.02.001.
Huettig, F., Mishra, R. K., & Olivers, C. N. (2012). On the mechanisms and representations of language-mediated visual attention. Frontiers in Cognition, 2, 394.
Huettig, F., Olivers, C. N. L., & Hartsuiker, R. J. (2011a). Looking, language, and memory: Bridging research from the visual world and visual search paradigms. Acta Psychologica, 137, 138–150. doi:10.1016/j.actpsy.2010.07.013.
Huettig, F., Rommers, J., & Meyer, A. S. (2011b). Using the visual world paradigm to study language processing: A review and critical evaluation. Acta Psychologica, 137(2), 151–171.
Huettig, F., Singh, N., & Mishra, R. K. (2011c). Language-mediated visual orienting behavior in low and high literates. Frontiers in Psychology, 2, 285. doi:10.3389/fpsyg.2011.00285.
Jackendoff, R. (1997). The architecture of the language faculty (Vol. 28). Cambridge: The MIT Press.
Jackendoff, R. (2011). What is the human language faculty?: Two views. Language, 87(3), 586–624.
Johansson, R., Holsanova, J., & Holmqvist, K. (2006). Pictures and spoken descriptions elicit similar eye movements during mental imagery, both in light and in complete darkness. Cognitive Science, 30(6), 1053–1079.
Jones, D. M., & Macken, W. J. (1993). Irrelevant tones produce an irrelevant speech effect: implications for phonological coding in working memory. Journal of Experimental Psychology. Learning, Memory, and Cognition, 19(2), 369–381.
Jesperson, O. (1924). The Philosophy of Grammar. University of Chicago Press.
Kamide, Y., Altmann, G. T., & Haywood, S. L. (2003). The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye movements. Journal of Memory and Language, 49(1), 133–156.
Klatte, M., Lachmann, T., Schlittmeier, S., & Hellbrück, J. (2010). The irrelevant sound effect in short-term memory: Is there developmental change? European Journal of Cognitive Psychology, 22(8), 1168–1191.
Klein, R. M. (2000). Inhibition of return. Trends in cognitive sciences, 4(4), 138–147.
Knoeferle, P., Crocker, M. W., Scheepers, C., & Pickering, M. J. (2005). The influence of the immediate visual context on incremental thematic role-assignment: Evidence from eye-movements in depicted events. Cognition, 95(1), 95–127.
Knoeferle, P., & Crocker, M. W. (2006). The coordinated interplay of scene, utterance, and world knowledge: evidence from eye tracking. Cognitive Science, 30(3), 481–529.
Kukona, A., Tabor, W. (2011). Multi-word bottom-up effects in the visual world paradigm. Poster presented at the 17th Annual Architectures and Mechanisms for Language Processing Conference, Paris, France.
Lavie, N. (2005). Distracted and confused?: selective attention under load. Trends in Cognitive Sciences, 9, 75–82.
Land, M. F., & Hayhoe, M. (2001). In what ways do eye movements contribute to everyday activities? Vision Research, 41(25), 3559–3565.
Landau, B., Jackendoff, R. (1993). What and where in spatial language and spatial cognition. Behavioral and brain sciences, 16, 217–217.
Logan, G. D. (1978). Attention in character classification: Evidence for the automaticity of component stages. Journal of Experimental Psychology: General, 107, 32–63.
Lupyan, G., & Swingley, D. (2012). Self-directed speech affects visual search performance. The Quarterly Journal of Experimental Psychology, 65(6), 1068–1085.
Lupyan, G., Thompson-Schill, S. L., & Swingley, D. (2010). Conceptual penetration of visual processing. Psychological Science, 21, 682–691.
Lupyan, G., & Ward, E. J. (2013). Language can boost otherwise unseen objects into visual awareness. Proceedings of the National Academy of Sciences, 110(35), 14196–14201.
Mani, N., & Huettig, F. (2012). Prediction during language processing is a piece of cake—But only for skilled producers. Journal of Experimental Psychology: Human Perception and Performance, 38(4), 843–847.
Mani, N., Durrant, S., & Floccia, C. (2012). Activation of phonological and semantic codes in toddlers. Journal of Memory and Language, 66(4), 612–622.
Marian, V., & Spivey, M. (2003a). Bilingual and monolingual processing of competing lexical items. Applied Psycholinguistics, 24, 173–193.
Marian, V., Spivey, M. (2003b). Competing activation in bilingual language processing: Within and between-language competition. Bilingualism: Language and Cognition, 6, 97–115.
Marslen-Wilson, W., & Zwitserlood, P. (1989). Accessing spoken words: The importance of word Knoeferle, P., Crocker, M. W., Scheepers, C., Pickering, M. J. (2005). The influence of the immediate visual context on incremental thematic role-assignment: Evidence from eye-movements in depicted events. Cognition, 95(1), 95-127.onsets. Journal of Experimental Psychology: Human Perception and Performance, 15(3), 576–585.
Marr, D. (1982). Vision. New York: W. H. Freeman and Co.
Mayberry, M. R., Crocker, M. W., & Knoeferle, P. (2009). Learning to attend: A connectionist model of situated language comprehension. Cognitive science, 33(3), 449–496.
McQueen, J. M., & Viebahn, M. C. (2007). Tracking recognition of spoken words by tracking looks to printed words. Quarterly Journal of Experimental Psychology, 60(5), 661–671. doi:10.1080/17470210601183890.
Meier, B. P., Schnall, S., Schwarz, N., & Bargh, J. A. (2012). Embodiment in social psychology. Topics in cognitive science, 4(4), 705–716.
Meyer, A. S., Sleiderink, A. M., & Levelt, W. J. (1998). Viewing and naming objects: Eye movements during noun phrase production. Cognition, 66(2), B25–B33.
Mishra, R. K. (2009). Interface of language and visual attention: Evidence from production and comprehension. Progress in Brain Research, 176, 277–292.
Mishra, R. K., & Marmolejo-Ramos, F. (2010). On the mental representations originating during the interaction between language and vision. Cognitive Processing, 11(4), 295–305.
Mishra, R. K., & Singh, N. (2010). Online fictive motion understanding: An eye-movement study with Hindi. Metaphor & Symbol, 25(3), 144–161.
Mishra, R. K., Singh, N., Pandey, A., & Huettig, F. (2012a). Spoken language-mediated anticipatory eye movements are modulated by reading ability: Evidence from Indian low and high literates. Journal of Eye Movement Research, 5(1), 1–10.
Mishra, R. K., Olivers, C. N., Huettig, F. (2012). Spoken language and the decision to move the eyes: To what extent are language-mediated eye movements automatic?. In Progress in Brain Research, Vol. 202, Decision Making: Neural and Behavioural Approaches.
Mishra, R. K., Huettig, F., & Olivers, C. N. (2013). Automaticity and conscious decisions during language-mediated eye gaze in the visual world. In N. Srinivasan & V. S. C. Pammi (Eds.), Progress in Brain Research: Decision Making: Neural and Behavioral Approaches. Amsterdam: Elsevier.
Mishra, R. K., & Singh, N. (2014a). Language non-selective activation of orthography during spoken word processing in Hindi-English sequential bilinguals: an eye tracking visual world study. Reading and Writing, 27(1), 129–151. doi:10.1007/s11145-013-9436-5.
Mishra, R. K., & Singh, S. (2014b). Activation of shape and semantic information during ambiguous homophone processing: eye tracking evidence from Hindi. Cognitive Processing, 15(4), 451–465.
Moores, E., Laiti, L., & Chelazzi, L. (2003). Associative knowledge controls deployment of visual selective attention. Nature Neuroscience, 6, 182–189.
Moors, A., & De Houwer, J. (2006). Automaticity: a theoretical and conceptual analysis. Psychological Bulletin, 132(2), 297–326.
Noudoost, B., Chang, M. H., Steinmetz, N. A., & Moore, T. (2010). Top-down control of visual attention. Current Opinion in Neurobiology, 20(2), 183–190.
O’Regan, J. K., & Noë, A. (2001). A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences, 24(5), 939–972.
Posner, M. I. (2012). Attention in a social world. New York: Oxford University Press.
Pylyshyn, Z. W., & Storm, R. W. (1988). Tracking multiple independent targets: Evidence for a parallel tracking mechanism*. Spatial Vision, 3(3), 179–197.
Pylyshyn, Z. (1999). Is vision continuous with cognition? The case for cognitive impenetrability of visual perception. Behavioural and brain sciences, 22(3), 341–365.
Pylyshyn, Z. (2007). Things and places: How the Mind connects with the World. Cambridge: MIT Press.
Pylyshyn, Z. (2009). The empirical case for bare demonstratives in vision. Compositionality, Context and Semantic Values. Studies in Linguistics and Philosophy Volume 85, pp 254–274.
Rayner, K., & Duffy, S. A. (1986). Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity. Memory & Cognition, 14(3), 191–201.
Richardson, D., & Matlock, T. (2007). The integration of figurative language and static depictions: An eye movement study of fictive motion. Cognition, 102(1), 129–138.
Richardson, D. C., Altmann, G. T., Spivey, M. J., Hoover, M. A. (2009). Much ado about eye movements to nothing: a response to Ferreira et al.: Taking a new look at looking at nothing. Trends in Cognitive Sciences, 13(6), 235–236.
Rolfs, M., Jonikaitis, D., Deubel, H., & Cavanagh, P. (2011). Predictive remapping of attention across eye movements. Nature Neuroscience, 14(2), 252–256.
Rommers, J., Meyer, A. S., Praamstra, P., & Huettig, F. (2013). The contents of predictions in sentence comprehension: Activation of the shape of objects before they are referred to. Neuropsychologia, 51(3), 437–447.
Salverda, A. P., & Altmann, G. (2011). Attentional capture of objects referred to by spoken language. Journal of Experimental Psychology: Human Perception and Performance, 37(4), 1122–1133.
Salverda, A. P., Brown, M., & Tanenhaus, M. K. (2011). A goal-based perspective on eye movements in visual world studies. Acta Psychologica, 137(2), 172–180.
Saslow, M. G. (1967). Effects of components of displacement-step stimuli upon latency for saccadic eye movement. JOSA, 57(8), 1024–1029.
Shtyrov, Y., Kimppa, L., Pulvermüller, F., Kujala, T.(2010). Event-related potentials reflecting the frequency of unattended spoken words: a neuronal index of connection strength in lexical memory circuits?. Neuroimage, 15:55(2), 658–68.
Soto, D., & Humphreys, G. W. (2007). Automatic guidance of visual attention from verbal working memory. Journal of Experimental Psychology: Human Perception and Performance, 33(3), 730–737.
Spivey, M. J., & Marian, V. (1999). Cross talk between native and second languages: Partial Activation of an Irrelevant Lexicon. Psychological Science, 10(3), 281–284.
Spivey, M. J., & Geng, J. J. (2001). Oculomotor mechanisms activated by imagery and memory: Eye movements to absent objects. Psychological Research, 65(4), 235–241.
Spivey, M. J., Tyler, M. J., Eberhard, K. M., & Tanenhaus, M. K. (2001). Linguistically mediated visual search. Psychological Science, 12(4), 282–286.
Spivey, M. J., & Gonzalez-Marquez, M. (2003). Rescuing generative linguistics: Too little, too late? Behavioral and Brain Sciences, 26(06), 690–691.
Spivey, M. J., & Dale, R. (2006). Continuous dynamics in real-time cognition. Current Directions in Psychological Science, 15(5), 207–211.
Spivey, M. J. (2007). The continuity of mind. Oxford, England: Oxford University Press.
Sunderman, G. L., & Priya, K. (2012). Translation recognition in highly proficient Hindi-English bilinguals: The influence of different scripts but connectable phonologies. Language and Cognitive Processes, 27(9), 1265–1285.
Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., & Sedivy, J. E. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 1632–1634.
Telling, A. L., Kumar, S., Meyer, A. S., & Humphreys, G. W. (2010). Electrophysiological evidence of semantic interference in visual search. Journal of Cognitive Neuroscience, 22(10), 2212–2225. doi:10.1162/jocn.2009.21348.
Trueswell, J. C., & Tanenhaus, M. K. (2005). Approaches to studying world-situated language use: Bridging the language-as-product and language-as-action traditions. Cambridge: The MIT Press.
Vecera, S. P., & Farah, M. J. (1994). Does visual attention select objects or locations? Journal of Experimental Psychology: General, 123(2), 146–160.
Vu, H., Kellas, G., & Paul, S. T. (1998). Sources of sentence constraint on lexical ambiguity resolution. Memory & cognition, 26(5), 979–1001.
Weber, A., & Cutler, A. (2004). Lexical competition in non-native spoken- word recognition. Journal of Memory and Language, 50, 1–25.
Wolfe, J. M. (1994). Guided Search 2.0: A Revised Model of Visual Search. Psychonomic Bulletin & Review, 1(2), 202–238.
Wolfe, J. M., Horowitz, T. S., Kenner, N., Hyle, M., & Vasan, N. (2004). How fast can you change your mind? The speed of top-down guidance in visual search. Vision Research, 44(12), 1411–1426.
Yee, E., & Sedivy, J. C. (2006). Eye movements to pictures reveal transient semantic activation during spoken word recognition. Journal of Experimental Psychology. Learning, Memory, and Cognition, 32(1), 1–14.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2015 Springer India
About this chapter
Cite this chapter
Mishra, R.K. (2015). Attention, Language and Vision. In: Interaction Between Attention and Language Systems in Humans. Springer, New Delhi. https://doi.org/10.1007/978-81-322-2592-8_7
Download citation
DOI: https://doi.org/10.1007/978-81-322-2592-8_7
Published:
Publisher Name: Springer, New Delhi
Print ISBN: 978-81-322-2591-1
Online ISBN: 978-81-322-2592-8
eBook Packages: Behavioral ScienceBehavioral Science and Psychology (R0)