Abstract
Three eye-tracking experiments investigated the impact of the complexity of the visual environment on the likelihood of word–object mapping taking place at phonological, semantic and visual levels of representation during language-mediated visual search. Dutch participants heard spoken target words while looking at four objects embedded in displays of different complexity and indicated the presence or absence of the target object. During filler trials the target objects were present, but during experimental trials they were absent and the display contained various competitor objects. For example, given the target word “beker” (beaker), the display contained a phonological (a beaver, bever), a shape (a bobbin, klos), a semantic (a fork, vork) competitor, and an unrelated distractor (an umbrella, paraplu). When objects were presented in simple four-object displays (Experiment 2), there were clear attentional biases to all three types of competitors replicating earlier research (Huettig and McQueen 2007). When the objects were embedded in complex scenes including four human-like characters or four meaningless visual shapes (Experiments 1, 3), there were biases in looks to visual and semantic but not to phonological competitors. In both experiments, however, we observed evidence for inhibition in looks to phonological competitors, which suggests that the phonological forms of the objects nevertheless had been retrieved. These findings suggest that phonological word–object mapping is contingent upon the nature of the visual environment and add to a growing body of evidence that the nature of our visual surroundings induces particular modes of processing during language-mediated visual search.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
See Huettig and McQueen (2007) for a detailed description of the materials and the results of seven norming studies. Five of the original item sets were removed from both Experiment 1 and all subsequent experiments, because they contained pictures of body parts present in the human-like characters.
- 2.
Prior to Experiment 1 (and Experiment 2) participants carried out an object naming task during which their eye movements were recorded. The task was independent of the subsequent main experiment and required participants to look at one object at a time presented at the centre of the computer screen and name it as fast as possible. Sixty objects which were not used in the main experiment had to be named. The task lasted around 5 min and we observed no obvious impact on participants’ performance in either Experiment 1 or 2 nor did they report anecdotal effects.
- 3.
There was one item on which more than 50 % of the participant sample had responded incorrectly. This item was removed from further analyses, and was removed from the subsequent experiments.
- 4.
Note that our main aim interest was not in the exact timing of the shifts to semantic and shape competitors. What is clear from the data (see Fig. 3.2) is that participants started to shift their eye gaze to both competitors after the target word had been heard.
- 5.
Due to an error, one experimental item had to be removed from the analysis.
References
Allopenna, P. D., Magnuson, J. S., & Tanenhaus, M. K. (1998). Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory and Language, 38(4), 419–439.
Andersson, R., Ferreira, F., & Henderson, J. M. (2011). I see what you’re saying: The integration of complex speech and scenes during language comprehension. Acta Psychologica, 137(2), 208–216.
Boyce, S. J., & Pollatsek, A. (1992). Identification of objects in scenes: The role of scene background in object naming. Journal of Experimental Psychology. Learning, Memory, and Cognition, 18(3), 531–543.
Cooper, R. M. (1974). Control of eye fixation by meaning of spoken language: New methodology for real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6(1), 84–107.
Cree, G. S., & McRae, K. (2003). Analyzing the factors underlying the structure and computation of the meaning of chipmunk, cherry, chisel, cheese, and cello (and many other such concrete nouns). Journal of Experimental Psychology: General, 132(2), 163–201.
Dahan, D., & Tanenhaus, M. K. (2005). Looking at the rope when looking for the snake: Conceptually mediated eye movements during spoken-word recognition. Psychonomic Bulletin and Review, 12(3), 453–459.
De Graef, P. (1998). Prefixational object perception in scenes: Objects popping out of schemas. In G. Underwood (Ed.), Eye guidance in reading and scene perception (pp. 313–336). Oxford, UK: Elsevier.
Duñabeitia, J. A., Avilés, A., Afonso, O., Scheepers, C., & Carreiras, M. (2009). Qualitative differences in the representation of abstract versus concrete words: Evidence from the visual-world paradigm. Cognition, 110(2), 284–292.
Fiser, J., & Aslin, R. N. (2002). Statistical learning of higher-order temporal structure from visual shape sequences. Journal of Experimental Psychology: Learning, Memory, and Cognition, 28(3), 458–467.
Frost, R. (1998). Toward a strong phonological theory of visual word recognition: True issues and false trails. Psychological Bulletin, 123(1), 71–99.
Han, S. W., & Kim, M. S. (2009). Do the contents of working memory capture attention? Yes, but cognitive control matters. Journal of Experimental Psychology: Human Perception and Performance, 35(5), 1292–1302.
Huettig, F., & Altmann, G. T. M. (2004). The online processing of ambiguous and unambiguous words in context: Evidence from head-mounted eye-tracking. In M. Carreiras, & C. Clifton (Eds.), The on-line study of sentence comprehension: Eyetracking, ERP and beyond (pp. 187−207). New York: Psychology Press.
Huettig, F., & Altmann, G. T. M. (2005). Word meaning and the control of eye fixation: Semantic competitor effects and the visual world paradigm. Cognition, 96(1), 23–32.
Huettig, F., & Altmann, G. T. M. (2007). Visual-shape competition during language-mediated attention is based on lexical input and not modulated by contextual appropriateness. Visual Cognition, 15(8), 985–1018.
Huettig, F., & McQueen, J. M. (2007). The tug of war between phonological, semantic and shape information in language-mediated visual search. Journal of Memory and Language, 57(4), 460–482.
Huettig, F., & McQueen, J. M. (2011). The nature of the visual environment induces implicit biases during language-mediated visual search. Memory and Cognition, 39(6), 1068–1084.
Huettig, F., Olivers, C. N. L., & Hartsuiker, R. J. (2011a). Looking, language, and memory: Bridging research from the visual world and visual search paradigms. Acta Psychologica, 137(2), 138–150.
Huettig, F., Quinlan, P. T., McDonald, S. A., & Altmann, G. T. M. (2006). Models of high-dimensional semantic space predict language-mediated eye movements in the visual world. Acta Psychologica, 121(1), 65–80.
Huettig, F., Rommers, J., & Meyer, A. S. (2011b). Using the visual world paradigm to study language processing: A review and critical evaluation. Acta Psychologica, 137(2), 151–171.
Huettig, F., Singh, N., & Mishra, R. (2011c). Language-mediated visual orienting behavior in low and high literates. Frontiers in Psychology, 2, 285.
Knoeferle, P., & Crocker, M. W. (2007). The influence of recent scene events on spoken comprehension: Evidence from eye movements. Journal of Memory and Language, 57(4), 519–543.
Landauer, T. K., & Dumais, S. T. (1997). A solution to plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211–240.
McClelland, J. L., & Elman, J. L. (1986). The trace model of speech perception. Cognitive Psychology, 18(1), 1–86.
McQueen, J. M., & Huettig, F. (2014). Interference of spoken word recognition through phonological priming from visual objects and printed words. Attention, Perception, and Psychophysics, 76, 190–200.
Saslow, M. G. (1967). Latency for saccadic eye movement. Journal of the Optical Society of America, 57(8), 1030.
Spivey, M. J., Richardson, D. C., & Fitneva, S. A. (2004). Thinking outside the brain: Spatial indices to visual and linguistic information. In J. Henderson & F. Ferreira (Eds.), The interface of language, vision, and action: Eye movements and the visual world (pp. 161–190). San Diego: CA: Academic Press.
Stolz, J. A., & Besner, D. (1998). Levels of representation in visual word recognition: A dissociation between morphological and semantic processing. Journal of Experimental Psychology: Human Perception and Performance, 24(6), 1642.
Strik, J. A., & Underwood, G. (2007). Low-level visual saliency does not predict change detection in natural scenes. Journal of Vision, 7(10), 1–10.
Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268(5217), 1632–1634.
Van Orden, G. C., Johnston, J. C., & Hale, B. L. (1988). Word identification in reading proceeds from spelling to sound to meaning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14(3), 371.
Võ, M. L.-H., & Wolfe, J. M. (2013). Differential electrophysiological signatures of semantic and syntactic scene processing. Psychological Science, 24(9), 1816–1823.
Yee, E., Overton, E., & Thompson-Schill, S. L. (2009). Looking for meaning: Eye movements are sensitive to overlapping semantic features, not association. Psychonomic Bulletin and Review, 16(5), 869–874.
Yee, E., & Sedivy, J. C. (2006). Eye movements to pictures reveal transient semantic activation during spoken word recognition. Journal of Experimental Psychology. Learning, Memory, and Cognition, 32(1), 1–14.
Acknowledgments
We thank Anna Gastel for drawing the semi-realistic scenes, Neil Bardhan for providing the artificial shapes and Johanne Tromp for assistance in running the experiments.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer India
About this chapter
Cite this chapter
Hintz, F., Huettig, F. (2015). The Complexity of the Visual Environment Modulates Language-Mediated Eye Gaze. In: Mishra, R., Srinivasan, N., Huettig, F. (eds) Attention and Vision in Language Processing. Springer, New Delhi. https://doi.org/10.1007/978-81-322-2443-3_3
Download citation
DOI: https://doi.org/10.1007/978-81-322-2443-3_3
Published:
Publisher Name: Springer, New Delhi
Print ISBN: 978-81-322-2442-6
Online ISBN: 978-81-322-2443-3
eBook Packages: Humanities, Social Sciences and LawSocial Sciences (R0)