Tactile stimulation disambiguates the perception of visual motion paths
- 186 Downloads
Although visual perception traditionally has been considered to be impenetrable by non-visual information, there are a rising number of reports discussing cross-modal influences on visual perception. In two experiments, we investigated how coinciding vibrotactile stimulation affects the perception of two discs that move toward each other, superimpose in the center of the screen, and then move apart. Whereas two discs streaming past each other was the dominant impression when the visual event was presented in isolation, a brief coinciding vibrotactile stimulation at the moment of overlap biased the visual impression toward two discs bouncing off each other (Experiment 1). Further, the vibrotactile stimulation actually changed perceptual processing by reducing the amount of perceived overlap between the discs (Experiment 2), which has been demonstrated to be associated with a higher proportion of bouncing impressions. We propose that tactile-induced quantitative changes in the visual percept might alter the quality of the visual percept (from streaming to bouncing), thereby adding to the understanding of how cross-modal information interacts with early visual perception and how this interaction influences subsequent visual impressions.
KeywordsBouncing/streaming illusion Tactile transients Illusory crescent Visual-tactile interaction
Visual perception traditionally was thought to be almost impenetrable by non-visual information and thus immune to influences of information from different modalities such as audition and touch. However, although visual information may dominate information from other modalities (e.g., Rock & Victor, 1964; Warren, Welch, & McCarthy, 1981), there is neuropsychological (e.g., Falchier, Clavagnier, Barone, & Kennedy, 2002) as well as behavioral (e.g., Shams, Kamitani, & Shimojo, 2000) evidence that non-visual information is capable of altering visual processing. For instance, Sekuler, Sekuler, and Lau (1997) showed that a brief “click” that coincides with the moment of overlap of two moving discs switches the dominant impression of the event from two discs streaming past each other streaming to two discs bouncing off each other. To date, the mechanism driving this qualitative shift in the visual impression is not yet understood, although perceptual (Meyerhoff & Scholl, 2018), attentional (Grassi & Casco, 2009), and cognitive (Grassi & Casco, 2010; Grove, Ashton, Kawachi, & Sakurai, 2012a) mechanisms have been discussed recently.
Beyond audiovisual interactions, vision could also be cross-modally affected by touch. A central challenge of studying cross-modal interactions, however, is to ensure that observed effects actually stem from an integration of the sensory information and not just from a shift in the response criterion (e.g., Odgaard, Arieh, & Marks, 2003). One promising way to isolate perceptual processes is to study cross-modal interactions with illusory percepts (e.g., Shimojo & Shams, 2001) or changes in perceptual performance. Regarding tactile-visual interactions, such changes in performance have been reported for localization tasks (rubber hand illusion; Pavani, Spence, & Driver, 2000) and temporal order judgments (temporal ventriloquism, Keetels & Vroomen, 2008). Further, a compelling demonstration of perceptual interactions between vision and touch comes from the touch-induced flash illusion (Violentyev, Shimojo, & Shams, 2005). In this study, the observers were more likely to report the perception of two visual flashes when two rather than one tactile transients coincided with the presentation of one visual flash. Because the coinciding tactile information impaired the sensitivity of the participants to distinguish between trials with one or two (physical) flashes, Violentyev et al. attributed the quantitative change in the visual impression to perceptual processing.
The current project
Most studies addressing perceived bouncing have studied audio-visual stimuli. However, as interactions between objects in the real world typically occur with coinciding auditory information, the visual impression of the discs bouncing off each other might be influenced from previous exposure (rather than perception; see Rips, 2011). In order to study the impact of coinciding transients themselves, we therefore investigated if (and how) coinciding tactile transients induce bouncing impressions. Given the large differences between tactile and auditory modality in terms of cortical organization (e.g., Rauschecker & Scott, 2009) as well as perceived simultaneity (Fujisaki & Nishida, 2009), this approach is also promising with regard to tracing down the perceptual root of perceived bouncing versus streaming. Our aims were twofold. First, we aimed to demonstrate that coinciding tactile information is capable of inducing qualitative changes in the visual impression of the bouncing/streaming display (Experiment 1). Second, we aimed to show that coinciding tactile information also alters perceptual processing in a similar task in order to isolate perceptual processing from (potential) cognitive influences such as biases or inferences (Experiment 2).
The effect of cross-modal information on perceived bouncing/streaming mostly has been studied with coinciding auditory information (Grove & Sakurai, 2009; Watanabe & Shimojo, 2001; Zhou, Wong, & Sekuler, 2007). In this experiment, we investigated a tactile-visuo variant of the bouncing/streaming illusion in order to study the modality-specifity of the corresponding qualitative change in the visual impression. A similar approach has been reported in the dissertation of Watanabe (2001; for reviews of his data see Shimojo & Shams, 2001; Watanabe and Shimojo, 2005). If coinciding transients in general elicit the impression of discs bouncing off each other, we should observe the typical bouncing vs. streaming illusion also with the vibrotactile transients in this experiment (thus providing an independent replication of Watanabe, 20011).
The final sample of Experiment 1 consisted of 24 students from the University of Trier (19–29 years old, 22 females). Data from one additional participant was replaced because this participant did not vary her responses.2 All participants reported normal or corrected-to-normal vision.
Apparatus, stimuli, and procedure
The experiment took place in a completely darkened room (no daylight, black furniture) all sources of illumination except the stimuli were eliminated. All stimuli were programmed using the PsychoPy libraries (Pierce, 2007). The experiments were presented on a 24-in. TFT screen (60 Hz, 1,920 × 1,200 pixels) at a restricted viewing distance of 73 cm. The vibrotactile stimulation was administered by a tactor (C-2, Engineering Acoustic, Inc.) that was attached to the heel of the non-dominant hand of the participants. The tactor was controlled by a standard amplifier (Power Amplifier Module PM40C, t.amp). In order to prevent any influence of the noise of the vibrating tactors (250 Hz, 0.3 mm peak-to-peak amplitude), our participants wore headphones playing white noise (95 dB) as well as ear plugs (29 dB reduction).
The results of Experiment 1 show that brief vibrotactile stimulations increase the probability of a visual bouncing impression. However, this result was obtained from explicit verbal reports regarding the perceptual impression. In research on auditory-induced bouncing, Grove et al. (2012) have challenged the attribution of changes in reported bouncing/streaming to perceptual processing. When they asked participants to distinguish between trials with objectively bouncing and streaming pairs of objects (implemented by differently dense moving patches), coinciding auditory information altered the response criterion of the participants toward bouncing reports but had no effect on the sensitivity to discriminate between objective bouncing and streaming (see also Grassi & Casco, 2012). From these results, Grove et al. (2012) concluded that the coinciding tones affected decisional processes rather than perception. Recently, however, Meyerhoff and Scholl (2018) showed that coinciding tones reduce the perceived amount of overlap between the moving discs (i.e., auditory-induced illusory crescents). Because less overlap between the moving discs also comes along with a larger proportion of bouncing impressions (e.g., Grassi & Casco, 2012; Scholl & Nakayama, 2004), the shift in response bias observed by Grove et al. (2012) might in fact be deeply perceptual in nature. In order to provide direct evidence for our suggestion that coinciding vibrotactile transients alter perceptual processing of bouncing/streaming impressions, we therefore tested whether vibrotactile transients also decrease the amount of perceived overlap between the moving discs.
The final sample of Experiment 2 consisted of 21 students from the University of Trier (18–33 years old, 17 females). Data from three additional participants were removed (one equipment failure, one participant switched off the white noise during the experiment, and one participant failed to comply with the task instructions).3
Apparatus, stimuli, and procedure
The results of our experiments provide two straightforward insights into tactile-visual interactions. First, a brief vibrotactile stimulation alters the quality of the visual impression (from streaming to bouncing; Experiment 1; see also Watanabe, 2001). Second, the coinciding tactile stimulation indeed is capable of altering perceptual processing. This conclusion stems from the results of Experiment 2, in which we used an implicit dependent variable to assess perception. Whereas the open responses of Experiment 1 could be explained in terms of shifts in the response criterion or even task demands, such cognitive influences on the adjusted overlap are unlikely due to the less explicit response format as well as the continuous repetition of the display until the participants were satisfied with their response. Thus, the dependent measure in this experiment allowed for a continuous perceptual comparison between the perceived overlap in the dynamic event and the adjusted overlap of the probe discs. We argue that this likely captures perceptual processing. [Of course, we did not mention the concepts under investigation such as bouncing, streaming, or causality in the instructions of Experiment 2.]
The two experiments differ not only in the experimental task at hand, but also in surface features such as the number of simultaneously moving objects or the speed of the moving object. Nevertheless, their spatiotemporal structure is remarkably similar: In both experiments, two objects approach each other until they fully overlap and then move apart again (see also Michotte, 1963). Most importantly, however, they pose the same question for the visual system, namely, establishing object correspondence (i.e., which object went where). Given this structural similarity, we would like to interpret our results as a first insight into how the qualitative change in the visual impression might arise on a perceptual level (for converging evidence that misperceptions of the actual overlap also arise in the bouncing/streaming displays, see Grassi & Casco, 2012; Kawachi, 2016). Because previous research has demonstrated that less (objective) overlap between the discs comes along with a higher proportion of bouncing reports (Scholl & Nakayama, 2004), it seems likely that the quantitative shift in perceived overlap induced the qualitative shift in the visual impression. As such, the bouncing/streaming illusion is a bistable stimulus in which observers could see bouncing or streaming but not any transitions in between (similar to other bistable stimuli such as the Necker cube). The perceived overlap between the moving discs, however, might alter the probability of one or other perceptual impression. Of course, further research is necessary in order to fully establish the proposed connection between illusory crescents and perceived bouncing.
From coincidence to bouncing
A central challenge for the visual system is to maintain the correspondence of objects when they change their location over time. In order to solve this task, the visual system relies on information from surface features such as color and shape (Feldman & Tremoulet, 2006), occlusion cues (Remijn & Ito, 2007), or the spatial alignment (Kawabe & Miura, 2006). Our results show that the visual system also incorporates tactile signals in order to disambiguate the visual percept. Although we did not directly compare auditory and tactile transients in our study (but see Watanabe & Shimojo, 2005), our results from tactile stimulation closely match findings from audio-visual as well as purely visual experiments exploring the impact of coinciding transients on the bouncing/streaming impression (Sekuler et al., 1997; Watanabe & Shimojo, 1998) as well as illusory crescents (Meyerhoff & Scholl, 2018; Scholl & Nakayama, 2004).
One possible interpretation of these similar findings across different modalities is that the visual system might interpret any kind of coinciding information as heuristic in favor of a causal interaction, which in turn increases the likelihood for the reversal of the motion paths that are perceived as two discs bouncing off each other. From an ecological point of view, such a perceptual heuristic seems plausible as most causal interactions between objects come along with coincident sounds (i.e., colliding billiard balls) or tactile vibrations (i.e., putting a mug on the table). Our observation that tactile information disambiguates stimuli when there is no naturalistic link between the stimuli (i.e., normally, we hear but do not feel the collision of external objects) further indicates that such a heuristic would work on an abstract level rather than with concrete situations and experiences.
The influence of attentional processing
An alternative to explain tactile-induced illusory crescents as well as tactile-induced perceived bouncing comes from cross-modal attentional processing. Indeed, the cross-modal consistency of bouncing impressions as well as illusory crescents between our experiments with tactile stimulation and previous work with auditory or visual stimulation (e.g., Watanabe & Shimojo, 1998; 2005) leans toward a general mechanism (rather than modality-specific mechanisms) that alters the visual percept (see Grove, Kawachi, & Sakurai, 2012b; Watanabe, 2001; Zhou et al., 2007). A good candidate for such a cross-modal mechanism is attentional processing because previous research has identified cross-modal interactions between the tactile and the visual modality (e.g., Frings, Amendt, & Spence, 2011; van der Burg, Olivers, Bronkhorst, & Theeuwes, 2009). Within the visual domain, Watanabe and Shimojo (1998) have reported that bouncing impressions become more likely if attention is distracted from the bouncing/streaming display by coinciding visual transients or dual tasks. Therefore, it is possible that any coinciding transient distracts attention from the bouncing/streaming display. Such an attentional disruption might result in missing the central frame which in turn induces illusory crescents as well as bouncing impressions.
Coinciding vibrotactile information induces perceived bouncing as well as illusory crescents between overlapping discs. Our study suggests that the qualitative change in the visual impression is accompanied by a quantitative change in the perceived overlap between the moving discs (i.e., touch-induced illusory crescents). This matches remarkably well with previous findings from studies with coinciding auditory information which in turn supports the idea that perceived bouncing stems from a modality-independent mechanism such as heuristical processing, or the distraction of attention, or a combination of both.
We became aware of this study, which is included in the dissertation of Katsumi Watanabe and the cited book chapter, only after the initial submission of this manuscript. We would like to thank both reviewers who brought this to our attention. In fact, the logic of both experiments is highly similar and both experiments differ only in the specific parameters such as object size and speed.
Including this participant in the analysis had no effect on the results.
Including these participants in the analysis had no effect on the results.
Note that fast speeds can result in motion blur on the display, which might interfere with the perception of the absolute overlap between the discs. Critically, however, our main interest in this experiment focuses on the difference between the condition with and without coinciding tone. As blur would occur equally across conditions, it therefore cannot explain our main results.
The data of the reported experiments is available at https://osf.io/nz4v3/. We would like to thank Moritz Breit for his help with the collection of the data.
- Frings, C., Amendt, A., & Spence, C. (2011). When seeing doesn't matter: Assessing the after-effects of tactile distractor processing in the blind and the sighted. Journal of Experimental Psychology: Human Perception and Performance, 37, 1174–1181. https://doi.org/10.1037/a0022336 CrossRefPubMedGoogle Scholar
- Michotte, A. (1963). The perception of causality. Oxford: Basic Books. [English translation by Miles T. & Miles E.; originally published as Michotte, A. (1946). La perception de la causalité. Louvain: Institut Supérior de Philosophie]Google Scholar
- Watanabe, K. (2001). Crossmodal interaction in humans. Unpublished dissertation, California Institute of Technology: Pasadena, CA.Google Scholar