Reorganization of spatial configurations in visual working memory

Abstract

Human beings have to constantly process multiple objects in visual working memory (VWM). Positional relations to other objects known as spatial configurations contribute significantly to the organization of information in VWM. The aim of our study was to clarify whether spatial configurations can be reorganized to a subset of objects during maintenance. Participants were shown an array of objects, and afterwards the objects disappeared. A valid cue was shown either during encoding or maintenance to highlight the side of the following probed object. Afterwards, the objects reappeared and participants were instructed to detect whether or not a particular object changed its location. We manipulated the configurations at retrieval regarding the number of objects, ranging from all objects to a single object. Our first and second experiment investigated reorganization for a number of six and 12 objects, respectively. In the third experiment, we used a retro cue only and manipulated eye movements (free view vs. enforced fixation). While showing that reorganization of spatial configurations during maintenance is possible in principle, we found some boundary conditions. There was no spatial configuration effect when participants had to fixate. Thus, eye movements are required for a configuration effect to occur.

Background

Fifty years ago, the multistore model of memory was proposed, which distinguishes between three different memory stores. These stores differ in capacity, duration, and chronological order, and others. Three stores were postulated: sensory, short-term memory, and long-term memory (Atkinson & Shiffrin, 1968). With our present research, we focused on the short-term store. Already, back then, the term working memory was used to describe the short-term store (Atkinson & Shiffrin, 1968). Later, more detailed differentiations were made, including several types of working memory (Baddeley & Hitch, 1974). Neuroimaging studies lead to similar assumptions. Short-term and working memory are distinguished nowadays because they claim different areas of the prefrontal cortex. Working memory leads to the ability to simultaneously manipulate the information stored and work with it. Short-term memory, however, is only a short-term store, but does not allow the organization and processing of the stored content (Diamond, 2013). This leads to the assumption that working memory can be seen as the tool you can use to access content of the short-term store. With our present research, we extend the understanding of processing mechanisms regarding the working memory system by studying the reorganization of spatial configurations during maintenance in the short-term store.

In visual working memory (VWM), single objects are not processed individually, but the relations to other objects, known as configurations, have a strong impact. These spatial configurations contribute significantly to the organization of information in the VWM. For example, viewers automatically process the spatial configuration of multiple objects, even if they are asked to independently memorize the locations of objects (Jiang, Olson, & Chun, 2000). Previous work on spatial configurations focused exclusively on the encoding and retrieval of information (e.g., Dent, 2009; Jiang, Chun, & Olson, 2004; Jiang et al., 2000; Papenmeier, Huff, & Schwan, 2012). But how are spatial configurations represented in VWM, and what operations can observers perform on memorized configurations? With the present set of experiments, we studied this question by investigating whether observers can reorganize spatial configurations during maintenance. We refer to encoding as the time when objects are visible, and to maintenance as the time when objects are no longer visible, after encoding and before retrieval. By addressing this research question, we want to clarify whether spatial configurations determine the encoding of information alone, or whether they also affect maintenance of information in VWM. In the following, we will first review the literature on the influence of spatial configurations during encoding and retrieval.

In an influential work on the importance of spatial configurations for organizing VWM, participants were asked to individually memorize the locations of several presented objects and to ignore the spatial configuration of all objects (Jiang et al., 2000). Although the participants were asked to detect the location change of a single object, they typically failed to memorize single objects only, but rather a spatial configuration including all objects memorized (Jiang et al., 2000; Papenmeier et al., 2012). Thus, they observed an improved detection performance for the location change of a single object if it was presented within the global spatial configuration of all objects memorized, but not alone. This was not caused by the presence of additional reference points in the displays only, because the presentation of a partial configuration (e.g., three of six memorized objects) did not improve the detection of the displacement of the individual object (Jiang et al., 2000; Papenmeier et al., 2012). Such configuration effects are not limited to the maintenance of abstract stimuli such as squares (Jiang et al., 2000; Papenmeier et al., 2012); they are also evident in the presentation of natural stimuli, such as images of lamps, cars, or chairs (Hollingworth, 2007; Papenmeier & Huff, 2014).

Importantly, spatial configurations are formed only on the basis of relevant objects and not on all objects visible on the display. This was shown in an experiment (Jiang et al., 2000) in which participants were instructed to memorize a color-coded subset of the visible objects only. The change in the location of the remaining objects between encoding and retrieval had no effect on the memory performance for the objects memorized. Thus, the spatial configuration effects are not caused by a perceptual snapshot of the entire scene. Nonetheless, multiple studies suggest that spatial configurations might be rigid representations based on the global perceptual grouping of the encoded objects. First, the change of task-irrelevant perceptual grouping stimuli on the memorized objects between encoding and retrieval has been shown to affect memory performance (Jiang et al., 2004). The formation of spatial configurations is, thus, closely linked to the perceptual grouping of the encoded objects. Second, whereas the presence of the global spatial configuration of all encoded objects at retrieval results in a memory benefit, the presentation of partial configurations at retrieval does not. This suggests that a partial configuration cannot access the represented overall spatial configuration, even if the partial configuration contains four out of six memorized objects (Papenmeier et al., 2012). Third, the view dependence of contextual information (Papenmeier & Huff, 2014; see also Wood, 2011) also supports the idea of a rigid perceptual snapshot of the encoded objects. In this study, participants were shown several objects lying on a virtual table, and one object was highlighted at retrieval for a location-change-detection task. Spatial configurations (full, no) and the point of view (0°, 60°) in the test image were manipulated. Configuration effects disappeared with a change in point of view of 60 degrees (Papenmeier & Huff, 2014).

Are spatial configurations, thus, only represented as rigid images of the encoded perceptual grouping, or can spatial configurations in VWM be reformed and reorganized? The retro-cue paradigm has established as a successful method of studying memory manipulation during maintenance (Griffin & Nobre, 2003; Landman, Spekreijse, & Lamme, 2003; Lepsien, Griffin, Devlin, & Nobre, 2005; Makovski & Jiang, 2007; Makovski, Sussman, & Jiang, 2008; Sligte, Scholte, & Lamme, 2008; Souza & Oberauer, 2016; Williams & Woodman, 2012). During the delay period—that is, after the stimuli have disappeared—participants are presented with an indication of which of the objects will be interrogated. This leads to a deliberate shift of attention to the highlighted objects. This leads, on the one hand, to a more stable representation of the highlighted objects (Sligte et al., 2008) and, on the other hand, to the deletion of the originally memorized and now irrelevant objects from memory (Williams, Hong, Kang, Carlisle, & Woodman, 2013). Previous results from retro-cue research show that directing attention to a single object during maintenance reduces context effects at retrieval (Bae & Luck, 2017; Sligte et al., 2008). In Bae and Luck’s (2017) study, for example, participants had to memorize the orientations of two sequentially presented objects. The reproduction of the orientation of objects had influenced each other. If one object was cued, however, the influence of the uncued object on the cued object was reduced.

By applying the retro-cue paradigm to spatial configurations, we aim to investigate whether directing attention to multiple objects of a spatial configuration during maintenance results in reorganization. We refer to reorganization as the process in which a global spatial configuration of objects is encoded in the beginning, and the number of objects and their relations are updated into the cued partial configuration during maintenance. In the following, we present three experiments investigating the reorganization of spatial configurations during maintenance. In Experiments 1 and 2, we do so by comparing the effects of a retro cue during maintenance with the effects of a cue during encoding. In Experiment 3, we further investigate set-size differences and eye movements to gain a deeper understanding of the retro-cueing effects and thus the structure of VWM.

Experiment 1

In Experiment 1, we compared the effect of attentional cues that appeared during either maintenance or encoding. If spatial configurations can be reorganized to a subset of objects during maintenance, directing attention to a subset of objects during either encoding or maintenance should cause an increased location-change-detection performance for single object displacements by the presence of a partial configuration encompassing the cued objects at retrieval. If, however, a reorganization of spatial configurations to a subset of objects during maintenance is not possible, only the presence of the complete configuration containing all encoded objects at retrieval should cause a configuration effect despite attention cues during maintenance. Nevertheless, attention cues during encoding should still result in configuration effects by partial configurations encompassing the cued objects only at retrieval, because only those objects should be encoded in the first place.

Method

We performed the method including sample size, experimental procedure, stimuli characteristics as well as the analysis as we had preregistered on the Open Science Framework: https://osf.io/cwr9t/.

Participants

We used the R-Package Powerbydesign (Papenmeier, 2016) to conduct a power analysis. Our goal was to obtain at least a power of .90 at the standard of .05 alpha error probability. The group means and standard deviations used in this power analysis were taken from previous data of our lab and led to a sample size of 28 participants. We preregistered the following exclusion criteria: Participants identified as not doing the task (always pressing the same button or a performance level that does not deviate from chance) as well as any participants not completing the whole experiment were supposed to be removed from the data set and replaced by new participants. No participants were excluded for this particular experiment. Eventually, 28 participants took part in this experiment, receiving course credit or monetary compensation of 2€ per 15 min. All participants had normal or corrected-to-normal vision, and their age ranged from 19 to 32 years (M = 24.3 years, SD = 3.5). Up to three participants were tested at the same time on different computers. Simultaneous testing and monetary compensation were consistent throughout all experiments. The research was conducted in accordance with APA standards for ethical treatment of participants and with approval of the Institutional Review Board of the University of Tübingen.

Stimuli and materials

We presented six squares on a 24-in. computer screen (Fujitsu Display B24-8 TE Pro) with an unrestricted viewing distance of about 57 cm using PsychoPy 1.85.6 (Peirce, 2007). Each square was gray (RGB color hex code: #777777) and measured 1.5° × 1.5° (degrees of visual angle). At the beginning of each trial, participants were told to view a centric fixation cross. The objects could appear in 20° × 20° area at the center of the screen. Furthermore, we generated random object locations for each trial with the same number of squares equally placed on each side, with a minimum center-to-center distance of 1.5 times the diameter of a square. In order to guarantee that all locations used within a trial—including the new location of the object probed in change trials—were subject to the same restrictions, such as minimum distance, we programmatically generated four object locations on each side of the array, with only three being visible for each side during encoding. In change trials, the invisible fourth location at the cued side was made visible at retrieval and the original location of the object probed was made invisible. The luminance was as follows: square: 28.5 cd/m2, cue: 118.7 cd/m2, fixation cross: 0.1 cd/m2, red probe: 29.1 cd/m2, array border: 0.1 cd/m2 and background: 152.4 cd/m2.

Procedure

Each trial consisted of three phases (see Fig. 1). In the first phase (encoding), six squares had to be encoded. Three squares were presented on each side for 2,000 ms. Thereafter, objects disappeared and left a blank image. The second phase (maintenance) lasted 2,000 ms. Participants had to maintain the object locations. Afterwards they reappeared, and one object was marked with a red frame (RGB color hex code: #FF0000) during this last phase (retrieval). Participants had to press one of two buttons on a keyboard as to whether or not the location of this object had changed. We used a cue of 100 ms, which covered either the whole left-hand or right-hand side of the stimulus array, either during encoding or maintenance, to shift attention to a subset of objects. Participants were told that the cue would be valid as the object probed would appear on this side. Each cue (RGB color hex code: #E6E6E6) occurred 1,000 ms after the start of the respective phase. Furthermore, the number of objects varied at retrieval, as four different spatial configurations were used. The configurations were defined as follows:

  • Complete: presence of object probed and all other objects that were present during encoding

  • Congruent: presence of object probed and all other objects that were cued

  • Incongruent: presence of object probed and all other objects that were not cued

  • No: presence of object probed alone

Fig. 1
figure1

Experimental procedure and manipulations in Experiment 1

Please note that the notion of configuration refers to the number of objects present during retrieval. Whereas all visible nonprobed objects were always located at their old locations, the object probed could change its location between encoding and retrieval, which is a prerequisite to measure participants’ location-change-detection performance.

In a nutshell, we manipulated the cue side (left/right), the moment of cue presentation (during encoding/maintenance), the location of the object probed at retrieval (new/old), and the configuration of the objects at retrieval (complete/congruent/incongruent/no). The cue was 100% valid, thus the object probed at retrieval was always one of the objects indicated by the cue. Participants were assigned to all conditions (within-subjects design). Trials were presented in randomized order with the restriction that the experiment consisted of five blocks containing 64 trials each leading to 320 trials in total. Each condition occurred equally often within each block and also within the whole experiment. Thus, we showed 10 trials per experimental condition (configuration, cue presentation, location, cue side). The experiment duration was one hour.

Design and analysis

We compared location-change-detection performance as indicated by the sensitivity measure (d′) across conditions using a 2 (moment of cue presentation: encoding, maintenance) × 4 (configuration: complete, congruent, incongruent, no) repeated-measures ANOVA. We calculated sensitivity (according to signal detection theory) as a dependent measure across the responses to the old/new probe location trials. That is, sensitivity d' is defined as ⅆ = Φ−1(phits) − Φ−1(pfa) with phits being the proportion of hits and pfa being the proportion of false alarms (Stanislaw & Todorov, 1999). Hits refer to the accurate detection of old locations, and false alarms refer to responding “old” to a new location. Note that sensitivity cannot be calculated for either phits or pfa having values of 0.0 and 1.0. Thus, we corrected such values to the proportions equaling half a trial correct or half a trial incorrect, respectively.

Results and discussion

All analyses have been preregistered. We performed a repeated-measures ANOVA, with sensitivity (d′) as the dependent variable and cue presentation and spatial configuration as the independent variables (see Fig. 2). There was a significant main effect for moment of cue presentation, F(1, 27) = 12.20, p = .002, ηp2 = .31, and a significant main effect for configuration, F(3, 81) = 17.66, p < .001, ηp2 = .40. There was no significant interaction effect, F(3, 81) = 0.18, p = .908, ηp2 = .01. We further used t tests to investigate the main effect of configuration (see Table 1).

Fig. 2
figure2

Sensitivity (d′) across all participants for Experiment 1. Error bars indicate the standard error of the mean (SEM)

Table 1 The p values and t tests for configurations in Experiment 1

The results of the t tests (see Table 1) indicate a configuration effect due to a higher location-detection performance for the complete configuration compared with no configuration. Most importantly, the results also indicate a reorganization effect during maintenance, as the complete and the congruent configurations are comparable, and there is a significant difference between the congruent configuration and no configuration.

First, we could replicate previous findings. When the cue appeared during encoding, we observed configuration effects of the cued partial configuration. More interestingly, presenting a cue during maintenance caused a pattern of results similar to presenting the cue during encoding. Thus, we conclude that it is possible to reorganize spatial configurations to a subset of objects by shifting attention to these objects during maintenance. Although we had no specific predictions regarding the incongruent condition, one might have expected a performance similar to that in the no condition. However, location-change-detection performance was lower for the incongruent configurations than for the no configuration condition. This result will be discussed across all experiments in the General Discussion.

Experiment 2

In Experiment 1, participants showed very high sensitivity values across all conditions. Thus, we conclude that the location-change-detection task was rather easy with our set size of six objects. In order to further investigate the reorganization of spatial configurations under more challenging conditions, we conducted a second experiment with a doubled set size of 12 objects.

Method

We performed the method including sample size, experimental procedure, stimuli characteristics as well as the analysis as we had preregistered on the Open Science Framework: https://osf.io/6mkft/.

Participants

We used the same power analysis metrics (Papenmeier, 2016) as in Experiment 1. Subsequently, 28 participants were invited for this experiment and received course credit or monetary compensation. We applied the same exclusion criteria as in Experiment 1, and two participants had to be replaced because of a performance level that did not deviate from chance, leading to the participation of 30 participants but a final sample of 28, as previously appointed. Participants of the final sample had normal or corrected-to-normal vision, and their age ranged from 19 to 32 years (M = 23.4 years, SD = 3.0).

Stimuli and materials

We used an experimental setup similar to that of Experiment 1, and thus there were also 10 trials per experimental condition (configuration, cue presentation, location, cue side). Because of the limited space, we reduced the dimension of the squares to 1° × 1° to ensure enough possible locations for all randomized objects. Furthermore, we used a minimum center-to-center distance of twice the square diameter.

Procedure and analysis

We used the same procedure and analysis as in Experiment 1, except with 12 instead of six objects. The experiment duration was 1 hour.

Results and discussion

All analyses have been preregistered. We performed a repeated-measures ANOVA, with sensitivity (d′) as the dependent variable and cue presentation and spatial configuration as the independent variables (see Fig. 3). There was a significant main effect for moment of cue presentation, F(1, 27) = 38.78, p < .001, ηp2 = .59, and a significant main effect for configuration, F(3, 81) = 23.38, p < .001, ηp2 = .46. There was also a significant interaction between moment of cue presentation and configuration, F(3, 81) = 5.92, p = .001, ηp2 = .18. Because of the significant interaction effect, we further investigated the configuration effects with separate t tests for the two levels of moment of cue presentation (see Tables 2 and 3).

Fig. 3
figure3

Sensitivity (d′) across all participants for Experiment 2. Error bars indicate the standard error of the mean (SEM)

Table 2 The p values and t tests for configurations in Experiment 2: Encoding
Table 3 The p values and t tests for configurations in Experiment 2: Maintenance

The results for the t tests (see Table 2) indicate a configuration effect during encoding due to a higher location-change-detection performance for the complete configuration compared with no configuration. Further, the results also indicate that observers used the cue during encoding, as the complete and the congruent configuration are comparable, and there is a significant difference between the congruent configuration and no configuration.

The results for the t tests (see Table 3) also indicate a configuration effect during maintenance. As participants’ location-change-detection performance was significantly lower with a single object compared with the complete condition, spatial configurations contributed to the memory representation. However, we did not observe a reorganization of spatial configurations during maintenance as indicated by the comparable performance in the congruent and the no condition, and the lower performance in the congruent than in the complete condition.

These results confirm previous findings that the formation of spatial configurations during encoding can include a subset of objects by shifting attention to these objects. However, in contrast to Experiment 1, reorganization was not possible during maintenance. Because of these differences, we ran a cross-experiment analysis.

Cross-experiment analysis

We compared location-change-detection performance as indicated by the sensitivity measure across conditions and experiments using a 2 (cue presentation: encoding, maintenance; within) × 4 (configuration: complete, congruent, incongruent, no; within) × 2 (set size: six, Experiment 1; 12, Experiment 2; between) mixed ANOVA. Most importantly, there was no significant three-way interaction of moment of cue presentation, configuration, and set size, F(3,162) = 2.26, p =.083, ηp2 = .04. There were three significant main effects with ps < .001, ηp2s between .42 and .49 for moment of cue presentation, configuration, and set size, as well as an interaction effect for moment of cue presentation and configuration, F(3,162) = 3.59, p =.015, ηp2 = .06. All other effects were not significant.

Taken the first two experiments into account, it was possible for participants to shift their attention to a relevant partial configuration during encoding with both six and 12 objects. During maintenance, a reorganization was possible with six objects, but not with 12 objects. However, we did not observe a significant three-way interaction of moment of cue presentation, configuration, and set size in our cross-experiment analysis. Thus, we cannot draw the clear conclusion that set size influences the reorganization process during maintenance based on our first two experiments. Therefore, we performed a third experiment that was specifically designed to investigate this influence.

Experiment 3

Because of the divergent results from the first two experiments and the cross-experiment analysis, we ran a third experiment that was designed to investigate the influence of set size on the reorganization of spatial configurations during maintenance. Because we were concerned that set size might be confounded with eye movements—larger set sizes elicit more eye movements (Zelinsky, 2001)—we introduced an additional condition with enforced fixations in Experiment 3.

Method

We performed the method including sample size, experimental procedure, stimuli characteristics as well as the analysis as we had preregistered on the Open Science Framework: https://osf.io/738nh/.

Participants

We doubled the number of participants due to the between design. Thus, 56 participants took part in this experiment in exchange for course credit or monetary compensation. We applied the same exclusion criteria as before, and, furthermore, participants had to have normal vision for a precise eye tracking. Moreover, participants that could not be calibrated by the eye tracker were removed and replaced by new participants. Fifteen participants had to be replaced, leading to the participation of 71 subjects, but a final sample of 56, as previously appointed. Nine participants had to be replaced because of a performance level that did not deviate from chance, and six participants could not be calibrated. Participants of the final sample had normal vision, and their age ranged from 18 to 30 years (M = 22.5 years, SD = 2.8).

Stimuli and materials

We used an experimental setup similar to that of Experiment 2. In contrast to Experiments 1 and 2, we tested all Experiment 3 participants with a computer (Dell Precision M4800) that was accompanied by an SMI iView RED250 mobile eye tracker with a 250-Hz sampling rate to record eye movements, and we presented our stimuli on the corresponding 16-in. display. Six participants were tested with a sampling rate of 60 Hz instead of 250 Hz because of a software issue. The size of stimuli on the display was identical to Experiment 2. In the present experiment, we only varied the instructions between the groups (eye movements as usual vs. maintain fixation). Thus, participants saw the same material on the same equipment in both conditions. The only difference was the possibility of eye movements. Participants’ heads were positioned in a chin rest.

Procedure

We used the procedure of Experiment 2 with the following changes. First, cues were presented during maintenance only. Second, we manipulated set size (6/12) within participants. Third, we manipulated fixation enforcement between participants: One group had to fixate on the center of the screen throughout a trial, and the other group was allowed to move their eyes freely. For the first group, fixations were monitored throughout a trial, and a trial was aborted and repeated at the end of a block if a participant did not fixate on the center of the screen. A fixation cross was shown during the trial, and if gaze samples recorded were outside of an invisible surrounding circle with a radius of 1.5° for more than 250 ms, then a trial was aborted and repeated. Participants started each trial by looking at the fixation cross for 500 ms in both groups. Each condition occurred equally often within each block, and also within the whole experiment. Thus, we showed 10 trials per experimental condition (configuration, set size, location, cue side). The experiment duration was between 1 and 1.5 hours.

Design and analysis

We compared location-change-detection performance as indicated by the sensitivity measure across conditions using a 2 (set size: 6, 12; within) × 4 (configuration: complete, congruent, incongruent, no; within) × 2 (fixation: enforced, free-view; between) mixed ANOVA. We manipulated fixation between participants in order to avoid potential carryover effects. In particular, we wanted to ensure that participants move their eyes freely when allowed to do so.

Results and discussion

All analyses have been preregistered. We performed a mixed ANOVA, with sensitivity (d′) as the dependent variable and fixation, set size, and spatial configuration as the independent variables. There was a significant main effect for set size, F(1, 54) = 164.16, p < .001, ηp2 = .75, and a significant main effect for configuration, F(3, 162) = 16.96, p < .001, ηp2 = .24. There was no significant main effect for fixation, F(1, 54) = 0.54, p = .464, ηp2 = .01. There was also a significant interaction effect for fixation and configuration, F(3, 162) = 3.42, p = .019, ηp2 = .06, indicating that the location-change-detection performance in the two fixation groups was differentially affected by the presence of spatial configurations at retrieval. Because of the significant interaction effect of fixation and configuration, we further investigated the configuration effects with separate t tests for both experimental groups (see Tables 4 and 5).

Table 4 The p values and t tests for configurations in Experiment 3: Enforced fixation
Table 5 The p values and t tests for configurations in Experiment 3: Free view

We found no significant results for all remaining interaction effects of the mixed ANOVA. In detail, there were no significant effects for fixation and set size, F(1, 54) = 0.47, p = .495, ηp2 = .01, configuration and set size, F(3, 162) = 1.25, p = .294, ηp2 = .02. and for the three-way interaction of fixation, set size, and configuration, F(3, 162) = 0.98, p = .404, ηp2 = .02.

In an exploratory analysis, we analyzed the eye-tracking data of the free-view group (see Table 6). We used the high-speed event-detection algorithm provided by the IDF Event Detector (SMI, Version 3.0.20) with the following detection parameters: minimum duration of 22 ms, peak velocity threshold of 40°/s, and a minimum fixation duration of 50 ms. This analysis included the 22 participants of the free-view group that we tracked with a sampling rate of 250 Hz. There were significant differences between the set sizes in number of saccades, t(21) = −3.54, p = .002, and saccade amplitudes, t(21) = 3.21, p = .004. That is, larger set sizes elicited more, but shorter, saccades. Despite this difference in eye movements in the free-view group, set size did not modulate the configuration effects on memory in Experiment 3. That is, we observed reduced contextual processing in the absence of eye movements only (enforced fixation).

Table 6 Saccade parameters for Experiment 3 per trial, mean (SD): Free view

In a nutshell, free-viewing participants could reorganize their spatial configurations, regardless of set size, during maintenance (see Figs. 4 and 5). Interestingly, the results of the enforced fixation do not show a configuration effect at all (see Fig. 4). Because of the fixation, eye movements were not possible, and it seems that this handicap interfered a configuration process in general. Previous research suggests that spatial representations are linked to eye movements (Bays, Catalao, & Husain, 2009; Bays & Husain, 2008; Berman & Colby, 2009; Deubel & Schneider, 1996; Hoffman & Subramaniam, 1995; Kalogeropoulou, Jagadeesh, Ohl, & Rolfs, 2017; Ohl & Rolfs, 2017; Van der Stigchel & Hollingworth, 2018). In early studies using a dual-task paradigm, including a saccade assignment and a letter-discrimination task, it was discovered that saccade target selection and object recognition deploy a common attentional mechanism (Deubel & Schneider, 1996). Recent studies underline the theory that eye movements are needed for functioning working memory. That includes mechanisms such as target selection, maintenance, and feature conservation during and after a saccade as well as gaze correction during item search at retrieval (Van der Stigchel & Hollingworth, 2018). Our results provide evidence that there might also be the reverse relation between eye movements and working memory. That is, the construction of certain spatial representations in working memory—spatial configurations, in our study—might be needed for eye movements. This is in accordance with evidence showing that eye movements influence the process of spatial memory components and that the distances and relations to other objects are important for visual perception and subsequent memory processes (de Vito, Buonocore, Bonnefon, & Della Sala, 2014; Laeng & Teodorescu, 2002). We conclude that eye movements are necessary for the formation of spatial configurations in VWM that are a prerequisite for reorganization to occur. Thus, there might be a strong bidirectional relation of eye movements and working memory.

Fig. 4
figure4

Sensitivity (d′) across enforced fixation group for Experiment 3. Error bars indicate the standard error of the mean (SEM)

Fig. 5
figure5

Sensitivity (d′) across free-view group for Experiment 3. Error bars indicate the standard error of the mean (SEM)

General discussion

We conducted three experiments investigating whether spatial configurations can be reorganized during maintenance. We focused on the comparison of encoding and maintenance, set size differences, and eye movements to better understand the underlying processes.

First, our results indicate that internal spatial configurations could be reorganized during maintenance with six objects. Second, we wanted to validate and better understand these findings by doubling the number of objects in the display. Although the results indicate no reorganization of spatial configurations during maintenance with 12 objects, there was no interaction of configurations and set size in a cross-experimental analysis. Finally, with our third experiment, it was shown that spatial configurations could be reorganized regardless of set size during maintenance, when people could move their eyes freely. Interestingly, there was no configuration effect at all when participants could not move their eyes.

By using a cue during encoding, we replicated previous findings that directing attention to a relevant subset of a global spatial configuration resulted in the representation of the relevant partial configuration (Jiang et al., 2000), irrespective of set size. However, the process leading to this effect might differ from previous research. In contrast to previous research, the relevant subset was not indicated right from the start of a trial but with a cue occurring 1 second after stimulus onset in our experiment. Thus, one cannot argue that participants looked at the relevant stimuli right from the start of trial in our experiments. Rather, it seems plausible that they first encoded the global configuration that was then updated based on the cue. This idea is further supported by the fact that the cue occurred in only 50 % of trials during encoding, such that participants could not wait for the cue to start encoding. Finally, in contrast to previous research (Jiang et al., 2000), the manipulation of the irrelevant objects affected memory performance in our experiments. That is, the presence of the incongruent configuration at retrieval led to a reduced memory performance compared with no configuration. If participants had just looked at or encoded the relevant stimuli, the incongruent configuration and no configuration conditions should be on the same level.

By using a cue during maintenance, we found that, in principle, spatial configurations can be reorganized during maintenance. Because there was no hint of the probed object’s side before the occurrence of the retro cue, we argue for a reorganization of the global spatial configuration held in memory. As described in the beginning, we refer to reorganization as the process in which a global spatial configuration of objects is encoded in the beginning and the number of objects and their relations are updated into the cued partial configuration during maintenance. The pattern of our results indicate such a reorganization process. Although the performance with the congruent configuration was comparable to the complete configuration, the presence of the incongruent configuration at retrieval led to a reduced memory performance compared with no configuration. If participants were not reorganizing to the relevant stimuli, the congruent configuration, incongruent configuration, and no configuration conditions should be on the same level. However, we also observed some boundary conditions, such as reorganization not occurring with 12 objects in the second experiment or no configuration effects occurring with enforced fixation in the third experiment. Concluding, while showing that reorganization of spatial configurations during maintenance is possible, we also came across some boundary conditions that require future research. Moreover, future research should also investigate the process driving the reorganization of spatial configurations—that is, whether reorganization is caused by strengthening the representation of the attended objects, forgetting the unattended objects, or some other process. Furthermore, participants might have learned that we always cued one side of the screen across trials, resulting in the encoding of two separate configurations (left-hand vs. right-hand side) into VWM. The retro cue could then lead to the removal of the uncued configuration instead of a reorganization of the global spatial configuration. Investigating this idea further by future research would provide interesting insights because this would suggest that observers can maintain multiple separate spatial configurations concurrently in VWM instead of only one global spatial configuration, as previously proposed (e.g., Jiang et al., 2000; Papenmeier & Huff, 2014).

We used the dependent measure sensitivity (d′) as an indicator of memory performance. This procedure is well established in working-memory and change-detection-task paradigms (e.g., Boduroglu & Shah, 2009; Haatveit et al., 2010; Macmillan & Creelman, 2005; Stanislaw & Todorov, 1999; Turker & Swallow, 2019). Based on this measure, we conclude that the reorganization of spatial configurations is possible under certain conditions because the memory performance for the probed object was boosted if it was presented either within the complete or the congruent spatial configuration. In order to get a more detailed picture of the underlying mechanisms, we also provided hits, false alarms, and proportion correct for every configuration condition in each experiment (see Table 7 in the Appendix). This shows that the manipulation of spatial configurations had a greater influence on hits than false alarms. As hits refer to an “old” response on an “old” trial, participants particularly benefited from the presence of a meaningful configurational context if the object probed was presented at an old location. This stands in line with our results and conclusions that complete configurations as well as reorganized partial configurations are represented in VWM.

We varied set size in order to investigate its impact on reorganizing spatial configurations during maintenance. Location-change-detection performance dropped with increased set size in our experiments, replicating previous findings that sensitivity regarding location-change detection drops when more objects are present (Jiang et al., 2000; Luck & Vogel, 1997). Importantly, however, there was no significant interaction of spatial configuration and set size. Thus, it remains an open question whether there is a capacity limit for either spatial configurations in general or the reorganization of spatial configurations in particular. Especially when cued during encoding, it could be argued that the effective set sizes were three and six, not six and 12. Nonetheless, only 50% of the trials were cued during encoding, and the cue occurred 1 second after stimulus onset, so it seems plausible that participants encoded all stimuli before cueing, as they did not know the relevant stimuli until the (retro) cue occurred. Therefore, we defined set size as the number of objects visible at stimulus onset. In general, future research should still pay attention to set size when investigating potential capacity limits for reorganization effects.

We observed strong effects of the retro cue on the incongruent condition. The (retro) cue shifted the attention to one side of the whole configuration during encoding or maintenance. Based on previous findings that directing attention to a single object during maintenance eliminates configuration effects at retrieval (Sligte et al., 2008), one might have expected a similar location-change-detection performance with incongruent configurations and no configurations. However, when incongruent objects showed up at retrieval, we observed an interference, as people’s location-change-detection performance was lower than in all other conditions. This corresponds to previous findings showing that cues and attention shifts influence the strength of interobject dependencies of working memory representations (Bae & Luck, 2017). Our findings led to similar assumptions, as it seems that a reorganization into the relevant partial configuration led to a disruption when the other objects reappeared. Thus, future research investigating the reorganization effect might benefit from not only focusing on congruent configurations but also from explicitly investigating the process underlying the interference caused by incongruent configurations.

The reliance of spatial configuration effects on eye movements was an interesting and unexpected finding of our third experiment. If spatial configurations in VWM were a rigid perceptual snapshot as proposed before (e.g., Higuchi & Saiki, 2017; Papenmeier & Huff, 2014; Wood, 2011), they should also have occurred with enforced fixation. However, configuration effects occurred only when people could move their eyes freely. Thus, spatial configurations required eye movements and exploration regarding the stimulus material. We will elaborate on this surprising finding by discussing literature on the relation between eye movements and VWM. One the one hand, eye movements are in the need of a functioning working memory. That includes mechanisms such as target selection, maintenance, and feature conservation during and after a saccade as well as gaze correction during item search at retrieval (Van der Stigchel & Hollingworth, 2018). This matches with neuroimaging outcomes, that visual processing parts of the brain are activated as well when it comes to spatial representations and selective attention (Berman & Colby, 2009). On the other hand, there is empirical evidence for the association of eye movements and VWM resources and processes (Bays et al., 2009; Bays & Husain, 2008; Deubel & Schneider, 1996; Hoffman & Subramaniam, 1995; Kalogeropoulou et al., 2017; Ohl & Rolfs, 2017). Using a dual-task paradigm including a saccade assignment and a letter-discrimination task, it was discovered that saccade target selection and object recognition deploy a common attentional mechanism (Deubel & Schneider, 1996). In particular, there is also an association of eye movements with the capacity of the VWM resources, which are modified by selective attention as well as by visual direction (Bays & Husain, 2008). Some research even suggests that eye movements determine the content of VWM (Ohl & Rolfs, 2017). Summarizing, eye movements can both interrupt and stabilize working memory content. However, this previous research focused on VWM for single object features. Our results extend these findings by showing that eye movements are required for processing multi-object spatial configurations. Future research should further investigate the influence of eye movements on spatial configurations in VWM in order to elevate our understanding of the underlying processes.

Conclusion

The reorganization of spatial configurations is possible during maintenance. However, we also came across some boundary conditions that require future research. In particular, eye movements and their relation to spatial configurations and the reorganization process need further investigation. Our findings stand in contrast to the theory of a rigid memory representation, such as spatial configurations as perceptual snapshots (e.g., Higuchi & Saiki, 2017; Papenmeier & Huff, 2014; Wood, 2011). The multistore model of memory provided a first important step into the postulation of the memory’s structure (Atkinson & Shiffrin, 1968). Their theory suggests that the short-term store consists of multiple substores, such as the auditory–verbal–linguistic store or the visual store. Our experiments were designed to investigate the latter one. By doing so, we provided evidence that the visual store does not consist of slots storing visual object locations independently. Instead, the global spatial configuration of memorized locations is also stored in visual short-term memory. With the retro-cue paradigm, we also investigated the processes operating on these configurations. In line with current interdisciplinary research, in which working memory can be described as the tool that you can use to access content of the short-term store (Diamond, 2013), we observed evidence for the reorganization of spatial configurations maintained in memory. Taken together, there is evidence that the original short-term store consists of different substores, and there are several ways to reorganize and update its contents.

References

  1. Atkinson, R. C., & Shiffrin, R. M. (1968). Human memory: A proposed system and its control processes. In K. W. Spence & J. T. Spence (Eds.), The psychology of learning and motivation: Advances in research and theory (Vol. 2, pp. 89–195). London, UK: Academic Press.

    Google Scholar 

  2. Baddeley, A. D., & Hitch, G. (1974). Working memory. In G. H. Bower (Ed.), The psychology of learning and motivation: Advances in research and theory (Vol. 8, pp. 47–89). New York, NY: Academic Press.

    Google Scholar 

  3. Bae, G.-Y., & Luck, S. J. (2017). Interactions between visual working memory representations. Attention, Perception, & Psychophysics, 79(8), 2376–2395. https://doi.org/10.3758/s13414-017-1404-8

    Article  Google Scholar 

  4. Bays, P. M., Catalao, R. F. G., & Husain, M. (2009). The precision of visual working memory is set by allocation of a shared resource. Journal of Vision, 9(10), 7.1–711. https://doi.org/10.1167/9.10.7

    Article  Google Scholar 

  5. Bays, P. M., & Husain, M. (2008). Dynamic shifts of limited working memory resources in human vision. Science, 321, 851–854.

    Article  Google Scholar 

  6. Berman, R., & Colby, C. (2009). Attention and active vision. Vision Research, 49(10), 1233–1248. https://doi.org/10.1016/j.visres.2008.06.017

    Article  PubMed  PubMed Central  Google Scholar 

  7. Boduroglu, A., & Shah, P. (2009). Effects of spatial configurations on visual change detection: An account of bias changes. Memory & Cognition, 37(8), 1120–1131. https://doi.org/10.3758/MC.37.8.1120

    Article  Google Scholar 

  8. de Vito, S., Buonocore, A., Bonnefon, J.-F., & Della Sala, S. (2014). Eye movements disrupt spatial but not visual mental imagery. Cognitive Processing, 15(4), 543–549. https://doi.org/10.1007/s10339-014-0617-1

    Article  PubMed  PubMed Central  Google Scholar 

  9. Dent, K. (2009). Coding categorical and coordinate spatial relations in visual–spatial short-term memory. Quarterly Journal of Experimental Psychology, 62(12), 2372–2387. https://doi.org/10.1080/17470210902853548

    Article  Google Scholar 

  10. Deubel, H., & Schneider, W. (1996). Saccade target selection and object recognition: Evidence for a common attentional mechanism. Vision Research, 36(12), 1827–1837.

    Article  Google Scholar 

  11. Diamond, A. (2013). Executive functions. Annual Review of Psychology, 64(1), 135–168. https://doi.org/10.1146/annurev-psych-113011-143750

    Article  PubMed  PubMed Central  Google Scholar 

  12. Griffin, I. C., & Nobre, A. C. (2003). Orienting attention to locations in internal representations. Journal of Cognitive Neuroscience, 15(8), 1176–1194.

    Article  Google Scholar 

  13. Haatveit, B., Sundet, K., Hugdahl, K., Ueland, T., Melle, I., & Andreassen, O. (2010). The validity of d prime as a working memory index: Results from the “Bergen n-back task.” Journal of Clinical and Experimental Neuropsychology, 32(8), 871–880. https://doi.org/10.1080/13803391003596421

    Article  PubMed  PubMed Central  Google Scholar 

  14. Higuchi, Y., & Saiki, J. (2017). Implicit learning of spatial configuration occurs without eye movement. Japanese Psychological Research, 59(2), 122–132. https://doi.org/10.1111/jpr.12147

    Article  Google Scholar 

  15. Hoffman, J. E., & Subramaniam, B. (1995). The role of visual attention in saccadic eye movements. Perception & Psychophysics, 57(6), 787–795. https://doi.org/10.3758/BF03206794

    Article  Google Scholar 

  16. Hollingworth, A. (2007). Object-position binding in visual memory for natural scenes and object arrays. Journal of Experimental Psychology: Human Perception and Performance, 33(1), 31–47. https://doi.org/10.1037/0096-1523.33.1.31

    Article  PubMed  PubMed Central  Google Scholar 

  17. Jiang, Y., Chun, M. M., & Olson, I. R. (2004). Perceptual grouping in change detection. Attention, Perception, & Psychophysics, 66(3), 446–453.

    Article  Google Scholar 

  18. Jiang, Y., Olson, I. R., & Chun, M. M. (2000). Organization of visual short-term memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 26(3), 683–702. https://doi.org/10.1037//0278-7393.26.3.683

    Article  PubMed  PubMed Central  Google Scholar 

  19. Kalogeropoulou, Z., Jagadeesh, A. V., Ohl, S., & Rolfs, M. (2017). Setting and changing feature priorities in visual short-term memory. Psychonomic Bulletin & Review, 24(2), 453–458. https://doi.org/10.3758/s13423-016-1094-6

    Article  Google Scholar 

  20. Laeng, B., & Teodorescu, D.-S. (2002). Eye scanpaths during visual imagery reenact those of perception of the same visual scene. Cognitive Science, 26(2), 207–231.

    Article  Google Scholar 

  21. Landman, R., Spekreijse, H., & Lamme, V. A. (2003). Large capacity storage of integrated objects before change blindness. Vision Research, 43(2), 149–164.

    Article  Google Scholar 

  22. Lepsien, J., Griffin, I. C., Devlin, J. T., & Nobre, A. C. (2005). Directing spatial attention in mental representations: Interactions between attentional orienting and working-memory load. NeuroImage, 26(3), 733–743. https://doi.org/10.1016/j.neuroimage.2005.02.026

    Article  PubMed  PubMed Central  Google Scholar 

  23. Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions. Nature, 390(6657), 279–281.

    Article  Google Scholar 

  24. Macmillan, N. A., & Creelman, C. D. (2005). Detection theory: A user’s guide (2nd ed). Mahwah, NJ: Erlbaum.

    Google Scholar 

  25. Makovski, T., & Jiang, Y. V. (2007). Distributing versus focusing attention in visual short-term memory. Psychonomic Bulletin & Review, 14(6), 1072–1078.

    Article  Google Scholar 

  26. Makovski, T., Sussman, R., & Jiang, Y. V. (2008). Orienting attention in visual working memory reduces interference from memory probes. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34(2), 369–380. https://doi.org/10.1037/0278-7393.34.2.369

    Article  PubMed  PubMed Central  Google Scholar 

  27. Ohl, S., & Rolfs, M. (2017). Saccadic eye movements impose a natural bottleneck on visual short-term memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 43(5), 736–748. https://doi.org/10.1037/xlm0000338

    Article  PubMed  PubMed Central  Google Scholar 

  28. Papenmeier, F. (2016). Powerbydesign: Power estimates for ANOVA designs. (Version R Package Version 1.0.4.) [Computer software]. Retrieved from https://CRAN.R-project.org/package=powerbydesign

  29. Papenmeier, F., & Huff, M. (2014). Viewpoint-dependent representation of contextual information in visual working memory. Attention, Perception, & Psychophysics, 76(3), 663–668. https://doi.org/10.3758/s13414-014-0632-4

    Article  Google Scholar 

  30. Papenmeier, F., Huff, M., & Schwan, S. (2012). Representation of dynamic spatial configurations in visual short-term memory. Attention, Perception, & Psychophysics, 74(2), 397–415. https://doi.org/10.3758/s13414-011-0242-3

    Article  Google Scholar 

  31. Peirce, J. W. (2007). PsychoPy—Psychophysics software in Python. Journal of Neuroscience Methods, 162(1–2), 8–13. https://doi.org/10.1016/j.jneumeth.2006.11.017

    Article  PubMed  PubMed Central  Google Scholar 

  32. Sligte, I. G., Scholte, H. S., & Lamme, V. A. F. (2008). Are there multiple visual short-term memory stores? PLoS ONE, 3(2), e1699. https://doi.org/10.1371/journal.pone.0001699

    Article  PubMed  PubMed Central  Google Scholar 

  33. Souza, A. S., & Oberauer, K. (2016). In search of the focus of attention in working memory: 13 years of the retro-cue effect. Attention, Perception, & Psychophysics, 78(7), 1839–1860. https://doi.org/10.3758/s13414-016-1108-5

    Article  Google Scholar 

  34. Stanislaw, H., & Todorov, N. (1999). Calculation of signal detection theory measures. Behavior Research Methods, Instruments, & Computers, 31(1), 137–149.

    Article  Google Scholar 

  35. Turker, H. B., & Swallow, K. M. (2019). Attending to behaviorally relevant moments enhances incidental relational memory. Memory & Cognition, 47(1), 1–16. https://doi.org/10.3758/s13421-018-0846-0

    Article  Google Scholar 

  36. Van der Stigchel, S., & Hollingworth, A. (2018). Visuospatial working memory as a fundamental component of the eye movement system. Current Directions in Psychological Science, 27(2), 136–143. https://doi.org/10.1177/0963721417741710

    Article  PubMed  PubMed Central  Google Scholar 

  37. Williams, M., Hong, S. W., Kang, M.-S., Carlisle, N. B., & Woodman, G. F. (2013). The benefit of forgetting. Psychonomic Bulletin & Review, 20(2), 348–355. https://doi.org/10.3758/s13423-012-0354-3

    Article  Google Scholar 

  38. Williams, M., & Woodman, G. F. (2012). Directed forgetting and directed remembering in visual working memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 38(5), 1206–1220. https://doi.org/10.1037/a0027389

    Article  PubMed  PubMed Central  Google Scholar 

  39. Wood, J. N. (2011). A core knowledge architecture of visual working memory. Journal of Experimental Psychology: Human Perception and Performance, 37(2), 357–381. https://doi.org/10.1037/a0021935

    Article  PubMed  PubMed Central  Google Scholar 

  40. Zelinsky, G. J. (2001). Eye movements during change detection: Implications for search constraints, memory limitations, and scanning strategies. Perception & Psychophysics, 63(2), 209–225. https://doi.org/10.3758/BF03194463

    Article  Google Scholar 

Download references

Acknowledgments

We would like to thank Selina Galefski, Eleni Sianni, and Chiara Winter for their help during data collection.

Funding

This research project was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation), Project number 357136437.

Author information

Affiliations

Authors

Corresponding author

Correspondence to J. David Timm.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Table 7 Hit rates, false-alarm rates, proportions correct across all experiments (SEM in parentheses)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Timm, J.D., Papenmeier, F. Reorganization of spatial configurations in visual working memory. Mem Cogn 47, 1469–1480 (2019). https://doi.org/10.3758/s13421-019-00944-2

Download citation

Keywords

  • Short term memory
  • Attention
  • Context effects
  • Eye movements
  • Working memory