Electrophysiological correlates of encoding processes in a full-report visual working memory paradigm
- 212 Downloads
Why are some visual stimuli remembered, whereas others are forgotten? A limitation of recognition paradigms is that they measure aggregate behavioral performance and/or neural responses to all stimuli presented in a visual working memory (VWM) array. To address this limitation, we paired an electroencephalography (EEG) frequency-tagging technique with two full-report VWM paradigms. This permitted the tracking of individual stimuli as well as the aggregate response. We recorded high-density EEG (256 channel) while participants viewed four shape stimuli, each flickering at a different frequency. At retrieval, participants either recalled the location of all stimuli in any order (simultaneous full report) or were cued to report the item in a particular location over multiple screen displays (sequential full report). The individual frequency tag amplitudes evoked for correctly recalled items were significantly larger than the amplitudes of subsequently forgotten stimuli, regardless of retrieval task. An induced-power analysis examined the aggregate neural correlates of VWM encoding as a function of items correctly recalled. We found increased induced power across a large number of electrodes in the theta, alpha, and beta frequency bands when more items were successfully recalled. This effect was more robust for sequential full report, suggesting that retrieval demands can influence encoding processes. These data are consistent with a model in which encoding-related resources are directed to a subset of items, rather than a model in which resources are allocated evenly across the array. These data extend previous work using recognition paradigms and stress the importance of encoding in determining later VWM retrieval success.
KeywordsEEG Encephalography Attention
Visual working memory (VWM) refers to the ability to encode, maintain, and retrieve visually presented stimuli over a brief amount of time. A key feature of VWM is that it is capacity-limited, and an emphasis of recent cognitive neuroscience research has been to elucidate the neural bases of this capacity limitation. Many of the foundational studies in this area have focused on the maintenance phase of VWM (Cowan, 2001; Luck & Vogel, 1997; Todd & Marois, 2004; Vogel & Machizawa, 2004; for reviews, see Luck & Vogel, 2013; Postle, 2006; Stokes, 2015). However, this focus has tended to obscure the facts that maintenance depends on successful encoding of the to-be-remembered stimuli and that capacity limitations are partially attributable to events during encoding (Cohen, Sreenivasan, & D’Esposito, 2012; Emrich, Riggall, LaRocque, & Postle, 2013; Ester, Serences, & Awh, 2009; Gurariy, Killebrew, Berryhill, & Caplovitz, 2016; Harrison & Tong, 2009; Peterson et al., 2014). Moreover, experiments focusing on encoding-related processes commonly test retrieval by using recognition, yet it is well accepted that VWM performance can be influenced by the VWM task demands (Adam, Mance, Fukuda, & Vogel, 2015; Berryhill, Chein, & Olson, 2011; Duncan, Schramm, Thompson, & Dumontheil, 2012; Klein, Addis, & Kahana, 2005). An unintended consequence of this focus on maintenance is that we may know less than we realize about the neural mechanisms underlying encoding and the role they play in the overall limitations of VWM capacity.
Here we paired a full-report recall VWM paradigm with high-density electroencephalography (hdEEG) and used frequency domain analyses to probe the neural correlates of VWM encoding. Specifically, we addressed two questions regarding the role of encoding-related processes in the successful retrieval of information in VWM. First, we examined how resources are assigned to individual items within a VWM array as a function of whether or not the item was remembered. Second, we examined how task-specific retrieval demands influence how items are encoded into VWM. One point central to the questions raised here was that, given an array of to-be-remembered items, people often remember certain stimuli, whereas others are subsequently forgotten.
Two general models have been proposed to describe how neural resources may be allocated across items within a VWM array (Mance, Adam, Fukuda, & Vogel, 2014; Peterson et al., 2014). One model, which we will call the “all-or-nothing” model, proposes that on any given trial, neural resources are allocated in an all-or-nothing fashion. In other words, on correct trials participants successfully allocate neural resources to all items in a VWM array and are then able to recall all or most of the items. Importantly, this model predicts a uniform distribution of neural signatures across all items within a given trial, with greater amplitudes expected on trials in which more items are remembered. Alternatively, one could imagine that encoding-related resources might be distributed unevenly across an array, with only a subset of items receiving sufficient resources to ensure successful encoding, thereby increasing the likelihood of them being remembered. We will refer to this model as the “subset-of-items” model. The primary prediction born of this model is that for each trial, items that are successfully recalled will have larger frequency tag amplitudes than do items that are not successfully recalled, and there will be few trials in which all items are remembered.
To test these models, we applied a full-report VWM recall paradigm (Adam et al., 2015; Cabeza et al., 1997; Klein et al., 2005). Unlike the more traditional recognition paradigms, which measure memory performance for a single item in a display, a full-report recall paradigm allows us to measure VWM for each item in the display. We can test the predictions of the two models by examining the distribution of neural responses as a function of whether or not each item in the display was subsequently recalled. To address our second question—whether task-specific retrieval demands can influence the manner in which items are encoded into VWM—we applied two variants of the full-report recall paradigm. In the simultaneous paradigm, participants freely recalled items in any order they wished. In the sequential paradigm, participants were forced to recall items in a randomly determined order. We could thus determine whether the distribution of neural responses during encoding was influenced by these distinct retrieval demands.
Here we leveraged classic frequency domain EEG techniques—frequency tagging of the visual evoked potentials in response to rapid serial stimulus presentations (for a recent comprehensive review, see Norcia, Appelbaum, Ales, Cottereau, & Rossion, 2015) and induced-power analyses (Canolty & Knight, 2010; Klimesch, 1999; for a review, see Ward, 2003). Frequency tags can be described as the neural response to visual stimulation presented at specific frequencies. For instance, a stimulus continually flickering at 5 Hz produces a corresponding 5-Hz steady-state oscillation detectible in the EEG signal. This signal can be detected in frequency space, revealing a “frequency tag” at 5 Hz. This technique is useful for identifying the consequences of attentional selection on visual stimuli. For example, when focused attention is applied to a flickering stimulus, the corresponding frequency tag increases in amplitude as compared to an unattended stimulus (Hillyard & Anllo-Vento, 1998; Morgan, Hansen, & Hillyard, 1996; Müller, Teder- Sälejärvi, & Hillyard, 1998; Muthu, Suzuki, Joon Kim, Grabowecky, & Paller, 2007; Zhang, Jamison, Engel, He, & He, 2011). If there multiple stimuli are flickering at different rates, the frequency tags for each stimulus are evident in the EEG (Appelbaum, Wade, Petter, Vildavski & Norcia, 2006, 2008; Zhang et al., 2011). This offers the tremendous advantage of permitting the measurement of individual items rather than measuring an aggregate response to all stimuli. We previously had leveraged this technique to investigate the neural correlates of VWM encoding while measuring behavioral performance within recognition paradigms. We found that the frequency tag amplitudes for later-remembered items were significantly greater than those for later-forgotten items (Peterson et al., 2014). We also found that on correct trials, the frequency tag amplitudes were significantly greater when the array contained two rather than four items (Gurariy et al., 2016). Together, these observations demonstrated the effectiveness of the frequency-tagging technique for identifying the neural correlates of VWM performance during encoding.
An analysis that is complementary to frequency tagging is to assess changes in induced power. Changes in induced oscillatory power within specific frequency bands (e.g., alpha, theta) can be interpreted as the results of task-specific neural processes and/or as a mechanism for cortico-cortical and cortico-subcortical communication (Canolty & Knight, 2010). Induced oscillations have been examined in WM paradigms in order to examine WM capacity, retrieval, task difficulty, attention, and executive control (for reviews, see Klimesch, 1999; Ward, 2003). However, the literature has revealed findings that have implicated various frequency bands in specific WM task demands. For example, some studies have reported increased theta power as a function of increased WM load (Deiber et al., 2007; Jensen & Tesche, 2002) and as a function of WM task demands or characteristics (Caplan, Madsen, Raghavachari, & Kahana, 2001; Kahana, Seelig, & Madsen, 2001; Kahana, Sekuler, Caplan, Kirschen, & Madsen, 1999). Others studies have reported increased theta power as well as decreased alpha power as WM demands increase (Gevins, Smith, McEvoy, & Yu, 1997) and when task difficulty and attentional engagement increase (Klimesch, 1999; Klimesch, Doppelmayr, Schimke, & Ripper, 1997). In contrast, others have reported increased alpha band power with increased WM load (Adam et al., 2015; de Vries, van Driel, & Olivers, 2016; Erickson, Albrecht, Robinson, Luck, & Gold, 2017; Fukuda, Kang, & Woodman, 2016; Fukuda, Mance, & Vogel, 2015; Fukuda & Woodman, 2017; Jensen, Gelfand, Kounios, & Lisman, 2002; Klimesch, Doppelmayr, Schwaiger, Auinger, & Winkler, 1999; Krause, Lang, Laine, Kuusisto, & Pörn, 1996). Here we included measurements of induced oscillatory power in order to identify the effects of WM performance on induced oscillatory frequency bands.
As a preview of what will be described in detail in the following sections, participants viewed arrays of four items and remembered item–location conjunctions. Unique frequencies were assigned to each stimulus, allowing the frequency tag amplitude to be calculated for each item, either correct or incorrect, at each hdEEG electrode. The results indicated that during encoding, higher-amplitude frequency tags are observed for correctly recalled items than for those later recalled incorrectly, but there were few trials in which most of the items were retrieved. These results were consistent with the “subset-of-items” model, and were true for both the sequential and simultaneous full-report paradigms. Moreover, during encoding there were distinct patterns of induced power as a function of how many items were remembered on a given trial. Further highlighting the importance of encoding on VWM performance, these patterns depended on the retrieval task.
Twenty-six adults participated in this study (13 males, 13 females; 19–35 years of age). Six datasets (i.e., six participants) were excluded because of excessive artifacts in the hdEEG data. All participants had normal or corrected-to-normal vision and no history of neurological disorders. The Institutional Review Board at the University of Nevada, Reno, approved all experimental protocols. Each participant gave written, informed consent.
Stimulus apparatus and display
The stimuli were presented on a Mitsubishi Diamond Pro270 (20 in., 1,024 × 768, 120 Hz) monitor run by a Mac Mini (2.6 GHz, 4GB 1600 MHz DDR3 RAM) computer with an Intel High Definition (HD) 4000 graphics processor (768 MB of DDR3 SDRAM). Stimuli were presented with the Psychophysics Toolbox (6.1; Brainard, 1997; Pelli, 1997) for Matlab (R2012b, The Mathworks Inc., Natick, MA).
Electrophysiological apparatus and recordings
The hdEEG data were collected using a Geodesic system 300 with a HydroCell Geodesic Sensor Net 130, 256-electrode cap at a sampling rate of 1000 Hz. The hdEEG was amplified using an EGI Net Amps Bio Amplifier with a gain factor of 20,000 per channel and digitized using a Texas Instruments analog-to-digital (AD) converter (ADS 1251, 24 bit, 20 kHz). The data were then recorded using the Net Station 5.0 software package (Electrical Geodesics, Inc., Eugene, OR, USA). Electrical impedances for the majority of the electrodes were kept below 50 Ω. Synchrony between the stimulus presentation and the EEG recording, as well as the flicker rate of the stimuli, was verified using a photodiode attached to the monitor. The photodiode output was amplified using a transimpedance amplifier and digitized with the EEG data via one channel of the AD converter.
Task design and procedure
Participants performed a VWM full-report paradigm with a set size of four items. Each trial started with a gray fixation screen (15.63 cd/m2) presented for 600 ms, plus or minus a random jitter between 0 and 100 ms, that contained a central fixation point of approximately 0.3 × 0.3 deg of visual angle) along with four black squares of approximately 1.2 × 1.2 deg of visual angle. The black squares (0.1538 cd/m2) were placed at the center of each quadrant, with the addition of random jitter (± 1.5 deg). They were presented first in order to denote the locations of the four stimuli and to prevent any contamination from visual-onset-evoked potentials in the hdEEG recordings. During encoding (1,000 ms), the squares were replaced with the VWM stimuli. These stimuli were pseudorandomly chosen from a set of ten polygons.
Before they began the experiment, participants were instructed not to blink, make jaw movements, or move their eyes away from central fixation and were shown how these movements would affect the EEG trace. Each participant was asked to blink during retrieval. They were also instructed that they would have two recall tasks during the experiment, and they completed five practice trials of each recall task to familiarize them with that task.
Behavioral data analysis
Behavioral accuracy was calculated separately for simultaneous and sequential recall for each participant. Participants correctly recalled between zero and four items. Accuracy was calculated as the percentage of correct responses across all trials for each recall type and averaged across participants. Accuracy was also calculated for the order of items retrieved. For example, if a participant correctly placed the first item but incorrectly placed the second item, accuracy was considered 100% for the first item and 0% for the second item. The percentages of trials on which a participant correctly recalled one, two, three, or four items were also tabulated.
Electrophysiological data analysis
The hdEEG data were processed and analyzed using a combination of functions from the Net Station 5.0 software package (Electrical Geodesics, Inc., Eugene, OR, USA), the Brainstorm software package (Tadel, Baillet, Mosher, Pantazis, & Leahy, 2011: http://neuroimage.usc.edu/brainstorm), and custom-made Matlab scripts. The data from both correct and incorrect trials were analyzed. First, in Net Station the data were high-pass filtered (>0.5 Hz) in order to remove slow-drift noise. The filtered data were then segmented into 1,000-ms epochs that were time-locked to stimulus onset and lasted until stimulus offset, which was the duration of the encoding period (150 total epochs per participant, 75 epochs of each recall type). To ensure the accuracy of the data segmentation, event triggers generated in Matlab were sent from the presentation computer to the acquisition computer at the start of every encoding period. The offset periods between trigger generation in Matlab and trigger registration in NetStation were measured using a photodiode. This offset was accounted for during data segmentation. Custom-made Matlab scripts sorted the individual epochs by recall type (simultaneous, sequential) and by the number of items correctly recalled.
Artifact detection and subject exclusion
After segmentation, the data epochs were cleaned of artifacts (e.g., eye movements, eye blinks, and bad channel detection) using NetStation’s built-in tools. An epoch was marked as bad and removed if it contained more than 20% bad channels or contained an eye movement or eye blink. Eye blinks and eye movements were detected using NetStation’s built-in artifact detection tool using minimum and maximum detection thresholds of 55 and 140 μV, respectively. When possible, if an epoch had one or more bad channels, these channels were replaced by interpolating the data from surrounding channels. Additionally, if there was an epoch in which up to 20% of the channels were bad, that epoch was removed. Bad channels were detected using NetStation’s built-in artifact detection tool using a minimum/maximum detection threshold of 200 μV. Participants who had more than 25% bad epochs were removed from the study. A total of six participants were excluded from the analysis because they exceeded the allowable number of bad epochs.
After excluding the participants who were rejected, an average of 7.1 trials (SD = 8.8), were excluded per participant, on the basis of artifact detection or excessive bad channels.
Since this was a full-report design, each trial consisted of four items that could be either correctly or incorrectly recalled. This is different from a recognition paradigm in which each trial probes one item. Due to this fact, for this analysis the trials were sorted on the basis of the accuracy of the individual items at their specific frequencies. For example, when examining the 3-Hz frequency tag amplitudes, we used trials in which the participant correctly recalled the 3-Hz item as compared to trials in which they incorrectly recalled the 3-Hz item, regardless of accuracy for the other items. To account for the unequal numbers of correct and incorrect epochs (e.g., a greater number of 3-Hz correct than of 3-Hz incorrect), we implemented a permutation analysis in which, on each of 100,000 iterations, equal numbers of correct and incorrect epochs would be pseudorandomly selected. For example, a participant might have recalled the 3-Hz item correctly 30 times (30 correct trials) and erred 20 times (20 incorrect trials). Given this case, our permutation analysis would pseudorandomly select 20 of the correct epochs to match with the 20 incorrect epochs. Importantly, we ensured that the numbers of correct and incorrect trials were matched to the type that included the smallest number of trials for each frequency (i.e., 20 trials were selected from both groups if one of the groups had either 30 correct and 20 incorrect or 20 correct and 30 incorrect as the minimum number of trials in both cases is 20). Due to the complexity of this permutation analysis, we validated our process by first simulating EEG datasets in which we embedded known signals at each of our target frequencies, and demonstrated that these known signals could be accurately recovered. Specifically, using Matlab we generated simulated EEG signals for which we controlled the amplitudes of the embedded frequency tag specific oscillations as well as a small amount of added noise. We then subjected these data to our permutation analysis to ensure that the amplitudes recovered from the analysis matched the known embedded frequencies.
For each of the 100,000 iterations of the permutation analysis, we computed the time domain average of the trial-number-matched epochs (correct, incorrect) for each of the target frequencies. A fast Fourier transform (FFT) was then applied to each average in order to extract the frequency tag amplitude corresponding to the frequency of the presented item (3, 5, 12, or 20 Hz) as a function of whether it was correctly or incorrectly recalled.
To control for outliers, the process of matching the numbers of correct and incorrect items, averaging across correct and incorrect items, applying the FFT, and calculating the index was repeated for each of the 100,000 iterations of the permutation analysis. These index values were averaged across iterations and then frequencies, for each electrode and participant. See Fig. 1 for a visual representation of the data analysis using hypothesized EEG data and frequency tags.
A group-level analysis of these indices was performed separately at each electrode using repeated measures, one-sample, two-tailed t tests against a test value of 0. The significance of these tests was evaluated using an alpha of .05. Additionally, to correct for multiple comparisons, we applied a false discovery rate correction (q = .10).
This orthogonal analysis permitted us to examine the task-specific effects of induced power. First, we segmented the data on the basis of the number of correct responses, regardless of flicker frequency. For example, we contrasted “good” trials, in which participants correctly recalled three or four of the items, with “bad” trials, in which participants correctly recalled one or two of the items. Trials in which participants correctly recalled zero items were excluded, since they made up a relatively small number of trials (6.25 trials per condition averaged across participants, with a standard deviation of 5.84) and due to the possibility that participants might not have actually performed the task during these trials. We performed the analysis in this way, instead of breaking the data into bins for one, two, three, and four correct, for two reasons. First, this procedure had been used in previous studies that had looked at neural activity as a function of “good” or “bad” trials (Mance et al., 2014). Second, this analysis required trials to be segmented into groups based on the number of items correctly recalled (one, two, three, or four). However, the numbers of trials in which participants correctly recalled one or four items were significantly smaller than those in which participants recalled two or three items. This difference would lead to data loss during the permutation procedure. As a result, the number of trials that would have gone into the permutation analysis would be much smaller than in the evoked analysis. Therefore, we parsed trials into “good” and “bad” groups. After segmenting the trials, we performed an FFT and extracted amplitudes across the range of frequency power bands (delta, theta, alpha, and beta) for each individual trial. We then averaged these frequencies across trials. This contrasted with the permutation analysis explained above, in which we averaged across trials before taking the FFT; this procedural difference allowed for the extraction of induced power that would have been lost due to variations in phase (Bastiaansen & Hagoort, 2003). This is an important difference to note between the two analyses: By using frequency tagging, we were explicitly entraining neural activity to that of an externally flickering stimulus, which allowed us to examine the neural signature specific to each stimuli. In contrast, the induced-power analysis examined changes in neural oscillations that were independent of the external stimulus presentation (Bastiaansen & Hagoort, 2003). Next we performed a permutation analysis, repeating this process 25,000 times to control for uneven numbers of trials. We performed the analysis only 25,000 times because of the increased computational load of performing the FFT on each individual trial, as opposed to on the average of trials. We again calculated an index for each frequency within each power spectrum between the good and bad trials. These index values were then averaged across the frequency bands: delta (1–4 Hz), theta (5–7 Hz), alpha (8–15 Hz), and beta (16–31 Hz).
A group-level analysis of these indices was performed separately at each electrode using repeated measures, one-sample t tests against a test value of 0. The significance of these tests was evaluated using an alpha of .05. Additionally, to correct for multiple comparisons, we applied a false discovery rate correction (q = .10).
Accuracy for response order showed declines with each subsequent response for each task [simultaneous: 1st: 87.65% (1.76), 2nd: 72.35% (2.82), 3rd: 44.21% (4.70), 4th: 22.74% (3.05); sequential: 1st: 62.95% (3.21), 2nd: 45.82% (0.96), 3rd: 41.96% (3.50) 4th: 40.42% (3.56)] (Fig. 2B). A repeated measures analysis of variance (ANOVA) was calculated to determine differences between task types (sequential, simultaneous) and response orders. The ANOVA reported significant main effects of both task [F(1, 19) = 25.59, p < .0001] and response order [F(3, 57) = 133.69, p < .0001], as well as a significant interaction [F(3, 57) = 58.14, p < .0001]. Performance was better for early responses, and the slope of performance was steeper in the simultaneous than in the sequential task; see Fig. 2.
Another strength of the full-report paradigm is that it allowed us to examine the distribution of correct and incorrect items across trials. For example, the distribution of correct and incorrect items across trials might be consistent with predictions made by the “all-or-nothing” model, in which the distribution would be much more bimodal, with the majority of trials containing zero or four items correct. Alternatively, the distribution might be more consistent with predictions made by the “subset-of-items” model, in which the majority of trials would contain two correct and two incorrect. To evaluate these possibilities, we calculated the percentages of trials in which participants correctly recalled zero, one, two, three, and four items for each of the two recall types (Fig. 2C). A repeated measures ANOVA was calculated to determine the effect of task (sequential, simultaneous) on the number of items recalled correctly. There were significant main effects of both task [F(1, 19) = 13.15, p < .0001] and number of items correctly recalled [F(3, 57) = 26.67, p < .0001], as well as a two-way interaction [F(3, 57) = 4.34, p = .008]. Again, performance was superior in the simultaneous task. On the majority of trials the participants remembered two items, and this was a higher proportion for the simultaneous-task trials. There were many fewer trials on which participants remembered either all four or zero items.
Remembered versus forgotten items
Induced power at encoding is associated with memory load
Our overall goal was to better understand VWM capacity limitations arising during encoding. We paired two full-report VWM tasks with a frequency-tagging hdEEG paradigm. This provided behavioral and neural responses specific to each stimulus item and improved upon our previous work using VWM recognition (Peterson et al., 2014). Consistent with these previous data, the frequency tag amplitudes at encoding were significantly higher for correctly retrieved items compared to items later forgotten. In these data, this meant that for a given stimulus array the frequency tag amplitudes varied as a function of later VWM accuracy. We also replicated and extended our earlier findings in low-density EEG data (Peterson et al., 2014) by showing that this frequency tag amplitude difference (Correct > Incorrect) is evident across the whole brain rather than being driven by a limited number of electrode sites. In a subsequent analysis examining induced power, we were able to demonstrate differences as a function of the full-report retrieval task. For the sequential task, there were significant differences in alpha, theta and beta frequency bands that were unmatched in the simultaneous task. These induced power observations were also associated with higher amplitude predominantly for trials with more correct responses compared to those with fewer correct answers (More Correct > Fewer Correct). In short, these data add to the evidence supporting a key role of encoding in determining VWM capacity limitations.
Our previous work had shown that frequency tags were significantly higher for probed and unprobed items on correct trials compared to incorrect trials. These results left open a mystery regarding whether the frequency tags of correct trials reflected greater resource availability or whether participants simply held the correct answer for each of the unprobed items as well as for the probed item. In other words, because we did not collect any accuracy information for the unprobed items, we could not draw conclusions about the allocation of resources to those items. The present data clarify this mystery by showing that the amplitude of the frequency tag is higher for correct items and significantly lower for incorrect items, consistent with our original data. The present data also show that encoding is not “all-or-none,” but rather, encoding resources can be unevenly distributed across individual items and those who receive more, are more likely to be subsequently remembered. A recent paper employed a full-report VWM design and measured neural activity using EEG (Mance et al., 2014). These authors were interested in VWM capacity limitations and, in particular, whether VWM capacity limitations could be attributed largely to attentional lapses, in which neural resources are allocated to either the entire stimulus set or none of the stimuli (Rouder et al., 2008), or to failures of attentional control, in which neural resources are only allocated to a subset of stimuli. Consistent with our results, their data indicate that VWM capacity limitations were primarily due to failures of attentional control rather than lapses. We also note that we presented stimuli for a long encoding duration (1,000 ms) to sufficiently power frequency tag analyses. This slow encoding duration is considerably longer than in most VWM studies (100–250 ms) (for a review, see Luck & Vogel, 2013). In other words, participants had more encoding time than in standard VWM tasks and they still generally could not remember all four stimuli.
Furthermore, Mance et al. (2014) included spectral power analyses of their EEG data, examining induced power in four frequency bands: delta, theta, beta, and alpha. They observed a relationship between high frontal theta, and reduced posterior alpha with successful WM performance. Additional studies have shown similar patterns of increased theta power along frontal and midline areas and/or decreased alpha power in more posterior regions as a result of increased attentional allocation, WM task demands, and WM load (Caplan et al., 2001; Deiber et al., 2007; Gevins et al., 1997; Jensen & Tesche, 2002; Kahana et al., 2001; Kahana et al., 1999; Klimesch et al., 1997; for a review, see Hsieh & Ranganath, 2014). However, several earlier studies show the opposite pattern, citing increased alpha power as WM load increased (Jensen et al., 2002; Klimesch et al., 1999; Krause et al., 1996). The induced power analysis presented here is consistent with this second set of findings. As we noted above, for the sequential full-report task, alpha band power (as well as theta and beta) was significantly stronger during trials when participants correctly recalled three or four items compared to one or two items. This increase in power is widespread. One important note is the lack of studies explicitly examining induced power during VWM encoding. Of the studies listed above only three explicitly examine the encoding phase (Jensen & Tesche, 2002; Klimesch et al., 1999; Krause et al., 1996) and the induced power analysis presented in these studies is consistent with our results: increased alpha band power during encoding as a function of number of items correctly recalled. More recently, studies have begun using alpha power suppression as a measure of WM capacity (Adam et al., 2015; de Vries et al., 2016; Erickson et al., 2017; Fukuda et al., 2016; Fukuda et al., 2015; Fukuda & Woodman, 2017). Overall, these studies report a decrease in alpha power during the VWM maintenance phase that corresponds to the number of items currently held in VWM. However, even within these studies, during the encoding stage, we see a clear increase in alpha power as a function of VWM load. One consistent result from these studies presented is different patterns of activity emerge during encoding, maintenance, and retrieval across alpha and theta bands as a function of task demands and VWM load.
The results presented here also demonstrate notable differences in the power analyses as a function of VWM retrieval task demands, as has been noted previously (Mance et al., 2014; for related patient work, see Berryhill et al., 2011). These results demonstrate that task demands at retrieval can have a measureable impact on how an item is encoded into WM. This is notable as it shows that, not only can encoding related processes effect how or if an item enters into WM, but expectations of how that item will be used also affect how the item is encoded. Importantly, the present study uses a different paradigm from the cited studies as it compares two different types of recall: simultaneous and sequential. Therefore, a discussion on why we might see differences in both the behavioral and neural data between the two recall tasks is warranted. One possible explanation is that participants may be using different strategies for each recall task. For example, perhaps participants hold in VWM the two left items and elect to ignore the two right items. This strategy would be more effective in the simultaneous because participants can immediately report the two items held in memory. However, in the sequential condition, in which response order is chosen randomly for the participant, this strategy would be less effective. This is because on approximately half the trials the participant must report the two left items last, leaving the representations of the two items subject to interference or decay over time.
For the participant to maintain high accuracy, it is advantageous for them to encode as many items as possible during the sequential condition. Thus, the differences in task demands may increase the like strategy used seem to be the cause of the differences seen in both the neural and behavioral data. In the behavioral data, we see patterns of results consistent with the use of two different encoding strategies. In the simultaneous condition, accuracy falls off in a linear fashion, where performance accuracy falls with the reporting of each additional item. Additionally, note that the accuracy for the first two items is fairly high but falls off significantly for the second two items. This is consistent with a strategy in which the participant remembers as many items as they can accurately recall. For example, a participant might only remember two or three items regardless of their position. Alternatively, in the sequential condition, accuracy is relatively high for the first item recalled, but lower for the last three items. Importantly, the accuracy for the last three items is almost identical. This is constant with a strategy in which the participant must try to remember all the items present. For example, because the participant does not know which items they will be probed on, they must try to remember all items equally. In the neural data, a similar pattern is found. If one assumes that during the sequential condition participants are attempting to remember all items instead of a subset of items, one would expect greater activity in the sequential compared to the simultaneous condition, which is seen in both the evoked and induced power analyses.
Importantly, one potential confound can emerge when examining the induced power as a function of “good” and “bad” trials, particularly in the simultaneous recall condition. For example, during the simultaneous condition, participants might encode two items strongly and two items weakly. These participants may use a strategy in which they report the weakly encoded items first to stave off forgetting. As a result, the participants may then forget the two remaining items that were initially strongly encoded. This may lead to the case in which, during the induced power analysis, trials may be sorted incorrectly as bad trials, despite the participant successfully encoding the first two items. This would then lead to a confound in the induced power analysis. The best way to correct for this potential confound would be to segment trials based on response order, and examine the frequency tags as a function of first, second, third, or fourth item recalled. However, this analysis cannot be conducted on the present dataset as there are few trials in which each specific frequency tagged item is chosen in each specific recall position (e.g., 3 Hz recalled first, second, third, or fourth). For example, trials must be segmented not only into groups based on frequency, but also further segmented into subgroups for each of four recall positions. In other words, each group used in the evoked analysis must be further broken down into four more groups, which leaves only around five trials in each group. More work is needed in the future in order to investigate this potential encoding strategy. These results should be considered for future research when interpreting effects of WM during maintenance. In combination, the insights provided by temporally sensitive evoked and induced EEG analyses are important in understanding the mechanism of VWM encoding and VWM in general.
In closing, we built upon past findings showing that encoding related activity contributes to WM capacity limitations (Gurariy et al., 2016; Peterson et al., 2014). Here, we show that within a given trial, frequency tags are larger for correctly recalled items than for incorrectly recalled items. This suggests that encoding related resources are allocated in a heterogeneous fashion across stimuli. Those stimuli that receive more resources are more likely to be later remembered. Furthermore, our induced-power results show that encoding related processes are clearly influenced by task-specific retrieval demands. Taken together our results demonstrate an important and complex role for encoding-related processes in overall VWM performance.
This research was funded by grants awarded to GPC and MEB from the National Institutes of Health: R15EY022775, 1P20GM103650 and the National Science Foundation: NSF OIA 1632849 and NSF OIA 1632738.
- Appelbaum, L. G., Wade, A. R., Pettet, M. W., Vildavski, V. Y., & Norcia, A. M. (2006). Cue-invariant networks for figure background processing in human visual cortex. Journal of Neuroscience, 26, 11695–11708. doi: https://doi.org/10.1523/JNEUROSCI.2741-06.2006 CrossRefPubMedPubMedCentralGoogle Scholar
- Deiber, M. P., Missonnier, P., Bertrand, O., Gold, G., Fazio-Costa, L., Ibañez, V., & Giannakopoulos, P. (2007). Distinction between perceptual and attentional processing in working memory tasks: A study of phase-locked and induced oscillatory brain dynamics. Journal of Cognitive Neuroscience, 19, 158–172. doi: https://doi.org/10.1162/jocn.2007.19.1.158 CrossRefPubMedGoogle Scholar
- Emrich, S. M., Riggall, A. C., LaRocque, J. J., & Postle, B. R. (2013). Distributed patterns of activity in sensory cortex reflect the precision of multiple items maintained in visual short-term memory. Journal of Neuroscience, 33, 6516–6523. doi: https://doi.org/10.1523/JNEUROSCI.5732-12.2013 CrossRefPubMedPubMedCentralGoogle Scholar
- Erickson, M. A., Albrecht, M. A., Robinson, B., Luck, S. J., & Gold, J. M. (2017). Impaired suppression of delay-period alpha and beta is associated with impaired working memory in schizophrenia. Biological Psychiatry, 2, 272–279. doi: https://doi.org/10.1016/j.bpsc.2016.09.003 PubMedPubMedCentralGoogle Scholar
- Gurariy, G., Killebrew, K. W., Berryhill, M. E., & Caplovitz, G. P. (2016). Induced and evoked human electrophysiological correlates of visual working memory set-size effects at encoding. PLoS ONE, 11, e0167022. doi: https://doi.org/10.1371/journal.pone.0167022 CrossRefPubMedPubMedCentralGoogle Scholar
- Peterson, D. J., Gurariy, G., Dimotsantos, G. G., Arciniega, H., Berryhill, M. E., & Caplovitz, G. P. (2014). The steady-state visual evoked potential reveals neural correlates of the items encoded into visual working memory. Neuropsychologia, 63, 145–153. doi: https://doi.org/10.1016/j.neuropsychologia.2014.08.020 CrossRefPubMedPubMedCentralGoogle Scholar
- Postle, B. R. (2006). Working memory as an emergent property of the mind and brain. Neuroscience, 139, 23–28. doi: https://doi.org/10.1016/j.neuroscience.2005.06.005 CrossRefPubMedGoogle Scholar