Keywords

1 Introduction

After more than two decades of technological improvements, virtual reality has evolved to an artificially created environment in which the user feels actually present [1]. Presence can be defined as the sense of “being there” in which the body behaves as if it is part of the illusion [2]. To a great extent, this feeling of presence is accomplished by using high-definition computer-generated graphics and stereoscopic 3D images that are presented on a head-mounted display (HMD), which moves along with the user’s head. Virtual reality found its way to everyday entertainment, but also to industrial, educational and medical settings [3, 4]. For example, doctors are trained to perform surgery using virtual reality, patients suffering from Parkinson’s disease learn to deal with their decreased postural control [5], and psychologists try to overcome the client’s social phobias with virtual environments built for exposure therapy [6]. Since the technology behind virtual reality (VR) and head-mounted display (HMD) devices is changing rapidly and the number of VR applications is growing every year, research on the user’s experience of being in the virtual environment itself and the methodologies to measure these experiences has become highly important too. However, it is important to clearly define what is meant with “experience”, which is a rather vague and broad concept. In the context of gaming, experience is defined as ‘an ensemble made up of the player’s sensations, thoughts, feelings, actions and meaning-making in a gameplay setting’ [7]. Moreover, considering that it points to an interaction between game, gamer and context, it is an agent-dependent and highly subjective concept [8]. Consequently, it is suggested that it is better to look at game experience as an underlying mechanism that make games motivating and fun.

Therefore, an important framework that is often referred to when studying the user’s experience in games is ‘flow theory’ [9]. Flow can be defined as a state of intense attentional focus, pleasurable feelings, and emotional rewards when engaged in a certain activity or task [e.g., 10,11,12]. Examples of activities that can produce a flow experience are dancing, making music, exercising, but also playing video games [13, 14]. Moreover, flow is often associated with a distorted perception of time and a loss of self-consciousness. It is also a highly individual-specific experience because activities that are perceived as enjoyable by some can be dreary for others. The key principle of flow is that it is determined by the sweet spot between skill and challenge [10, 11]. When a person’s skills are highly developed but the challenge is low, he or she becomes bored. In contrast, when skills are underdeveloped but the challenge is high, a feeling of frustration and anxiety might arise. Flow occurs when skills and challenge are perfectly aligned, leading to satisfaction and happiness. In order to create an optimal gaming experience related to flow, clear goals, feedback, sense of control and an appropriate balance between challenge and skills should be integrated [9, 15]. More specifically, the overriding goal of the game should be made clear from the beginning and intermediate goals should be presented at appropriate times [16, 17]. Feedback can consist of several elements: progress towards the goals, immediate feedback on in-game actions and the ability to know your status or score in the game at any given time [15]. Challenge is considered the most important aspect of a good game design. Difficulty levels of a video game should be variable to meet all players at the correct level of challenge. To create this balance between challenge and skill level, games typically start out with a beginner’s level which gradually increases in difficulty as the player’s skills progress [17, 18].

Interestingly, an important characteristic of flow during gaming is the high level of attentional focus. When playing a game, the player has to allocate attention to relevant stimuli (e.g. opponents, targets) but ignore irrelevant stimuli (e.g. a tree in the background). In the flow state, these irrelevant stimuli coming from surroundings are usually more ignored and less consciously processed compared to a state without feelings of flow [19]. This led to an interesting line of research using dual-task paradigms to measure flow in an indirect way [e.g., 20, 21]. The idea is that in addition to a primary task (i.e. playing the video game), a secondary task is implemented in which stimuli are presented that will attract the player’s attention. A good candidate for the secondary task is the classic oddball paradigm, originating from the field of cognitive psychology. In this paradigm, a series of auditory tones are presented at more or less the same pace. The crucial manipulation is that in one out of ten tones (by approximation), participants hear a deviant ‘oddball’ tone that is presented with a different tonal frequency and the participants are asked to press a button as soon as they hear it. The idea is that this oddball stimulus pulls attention away from the primary task (i.e. playing the game) and can therefore be an indirect marker of the amount of attention allocated to this primary task. Flow would be associated with increased attentional focus, so the response to the oddball tone should be slower. Interestingly, an oddball tone also elicits the so-called P300 component. This P300 event-related potential component reflects an increase in electrical activity on the midline posterior scalp surface around 300 ms after the presentation of the tone and has been related to cognitive surprise or stimulus categorization [e.g., 22]. Similar to the behavioral response and based on the assumption that there is a limited set of attentional resources [23], it follows that when people are actively engaged in the primary task (or video game), fewer attentional resources will be available for the processing of the auditory tones, which will be reflected by a decreased P300 response. Furthermore, because of this close relationship between flow and attention, Weber and colleagues approached flow from a neurocognitive point of view and proposed their ‘Synchronization theory of flow’, which can be directly applied to a media context [12]. In this theory, flow is considered as a synchronization phenomenon of different attentional and reward networks in the brain. Decades of research on brain oscillations has shown that when brain regions oscillate with similar frequencies (or “rhythms”), less energy is consumed. In other words, neural synchronization is energetically cheap [24]. This can explain why flow is rarely associated with feelings of exhaustion or burdensomeness, although the task or activity can be highly challenging [12]. Evidence for this theory was found in multiple experiments [20, 25, e.g., 26,27,28]. Weber et al. [25], for example, showed increased functional connectivity among attentional networks with decreasing distraction to stimuli in a secondary task.

However, in this study, our primary goal was to extend the methodology of the dual task-paradigm to the domain of virtual reality. This is especially interesting, because although questionnaires and interviews are quite common when measuring gaming experience [29], it can be argued that they are likely to disrupt the immersive experience in VR. For example, when a participant is playing a game in a virtual environment without any other avatars in the room, the voice of an experimenter or a pop-up message asking about his experience will likely decrease his or her feelings of presence. In this regard, the use of electrophysiological measurements like EEG (brain activity), GSR (galvanic skin response) and ECG (heart rate) can be very informative. These kinds of psychophysiological measures are not only less likely to disrupt the immersive experience in VR, they also reflect unconscious cognitive and affective processes that are impossible to measure with questionnaires or via interviews. However, a challenge where the field currently has to deal with is the fact that psychophysiological signals often become hard to interpret when studying real-life behavior like playing a game in VR. There are two main reasons for this. First, since it is often impossible to impose rigorous experimental control [30], the signal-to-noise ratio is too low to obtain reliable measurements. The solution for this is to increase the duration (and hence quantity) of the measurements, in order to obtain more data so the noise will be averaged out. A second reason relates to the fact that the visual stimulation in a virtual environment can be so complex and intense that it becomes hard to distinguish between different cognitive and affective processes [21]. On top of that, processes with increased activation (e.g. more attention to targets) and processes with decreased activation (e.g. less attention to background) might camouflage each other in the physiological signals, so they will not be picked up in the analysis. We wanted to check whether using the dual-task paradigm to indirectly measure attention allocated to the primary task would be a solution for these issues in the context of VR.

In order to compare different attentional states determined by the balance between the challenges and skills of an individual, we modified the game mechanics of a popular shooter game so that an experience of boredom, flow and frustration was created. In addition, we let participants experience all three conditions both with a regular PC gaming set-up (desktop computer connected to television screen; from now on referred to as ‘PC’) and a VR set-up (desktop computer connected to HTC Vive HMD; from now on referred to as ‘VR’). While participants played the shooter game with their right hand, they had to respond to oddball sounds with the left hand. Concretely, eighteen participants played the same game with subtle adaptations for eight minutes each in six different versions (fully counterbalanced within-subjects design; VR/PC x boredom/frustration/flow) while their EEG was recorded, heart rate was monitored, subjective flow experience was questioned and in-game performance was logged.

With respect to our predictions, we first wanted to make sure whether the participants did indeed experience the boredom, flow and frustration condition in the way we intended it, by analyzing their self-reports (i.e. flow questionnaire). Second, since previous research has shown that the response to the oddball sounds was delayed and that more detection errors were made when participants experienced flow compared to boredom and frustration [21], our main research question was whether we could replicate this effect and whether this effect would be larger for VR. The latter can be expected because immersion is higher in VR. Similarly, we hypothesized that in VR, compared to regular gaming on a PC set-up, participants would show a smaller P300 component, indicating increased attentional allocation to the game. Furthermore, we expected this effect to be driven by an experience of flow, compared to an experience of boredom or frustration. Third, with respect to brain oscillations, we expected to find changes in alpha power (8–13 Hz), where increased alpha power would be an indication of boredom and decreased alpha power an indication of attentional focus. In addition, increases in high-frequency power (beta and gamma) would also reflect more attentional allocation. However, we were not sure whether the auditory stimuli in the oddball paradigm would have an effect on the oscillatory activity related to the secondary task, making it hard to make exact predictions. Finally, we also included electrophysiological heart rate measurements (ECG) in order to measure arousal. Since increased flow and increased immersion in VR would likely lead to increases in autonomic arousal, we expected this to be reflected in the ECG signal.

2 Method

2.1 Participants

For this study, 18 participants were recruited through online sampling. There were more men (83%; M = 23 years old; SD = 2.1 years old) than women (17%; M = 25.67 years old; SD = 2.52 years old) and almost all of them (94.44%) were highly educated or still attending university. With respect to gaming experience, 61% classified themselves as casual gamers, 33% identified themselves as experts and 6% had never gamed before. In addition, 44% had already played the commercial version of presented game. All included subjects participated on a voluntary basis and signed an informed consent with ethical approval from the universal ethical committee.

2.2 Stimulus Material

Primary Task.

As a primary task, subjects were asked to play a custom-made, first-person shooter game based on the commercial success “Counter-Strike: Global Offensive” (CS: GO). We chose this type of game because it is straightforward to play and has clear goals and immediate feedback, which are important prerequisites for the experience of flow [31]. The subject could start the game by triggering the slide door with the inscription “Start”. In each condition, the player started in a practice room with three targets on the wall. Next, there were five different rooms with targets they had to shoot (see Fig. 1). These targets were cardboard cut-outs of enemies that had to be shot twice, in order to avoid random shooting and accidental striking. Players could only proceed to the next room when every target in a room was hit. Immediate feedback was provided (“Good job”) when shooting all targets in a room and the subject could keep track of his/her progress and munition on a scoreboard. After five rooms, players arrived in a sixth room with a final scoreboard and a portal to the first level again, allowing them to start all over.

Fig. 1.
figure 1

Basic floorplan of the game with target placement (upper left) and decoration (bottom left) and screenshot of gameplay depicted in two versions of the map (upper and bottom right).

Full size image

The three conditions of interest (boredom, flow and frustration) were operationalized by manipulating two different features of the game. First, the conditions differed in the speed at which the shooting targets moved. In the boredom condition, targets were stationary, making it rather easy to shoot them. In the flow condition, targets moved at a low speed from left to right and from back to forth. In the frustration condition, targets moved with high speed. Importantly, only in the flow condition, the speed of targets was adapted from room to room. By implementing this, we ensured that only in the flow condition subjects would make progress based on their skill level. A second manipulation concerned the amount of ammunition granted in each condition. In the boredom conditions, players had 12 bullets per target, which was plenty. In flow, we used a scheme for distributing ammunition throughout the levels, adapting it to the skills of the player. Players were only granted two bullets per target in the frustration condition, giving them no ammunition to spare. Because participants went through six gaming sessions of eight minutes each, two versions of the same map were alternately used, keeping elements like target placement, direction in which targets move, the amount of obstructions and the type of decoration as equal as possible (see Fig. 1).

Secondary Task.

The goal of the secondary task in this experiment was to draw attention away of the primary task. We chose for the oddball paradigm that was also used in the study of Núñez Castellar et al. [21] and Debener and collegues [32]. In this oddball paradigm, participants had to listen to auditory stimuli that were presented with random intervals of 960, 1060, 1160, 1260 and 1360 ms. Participants were instructed to react as fast and accurate as possible to the oddball sounds by means of a response box right below the keyboard that was used for the primary task. Importantly, the experimenter emphasized the importance of performing well on both the primary (i.e. the game) and secondary task. During each gaming session of eight minutes (six in total; two devices x three conditions), 320 sounds were presented. Each session consisted of 80% standard tones, 10% oddball sounds, and 10% novel sounds. The standard and oddball sounds were two sinusoids (350 Hz and 650 Hz) with a mean duration of 339 ms, whereas the 96 unique novelty sounds [33] had a random frequency with a mean duration of 338 ms. To avoid confounds, the low (350 Hz) and high (650 Hz) sinusoids were counterbalanced across participants alternating as standard or rare (oddball) sounds. For this study, only the standard and oddball sounds were analyzed.

2.3 Procedure and Design

In this study, we chose for a within-subjects design. This meant that for each participant, we could compare the three experience conditions (boredom, flow and frustration) for two different devices (2D and VR), without confounding effects of differential baselines (signal-to-noise ratio of the measurements or level of skills related to gaming performance) that we would have had with a between-subjects design. Concretely, participants had to play the three versions of the game (the boredom, flow and frustration version) both in virtual reality with the content presented on the HMD and in 3D (not stereoscopic) on a large television screen. To deal with the challenge of having delicate scalp measurements and a large head-mounted display with straps on top of the head, the game was programmed in a way that the viewing direction could also be controlled with the mouse and keyboard. By doing so, minimal head movement was necessary to play the game, while a certain level of immersion in the virtual environment was reached. With respect to technical specifications, an Alienware gaming PC, a 46-in. Phillips television screen and a HTC Vive HMD were used. This HMD offered a resolution of 2160 × 1200 (with 1080 × 1200 per eye), global lighting and AMOLED-displays of 90 Hz. The task was played with a standard keyboard and mouse. The game itself was programmed in Notepad++ in the object-oriented open source programming language Squirrel. In order to run the game, CS: GO was opened through Steam, an online gaming platform developed by Valve. To convert CS: GO to the virtual reality domain, VorpX, a 3D-driver for virtual reality headsets with full head tracking support, was used.

To avoid training effects, the order of conditions was counterbalanced across participants, whereas device and condition were additionally counterbalanced within participants. Participants were tested individually and were seated at approximately 1 m of the screen behind a table (Fig. 2). While mounting the EEG electrodes, participants were asked to fill out the informed consent. Next, participants were asked to focus on a fixation cross for six minutes, while alternating between no blinking and being relaxed for one minute. Afterwards, standardized instructions were given to the subjects and they got the chance to practice navigation and shooting in the game environment. In between the different runs, participants filled out the Flow Questionnaire (FQ; [34]), which we used to measure the subjective experience of flow during gaming.

Fig. 2.
figure 2

Illustration of the experimental set-up with the participants playing the game in the virtual reality condition. The computer on the left provides the oddball task, whereas the laptop in the middle is used to record the EEG and the gaming computer with flat screen and the HTC Vive on the right display the game.

Full size image

2.4 Electrophysiological Recordings and Preprocessing

EEG data was collected with a Biosemi ActiveTwo system (Biosemi, Amsterdam, Netherlands) using 64 Ag-AgCl scalp electrodes positioned according to the standard international 10–20 system. Because of the HMD, it was impossible to attach VEOG and HEOG electrodes, but we did measure the ECG signal with three additional external electrodes (attached to the left and right collarbone and the lower left ribcage). EEG signals were recorded with a sampling rate of 2048 Hz and pre-processed with a bandpass filter of 0.01–30 Hz. Data was processed and analyzed using EEGLAB [35] and ERPLAB [36]. The quality of the continuous EEG was manually examined and large episodes of random noise were deleted. Nevertheless, all datasets were eventually included in the current study. Electrode P2 showed excessive high frequency noise in most datasets and was therefore deleted and interpolated. The signal was re-referenced offline to frontal electrode FPz, but results were quite similar with AFz or the average of all electrodes as reference. In a next step, epochs, time-locked to the onset of the auditory stimulus (oddball or standard), were extracted with a time window of −200 to 2000 ms. In order to exclude remaining artefacts in the data, epoch rejection was applied, deleting epochs (equally distributed across conditions) containing activity below and above a threshold of −70 μV and 70 μV, respectively. Because we were mainly interested in central posterior regions, the artefact rejection was only applied to channels CPz, Pz and POz.

After calculating the average across epochs per condition, a grand average across subjects was computed. Whereas waveforms showing the different electrodes and topographies were based on the grand average, latency and amplitude measures were calculated per subject and condition. With respect to the P300 component, all plots and statistics were based on the central midline electrodes Cz, Pz and POz [37, 38]. We chose these electrodes because the P300 is most often measured in parietal and central regions and less often in frontal electrodes (although a difference can be made between the P3a and P3b, which we will not cover in this study, but see Polich [37] for an overview).

3 Results

3.1 Questionnaires

Flow Questionnaire (FQ).

The flow questionnaire as used by Sherry and colleagues [34] consists of 12 items measuring three subscales: easiness (similar to boredom), flow and difficultness (similar to frustration). Participants completed this questionnaire after each gaming session, allowing us to validate whether feelings of boredom, flow and frustration were indeed experienced in the respective conditions. A repeated measures ANOVA with factors device (PC vs. VR), condition (boredom, flow, frustration) and subscale (boredom, flow, frustration) was conducted on the scores. Mauchly’s test indicated that the assumption of sphericity had been violated for the main effect of subscale, χ2(2) = 23.18, p < .001 and the interaction between condition and subscale, χ2(9) = 65.23, p < .001. Therefore, for these effects, Greenhouse-Geisser corrected tests are reported. The main effect of device was marginally significant, F(1, 17) = 3.66, p = .07, r = .42, whereas the main effects of condition and subscale were highly significant, F(2, 34) = 11.69, p < .001, r = .64 and F(1.13, 19.26) = 38.26, p < .001, r = .83, respectively. The interaction between device and condition was non-significant, F(2, 34) = .63, p = .53, r = .6, but the interaction between device and subscale did show a significant effect, F(2,34) = 5.47, p < .01, r = .49. Both the interaction between condition and subscale and the three-way interaction between device, condition and subscale showed a tendency toward significance, F(1.81, 30.80) = 3.05, p = .07, r = .39, and F(4, 68) = 2.42, p = .06, r = .35.

Taken together, the statistical analysis does not show convincing differences in flow between the different gaming sessions. Indeed, when looking at the FQ-scores, participants primarily had the feeling that the games were easy to play. In the PC condition, the score on the subscale ‘easy’ was even higher than the score on subscale ‘flow’, t(17) = 2.43, p = .03, indicating that the operationalization of flow might have been suboptimal. However, when comparing the subjective experience of flow between VR and PC gaming, participants reported significantly more flow during VR gaming compared to PC gaming, t(17) = 3.46, p < .01 (Table 1).

Table 1. Subjectively reported experience of flow, as measured with the Flow Questionnaire. The bold numbers indicate the condition that should have had the highest score based on the experimental manipulation of the gaming experience.
Full size table

3.2 Behavior

A two-way repeated measures ANOVA was conducted to compare the effect of device and condition on the mean reaction times measured in the conditions boredom, flow and frustration with both PC and VR. There was no significant main effect of device, F(1, 17) = .02, p = .905, r = 0, nor was there a main effect of condition, F(16, 2) = .514, p = .603, r = .17. In addition, no interaction effect was found between device and condition, F(2, 16) = 1.51, p = .239, r = .28. A non-parametric Friedman test of differences among repeated measures was conducted for the error rate and rendered a Chi-square value of 3.84, which is non-significant (p > .5). To detect possible differences between error rates in the conditions, a post-hoc test was administered for every feasible pair. A Wilcoxon Signed-Ranks Test indicated no significant differences in error rates between conditions and devices (all ps > .5) (Table 2).

Table 2. Means and standard deviations for reaction times (RT in seconds) and error rates (ER) related to detecting oddball sounds.
Full size table

3.3 P300

In order to make the P300 as comparable as possible across conditions in terms of baseline, standard trials were subtracted from oddball trials. For the statistical analysis, we took the mean amplitude at three posterior midline locations (Cz, Pz and POz) between 500 and 1500 ms after the onset of oddball or target sound. This allowed us to statistically validate any differences between the conditions on different topographical locations of the scalp surface. A repeated-measures ANOVA with factors electrode (Cz, Pz, POz), device (PC, VR) and condition (Boredom, Flow, Frustration) was performed on the data. Mauchly’s test indicated that the assumption of sphericity had been violated for the main effect of electrode, χ2(2) = 19.38, p < .001, the interaction between electrode and condition, χ2(9) = 47.7, p < .001, the interaction between electrode and device, χ2(2) = 13.27, p = .001, and the interaction between electrode, condition and device, χ2(9) = 31.52, p < .001. Therefore, for these effects, Greenhouse-Geisser corrected tests are reported.

First, the main effect of electrode was highly significant, F(1.14,17.15) = 9.05, p < .01, r = .61, whereas the main effects of condition and device were not, F(2, 30) = .24, r = .13 and F(1, 15) = .34, p = .57, r = .15, respectively. Second, the two-way interactions between factors electrode and condition and factors condition and device were also not significant, F(1.89, 28.42) = 1.02, p = .37, r = .25 and F(2, 30) = 1.85, p = .18, r = .33, respectively, just like the two-way interaction between electrode and device, F(1.24, 18.61) = 3.11, p = .09, r = .13. Finally, the three-way interaction between electrode, device and condition was also non-significant, F(2.1, 31.49) = .23, p = .80, r = .12.

Because the two-way interaction between factors electrode and device unexpectedly showed a small tendency towards significance, we decided to do the comparison across conditions for each electrode separately. Results did not change for electrodes Pz and POz, but we now did find a significant main effect of device for measurements taken at electrode Cz, F(1, 17) = 6.71, p = .02, r = .53. As predicted, the P300 response decreased when participants were playing in VR (M = 3.27 µV, SD = 2.59 µV) compared to playing on the PC (M = 1.31 µV, SD = 2.49 µV; see Fig. 3). The main effect of condition was also significant, F(2, 34) = 3.54, p = .04, r = .42, whereas the interaction between device and condition was not, F(2, 34) = .15, p = .86, r = .09.

Fig. 3.
figure 3

The mean P300 component or the neural counterpart of detecting an oddball sound. The left side of the figure indicates that the amplitude of the P300 response was decreased when gaming in VR. The right side shows the topography and location where the P300 was maximal.

Full size image

Focusing on electrode Cz, follow-up paired-samples t-tests gave some indication that this main effect of device was driven by the ‘flow’ condition, showing a marginally significant difference in P300 amplitude between 2D and VR, t(17) = 2.05, p = .06 (see Fig. 4), whereas the difference between 2D and VR for the boredom and frustration condition was not significant, t(17) = .94, p = .36 and t(17) = 1.09, p = .29.

Fig. 4.
figure 4

The mean P300 component, split out for the different conditions (from left to right: boredom, flow and frustration). Although not significant, the plot suggests that the difference between regular and VR gaming is driven by the flow condition.

Full size image

However, these findings should be interpreted with caution, since the topographical map in Fig. 3 makes it clear that the P300 component is more centered around Pz than around Cz, indicating that the significant main effect of device at Cz might not only reflect the P300 component.

When we correlated the mean P300 component across all conditions with the mean in-game performance score (average number of target hits), we found a highly significant negative correlation (r = −.60, p < .01, see Fig. 5). This means that when participants performed well, their P300 response was decreased and they were less distracted by the oddball sounds. All other correlations (corrected for multiple testing), did not reach significance (all ps > .5).

Fig. 5.
figure 5

Correlation between the mean in-game performance (number of targets hit) and the size of the P300 component.

Full size image

3.4 Brain Oscillations

A repeated-measures ANOVA with factors frequency band (theta, alpha, low-beta, mid-beta, high-beta and gamma), device (PC and VR) and condition (boredom, blow, frustration) was performed on the power spectral density of gamma band oscillations, measured and collapsed across Cz, Pz and POz. Unsurprisingly, the main effect of frequency band was highly significant, F(2.34, 39.73) = 35.11, p < .001, r = .82. The main effects of device and condition were not significant, F(1, 17) = .38, p = .55, r = .15, and F(1.05, 17.77) = .94, p = .35, r = .23, respectively. With respect to the two-way interactions, we found a highly significant interaction between frequency band and device, F(2.17, 36.85) = 9.58, p < .001, r = .60. The interactions between frequency band and condition and between device and condition did not reach significance, F(2.21, 37.53) = 2.58, p = .08, r = .36, and F(1.04, 17.61) = 1.14, p = .30, r = .25, respectively. Finally, the three-way interaction between frequency band, device and condition was also not significant, F(2.76, 46.87) = 1.34, p = .27, r = .27. Because of the highly significant interaction between frequency band and device, we did some additional post-hoc analyses. Whereas the difference between PC and VR gaming was not significant for oscillatory power in the theta, alpha, low-beta and mid-beta frequency ranges (all ps > .1), there was a significant increase for high-beta and gamma power spectral density in VR gaming compared to regular 2D gaming, t(17) = 2.23, p = .04 and t(17) = 2.39, p = .03, respectively (see Fig. 6).

Fig. 6.
figure 6

Brain oscillations. The figure indicates that only high-frequency oscillatory activity was significantly increased when gaming in VR. There was no interaction with condition (boredom, flow and frustration)

Full size image

Regarding the engagement index (beta power/alpha power), a repeated-measures ANOVA with factors device (PC and VR) and condition (Boredom, Flow, Frustration) was performed. The main effect of device was not significant, F(1, 17) = 1.15, p = .30, r = .82, just like the main effect of condition, F(2, 34) = 1.04, p = .36, r = .78. Also the interaction between device and condition did not reach significance, F(2, 34) = .95, p = .40, r = .75. Therefore, we did not explore this engagement index in further detail.

3.5 ECG

A repeated-measures ANOVA with factors device (PC and VR) and condition (boredom, flow, frustration) was performed on the average heart rate data (measured in beats per minute). The main effect of device was not significant, F(1, 17) = .84, p = .37, r = .22. There was no difference in average heart rate when participants were playing in VR (M = 80.63 bpm, SD = 8.08 bpm) compared to playing on the PC (M = 78.69 bpm, SD = 13.04 bpm; see Fig. 7). The main effect of condition also lacked significance, F(2, 34) = .06, p = .94, r = .06, just like the interaction between device and condition, F(2, 34) = .01, p = .99, r = .03.

Fig. 7.
figure 7

Average heart rate, measured in beats per minute, based on the ECG signal.

Full size image

4 Discussion

In this study, we expanded the work of Weber and Huskey [20] and Núñez Castellar et al. [21] and used their innovative dual-task approach to measure attentional allocation while gaming under conditions of boredom, flow or frustration. Importantly, we made a well-controlled comparison between regular PC and VR gaming in order to find out whether the dual-task approach could also be extended to more immersive technologies. The dual-task approach consisted of letting participants perform a primary task (shooter game) while asking them to respond to oddball sounds in a secondary task (oddball task). Every participant played six games in total (VR/PC x boredom/flow/frustration) and each game lasted eight minutes on average. Our first goal was to validate whether participants did indeed experience feelings of boredom, flow and frustration like we intended them to do. A second goal was to replicate the finding that during flow, participants were slower and less accurate to respond to oddball sounds than during boredom or frustration [e.g., 21] and to find out whether this effect would interact with the type of device (PC or VR). A third goal was to extend behavioral markers of attentional allocation during gaming to psychophysiological markers like the P300 ERP component, brain oscillations and the ECG signal.

Unfortunately, participants did not subjectively reported more feelings of flow in the flow condition compared to the boredom and frustration condition during both PC and VR gaming. Their predominant feeling was that of boredom in all conditions, indicating that the game might have been too easy to play. However, when comparing feelings of flow between PC and VR gaming, significantly more flow was experienced when gaming in VR. Hence, although our design was more ecologically valid (closer to real-life) than that of previous research (different game levels in each condition instead of repetitions of levels in boredom and frustration condition, e.g. Núñez Castellar et al. [21]), these results indicate that the way in which we created feelings of boredom, flow and frustration still needs some adaptations and tweaking in order to get the experience right. Besides game mechanics, another reason why we failed to create experiences of boredom, flow and frustration might be related to how proficient participants were at playing games. Both experienced and non-experienced gamers participated, introducing potential confounds like having experience with being in a VR environment or not. This suboptimal experience of flow was also reflected in the behavioral data, for which we did not find significant effects of condition, device or their interaction. In contrast to Núñez Castellar et al. [21], reaction times to the oddball tones were surprisingly similar (around 742 ms) in all six conditions and we did not even observe a difference between PC and VR gaming. Given the results of the subjective and behavioral data, one could argue that it does not make sense to interpret the associated psychophysiological signals because the assumption that participants experienced boredom, flow and frustration in the respective conditions was not met. However, research in the past has often showed large dissociations between subjective and behavioral outcomes on the one hand and more unconscious, implicit physiological outcomes on the other hand [39] and the interpretation of these findings will at least convey interesting information for follow-up research. Therefore, we still think it is valuable to report our findings.

With respect to the neural processing of the oddball sound of the secondary task, we expected the largest decrease in P300 amplitude when participants experienced flow in VR. As expected, we observed a clear P300 response starting around 400 ms after the oddball sound, reaching its optimum at its typical posterior midline location around electrode Pz [e.g., 38]. Although we did not find significant main effects of condition or device, we did observe that the P300 was significantly decreased when playing in VR compared to playing with a regular PC set-up. However, this effect was driven by activity around the more anterior region (Cz), which did not entirely fit with the topographical location of the component. Furthermore, there was some indication that the decreased P300 for VR compared to PC was driven by the flow condition, but this effect should be interpreted with caution (marginally significant effects). Interestingly, we also observed a highly significant negative correlation between the average in-game performance score of a participant and the mean amplitude of the P300 (both collapsed across the six games). This basically means that the better a participant performed, the less he or she was distracted by the oddball sounds. Assuming this correlation can be replicated, this indicates that the P300 component can be used as an indirect marker of in-game performance, which might be of interest to game- or VR developers. Taken together, the data suggests that using the P300 as indirect marker of attentional allocation might be promising in future research but that the signal-to-noise ratio in this study might not have been high enough to find reliable effects. Although we used a within-subjects design to compare conditions, eighteen participants just might not have been enough.

When looking at brain oscillations, we found that there was a significant increase in high frequency activity (high beta and gamma; 21–100 Hz) for the VR conditions. There was no interaction with condition or an interaction between condition and device. We can think of two reasons for this finding. A first one relates to motor-related brain activity: since it is known that beta and especially high-beta activity is associated with brain regions responsible for muscle activity [40], it is clear why we would observe more beta in the VR condition, where participants tended to look around more than during PC gaming. A second reason, however, directly relates to attentional focusing. Previous research has shown that gamma activity can reflect higher-order cognitive processes like mental effort and concentration [41]. Therefore, it is likely that participants were able to focus more on the game in VR because of the increased immersion and presence. An interesting discrepancy with previous research relates to the alpha frequency band (8–13 Hz). Whereas Núñez Castellar et al. [21] showed a significant alpha power increase in the flow condition, likely related to reward-related processes, we did not observe this difference. Another caveat we have to make is that we only looked at brain oscillations in posterior regions (Cz, Pz and Poz). Previous research has shown that different frequency bands reflect different processes depending on the location of the measurement, but we did not this into account [e.g., 42]. Nevertheless, in contrast to measuring the P300 component, for which a secondary oddball task is required, looking at brain oscillations is a very straightforward and easy procedure. Therefore, we definitely think brain oscillations are a promising way to assess user experience during VR gaming in the future (given adapted HMDs that include EEG electrodes). Furthermore, there are still a lot of interesting research questions related to the aforementioned Synchronization theory of flow [12], which considers flow as a synchronization phenomenon of different attentional and reward networks in the brain. It would be interesting to investigate whether the observed increases in high beta and gamma are related to these attention- and reward-related networks. Finally, we measured the ECG signal and performed an analysis on the average heart rate per gaming session (beats-per-minute). This analysis did not reveal any interesting effects. However, there is some research suggesting a promising role for heart-rate-variability analyses in gaming research [43].

In sum, our research suggests that psychophysiological measures are promising tools to quantify attentional allocation in VR, but more research is needed to clarify whether and how this translates to flow.