Keywords

1 Introduction

Emotions in humans are defined as states of readiness that are used to prepare behavioral responses, foster interactions with the socioeconomic environment, and enable us for making advantageous decisions [19]. Nevertheless, emotions are not always accurately processed, and people overwhelmed by them may take disadvantageous decisions. Therefore, it is no surprise that previous research established that emotions impair or facilitate advantageous decision-making performance [8]. Such emotions may be classified through their two independent components, which Russell defined as valence and arousal [18]. In such a model, valence is defined as an emotional experience of certain situations, and those situations are evaluated in a positive or negative aspect. On another hand, arousal defines the level of excitement in certain situations. In that regard, difficulty and performance on the decision task have been correlated with physiological arousal, which has been generally validated in the models of emotions [4].

Previous investigations have found that economic decision-making may be considered biased through high emotions and arousal [1]. Such investigations were found to be highly context-dependent. Thus serious games emerged as a useful tool that can deliver the needed context [20]. Serious games may be defined as games whose purpose is other than entertainment alone, and it can be considered as a ‘serious’ investigation of certain aspects of human endeavor [12]. Through their inherent ability to provide decision choices with immediate feedback [22], such decision-making choices in serious games could be designed with the financial context in mind. Furthermore, they could provide an application for practicing emotion-regulation through psychophysiology that can be implemented as a method of biofeedback [17]. Therefore, serious games might provide an interactive context for the on-line perception of emotional responses through biofeedback, which reflects the changes in the individuals’ physiology [25].

In contrast to the traditional methods of applying sensors to the skin, eyetracking enables the collection of physiological data unobtrusively and remotely [16]. The expensive hardware and sensing conditions required, restrict the application of directly-attached sensors in the industrial domain [15]. Taking this limitation into consideration, contactless, non-invasive, and unobtrusive methods for acquiring physiological signals are required to evaluate human behavior in such applications [23]. Therefore, this paper sets out to investigate the potentials of using pupil diameter (PD) measure as a remote device that has the benefits of providing contactless biofeedback for serious games.

Fig. 1.
figure 1

The underlying mechanism of the activation of SNS and PNS on the PD and HR physiology [9].

Full size image

2 Background

It is argued that the underlying activation of autonomic nervous system (ANS) is a part of the generative process of emotions and that physiology allows for the investigation of the complex concept of arousal, which is reflected in the underlying mechanism of the activation of sympathetic nervous system (SNS) and parasympathetic nervous system (PNS) [21]. The authors emphasize the interdependence of such activation in the underlying mechanism of emotions. Both of the mentioned ANS branches are associated with specific modalities of physiology, such as heart rate (HR) and PD [5]. Nevertheless, it is unique that the underlying activation of ANS is observed clearly and separately in the physiological measure of PD. Therefore, pupillometry and PD are associated with both the SNS (dilation) and PNS (constriction) tonus and their balance [5]. Similarly, HR is also associated with both branches of ANS activation and allows for their investigation [13]. Both of these mechanisms have been illustrated on Fig. 1.

There are two independent muscles controlling PD, and both are associated with the two competing branches of the ANS [10]. More specifically, the iris sphincter muscle is innervated by the PNS, while the SNS innervates the iris dilator muscle. Therefore, evidence suggests that PD is a measure of the activation of both the SNS and PNS influencing the contraction of the sphincter and dilator muscles. Therefore, the minimal constricted PD is associated with SNS activity, while the maximal dilated PD is associated with PNS activity [24]. Nevertheless, evidence suggests a complex interdependence of PNS and SNS influence on pupillary muscles, as the PD might be associated with the effect of inhibition of the PNS tone together with the SNS activation [5]. Previous research found strong correlations between PD and heart-rate variance (HRV) spectrum in the low-frequency (LF) (0.04–0.15 Hz) and high-frequency (HF) (0.15–0.45 Hz) bands [16].

HR and its measure of HRV have been validated as suitable physiological measures of arousal, commonly used in serious games. The information regarding health, emotional, and cognitive state of a person could be inferred through physiological measurements [16]. Therefore, the possibility of acquisition of such measurements with a contactless method might have an important application in many fields, one of which is in serious games [16]. Ample evidence suggests that PD changes reflect higher cognitive processes, such as information-processing load, and physiological arousal in reaction to the stimuli [3]. Contactless sensors offer an added advantage over the traditional ones because they eliminate the possibility of motion artifacts that may occur with electrode measurements [16]. Spontaneous fluctuations in pupillary diameter (SFPD) are innervated by the pupillary contraction and dilation which are under control of the ANS, and therefore correlated with HRV [16].

HRV indices reflect the autonomic balance, where HF band is associated with PNS activity, while the LF band is a complex interference of both SNS and PNS influences in both efferent and afferent direction. Moreover, the LF band is associated with the vascular system resonance. This association motivates the usage of LF/HF ratio as a measure of physiological arousal. Moreover, similar effects were also found for the HRV indices in electrocardiogram (ECG) data, where increased SNS activity was found to increase HR, while PNS decreases it [10], as illustrated on Fig. 1. Variations in the interbeat intervals of the successive heartbeats were associated with HRV, and it has been regarded as a quantitative marker of cardiovascular regulation by the ANS. Such marker is used in physiology as the measure of activation of the SNS - which reduces HRV, and with the activation of the PNS - which increases HRV [16]. This activation and inhibition mechanism of the two branches of ANS causes SFPD, termed ‘hippus,’ which are the target of the investigation in this study [16].

Physiological recordings of arousal are assessed against the baseline measurements before the onset of the task, and they reflect the resting-state of the ANS [11]. It is accepted that bodily states shape emotions, and their physiological responses provide an objective insight into their workings [6]. This renders physiology as an objective measure for evaluating decision performance in games [14].

More specifically, this paper sets out to investigate how the interdependent and competing activation of the two branches of ANS, mainly PNS and SNS, are reflected on the pupillary muscles and affect the performance on decision-making task in serious games, through the measurements of PD and HR. The proposed question will be validated through the usage of ground truth ECG device measuring arousal through HRV. Furthermore, the effects of such arousal inferred through PD will be investigated in the financial decision-making task in the serious game.

3 Methodology

The 21 students of Blekinge Institute of Technology were randomly-recruited and participated in the study. They were aged between 20–24 years, while 14 were males and 7 females. No major psychiatric, medical disorders or ophthalmological problems other than corrected vision was reported. The participants signed the informed consent to take part in the experiment after they received complete information on the aims and experimental conditions of the study.

3.1 Experimental Setup

A constant temperature of 23 \(^{\circ }\)C ± 1 \(^{\circ }\)C and artificial fixture light were controlled throughout the experiment. The experiment was performed in the sound-attenuated room, where the participants were seated in a fixed chair in front of a screen at a 50–60 cm distance. The screen was equipped with the Tobii T60 eye-tracker recording PD data with a frequency of 60 Hz. Furthermore, physiological Movisens ekgMove sensor was applied using a chest band holding two contact electrodes positioned at the left and right lowest rib points, which provided a minimal obtrusive measurement of ECG data. Biofeedback arousal data based on HRV was transferred to the serious game application using Bluetooth. Baseline period for both of the physiological modalities was recorded for five minutes in a resting state before the onset of the game. Both of the physiological datasets were analyzed offline.

Fig. 2.
figure 2

The screenshot of the challenging decision-making task in the Auction Game, depicting the players’ physiological arousal state through biofeedback meter at the top right [2].

Full size image

The serious game used in this study was the Auction Game [2] since it presents a challenging financial decision-making task to the participants, see Fig. 2. The game was connected with the ECG sensor for the on-line biofeedback based on physiological arousal, inferred from the HRV data. Such coupling provided a reliable measure of physiological arousal in a stressful environment [2], which was used as the ground truth for the data analysis. The inferred physiological arousal continuously adjusted the difficulty of the decision-making task in the game through the biofeedback method.

In the game, the participants were presented with a buy or sell decision for given trading stock. To make a decision, participants had to calculate the mean value from three given price estimations and click on the buy or sell button based on the offered price for a stock. Price estimations were directly linked to the physiological arousal level, such that they deviated from the correct price with higher variance, the more aroused the participant was. Thus, lower physiological arousal would make the variance of price estimations closer to the correct price, so that a buy or sell decision could have been easier. The serious game was linear, which meant that there was always just one possible correct decision to be made in each trial. To promote a more significant challenge, higher physiological arousal also reduced the decision time, while the task became more challenging at subsequent trials as the decision period was further reduced, forcing a quick decision.

3.2 Data Analysis

Regarding ECG data, the raw signal from the device was amplified, after which it was band-pass filtered at 10–40 Hz with 16-bit digitization. Furthermore, the signal was then smoothed using a 10 ms moving average window, while the R-peaks (heartbeats) were identified using the OSEA algorithm [7]. The heuristic rules were applied in detection to avoid missing R-peaks or detecting multiple peaks for a single heartbeat. The final data consisted of interbeat intervals measured between successive heartbeats to obtain the HR and HRV parameters accordingly.

Regarding PD data, offline analysis of the eye-tracker data in millimeters was used to obtain the power of LF and HF bands that correlate to HRV parameters. The data were corrected for the short and long blink periods. A linear interpolation was then applied to the short blink periods. Both HRV and PD values were normalized for each participant by subtracting the data from the resting-state baseline measurements before the task. Following the method given by Park et al. [15] for extracting HRV indices of LF/HF ratio, the PD data were band-pass filtered for the LF range (0.04–0.15 Hz) and the HF range (0.15–0.4 Hz). Afterward, the signal was processed using FFT analysis (i.e., using the Hanning window of size 180 samples and the resolution of 1 sample) to extract the powers in the LF and HF bands, which were used to infer the HRV index of LF/HF.

4 Results

In order to investigate the potentials of using PD as a remote measure for providing contactless biofeedback for serious games based on physiological arousal, PD and HR values were recorded during the performance on decision-making task in serious games. Pearson product-moment correlation and ANOVA comparisons were used to determine the relationship between several variables determining performance in the Auction Game and normalized PD values during the task. Decision performance was assessed throughout the game as: time needed to reach a decision; final game score (money earned); and the level reached in the game.

Fig. 3.
figure 3

The difference between the arousal levels based on the ground truth HRV data, in regards to the LF/HF ratio acquired through PD, on a decision-making task in the Auction Game. There is a significant difference between the highest and the lowest arousal condition with 95% confidence interval at the <.001 probability level. The line illustrates a significant negative correlation between the mentioned data.

Full size image

Regarding the relationship between arousal parameters of LF/HF acquired through PD data and the ground truth HRV acquired through ECG data, there was a significant difference between the arousal levels (F(4,1509) = 11.5157, p < .001), as depicted on Fig. 3. A Tukey posthoc test revealed that the LF/HF ratio for the highest arousal condition 5 based on the ground truth ECG data (−2.004 ± 1.321, p < .001) was significantly different than the ratio for lowest one 1 (−1.675 ± 1.014). There was no statistically significant difference between the other groups (p = .169). Furthermore, a significant negative correlation between the LF/HF ratio and the arousal conditions was found (r = −.112, n = 1514, p < .001), as illustrated with a line on Fig. 3. These results provided evidence for validation that the PD data, more specifically the LF/HF parameter, indeed provides a measure of physiological arousal. More specifically, the higher LF/HR ratio is correlated with lower arousal, and therefore, a relaxed emotional state.

Fig. 4.
figure 4

The difference between the successful and the unsuccessful decision trials arousal, in regards to the LF/HF ratio acquired through PD, on a decision-making task in the Auction Game. The successful trials had significantly higher LF/HF ratio, than the unsuccessful ones with 95% confidence interval at the .010 probability level.

Full size image

There were a strong, negative correlations between the LF/HF ratio and the time needed to reach a decision on each trial (r = −.118, n = 1477, p < .001), as well as, between the LF/HF ratio and the total score (money gained) in the game (r = −.069, n = 1477, p < .008). This evidence was further supported through a significant difference found (F(1,1475) = 6.576, p = .010) between the arousal LF/HF ratio on the successful decision trials (−1.476 ± 0.899), which was significantly higher than on the unsuccessful ones (−1.607 ± 0.978), as depicted on Fig. 4. These findings lend support to the idea that the LF/HF ratio as an index of arousal influences decision performance in serious games, where participants in the more aroused emotional states take longer to reach a decision, and an unsuccessful one at that. These participants also had a lower overall score in the game. More importantly, this information was inferred from the LF/HF ratio acquired from the PD, which gives further evidence for its usage as a remote measure of the emotional state of arousal.

Fig. 5.
figure 5

The difference between the participants who had successfully finished the game and the ones that have not, in regards to the LF/HF ratio acquired through PD, on different levels in the Auction Game. The participants who had successfully finished the game had significantly higher LF/HF ratio than the ones that had not, at the < .001 probability level.

Full size image

Finally, there was a significant difference in LF/HF ratio found (F(1,51) = 11.344, p < .001) between the participants who had successfully finished the game (−1.024 ± 0.415, p < .001), compared to a lower value for the ones that have not (−1.924 ± 1.174), as illustrated in Fig. 5. This finding suggests that the participants who managed to finish this challenging game were able to put themselves in a different emotional state of arousal throughout the game, which enabled them to succeed in the game, compared to the ones that have not reached the end.

5 Discussion

The PD data suggests that physiological arousal influences human financial decision-making, as has been previously validated in the economic field, as well as in the field of serious games through the use of other physiological sensors. This study found that the participants in lower arousal states reach their decisions faster, and make more correct decisions which result in higher overall score (money gained) throughout the game. This evidence gives support to the previous finding, as well as gives evidence that PD may be used as a remote contactless measure of arousal to observe the similar effects. If one considers that such remote measure is less obtrusive and has a lower impact on human behavior in the experimental sessions, one can argue that this might be a more ecologically valid measure to infer arousal in user experiments on decision-making in serious games.

The suggestion that PD might be used as a reliable measure of arousal is further supported through the evidence on ground truth validation between the PD and ECG data. Previous evidence suggested synchronicity between human cardiac rhythm and SFPD through neural pathways [15]. They have found that the power in LF (0.04–0.15 Hz) and HF (0.15–0.4 Hz) bands in the ANS were found to be synchronized with the SFPD rhythm in the same frequency ranges respectively, which was validated in this study as well.

The evidence from neuroscience indicates that the same activation of the ANS should be observable on both ECG and PD data, which was validated through the findings in this study. Nevertheless, due to the interdependence of the PNS and SNS activation with their effect on physiological responses, PD provides a rare opportunity to observe distinct activation of the two different pupillary muscles. This information coupled with other evidence suggesting that also higher cognitive functions (e.g., memory load and cognitive processing) are also observable on the pupil data, gives a clear direction on the PD usage in a well-rounded biofeedback modality which provides a window into the mental processes.

Furthermore, it was observed that the participants in a lesser aroused states throughout the game were able to finish the game until the end, compared to the ones that were in a more aroused emotional state throughout the game. Since the stimuli for arousal states throughout the game were variable, due to biofeedback controlling the difficulty, it is unclear if the regulation of arousal made participants more successful at decision-making. Nevertheless, this finding lends further support to the evidence that physiological arousal influences the decision performance, but it also gives motivation that the participants who were able to regulate their emotional arousal were able to reach further in the game due to more advantageous decision performance.

The evidence suggests that both PD and HRV data on the task were correlated with decision performance in the serious game set in the financial context. As previous research established that both of these physiological measures are associated with the balance between activation of PNS and SNS, it can be suggested that both these physiological measures would be a useful measure to asses the decision-making performance of participants. These underlying factors contribute to arousal, which has already been validated as an influencing factor in decision-making performance and task difficulty [2]. As the complex interdependent activation of PNS and SNS is reflected in both PD and HR data, these findings validate the notion that these modalities are inferring the same activation and its effects on the decision performance scores in serious games.

6 Limitations

In biofeedback-enhanced serious games, it is hard to distinguish if the participants performed better on the decision-making task, or if the task got easier by regulating emotions and subsequently reducing the task difficulty. Such ambiguity in the experimental data is known as the biofeedback loop, and it applies to the evidence given in this study. One of the aspects of this ambiguity is reflected on how the differences in decision-making time affect the arousal states in participant since this time was controlled by the same arousal states elicited. Nevertheless, the inference of physiological arousal on decision performance in the Auction Game was found, regardless of its nature.

Moreover, inferring arousal through PD has a limitation where the SFPD activity during blinking periods is not available. Linear interpolation was applied to mitigate this limitation, but more extended blink periods greater than one second are going to have this limitation.

Furthermore, systematic carryover effects between the trials are observable in the physiological data, where the arousing stimuli of one trial influences pupil size on the next trial. Therefore, future work should study the duration of these carryover effects to design independent trials, as well as try to mitigate this limitation by providing emotion-regulation training before the task.

7 Conclusion

This paper gives evidence that PD might be used as a reliable measure of physiological arousal by enabling the collection of physiological data unobtrusively and remotely in serious games set in the financial decision-making context. This evidence has been compared against the ground truth ECG data. Moreover, it suggests that the interdependent and competing activation of the two branches of ANS, mainly PNS and SNS, are observable on both the ECG and PD data. The Auction Game in this study used an electrode contact ECG device as a biofeedback measurement. Nevertheless, this study validated the potential usage of a more unobtrusive PD measure for biofeedback, which would be beneficial to observe the significant effects of arousal on human decision-making inside of the serious game. Therefore, one might argue that potential future biofeedback could be implemented using the PD physiological modality for such serious games.

Furthermore, this paper gives evidence that physiological arousal inferred through the PD modality affects the performance on decision-making task in serious games set in the financial context. It was found that the participants with lower arousal took less time to reach their decision, and those decisions were more successful in comparison to the participants in high arousal. Therefore, such lower arousal participants were able to get a higher total score in the game. The evidence also suggested that the participants who were able to regulate emotions and have lower arousal throughout the game were able to reach the final level and finish the game.

Taking these finding together, it is suggested that PD might be reliably used as a biofeedback method for arousal in serious games. Future work should investigate this direction of application, as well as providing a method that would provide information on memory load, cognitive processing, and emotional states, all inferred from PD data. Furthermore, if one considers eyetracking as a device capable of inferring attention information, one can see how this modality is a promising device for an excellent biofeedback method on the workings of the mind.