Keywords

1 Introduction

Previous research has shown that the pupil diameter (PD) is inherently controlled by the Autonomic Nervous System (ANS) [1, 2]. There is evidence that, in constant light conditions, the pupil diameter is increased when a subject is presented with stress stimuli. The reasons behind this phenomenon lies in a mechanism that modifies the balance between the Sympathetic and Parasympathetic divisions of the ANS [2]. This article describes a subtask under the main project called “AffectiveMonitor” (Fig. 1) which is a system for the evaluation of a computer user’s affective state based on the Circumplex Model of Affect [3].

Despite all that, pupil diameter changes are not only caused by affective reactions, but also by the amount of light that falls upon the retina, causing the Pupillary Light Reflex (PLR), which can be viewed as a process to regulate the retina light flux [4], to occur. This effect causes the contraction of the pupil and is superimposed to the changes in pupil diameter caused by affective responses, and hindering our study. Thus, we seek to remove the PLR component from the pupil diameter signals we measure. In our previous work from the same research group [1], we have already presented our approach, using Adaptive Interference Canceller (AIC) to remove the PLR component from the PD signal. That previous work utilized the AIC canceller to implement a stress detector tested on the reactions of the subject to “Incongruent Stroop Segments”. The study showed promising results in the PD-based system’s performance as evaluated by the Receiver Operating Characteristic curve (ROC). The PD-based stress detector exhibits an area under the curve (AUROC) of 0.9331, indicating robust performance after the PLR was removed.

Fig. 1.
figure 1

An entire system including Kinect V2 (on top of the screen) and TM3 (in front of the computer)

Full size image

There are also other physiological signals that can act as an indicator of arousal changes such as the Galvanic Skin Response (GSR), the Blood Volume Pulse (BVP), the Heart Rate (HR), and etc.; however, the pupil diameter is more suitable to estimate the arousal level and assess the affective state of a computer user because it can be observed non-intrusively, which is critical due to the nature of the study itself. In this kind of experiment, it is highly desirable that the subject is to be at his/her normal state as much as possible, without any unnecessary distracting factor. This issue is also the reason why we chose to use the RGB camera from Kinect, which is already a part of the AffectiveMonitor system [3], to measure the illumination around the subject’s eye. Previously, a light meter was used to obtain illuminance signals to play the role of the required noise reference in the AIC algorithm. The light meter requires the placement of a sensor at the desired area where we would like to measure the illumination and it causes some distraction to the subject during the experiment.

In the following sections, we will discuss the AIC strategy in detail and describe how we obtain the illuminance signals around the eye area of the subject’s face using images from the RGB camera as a mean to measure the illumination.

Fig. 2.
figure 2

(from [1])

Diagram of Adaptive Interference Canceller (AIC)

Full size image

2 Methodology

2.1 Adaptive Interference Canceller

The Adaptive Interference Canceller (AIC) is a system that is often used in Digital Signal Processing (DSP) to remove an unwanted interference component that pollutes a signal of interest [5]. The best way to explain how the system work is to walk through its diagram (Fig. 2). The concept here is to measure the signal of interest s(k) that is corrupted with an uncorrelated noise z(k) as the primary input signal d(k). The reference input signal r(k) is a signal that is correlated with the corrupting noise z(k) but uncorrelated with our target signal s(k). The adaptive algorithm, the Least Mean Square (LMS), in this case, will adjust the parameters in an Adaptive Transversal Filter (ATF) to bring the reference input signal r(k) to be as close as possible to the interference signal z(k) in order to bring the error e(k), down to a minimum value (in a mean squares sense). By doing so, we can obtain our signal of interest, i.e., the filtered signal \(\hat{s}(k)\), with the attenuated interference signal. In order to apply the theory to our application, we can think of the pupil diameter signal (PD) obtained from the TM3 Eye-Gaze Tracker as the primary input signal d(k) while the measured illumination around the subject’s eye area, from the RGB camera (Kinect) is used as the reference input signal r(k). After the filtering process, we expect to obtain the output signal e(k) that mainly contains the Pupillary Affective Response (PAR) component without the Pupillary Light reflex (PLR), which is removed by the adaptive filter.

Fig. 3.
figure 3

Diagram showing the process of finding correlation between Kinect and LUX meter signals

Full size image

2.2 Kinect as LUX Meter

As we explained earlier, in the introduction section, the studies related to the evaluation of affective state require the subject to be in his/her normal condition as much as possible to minimize extraneous stimulation or distractions. We chose to utilize the RGB camera (Kinect) for an illumination measurement since Kinect is already a part in our system [3]. Firstly, we will briefly define a few terms used throughout this article to help understanding concepts which will be introduced in later sections.

Luminance: Measured in candela per square meter is the parameter perceived by humans as the brightness of a light source.

Fig. 4.
figure 4

Cropped video used to compute luminance around the eye area

Full size image

RGB Camera: Captures the incoming light rays and turns them into electrical signals enabling many pieces of electronic equipment to act as light detectors. The incoming color and brightness of an image are converted to numbers, preserving those characteristics of the image and breaking it up into millions of pixels, depending on the camera resolution.

For the purpose of eliminating the unwanted PLR factor from the pupil diameter signal using the RGB camera, we only compute the pixel values around the eye area, using a cropping rectangle image that always has its center between the left and the right eyes (Fig. 4).

The pixel values in an image from the RGB camera are proportional to the luminance, because the light sensors convert the intensity of light falling upon them to electrical signals whose strength depends on the brightness of the received light. That is why an RGB camera can act as a luminance meter [6]. Equation 1 is used to calculate the luminance from RGB values in the image, according to a color model based on human physiological characteristics [7]. Note that, R is Red, B is Blue, and G is green.

$$\begin{aligned} Y' = 0.299R'+0.587G'+0.114B' \end{aligned}$$
(1)

However, the sensitivity of the sensors may be different for different RGB cameras. The relationship shown above may vary depending on the camera specifications. For this reason, we need to find out if the luminance values measured via our implementation followed the same trends as the luminance values measured using a luminance meter. To verify this hypothesis, we performed simultaneous light measurements in our experimental setup using the RGB camera from Kinect and a stand-alone LUX meter (Extech 401036 Datalogging Light Meter), while introducing strong illumination changes. Subsequently, after some processing, we computed the correlation between the two signals. If our hypothesis is correct, the luminance values obtained from Kinect should have a high correlation with the luminance value measured from the lux meter. A summary diagram of how we confirm our hypothesis is shown in Fig. 3

Table 1. Correlation coefficient between Kinect and LUX meter
Full size table
Fig. 5.
figure 5

Pre-processing of luminance signals obtained from LUX meter (blue) and Kinect (red) (Color figure online)

Full size image
Fig. 6.
figure 6

Scatter plot and correlation (m = 8, b = –386)

Full size image

We can notice that the plots in Fig. 5a are not synchronized because of the different delays in the measurement systems. To circumvent this problem, we performed a correlation analysis to determine the delay time and then re-align the two signals. After the aligning process, now we can calculate the correlation between the two signals. Figure 5b shows the plot of luminance after the preprocessing and shifting of one signal to align it with the other. Then we computed the correlation between these signals. The correlation coefficients of illuminance signals measured from Kinect and the LUX meter are shown in Table 1. The pairs of measurements are shown in a scatter plot in Fig. 6, which also includes a “best fit” line. The result indicates strong correlation between the two signals, confirming that our hypothesis is correct and that we can use the luminance signal from our implementation as the reference input r(k) (see Fig. 2) to filter out the PLR from the pupil diameter measured signal.

2.3 Removing the Pupillary Light Reflex

As we have explained, we use an adaptive interference canceller (AIC) to filter out the Pupillary Light Response (PLR) and obtain a result, an output signal, containing only the Pupillary Affective Response (PAR). The first step is to pre-process the pupil diameter signal (PD); for instance, substituting missing samples due to eye blinks with an average value of the samples recorded in the neighborhood of the missing samples, and normalizing the pupil diameter as well as the illuminance signal before the filtering process. The signals we record are pupil diameter values obtained from the TM3 Eye-Gaze Tracking device from both left and right eyes containing about 7000 samples recorded at a sampling rate of 1 sample/sec. The illuminance signals are recorded using the RGB-camera (Kinect) at the same sampling rate. An example plot of pupil diameter signals after pre-processing along with the illuminance signal is shown in Fig. 7.

To develop the adaptive interference canceller (AIC), we follow the theory and practice from [8] as our guideline to implement an LMS adaptive filter. Both recorded pupil diameter signals that are impacted by the pupillary light response (d(k)) and the illuminance signal (r(k)) are normalized before they are processed by the LMS adaptive filter. There are two hyperparameters that affect the performance of the adaptive filter. They are the length of the delay line (L) and the learning rate (mu). The longer the delay line is, the slower and smoother the modified reference input signal (y(k)) becomes. In this case, we would like y(k) to imitate the PLR component in the primary input (d(k)) as much as possible so the output signal (e(k)) is only left with the PAR after y(k) is subtracted from d(k). The learning rate (mu) determines how fast the filter can adapt to its target. Setting the right learning rate (mu) is critical here since if it is set too low, the filter could have a degraded performance; while the system might be unstable if it is set too high. In our study, we set the delay line length (L) at 10 and the learning rate at 50. We will discuss our results in the next section.

Fig. 7.
figure 7

Plots from top to bottom: Pupil diameter (Left), Pupil diameter (Right), Illuminance

Full size image
Fig. 8.
figure 8

Plot of signals processed in removing PLR using AIC

Full size image

3 Result

Our results are shown in Fig. 8, which consists of two plots. The first one (Fig. 8a) shows signals d(k), r(k), y(k), and e(k), respectively, from top row to bottom row. Figure 8b shows some of these same signals superimposed for an easier visualization. Here each signal is shown in a different line style. The primary input signal (d(k)) is represented in solid black line at the top part of the graph; this signal is the left pupil diameter signal. Our output signal (e(k)) is also in solid black line but located at the bottom of the graph. The reference signal (r(k)), illuminance, is shown here in light color and, lastly, the modified reference signal (y(k)) is plotted in dotted line. It is possible to observe, in this figure, how each signal behaves based on the nature of the pupillary response. The basic idea is that when the illuminance is low, the pupil diameter is increased in order to adjust the amount of light that reaches the retina. That is why when the reference signal is lower, the pupil diameter signal is shifted upward.

The purpose of the implementation of the adaptive filter is to eliminate this effect and shift the pupil diameter down to the baseline when the interference from pupillary light response produced by illumination changes occur. In Fig. 8b, we observe that the output signal behaves as we expected. In these instances, the output signal in the plot did shift down to the base line while still preserving the PAR information. Hence, the LMS filter seems to be removing the influence of the pupillary light response from the pupil diameter signal.

4 Conclusion

In this paper, we described the processing of images from an RGB-camera for substitution of a LUX meter to measure the illuminance around the eyes of the subject. Our testing showed a correlation of 0.922 between the illuminance signals obtained by the LUX meter and the Kinect camera.

We then used the illuminance signal obtained through the RGB camera in Kinect as the noise reference signal in an adaptive interference canceller. In the canceller architecture the measured pupil diameter signal is the primary input and comprises both, the pupillary light response (interference) and the pupillary affective response (signal of interest). The behavior of the results obtained from the adaptive interference canceller seem to indicate that canceller is, in effect, compensating for pupil diameter signal shifts that are clearly occurring in response to illumination changes.