Keywords

1 Introduction

In this study we focus on the relationship between the pupillary responses and cognitive load. Pupil dilation can be measured continuously during processing of a task, therefore; it could be used as a robust measure for cognitive load [1, 2].

In our previous study [3] we showed that by applying a set of plain language standards (PLS) to online passages adopted from internet we can simplify the passages [4] so that there is less cognitive demand on the readers and hence the user’s performance in answering questions related to passages improved significantly. Additionally, we showed that readers who read the simplified version of the passage had shorter average fixations and exhibited a more efficient visual search behavior as compared to those who read the original version of the passage. In this study, we extended our previous work by investigating time series analysis of pupil dilation as a proxy for measuring cognitive effort. To this end, we conducted an exploratory analysis in order to understand how text simplification affected pupil dilation over time and whether or not this effect was consistent over different time intervals during reading. In a recent study, pupillary data during fixation were separated from pupillary data during saccade [5]. It was shown that there was a significant difference between PD during these two types of eye-movements. Grounded in this pervious results, we separated PD data during fixation (PD-fixation) from PD data during saccade (PD-saccade) and investigated the effect of text simplification on these two variables separately over time. Time series analysis was conducted on the same eye tracking data set reported in our previous study that showed text simplification was effective in reducing cognitive load (i.e., it improved performance significantly) [3]. In the following sections, we provide a brief review of related Pupillometry literature, explain the experimental set up used to collect the eye movement data, and discuss the time series analysis that was conducted to examine whether and how pupil dilation during reading was affected by cognitive load manipulated through text simplification.

2 Theoretical Background

Pupillary response is an involuntary reflex and the pupil size can range in diameter from 1.5 mm to more than 8 mm [6]. For more than twenty years psychologists have argued that changes in pupil dilation accompany cognitive processing. Many studies have validated this argument using a variety of tasks, such as sentence comprehension [7], mental arithmetic [8], as well as letter matching [9]. Zehui Zhan et al. [10] used pupil dilation to assess the reading ability of online learners.

A number of studies have shown that users’ pupils dilate when the difficulty of the task increases. For example Siyuan Chen et al. [11] manipulated the level of task difficulty in recalling number of player positions (defender and attacker) in a basketball game which was played in a computer based training application and showed that in some cases pupil size was larger in the more difficult task. Klinger et al. [12] measured pupil dilation during a mental multiplication and found a task difficulty effect on dilation magnitude. They showed that easy manipulation problems caused the smallest pupil dilations and hard problems caused the largest. Other studies also suggest that pupil dilation is a reliable proxy of cognitive load [13,14,15].

A recent study argues that because fixations and saccades reflect different types of processing, it is useful to examine pupil dilation during these events separately [5]. Fixations are relatively stable gazes that allow us to focus on objects that we like to inspect. Hence, fixations have been identified as reliable indicators of attention. Saccades are rapid movements of the eye when moving from one fixation to another. During saccades we are not able to process visual information [16, 17].

3 Methodology

This section provides a brief review of the laboratory experiment that was conducted to collect eye movement data used in this study. It also explains how the pupil data was prepared for time series analysis.

3.1 Eye-Tracking Experiment

A comprehensive set of plain language standards [3] were used to convert an actual online text passage about sports (18th grade reading level) to a simpler version (10th grade reading level). Each participant was randomly assigned to one of the two versions of the text passage, which was displayed on a computer screen. Participants were recruited from a pool of college students. Out of the 54 collected datasets, 26 were from participants assigned to read the original version of the text and the rest were from those who were assigned to read the simplified version of the same passage.

We used a commercially available eye tracking device, Tobii x300, to collect eye movement data. This remote eye tracking device can capture eye movements unobtrusively at the rate of 300 samples per second. The eye tracker was calibrated for each participant before starting the task. This process requires participants to observe a moving dot on the eye-tracking screen. Tobii software version 3.2.3, and I-VT filter with 30 °/sec saccadic velocity threshold was used to process raw gaze data into fixations and saccades.

3.2 Time Series Analysis of Pupil Dilation

Eye-movement data were individually saved in .csv format for further processing. Because the task was not time limited, the duration of reading was different among participants, which resulted in different number of data points for each participant. To facilitate the comparison of time series analysis, studies often equalize the number of data points by designing the task in a way to have pre-specified time windows [18,19,20]. While this approach is useful and relevant for many experiments, we were interested in examining reading behavior in a setting that allowed users take as much time they needed to read and understand the text. To compensate for the unequal number of data points in such a setting, we used cubic spline interpolation method [21] to construct equal size arrays of PD data for all participants. Interpolation is the estimation of intermediate values between precise data points [22]. This process allowed us to have an equal number of pupil data points for each of the participants in each of the two experimental groups (original and simplified conditions). To look at the changes in pupil dilation over time, average PD values for all participants were calculated for each data point.

As mentioned earlier, the results of a previous research [5] shows that pupil dilation during fixation is different from pupil dilation during saccade. Thus, in this study we examined both overall pupil dilation as well as pupil dilation during fixations and saccades. We also looked at changes in PD over three distinct reading periods, beginning, middle, and end. To do so interpolated data points were divided into three equal intervals.

4 Results

The comparison between PD when reading original and simplified passages was indicated in Fig. 1. PD-saccade and PD-fixation were not yet separated in this plot. Figure 1 displays the trend of overall PD over time. As shown in Fig. 1 the overtime PD trend is similar between reading the original and simplified version of the text except for the beginning and the end part of the graph. We used a t-test to see whether the overall averages for these two trends were different. The results of the two-sample t-test for the means of the overall PD during reading the original or simplified passages showed that there were no significant differences between the two trends (t-stat = 0.93, p = 0.35). In other words, no significant differences were detected in pupil dilation during reading original vs. simplified versions of the same text.

Fig. 1.
figure 1

Time series trend of pupil dilation, red: simplified, blue: original passage (Color figure online)

Full size image

Next, we refined the above analysis by investigating differences in the means of PD during three different time intervals (beginning, middle and end). A two-way factorial ANOVA was conducted to compare the effects of two independent variables (1) text simplification, and (2) time interval. Text simplification included 2 levels (1. Original and 2. Simplified) and time interval included 3 levels (1. Beginning, 2. Middle and 3. End). The results, shown in Table 2, indicated that text simplification did not have a main effect on pupil dilation (F (1,938) = 0.88, p = 0.35). The results, however, show that time interval did have a main effect on PD (F (2,938) = 17.44, p = 0.00). The interaction effect was almost significant (F (2,944) = 2.44, p = 0.09). By comparing the pairwise interactions between time intervals and text simplification, as shown in Table 3 and Fig. 2, we can see that there is a significant difference in PD between simplified and original versions in the last time interval. There are no significant differences between simplified and original versions in the beginning and the middle reading intervals (p-beginning = 0.32, p-middle = 0.63).

Fig. 2.
figure 2

The main effect of text simplification and time intervals and their interaction effect on the dependent variable PD

Full size image

Next, we ran the same analysis but this time we examine PD during fixation and saccades separately. Figures 3, and 4 show the time series trend of PD-Fixation and PD-Saccade when reading original versus simplified passages. These graphs show more nuanced differences between PD trends in the original simplified conditions.

Fig. 3.
figure 3

Time series plot of pupil dilation during fixation (blue = original, red = simplified) (Color figure online)

Full size image
Fig. 4.
figure 4

Time series plot of pupil dilation during saccade (blue = original, red = simplified) (Color figure online)

Full size image

A two-way ANOVA was performed to investigate the effects of (1) text simplification and (2) fixation/saccade separation on PD. The results of ANOVA in Table 4 show that PD is significantly affected by text simplification regardless whether it is measured during fixations or saccades (F (1, 1068) = 68.71, p = 0.00) and that PD values are significantly different during saccades and fixations regardless of text condition (original vs. simplified) (F (1, 1068) = 331.64, p = 0.00). There is no significant interaction effect between text simplification and fixation/saccade separation as indicated in Table 4 (F (1, 1072) = 1.63, p = 0.20). The graphical representation of this analysis is displayed in Fig. 5, which shows that no matter what type of passage the participant was reading (original or simplified) the average PD during fixation (blue line) was smaller than average PD during saccade (red line), and this difference remains almost the same either when reading the original passage or the simplified passage. Both PD-Fixations and PD-Saccades had larger average values in the simplified text condition. These results show that separating PD-Fixation and PD-Saccade provides additional information that is useful when performing time-series analysis of pupil dilation (Table 1).

Fig. 5.
figure 5

The main effect of text simplification and fixation/saccade separation and their interaction effect on the dependent variable PD (Color figure online)

Full size image
Table 1. Descriptive statistics and t-test for PD during reading original and simplified passages
Full size table
Table 2. ANOVA results comparing the means of PD during different intervals and among original and simplified passages
Full size table
Table 3. Pairwise comparison between different time intervals for overall PD
Full size table
Table 4. ANOVA results comparing the means of PD between fixation and saccade during original and simplified passages
Full size table

Having separated PD-Fixation from PD-Saccade, next we investigated this data using the three reading time intervals. The two-way ANOVA test was used for PD-Fixation and PD-Saccade to compare the overall effects of two independent variables (1) text simplification, and (2) time interval separation.

Table 5 displays the overall results of the ANOVA tests. The results show that PD values are significantly different both during fixations and saccades when people read original vs. simplified text passages (F(1,518) = 54.23, p = 0.00 for PD-fixation and F(1, 542) = 35.885, p = 0.00 for PD-saccade). The results show that PD values during fixations are also significantly different over the three time intervals (F(1,518) = 65.17, p = 0.00 for PD-fixation). Additionally, the differences in PD-fixations between the original and simplified conditions are significantly different in the three time intervals (F (2,524) = 10.13 and p = 0.00). The same is true for PD values during saccades (F(1,542) = 17.05, p = 0.00). The results show significant interaction effect between text simplification conditions and time intervals (F (2,548) = 75.59, p = 0.00).

Table 5. ANOVA results comparing the means of PD-Fixation and PD-Saccade within different time intervals
Full size table

These differences are further shown in Table 6, which displays the pairwise comparison between PD saccade/fixation for original and simplified passages among different time intervals. In other words, PD-fixation is significantly different between original and simplified passages in the beginning (p = 0.00), and end (p = 0.00) of the reading duration but not in the middle of the reading duration (p = 0.53). Identically, PD-saccade is also significantly different between original and simplified passages in the beginning (p = 0.00) and the end (p = 0.00). However, the difference in PD-saccade between original and simplified passages is also significant in the middle of reading (p = 0.00). Figures 6 and 7 display graphical interpretations of these results. As we can see in these figures, average PD-Fixation is larger in the simplified group compared to the original group at the beginning and the end intervals. In the middle interval Average PD-Fixation values are the same in both groups. While we observe a similar trend for PD-Saccade in the beginning and end intervals, in the middle interval average PD-Saccade for the original passage is significantly larger than average PD-Saccade for the simplified passage.

Table 6. Pairwise comparison between different time intervals for PD-Fixation and PD-Saccade
Full size table
Fig. 6.
figure 6

The main effect of text simplification and time intervals and their interaction effect on the dependent variable PD during fixation

Full size image
Fig. 7.
figure 7

The main effect of text simplification and time intervals and their interaction effect on the dependent variable PD during saccade

Full size image

These findings are consistent with previous literature [11,12,13,14,15] that identify pupil dilation as a reliable measure of cognitive load in cognitive tasks. In addition; these results indicate that separating pupil dilation during fixation and saccades can provide more nuanced information that is not available when considering only the overall PD. Furthermore, these results open new research questions in the field of pupillometry related to HCI research, which will be discussed in the next section of this paper.

5 Discussion and Conclusion

Time series analysis of eye-tracking data is important because it provides a continuous measure of eye-movement data, which allows us to examine moment by moment analysis of eye-movement data. In this study, we conducted time-series analysis of pupil dilation, which is considered a reliable measure of cognitive effort [11,12,13,14,15].

We investigated whether reducing cognitive load of readers by simplifying text passages can affect their pupil dilation during reading and whether this effect remained steady over different time intervals of reading. The simplified text passage used in this study was developed using a set of plain language rules [3]. The original passage, which was an actual news passage about sports, was simplified from 18th grade reading level to 10th grade reading level through systematic application of the plain language rules described in [3].

The time series PD analysis in our study was based on eye movement data of 54 participants, 26 reading the simplified version of the passage and 28 reading the original passage. Gaze data was recorded using Tobii X-300 eye-tracker with sampling rate of 300 Hz. Eye-movement data were then obtained from eye-tracking software (Tobii studio V3.2.3). Cubic splines interpolation method [21] was used to interpolate the data points and to make arrays of equal sizes of pupil data among participants in different conditions. For each cell in the array averages among all participants were calculated and used to compare the trend of PD over time between simplified and original passages.

The results of t-test comparing overall PD values between the original and simplified groups showed that text simplification did not significantly affect pupil dilation over time. However, when dividing the data points into three equally size intervals, the results showed that PD values were significantly different between the original and simplified groups in the last part of reading. Because pupil dilation is generally associated with increased cognitive load [20, 23], the results displayed in Fig. 2 suggest that participants were experiencing more cognitive load at the beginning of the task (compared to the two other time intervals) when they were familiarizing themselves with the text. The results also show that cognitive load was similar for the two text conditions (original vs. simplified) at the beginning and middle time intervals but it was significantly lower for people in the simplified text condition at the end interval. These results provide evidence that examining PD in various time intervals can provide additional information for understanding cognitive load.

Next, we examined PD for fixations and saccades separately. Our analysis, showing that the impact of text simplification on PD was significant when separating PD-Fixation from PD-Saccade, supported the study that argues examining PD during fixations and saccades separately is useful in refining the explanatory power of pupillemotry. Our results are consistent with the argument that the observed differences are due to differences in the nature of fixation and saccadic eye-movements [5]. Our results show that pupil dilation was slightly larger during saccades compared to pupil dilation during fixations (Fig. 5). The results also show that PD regardless whether it was measured during fixations or saccades was larger in simplified version of the text. Note that PD during fixations refers to visual information processing. Because during saccades we cannot process visual information (our eyes move too fast to be able to take foveal snapshots), PD during saccades may indicate cognitive processing beyond what is typically associated with attention measured as foveal processing of visual information. Given this interpretation, the results in Fig. 5, suggests higher cognitive activity during saccades compared to fixations regardless whether participants were reading the original or the simplified versions of the text. It also suggests higher cognitive activity in the group that were reading simplified text. Given that the performance for the same set of data indicated that people provided significantly more accurate answers to questions about the text in the simplified group [4], higher cognitive activity in the simplified group in this case may indicate higher level of engagement with the task.

Next, we examined PD during fixations and saccades over the three reading intervals: beginning, middle, and end. The results, displayed in Fig. 6 and Fig. 7, reveals different effects. During fixations, PD increases consistently over the three time periods when people read the original version of the text. However, when people read the simplified version, PD in the middle of the reading is significantly smaller than the two other intervals. During the saccades, PD values in the middle interval are higher than the two other intervals for the original version of the text while they are lower than the two other intervals for the simplified version of the text. These results indicate the presence of different types of activities in the middle of reading as represented by PD during saccades and fixations. While future experiments are needed to fully explain these differences, these results provide evidence for the usefulness of examining PD during fixations and saccades separately during various intervals.

Overall our findings are consistent with previous literature that have employed pupil dilation as a reliable measure of cognitive load. Additionally, our results indicate that investigating pupil dilation in different time intervals is useful in providing a better understanding of cognitive load and that separating the analysis of pupil dilation during fixations and saccades can provide additional useful information about cognitive load. These results provide a rationale for new research questions in the field of pupillometry related to HCI research. For example, why PD-Fixation and PD-Saccade show similar behavior at the beginning and end of the reading when comparing reaction to original and simplified passages, but show different behavior in the middle of reading? Is this a consistent behavior even when we test different passages or with a different population of readers?

6 Limitations and Future Research

As in any other research our study has some limitations, which provides opportunity for directing future research efforts. For example, future studies, including some of our own planned experiments, are needed to test passages with different content other than sports to see whether similar results are obtained. Expanding population to include older users or those with limitations in reading proficiency can also help to improve our understanding of the relationship between cognitive load and pupillometery.

7 Contributions

In this exploratory study, we set out to examine the impact of cognitive load (manipulated via text simplification) on pupillometry. In particular, we examined (1) how pupil dilation was affected when people read simplified vs. original versions of a passage, (2) whether this impact was different if we separate pupil dilation during fixations and saccades, and (3) whether pupillary responses were different during various time intervals. Our results provide evidence that all these types of exploratory investigations can provide useful information for refining our understanding of the relationship between cognitive load and pupillometry.