1 Introduction

Being an important part of visual perception and human behavior, eye movements have long been a subject of research interest. The first approaches to measure a person’s gaze direction date back to the early 1900s [74]. Until recently, these technologies were severely limited by the cost of the equipment required, a lack of precision, and poor usability and were only used in very specific niches of research. Over the last few years, however, with rapid advances in sensor technology and data processing software, eye tracking solutions have become easy to use, lightweight, efficient, and affordable and found increasing adoption in many fields, including gaming, marketing, automotive technology, military, and healthcare [26].

While alternativesFootnote 1 exist, the most popular method today is video-based eye tracking, where mathematical models are used to calculate a person’s gaze direction from video recordings, for example based on the shape and position of pupil and iris, or based on light reflection patterns in the eyes [2]. This method can not only be used in head-mounted devices, such as smart glasses and virtual reality headsets, but also through built-in cameras in laptops, tablets, and smartphones without requiring any additional hardware [45, 56]. With further improvements in cost and performance, eye tracking may soon be included as a standard feature in various consumer electronics, moving us towards a “pervasive eye tracking world” [58].

The many beneficial uses and enormous potentials of the rising technology have to be acknowledged and should be embraced. However, a more ubiquitous use of eye tracking will also raise serious privacy concerns – not only because gaze data may be collected and shared in non-transparent ways, but also because such data can unexpectedly contain a wealth of sensitive information about a user.

Drawing from a broad range of scientific disciplines, including neuroscience, human-computer interaction, medical informatics, affective computing, experimental economics, psychology, and cognitive science, this paper provides a structured overview and classification of sensitive pieces of information that can be disclosed by analyzing a person’s eye activities. According to the reviewed literature, eye tracking data may reveal information about a user’s biometric identity (Sect. 2.1), mental activities (Sect. 2.2), personality traits (Sect. 2.3), ethnic background (Sect. 2.4), skills and abilities (Sect. 2.5), age and gender (Sect. 2.6), personal preferences (Sect. 2.7), emotional state (Sect. 2.8), degree of sleepiness and intoxication (Sect. 2.8), and physical and mental health condition (Sect. 2.9). In order to take rapidly evolving technology trends and newly emerging privacy threats into account, we will consider not only proven and established approaches but also inference methods that are subject to ongoing research. Limitations of the presented methods and their practical applicability will be reflected upon in Sect. 3, followed by a conclusion in Sect. 4.

2 Inference of Personal Information from Eye Tracking Data

With reference to published research, filed patents, and existing commercial products, this section presents and categorizes personal information that can be inferred from eye tracking data. As a basis for potential inferences, eye tracking devices can record a large variety of gaze parameters.

Some of the most commonly measured eye movements are fixations, saccades, and smooth pursuit eye movements [85]. During a fixation, the eyes are relatively stable and focused on a specific position, allowing for information to be acquired and processed. Saccades are rapid eye movements from one fixation point to another, lasting 30 to 80 ms [87]. Smooth pursuit movements are performed when eyes are closely following a moving visual target. In addition to the spatial dispersion, duration, amplitude, acceleration, velocity, and chronological sequence of such eye movements, many eye trackers capture various other eye activities, including eye opening and closure (e.g., average distance between the eyelids, blink duration, blink frequency), ocular microtremors, pupil size, and pupil reactivity [19, 58]. Furthermore, most eye trackers videotape parts of the user’s face and may thereby capture additional information, such as the number and depth of wrinkles, and a user’s eye shape and iris texture [40]. Therefore, these parameters were also considered in our investigation into the richness and sensitivity of eye tracking data. Fig. 1 provides an introductory overview of common eye tracking measures and the categories of inferences discussed in this paper.

Fig. 1.
figure 1

Overview of sensitive inferences that can be drawn from eye tracking data.

Full size image

2.1 Biometric Identification

Due to differences in physical oculomotor structure and brain functioning, certain gaze characteristics are unique for every individual, similar to fingerprints, and can thus be exploited for biometric identification [9, 74, 76]. Among other methods, people can be told apart based on distinct patterns of pupil reactivity and gaze velocity [9], or by comparing their eye movement trajectories when they focus on a moving target – even if the eye activity is only recorded through an ordinary smartphone camera [55].

Aside from such gaze-based measures, the complex textures and color patterns in a person’s iris are also suitable for biometric identification. This approach, called iris recognition, is being used in a variety of real-world security and surveillance applications and has been recognized as “one of the most powerful techniques for biometric identification ever developed” [64]. Even though their iris scanning capability is usually not advertised, it should be understood that commodity eye trackers often record and process high-resolution images of the user’s iris, which can not only be used to uniquely identify the user but also to deceive iris-based authentication mechanisms and thereby steal the user’s identity [40].

In cases where a unique identification of an individual is not possible (e.g., because the person is not registered in the recognition system database), other attributes inferred from eye tracking data, such as age and gender (see Sect. 2.6), health condition (see Sect. 2.9), or ethnicity (see Sect. 2.4), can still help to classify the target person into a specific demographic group and thereby approximate the identity [74].

2.2 Monitoring of Mental Workload and Cognitive Processes

Certain patterns in eye movement, pupil dilation, and eye blinking have been recognized as reliable indicators of mental workload in people of any age [19, 63], sometimes offering higher accuracy than conventional methods like Electroencephalography [8]. Through eye tracking, it is also possible to distinguish a user’s moments of awareness from moments of distraction and mind wandering [31, 84].

Apart from detecting a user’s mental presence and measuring the mere intensity of cognitive processing, eye tracking can also provide insights into specific conscious and unconscious thought processes in a large variety of contexts. Among other mental tasks and activities, ocular measures have been used to study memory retrieval [19, 31], problem solving [31, 75], learning processes [44, 69], the formation of expectations [19, 27], internal reasoning [19], and mental computations [19, 31].

Eye tracking data can not only – to a certain extent – reveal what we remember, imagine, expect, and think about, but also our specific decision-making strategies [19, 28] and cognitive styles, i.e., individual differences in the way we acquire, process, and interpret information [72]. For example, people can be classified as field-dependent vs. field-independent (people of the latter type pay more attention to detail and exhibit a more analytical approach to processing visual information) [72], or as verbalizers vs. visualizers (people of the latter type can process visual information, such as images and diagrams, better than textual information) [44]. The gaze-based inference of such cognitive styles is feasible and can achieve high accuracies, as has been confirmed in a recent study by Raptis et al. [72].

Researchers from the field of cognitive science and experimental psychology have suggested that eye tracking data will not only be used for the real-time analysis but also for the prediction of human decisions and behavior [28].

2.3 Inference of Personality Traits

Experimental research has shown that it is possible to automatically infer personality traits from eye tracking data [34, 35, 42]. For example, gaze patterns captured during everyday tasks can be used to evaluate users along the so-called Big Five traits, namely openness to experience, conscientiousness, extroversion, agreeableness, and neuroticism [34, 42]. The gaze-based assessment of personality traits is possible not only in binary form (high vs. low) but also in the form of ranges. In [35], for instance, eye movement analysis was used for the automatic recognition of different levels of curiosity.

Besides the Big Five traits and curiosity, gaze metrics were found to be associated with various other personality traits, including emotional intelligence [54], indecisiveness [36], the tendency to ruminate [21], trait anxiety [42], sexual compulsivity [87], boredom susceptibility [70], and general aggressiveness [6]. Eye tracking has even been used to investigate people’s attachment styles in interpersonal relationships (e.g., secure, withdrawn, fearful, enmeshed) [81].

Based on data from 428 study participants, Larsson et al. [53] also suggest that some personality traits, including tendermindedness, warmth, trust, and impulsiveness, are genetically linked to certain iris characteristics, offering – besides gaze behavior – another potential ocular biomarker to analyze people’s personalities.

2.4 Inference of Cultural Affiliation and Ethnicity

It is widely agreed that culture fundamentally shapes human cognitive processing and behavior [11]. Studies have shown that intercultural differences are reflected in certain gaze characteristics [12, 24, 41, 61]. For example, people of different cultural background were found to exhibit discriminative eye-movement patterns when seeking information on search engine results pages [61], when exploring complex visual scenes [12, 24], and when viewing videos of actors performing cultural activities [41]. Some cultural biases in visual processing are so pronounced that they can still be measured when external stimuli draw attention in an opposite manner to the respective bias [24].

Additionally, eye movements can reveal a person’s knowledge of certain cultural practices. For instance, in an eye tracking study by Green et al. [27], Chinese infants exclusively predicted the goal of eating actions performed by an actor with chopsticks, whereas European infants only anticipated that food would be brought to the mouth when eating actions were performed with Western cutlery, as indicated by their predictive gaze shifts towards the actor’s mouth.

Some studies have also investigated how people of different “race”Footnote 2 differ in their viewing behavior [25, 33, 88]. Apart from the fact that video-based eye trackers can directly record the eye color, eye shape, and skin color of a user, it has been observed in eye tracking studies that test subjects view “other-race faces” differently than faces of their “own race” in terms of the facial features scanned (e.g., initial focus and greater proportion of fixation time on the eyes vs. nose and mouth) [25, 88]. Furthermore, researchers have observed characteristic changes in pupil size, which are attributed to elevated cognitive effort during face recognition, when people look at “other-race faces” [88]. Such differences have been reported, for example, between “Black and White observers” [33] and between “Western Caucasian and East Asian observers” [25] and could potentially allow inferences about the genetic and ethnic background of eye tracking users.

Eye tracking data may also allow inferences about a user’s native language. For instance, considerable differences in eye movement patterns during reading can be observed between native and non-native speakers of English [39]. Eye tracking can even be used to determine which specific words are difficult to understand for a person [51]. Among other things, such information could help in estimating a subject’s nationality or geographical origin.

2.5 Skill Assessment

Eye tracking has been used extensively in the study of human expertise and to discriminate between performance levels in a variety of areas [30, 31, 69, 75]. For example, gaze behavior can be analyzed to assess reading and listening comprehension skills [10, 92]. During a corresponding task or scenario, eye tracking can also be used to distinguish between experts and novices in chess [75], several sports [46], chemistry [69], mathematics [31], school teaching [14], and various medical skills, including surgery, nursing, anesthesia, and radiology [30].

Among other gaze characteristics, expertise is often associated with systematic eye movement patterns reflecting a specific task strategy [31], with the targeted inspection of important regions and task-relevant information [30, 75], and with more consistent gaze patterns over consecutive trials of a task [46].

In some fields, eye tracking has not only been used as a tool to discriminate between people of different skill levels, but also to predict people’s task performance and learning curves [52, 69] and to examine specific learning disabilities, such as mathematical difficulties and dyslexia [31, 85].

2.6 Age and Gender Recognition

Just like physical shape, skin texture, and cognitive abilities, human eyes and visual behavior are fundamentally affected by the aging process [20, 36]. For example, eye tracking studies found age-related differences in people’s visual explorativeness, pupil reactions to certain visual stimuli, and error rates in eye movement tasks [36, 42].

Furthermore, detailed frontal face images, which are typically required for video-based eye tracking, have already been used for automated age estimation, for instance based on wrinkles in the eye area [15]. Dynamic facial expressions, such as smiles, may also be analyzed to infer the age of test subjects [17]. Other parameters utilized for computerized age-group recognition include iris size and iris texture [20].

As with age, a person’s gender can be reflected in certain eye tracking measures. For instance, studies found systematic gender differences in people’s fixation distribution while viewing natural images (e.g., stills from romance films or wildlife documentaries) [68], during online shopping [38], when playing video games [42], and when viewing sexual stimuli [87]. Researchers have already used such differences in visual behavior to automatically classify the sex of test subjects [68].

2.7 Inference of Preferences and Aversions

Eye tracking is widely employed to investigate people’s interests, likes, and dislikes. Spontaneous attention to specific objects in a visual scene (e.g., in terms of frequency, duration, and sequence of gaze fixations) is regarded as a natural indicator of interest [19, 74, 87]. For data presentation and analysis, gaze fixations are commonly aggregated into heat maps to quickly identify potential regions and objects of interest [74].

Besides the focus of visual attention, other eye parameters, such as pupil dilation and blink properties, can also be used to analyze a person’s degree of interest and to distinguish between positive, neutral, and negative responses to visual stimuli [55]. Emotion detection from gaze data, which can assist in analyzing a user’s interests and preferences [55, 83], will be discussed in Sect. 2.8.

Among other things, eye tracking has been used to examine preferences for certain types of gambling [65], mobile apps [56], activities of daily living [86], types of food [32], colors, geometric shapes, and product designs [3], pieces of clothing, animals, video game characters, and items of furniture [83]. Beyond mere interest, existing research even suggests that people’s patterns of visual attention reflect their consumption and purchasing behavior [91].

Eye tracking has also been used extensively in the study of love and sexual desire. For example, researchers have analyzed pupillary responses and the allocation of visual attention to measure levels of sexual arousal and to investigate mating preferences towards specific facial characteristics, age groups, body shapes, body parts, and signs of social dominance [3, 87].

Apart from positive interests, visual attentional biases captured by eye trackers can also reflect a person’s phobias and aversions (e.g., fear of spiders) [3, 37]. Some interests and preferences can already be inferred from eye tracking data with high accuracy [56, 73, 87] and several patents have been filed in this field [3, 83].

2.8 Detection of Short- and Medium-Term User States

Moods and Emotions.

Eye tracking is increasingly used in the interdisciplinary field of affective computing, where systems are developed to automatically recognize human emotions based on physiological signals and behavioral cues [73, 83]. It has been shown that various ocular measures, including pupil size, blink properties, saccadic eye movements, and specific biases of visual attention, can contain information about a person’s emotional state [4, 23, 55, 59].

Gaze data can reflect emotional arousal and the valence of emotions (positive, negative, neutral) [19, 55] as well as more specific affective states, such as happiness and enthusiasm [83], acute stress and worry [59], humorous moods and disgust [73], curiosity [4], distress, nervousness, and hostility [23], fear, anger, sadness, and surprise [55].

Eye tracking can not only be used to detect emotions with high accuracy [73] but also to estimate the intensity of emotions [55, 83]. Based on gaze parameters, existing methods can even distinguish whether a user’s emotional response to a given stimulus is rational or purely instinctive [55].

Fatigue and Sleepiness.

For over two decades, there have been approaches to automatically derive a person’s level of sleepiness from certain ocular measures, such as blink rate, blink duration, average distance between the eyelids, fixation durations, and velocity of eye movements [57]. Recent studies have confirmed the suitability of eye tracking measures as indicators for sleepiness and fatigue [63, 89]. Corresponding methods have already been patented and achieve high accuracies – not only while the user is working on specific cognitive tasks, but also during everyday natural-viewing situations [57, 89].

Intoxication.

The consumption of alcohol and other recreational drugs can have measurable effects on various eye and gaze properties, such as decreased accuracy and speed of saccades, changes in pupil size and reactivity, and an impaired ability to fixate on moving objects [29, 67, 85].

Apart from alcohol, significant abnormalities in oculomotor functioning were found in people under the influence of nicotine, 3,4-methylenedioxymethamphetamine (“MDMA”), and tetrahydrocannabinol (“THC”) [29, 70].

Researchers have demonstrated the ability to differentiate between drug-impaired and sober subjects with high accuracy based on eye tracking data [29]. The magnitude of some ocular effects is closely associated with the amount of drugs consumed [85] and certain effects can even be detected at non-intoxicating doses [77]. In addition to pupillary changes and eye movement impairments, an attentional bias towards drug-related visual stimuli has been observed among intoxicated test subjects [67].

Not only a state of intoxication, but also an acute state of drug deprivation and craving can have a distinct effect on certain eye tracking parameters [29, 70].

2.9 Health Assessment

Physical Health.

Many diseases and medical conditions directly affect the eyes, or parts of the brain that are responsible for oculomotor function, and thereby cause gaze impairments [3, 19, 30]. Characteristic eye movement patterns were found, for example, in people suffering from concussion [43], fetal alcohol syndrome [3], irregular growth [3], chronic pain [22], neurocognitive impairment due to preterm birth [82], multiple sclerosis [3], Alzheimer’s disease [30, 43], Tourette syndrome [19], Parkinson’s disease [30], and various vision disorders (e.g., myopia, farsightedness, and blind spots) [3, 43].

As filed patents and published experimental studies show, eye movement analysis can be used to diagnose, monitor, prognose, and sometimes even predict various health disorders [30, 43] which can be subsumed under the umbrella term ETDCC (“Eye Tracking-Relevant Diseases, Conditions, and Characteristics”) [3].

Research has further demonstrated that certain patterns in gaze orientation and pupil reactivity to food-related stimuli (e.g., high vs. low calorie food images) can be indicative of overweight and obesity [32].

Mental Health.

Abnormal eye movements can be used as behavioral biomarkers for the diagnosis of various mental health problems [1, 5, 29]. Oculomotor dysfunctions and gaze peculiarities are found, for example, in sufferers of anxiety disorder [29], depression [1], bipolar disorder [30], borderline personality disorder [6], schizophrenia [5], obsessive–compulsive disorder [13], binge-eating disorder [79], ADHD [7], mild cognitive impairment [30], autism [43], and posttraumatic stress disorder [66].

Some common symptoms of mental disorders are irregularities in blink rate and blink duration [19], abnormal stability and dispersal of gaze fixations during free viewing [5], unusual biases of visual attention [66], impaired smooth pursuit eye-movement performance [85], eye contact avoidance, and abnormal distance between the eyelids [1].

Certain mental illnesses, including depression and schizophrenia, can already be detected automatically via eye tracking [1, 5, 30] and corresponding methods have been filed as patents [43]. Besides the possibility of binary classification (suffering vs. not suffering), some ocular measures are associated with the severity of mental disorders [19]. Not only acute disorders can be reflected in gaze data, but also past mental health issues and even the personal risk of future outbreaks [71, 78]. For example, researchers have observed characteristic gaze patterns in previously depressed individuals [78] and found biases in visual attention that were predictive of future depression scores at a delay of more than two years [71].

Substance Use Disorders.

Apart from acute states of intoxication (which we have discussed in Sect. 2.8), eye tracking data may contain information about a user’s longer-term drug consumption habits and addictions. Numerous eye tracking studies have reported a strong attentional bias towards drug-related visual cues in addicts of cocaine [16], alcohol [67], cannabis [90], and tobacco [18, 70].

Among other possible methods, such attentional biases can be detected by measuring how quickly, how often, and for how long a person’s eyes fixate on corresponding stimuli in comparison to neutral stimuli, or by testing the person’s ability to look away from drug-related stimuli on command [16, 18]. Significant biases have not only been observed in long-term addicts but also in habitual drug users without clinical symptoms of dependency [18, 67]. The strength of attentional biases towards drug-related visual cues was found to be correlated with scores on drug use scales, such as the Obsessive Compulsive Cocaine Scale [16] and with self-reported lifetime drug consumption [62]. Research has also shown that certain biases in visual attention can be predictive of craving and even relapse in drug addiction [16].

3 Discussion and Implications

As shown in the previous section, various kinds of sensitive inferences can be drawn from eye tracking data. Among other categories of personal data, recorded visual behavior can implicitly contain information about a person’s biometric identity, personality traits, ethnic background, age, gender, emotions, fears, preferences, skills and abilities, drug habits, levels of sleepiness and intoxication, and physical and mental health condition. To some extent, even distinct stages of cognitive information processing are discernable from gaze data. Thus, devices with eye tracking capability have the potential to implicitly capture much more information than a user wishes and expects to reveal. Some of the categories of personal information listed above constitute special category data, for which particular protection is prescribed by the EU’s General Data Protection Regulation (Art. 9 GDPR).

Of course, drawing reliable inferences from eye tracking data is not a trivial task. Many situational factors can influence eye properties and gaze behavior in complex ways, making it difficult to measure the effect of a particular action, internal process, or personal characteristic of the user in isolation [55]. Seemingly identical ocular reactions can result from completely different causes. For example, an intensive gaze fixation on another person’s face may indicate liking, aversion, confusion, recognition, and much more. Similarly, a sudden change in pupil size can be indicative of many different feelings or internal states, including physical pain, sexual arousal, interest, happiness, anger, or simply be a reaction to ambient events and conditions, such as noise or varying lighting [19, 55].

In spite of existing challenges and limitations, the reviewed literature demonstrates that there is considerable potential for inferences in many areas and that numerous research projects, patented systems, and even commercial products have already taken advantage of the richness of eye tracking data to draw inferences about individuals with high accuracy.

It should be acknowledged that many of the cited inference methods were only tested under controlled laboratory conditions and lack evaluation in real-world scenarios [4, 18, 27, 52, 65, 67, 69, 86, 88]. On the other hand, it may reasonably be assumed that some of the companies with access to eye tracking data from consumer devices (e.g., device manufacturers, ecosystem providers) possess larger sets of training data, more technical expertise, and more financial resources than the researchers cited in this paper. Facebook, for example, a pioneer in virtual reality and eye tracking technology, is also one of the wealthiest and most profitable companies in the world with a multi-billion dollar budget for research and development and a user base of over 2.3 billion people [93]. It seems probable that the threat of unintended information disclosure from gaze data will continue to grow with further improvements of eye tracking technology in terms of cost, size, and accuracy, further advances in analytical approaches, and the increasing use of eye tracking in various aspects of daily life.

In assessing the privacy implications of eye tracking, it is important to understand that, while consciously directed eye movements are possible, many aspects of ocular behavior are not under volitional control – especially not at the micro level [19, 55]. For instance, stimulus-driven glances, pupil dilation, ocular tremor, and spontaneous blinks mostly occur without conscious effort, similar to digestion and breathing. And even for those eye activities where volitional control is possible, maintaining it can quickly become physically and cognitively tiring [58] – and may also produce certain visible patterns by which such efforts can be detected. Hence, it can be very difficult or even impossible for eye tracking users to consciously prevent the leakage of personal information.

Though this paper focuses on privacy risks, we do not dispute the wide-ranging benefits of eye tracking. Quite the opposite: we believe that it is precisely the richness of gaze data and the possibility to draw insightful inferences from it that make the rising technology so valuable and useful. But to exploit this potential in a sustainable and socially acceptable manner, adequate privacy protection measures are needed.

Technical safeguards have been proposed to prevent the unintended disclosure of personal information in data mining, including specialized solutions for eye tracking data [58, 80]. These comprise the fuzzing of gaze data (i.e., inserting random noise into the signal before passing it down the application chain) and the utilization of derived parameters (e.g., aggregated values instead of detailed eye fixation sequences) [58]. Experiments have already shown that approaches based on differential privacy can prevent certain inferences, such as user re-identification and gender recognition, while maintaining high performance in gaze-based applications [80]. In addition to approaches at the technical level, it should also be examined whether existing laws provide for sufficient transparency in the processing of gaze data and for proper protection against inference-based privacy breaches. The promises and limitations of existing technical and legal remedies are beyond the scope of this paper but deserve careful scrutiny and will be considered for future work.

Even though eye tracking is a demonstrative example, the threat of undesired inferences is of course much broader, encompassing countless other sensors and data sources in modern life [47]. In other recent work, we have examined sensitive inferences that can be drawn from voice recordings [49] and accelerometer data [48, 50], for instance. In our view, the vast possibilities of continuously advancing inference methods are clearly beyond the understanding of the ordinary consumer. Therefore, we consider it to be primarily the responsibility of technical experts, technology companies, and governmental agencies to inform consumers about potential consequences and protect them against such covert invasions of privacy. Also, since it is unlikely that companies will voluntarily refrain from using or selling personal information that can be extracted from already collected data, there should be strong regulatory incentives and controls.

4 Conclusion

While the widespread adoption of eye tracking holds the potential to improve our lives in many ways, the rising technology also poses a substantial threat to privacy. The overview provided in this paper illustrates that, through the lens of advanced data analytics, eye tracking data can contain a rich array of sensitive information, including cues to a user’s biometric identity, gender, age, ethnicity, personality traits, drug consumption habits, moods and emotions, skills, preferences, cognitive processes, and physical and mental health condition. Since inference methods are often based on hidden patterns and correlations that are incomprehensible to ordinary consumers, it can be impossible for them to understand and control what information is revealed.

Although there is extensive literature on the analysis of eye tracking data, we believe that many possible inferences have not yet been investigated. Keeping track of the evolving possibilities of data mining methods in this field is therefore an important avenue for future research. This paper represents a crucial first step towards understanding the sensitivity of eye tracking data from a holistic perspective. The findings compiled herein are significant enough to warrant a warning to users whose privacy could be affected, as well as a call for action to the public and private actors entrusted with protecting user privacy in consumer electronics. Considering the rapid proliferation of eye tracking technology, existing technical and legal safeguards urgently need to be assessed regarding their ability to avert undesired inferences from gaze data, or to at least prevent the misuse of sensitive inferred information.