Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

It is increasingly common for people to stretch their limits in order to have more time for work, for family and for leisure activities [7]. This extra time is usually obtained at the expense of rest and relaxation time. This, together with inadequate sleep patterns, is one of the leading causes of fatigue.

Fatigue and its effects may not be immediate or may not even be visible, but there are many consequences on health and safety at work, as well as on labor productivity. This can be seen often in students preparing for exams, office workers, industrial workers, health care professionals, drivers, pilots or military personnel. Fatigue may even put people working in safety-sensitive jobs at risk, and any mistake on their part can lead to loss of lives [6, 18].

Mental fatigue can be seen as a state that involves a number of effects on a set of cognitive, emotional and motivational skills and usually results in overall discomfort, as well as the emergence of performance limitations. Some of these limitations imply that a tired person is often less willing to engage in tasks of effort, or perform the task of a conditioned form, well below their normal capacity [8]. Thus, mental fatigue may be characterized by a perception of a lack of mental energy. Persons who are affected by mental fatigue may feel like they have less energy than usual and are unusually tired and lethargic. Excessive activity and stimulation of the brain can cause a person to feel mentally exhausted, and the feeling is similar to what the body feels when a person is physically fatigued [19].

The effects of fatigue may occur at any moment and they may persist from only a few hours to several consecutive days. Depending on its duration and intensity, fatigue may make the carrying out of daily tasks increasingly difficult or even impossible. In severe or prolonged cases, it can cause illnesses such as depression or chronic fatigue syndrome [11, 18]. Thus, the detection and monitoring of fatigue are nowadays very important towards not only organizational performance but also worker well-being and health.

This paper describes an non-intrusive approach for fatigue monitoring and detection in real-time. It is especially suited for workplaces in which computers are used as it relies on the analysis of the workers interaction patterns with the computer. We conduct a case study to analyze how mental fatigue, mood and interaction performance interact. The main aim is to develop working and leisure environments that are aware of the user state and can react accordingly so as to improve the user’s performance and well-being.

1.1 Related Work

There are many factors, both internal and external, that modulate the onset and presence of fatigue. These include sleep deprivation, naps, noise, heat, mood, motivation, time of day, and workload, and of course the individual profile [3].

The user’s profile provides valuable information with respect to the potential level of fatigue. It can be seen as a predicted base level of fatigue in the sense that it establishes a baseline, according to the lifestyle of the individual [2]. These aspects have been thoroughly studied, mostly by psychologists, and encompass:

  • Age—Defines the mental age of the individual. It is important to understand the expected cognitive abilities of the individual, which may have a tendency to degrade with old age.

  • Gender—The mental states are different between men and women.

  • Professional occupation—Many occupations are intrinsically more tiresome or exhausting than others.

  • Consumption of alcohol and drugs—The use of certain substances for short or prolonged periods may cause dependencies and other effects that lead to a state of mental fatigue.

Mental fatigue is also affected by a number of other external factors. They may or may not be directly related to the individual’s behavior. They include:

  • The mood of the individual may influence decisively his or her mental state, with a particular effect on his or her motivation to work. Although tired, the individual may overcome (even if only temporarily) the effects of fatigue with a positive mood and motivation.

  • Stress may be defined as the demands placed upon the individual’s mind or body by external stimuli, requiring the individual to acclimatize to the dynamic requirements of the environment. However, these processes of acclimatization require an additional effort from the brain which, when prolonged over long or intense periods, will result in mental fatigue.

  • Mental Workload as a result of the relationship between the amount of mental processing capacity and the amount required by the task.

  • Sleepiness is often mistaken for mental fatigue or generalized as such. A difference exists and must be pointed out. However, the mistake is understandable since sleepiness is a symptom that is strongly connected to mental fatigue: it is one of the methods our brain uses to tell us that it is running out of resources. Sleepiness often results in a general loss of the individual vitality.

These factors can be assessed using validated tools such as the USAFSAM fatigue scale [15] for mental fatigue states, or as the NASA TLX [5] in the case of mental workload.

However, these instruments, based on the individual’s subjective interpretation of the symptoms, do not fully take into account inter-individual differences. There are instruments that help to account for individual differences, such as the Profile of Mood States (POMS) [4] and the State-Trait Anxiety Inventory (STAI)[17]. However, their use in complex systems can prove to be complicated and confusing, due to the same problems that can be observed in subjective measures of fatigue detection [4, 10].

The POMS can also be used to estimate fatigue with the Vigor-Activity and Fatigue-Inertia factors. The POMS measures five aspects of affect or mood [9]. It consists of 65 adjectives describing feeling and mood to which the subject responds according to a five-point scale ranging from “Not at all” to “Extremely”. Results are reported as six mood factors, namely:

  • Tension-Anxiety: Heightened musculoskeletal tension including reports of somatic tension and observable psychomotor manifestation;

  • Depression-Defection: Depression accompanied by a sense of personal inadequacy;

  • Anger-Hostility: Anger and antipathy toward others;

  • Vigor-Activity: Vigorousness, ebullience, and high energy;

  • Fatigue-Inertia: Weariness, inertia and low energy level; and

  • Confusion-Bewilderment: Bewilderment, muddleheadedness; appears to be an organized-disorganized dimension of emotion.

Because of its length, the POMS only results practical for occasional uses such as establishing a baseline and estimating the effects of excessive sleep deprivation or restriction.

Existing approaches on fatigue and mood analysis rely mostly on the described self-report mechanisms, which result unpractical for use in realistic scenarios. In fact, aspects such as fatigue, stress or mood change during the day as daily events occur. Following workers in real-time would imply to use these instruments multiple times during the day, with a significant impact on the workforce’s routines. On the other hand, in the last years there have been emerging electronic performance monitoring tools. However, these rely on traditional productivity indicators that essentially quantify the work produced. This kind of approaches is often dreaded by the workforce, who feel spied. Moreover, the pressure that emerges from the need to produce in order to perform often has the opposite effect, i.e., decreased productivity [1, 16].

To address these issues we propose a non-invasive framework that can perform an analysis of performance that is not dependent on productivity indicators and should thus be more acceptable by the workforce. Moreover, it can be used continuously throughout the day without any impact on working routines. This framework is described in the following section.

2 Framework

In this paper we propose a non-invasive, non-intrusive, real-time approach to assess mental fatigue through the analysis of keyboard and mouse interaction patterns. The analysis can happen directly from the usage of an individual’s computer as within the context of so-called desk-jobs. We build on the fact that computers are nowadays used as major work tools in many workplaces, to devise a non-invasive method for mental fatigue monitoring based on the observation of the worker’s interaction with the computer, specifically the aspects related to the use of the mouse and the keyboard.

This system allows the detection of behaviors associated to fatigue in a non-invasive and non-intrusive way with the aim to develop leisure and work context-aware environments that may improve quality of life, mental health and individual performance, as well as productivity in organizations. To achieve this purpose we follow the guiding lines of Ambient Intelligence (AmI) [14], in which the technological aspects are hidden in the environment and the user is placed in the middle of the paradigm. There is also a focus on non-intrusiveness, with acquisition of information taking place without the need for explicit or conscious user interactions.

Another major objective of the system is to support the decision-making processes of team managers or group coordinators. In this perspective, each element of a group/organization is seen as part of a whole which contributes to the general level of fatigue and the distraction of the group. Thus, the estimated fatigue level of the group is the average result of the fatigue level of its elements.

The developed system is also composed of a graphical web interface, which is available online and can be accessed remotely. It has different features according to the type of user. Specifically, a worker can analyze his data in real-time so as to know his current performance. He can also analyze personal historic data so as to know his evolution in the recent past (e.g. last day, last week). A higher-level login is also available in which similar features are provided but including data from all the users. The aim is that the group manager can, at a glance, have an estimation of the state of the workforce. This allows him to act both at a group level as well as at an individual level (e.g. advise someone to make a break when his individual performance drops significantly). The use of one or both types of login depends thus on the organization’s policy, i.e., they may allow each worker to decide on working schedules and pauses at will (in which case both the worker and the manager have logins to the web interface) or they may decide that only the manager can take such decisions (in which case only the manager has access to the web interface) (Fig. 1).

Fig. 1
figure 1

Interface of the performance monitoring web service

Full size image

3 Case Study

In order to test and validate the proposed system and approach, a case study was prepared using detection system models which were previously trained [12, 13]. To reach that goal we used two groups of volunteers who used the tool while they were providing feedback of their state of fatigue through USAFSAM Fatigue scale, as well as their mood through the POMS questionnaire.

Feedback values recorded during the monitoring period were used to validate the system and used to analyze the variation of moods.

The participants in the case study, fourteen in total (10 men, 4 women) were students from the University of Minho in the field of physics. Their age ranged between 18 and 25.

3.1 Methodology

The methodology followed to implement the study was devised to be as minimally intrusive as the approach it aims to support. Participants were provided with an application for logging the previously mentioned events of the mouse and keyboard. This application, which maintained the confidentiality of the keys used, needed only to be installed in the participant’s computer and would run in the background, starting automatically with the Operating System. The only explicit interaction needed from the part of the user was the input of very basic information on the first run, including some personal identifying and profiling information.

As mentioned, two different groups of users were selected (7 participants in each group) who underwent the experience in different periods of the week. The first group was monitored on a Monday morning (starting at 9 am), while the second did so on a Thursday in the afternoon (starting at 14 pm). Each session had a duration of 3 h. During the session, in addition to the collection of interaction features, user feedback regarding their state was collected hourly.

3.2 Results

During the two sessions the mental fatigue monitoring system was used in real time. It not only allows recording the interaction patterns with the mouse and keyboard, but also to calculate the estimated individual level of mental fatigue for each user, and the estimated average level of each group. These values were compared with the feedback given by users.

Table 1 summarizes the results, showing that the first group evidences a less fatigued state than the second one. This can be confirmed through data collected during the session. It is also possible to check a RMSE (Root-mean-square Error) of 0.2 for group 1 and 0.4 for group 2. Taking into account the scale used, this is an acceptable error.

Table 1 Average level of fatigue of the different groups in USAFSAM fatigue scale of the estimated values and feedback values
Full size table

In addition to the subjective level of fatigue through the USAFSAM fatigue scale it was used the POMS factors in order to observe the influence of fatigue on the moods of the different groups, and therefore the emergence of fatigue and loss of vigor.

Through Table 2 and Fig. 2 it is visible that the average values are different between the two groups. In addition, the T-test was used to validate the differences between the two groups, in order to determine if the two sets of data are significantly different from each other, a fact that was confirmed.

We can also observe that fatigue is higher in the second group and, on the contrary, vigor is lower in the second group. This is in accordance to the fatigue levels of each group.

Table 2 Average vales of Profile of mood states for group 1 and group 2, as well as the resulting p-value from the T-test
Full size table
Fig. 2
figure 2

Profile of mood states from groups 1 and 2: group 2 has more negative mood states

Full size image

4 Conclusion

This paper described a prototype for monitoring mental fatigue in real-time of an individual or a group of individuals. The aim is to detect behaviors associated with mental fatigue within a group, so as to estimate the average level of mental fatigue in the group. Such information may be extremely useful in scenarios of critical tasks, which require that the individual performing the task is fully rested, as well as to assess the mental state of working groups as one.

The results of the experiment demonstrate that it is possible to evaluate the average degree of mental fatigue in a group using the prototype presented. The prototype used in the experiment uses a fatigue detection model trained in a previous study, with data of a different group of users but with a similar profile. This also shows that the changes induced by fatigue on different users can be generalized, i.e., we all react similarly.

Within the context of the CAMCoF project, which supports this work, the long-term goal is to develop environments that are autonomous and take measures concerning their self-management to minimize fatigue and increase the performance and well-being of a group of individuals.