Keywords

1 Introduction

The increase of the population average age determines a greater incidence of neurodegenerative diseases that are specific of aging [1, 2]. Such diseases compromise the ability of attention, concentration, memory, reasoning, calculation, logic and orientation, thus producing effects on the patients and his/her family [3]. Dementia is not a specific disease but a descriptive term for a set of symptoms that can be caused by various disorders that affect the brain. It is commonly preceded by a pre-dementia stage, named Mild Cognitive Impairment (MCI), where elderly people have a cognitive decline greater than expected for their age [4]. Such a cognitive decline might not interfere notably with Activities of Daily Living (ADL) [5], but see [6].

Until a few decades ago, manifest dementia was considered a severe condition, however with the progress of research on the Central Nervous System and in the neuropsychological field, there was a greater knowledge of the characteristics of this pathology, a better differentiation of its forms and, as a consequence, a therapeutic approach. Though, clinical trials did not lead to important breakthroughs until now, thus rising a great interest for non-pharmacological treatments [7]. Among them, the use of the so-called exergames (video games for exercise), in general of Virtual Reality (VR) systems, i.e. enriched environments through which to assess and to stimulate cognitive functions [8], has gained popularity, as assistive technology [9].

In general, it is essential to use screening tools that can provide an early diagnosis. Currently, these tools include classical paper-pencil tasks, which try to evaluate the main cognitive functions that are compromised by the disease, through specific questions aimed at the patient. However, these tests have several limitations: they have a low specificity and sensitivity, especially in the early stages of the disease, and their scores are influenced by multiple factors, such as visual and auditory disturbances, patient education and psychological factors. For this reason, in the last few years researchers have been trying to develop further tools for the evaluation of cognitive functions that overcome these limits and are therefore able to actually measure what they are supposed to measure. This can be achieved by evaluating the symptoms, the impairments, the typical difficulties of the early stages of diseases through VR environments. Virtual Reality has several advantages with respect to traditional paper-pencil tests: ecological validity, i.e. similarity between the assessment environment and the real world; immediate performance feedback; personalization of the environment and task, thus producing engagement in participants [10,11,12].

1.1 Aim of This Paper

As we will describe in the next section, there are many approaches in the literature aiming to overcome the limits of the paper-pencil tests, usually considered in hospital and retirement homes to assess the cognitive status of elderly people.

Nevertheless, many of these approaches are still in the form of proofs-of-concept, and there is not an established use in the hospital practice.

This paper, starting from previous considerations about human-computer interaction (HCI) in immersive VR environments (see for example [13, 14]), has the following aims.

  • To propose two approaches that could be used to assess the following intellectual functions usually impaired by dementia: ability to concentrate and pay attention by using a non-immersive VR platform implementing a virtual supermarket; reasoning and visual perception difficulty in estimating distances and orientations by considering an immersive VR environment, where people play with a shape-sorter game.

  • To evaluate the proposed approaches by comparing the obtained results with the scores of two standard clinical paper-pencil tests: the Pfeiffer an the GPCog.

  • To put in evidence the limits and the potentialities of these two approaches, and to devise further steps in order to consider both the needs of overcoming the limitations of the standard tests, and the need of proposing non-invasive and natural HCI to patients.

  • To consider other data that such kind of systems could provide to therapists and doctors, e.g. the possibility of measuring and tracking the 3D pose of patients, thus opening the way to the further clinical evaluations.

1.2 State of the Art

In the last twenty years, neuropsychology has been used as a tool for the diagnosis and management of cognitive decay. In fact, early diagnosis is one of the main objectives of the neuropsychological investigation, since it allows significant therapeutic and assistance interventions both by the medical and psychological staff and by the family [15]. The main objective of a neuropsychological evaluation is to provide data and evidence that contribute to the diagnostic framework of a specific patient [16].

One of the most common and significant symptoms of dementia is the loss of memory, however, taken individually does not necessarily imply the presence of the disease but must be accompanied by the impairment of at least one of the following intellectual functions so that we can actually talk about dementia [17]: (i) Communication and language: difficulty in following a conversation or finding the right words to express a concept; (ii) Ability to concentrate and pay attention: difficulty in making decisions or solving problems; (iii) Reasoning and judgment: difficulty in carrying out complex actions (for example cooking a meal); (iv) Visual perception: difficulty in assessing distances and orientations.

Paper-Pencil Tests for Assessing the Cognitive Decay. Here, we consider the classical paper-pencil tests that are relevant to our research, only. The Mini-Mental State Examination (MMSE) is composed of eleven items [18]. It presents some limitations: a low level of sensitivity for mild degrees of impairment and a dependence on cultural background [19]. The Short Portable Mental Status Questionnaire (SPMSQ) or Pfeiffer test [20] was developed with the aim of providing the clinician with a simple tool to assess the presence and degree of cognitive function impairment. Through a short interview, the clinician is able to establish whether the organic mental syndrome and the impairment of cognitive functions is the patient’s main problem or whether it represents a complication of a medical or psychiatric disorder. The test consists of a list of ten questions that investigate some aspects of cognitive abilities: seven items are focused on orientation, two items evaluate long-term memory, and one item evaluates the concentration capacity. The patient’s cognitive status is evaluated on the total number of errors: from 0–2 errors (intact intellectual functions) to 8–10 errors (severe deterioration of intellectual functions). The test is affected by the scholar degree of patients [21]. The General Practitioner Cognitive Assessment of Cognition (GPCog) is a cognitive impairment screening test consisting of two different sections: a cognitive section and an informative one [22]. The first section is the actual cognitive test to be administered to patients, the second consists of an informative interview with a family member and is useful to improve the efficiency of the test. The cognitive test includes nine items that investigate temporal orientation, visuospatial functionality and memory. The evaluation range is as follows: the maximum score obtainable is 9 (no cognitive impairment); a score of less than 5 highlights the presence of cognitive impairment; whereas a score between 5 and 8 is considered borderline.

New Approaches to Assess Cognitive Decay. In the last few years, researchers have tried to devise alternative methods to traditional paper-pencil tests to determine the cognitive state of older people. The main objective is to design tests that allow making an early diagnosis of dementia in order to study the symptoms and their course over time. Since people with MCI have difficulty performing normal ADL [6, 23, 24], the new approaches are focused on the simulation of this type of activity to evaluate the behavior of patients in different daily situations. This is possible thanks to the use of both non-immersive and immersive VR platforms, which allows reproducing complex situations of daily life in which the psychopathological reactions and cognitive functions of patients can be evaluated more immediately than the classic paper-pencil tests. In this way, screening instruments with a greater ecological validity can be obtained [25]. Moreover, through virtual reality, it is possible to reproduce environmental and social situations that can stimulate the subject in a similar way to the corresponding context [26], allowing to modulate the intensity and duration of the experience according to the needs of patients.

VR environments and, in general, serious games can provide useful tools for neuropsychological assessing and training of elderly people, by involving physical exercise, social engagement, and positive emotions [12, 27, 28].

Nevertheless, few works aim to understand how people interact within such kinds of immersive or non-immersive VR environments. A non natural interaction (see [29]) could negatively affect the validity of the developed approaches.

Non-immersive Virtual Reality Systems. By considering non-immersive VR systems, in [30] the authors proposed a music video game with the aim of increasing positive emotions in patients. A tablet-based cooking game was presented in [31], where the patient by following a recipe plans the steps to do the recipe. With the aim to combine physical stimulation and cognitive one an exergame for elderly people has been proposed in [32]. In [33] the authors proposed a virtual loft and tasks involving subjects in simulated daily activities. Each task has been designed to evaluate different cognitive functions: executive functions (reasoning and planning), attention (selective and divisive), memory (short term, long term and perspective) and visuospatial orientation. In this VR system, the subject interacts with the 3D environment through the touch screen of a computer. The serious game ECO-VR consists of a virtual apartment and interactions with the virtual environment and objects take place through the use of a joystick and a standard screen [34]. Each task requires the simultaneous involvement of two or more cognitive functions. The game was tested on healthy elderly, the scores obtained (total score, completion time, strategies) were compared with the score acquired with the MMSE and the comparison showed a moderate correlation. RehabCity simulates a district where four buildings can be visited (a supermarket, a post office, a bank and a pharmacy) [35]. By accessing each individual building, it is possible to perform tasks of different complexity. Each task requires the involvement of different cognitive domains, such as visuospatial orientation, attention and executive functions. RehabCity is a multiplatform environment in which interaction takes place via a computer screen and a joystick. It is used for the rehabilitation of people with cognitive deficits and also used as a cognitive assessment tool. The game presented in [36] simulates a supermarket, where the main objective is to buy the products in the shopping list and pay the amount due to the cash desk. The game has been implemented on a tablet so the subject interacts with the virtual environment by simply touching the screen. The execution of the tasks requires the simultaneous involvement of many cognitive processes, therefore the authors considered the game too challenging for patients with dementia and it was tested only on healthy elderly people and patients with MCI.

Immersive Virtual Reality Systems. By moving towards immersive VR, we can consider the use of stereo displays to add realism to the observed scene, as in [37], though for an ecological visual interactions the positions of user’s eyes have to be considered [38]. In the immersive VR game presented in [39], the subject wears a headset that allows him to interface with the virtual world and interacts with it thanks to markers placed on the headset itself, and on a special glove. In this way, the system traces the position of the head and of the glove worn by the subject, so the latter can interact with the objects present in the scene through his/her hand. Furthermore, he/she can move in all directions within the supermarket through the use of a joystick. In this way, the patient is immersed in a virtual simulation that reflects different daily activities concerning shopping in a supermarket. Recently, a virtual shop was proposed to measure episodic memory for older adults [40]: the results show a negligible cybersickness symptoms and high levels of presence. Moreover, memory performances in the VR task are positively correlated with performance on a traditional memory task. The system proposed in [41] uses a treadmill to allow people to move in a scenario that simulates a fire evacuation drill consisting of six different scenarios of increasing difficulty, though the VR environments was displayed on a curved rear projection screen. The authors examined the relationships among the performances of three groups of participants in the VR setup and traditional neuropsychological tests employed to assess executive functions. A VR exergame for people with MCI is proposed in [42]: the results showed a new way of engaging older adults in physical activities that can tailored to their abilities. As a future work, the authors proposed an evaluation of the exergame against comparable human-guided movements.

2 Materials and Methods

2.1 Platforms and Virtual Environments

Both non-immersive and immersive virtual reality HCI platforms are considered in this experimental evaluation. Non-immersive platforms consist in PC and tablet applications, a comparison between two interaction modalities (i.e., interacting with a mouse and by using the touch screen) is considered in order to understand which is more suitable [13], especially targeting older people. Immersive platforms are based on head-mounted displays (HMD), where people can interact with both touchful or touchless devices [14].

The VR environments have been designed and implemented by using Unity3D platform, this allows a fast prototyping and an easy integration with the several interaction techniques we have considered. Also the deploy on different hardware platforms can be easily achieved.

As previously discussed, we implemented two scenarios for the assessment of the following cognitive capabilities: (i) attention and memory, with a “Supermarket Virtual Environment”; (ii) visual perception and reasoning, with a “Shape-sorter Immersive Virtual Environment”.

Supermarket Virtual Environment. By taking inspiration from the existing works in the literature, and by considering an everyday life task, we implemented a virtual supermarket for PC and tablet devices. The environment is composed of three shelves, containing several common life items. The user receives a random generated item list and he/she should select and put them in a virtual basket, then go to the cash desk in order to pay the given amount of money. The procedure is identical to the corresponding real life task.

The layout of the scene, the dimension of the items and of the text, and all what concerns the user interface have been discussed with geriatrics medical staff and tested by a set of over-65 people before starting the experiments. It is also worth noting that 3D models of real objects (e.g. real objects brands) have been downloaded and modified with SketchUp, in order to increase the realism of the task.

Figure 1 shows a sketch of the “Supermarket” virtual environment, and of the cognitive assessment procedure by using the developed tool.

Fig. 1.
figure 1

A sketch of the “Supermarket” experimental setup, based on a standard tablet.

Full size image

Shape-sorter Immersive Virtual Environment. To test the depth and shape perception, and the capability of planning simple re-ordering tasks, we devise an immersive virtual reality environment that consists of a large scale replica of the shape-sorter children game (see Fig. 2 for a sketch of the “Shape-sorter” immersive virtual environment and of the assessment procedure). The environment is composed of several solid shapes and of a structure with the corresponding holes. The user can interact with them, and he/she should place them in the correct location.

Fig. 2.
figure 2

A sketch of the “Shape-Sorter” experimental setup, based on an immersive VR technology.

Full size image

2.2 Participants

All the participants were informed about the aim of the experiment and the procedure, after that they signed a written consensus. No personal or sensitive data are stored.

Supermarket Virtual Environment. We have divided the experimental session in two phases: a preliminary assessment with 32 volunteer healthy people (ages between 21 and 78 years, average age 39.8 ± 18.8); and an experimental evaluation with 30 elderly volunteersFootnote 1 (ages between 60 and 94 years, mean age 74.6 ± 10.5), both healthy and affected by mild cognitive impairment.

Shape-sorter Immersive Virtual Environment. During this experimental session 35 volunteer subjects have been tested, including 30 adults (ages between 20 and 45 years (average age \(26.0\,{\pm }\,7.3\)) and 5 children and teenagers between 7 and 17 years old (average age \(10.6\,{\pm }\,4.6\)). It has been chosen to involve in the study a sample of children, although small, in accordance to two different motivations. First of all, for the purpose of determining whether the task is suitable even for people belonging to a low age range, both in terms of practical difficulties that the task itself requires and from a health point of view. Secondarily, in order to compare the technical and behavioral differences employed by the two categories of users to complete the task. It is worth noting that it is a preliminary analysis, prior to an effective use of such an immersive environment for the cognitive assessment of impaired people. For this reason, we have not included elderly people, since we are still evaluating the acceptance of the head-mounted displays.

2.3 Experimental Procedure

Supermarket Virtual Environment. By following the results shown in [13], we tested the “Supermarket” Virtual Environment by using the tablet device, thus inputs to the system are provided with the device touchscreen. The choice is given both by the fact that no significant differences between the use of a tablet with touchscreen and a PC with mouse are reported in [13], and by the fact that in an assistive context (i.e., a hospital or a retirement home) the use of portable device may be more practical. The experiment has been conducted as follows (see also Fig. 1 for a pipeline describing the experiment):

  • Interview with the subject. During such interview some data (e.g., age and schooling that have been stored in an anonymous way) have been recorded, together with the results of the following paper-pencil questionnaires: the Pfeiffer test and the GPCog.

  • Training phase, in order to allow the user to become familiar with the system.

  • Test execution.

Shape-sorter Immersive Virtual Environment. In [14], the authors showed that interaction with touchful devices, e.g. the controllers provided with the HTC Vive headset, are more robust and stable with respect to interaction with a touchless technology, e.g. by using a Leap Motion. For this reason, since our aim is to interact with the VR environment in a reliable way, in this experiment we followed the touchful approach. The immersive VR environment has been tested in laboratory conditions, only. The experiment has been conducted as follows (see also Fig. 2 for a pipeline describing the experiment):

  • Interview with the subject. During such interview some data (e.g., age and schooling that have been stored in an anonymous way) have been recorded.

  • Training phase, in order to allow the user to become familiar with the system.

  • Test execution.

2.4 Data Collection and Analysis

Supermarket Virtual Environment. Besides the possibility of a direct observation of the user performing the task by the doctors (or indirect by using RGB and RGBD sensors [43]), the developed system allows us to store the following data:

  • Given and actual shopping list, i.e. the list randomly generated and provided to the user, and the shopping cart he/she effectively composes.

  • List of errors in the “shopping task”, i.e. number and typology of errors: wrong items put in the shopping cart, numbers of item deleted and/or re-inserted in the shopping cart.

  • List of errors in the “payment task”, i.e. whether the payed amount of money is wrong. It is worth noting that the system allows for the possibility of receiving back the change.

  • List of errors in the “recall task”, i.e. the number of items in the shopping list that are not correctly remembered after having completed the task.

  • Partial time spent to complete the given tasks.

By defining as CI the number of bought items actually present in the shopping list, as WI the number of bought items that were not in the list, and DI the number of deleted items, we can define a score SS, empirically defined as follow:

$$\begin{aligned} SS=CI-\alpha WI - \beta DI, \end{aligned}$$
(1)

where \(\alpha \) and \(\beta \) are constant we defined in order to give different weights to errors of different importance.

In an analogous way, we can define:

$$\begin{aligned} PS= 10 - \gamma E, \end{aligned}$$
(2)

if the payed amount is correct, where E counts the number of times the user reset the payment tasks and \(\gamma \) is an empirical constant, or \(PS = 0\), if the payed amount is less than the required one.

Shape-sorter Immersive Virtual Environment. The immersive virtual environment and the tracking capabilities of the adopted system (in our prototype the HTC Vive lighthouse system), allows us to compute and store the following data:

  • Number of errors and number of repositioned items, i.e. the total number of objects that have not been put in the corresponding holes, and the number of times the experiment supervisor resets an item position.

  • Total completion time, i.e. the time to put in the right position all the elements.

  • Partial times, i.e. the times to put in the right position each item.

  • 3D positions of the user head, and of his/her hands (computed by using the HTC Vive sensors).

The availability of the 3D user’s position allows us to better analyze the behavior of the subject acting in the immersive VR scenario, in particular the planning of actions (reaching and grasping tasks) and his/her posture.

3 Results

3.1 Supermarket Virtual Environment

Figure 3 shows the collected data and the metrics computed for the “Supermarket” experiment. Results are divided by considering separately healthy subjects (Healthy) and the one affected by mild cognitive impairment, i.e. MCI. By considering the shopping task, both the average score and the average time (Fig. 3(a)(b)) differ between the two groups (\(p<0.05\), t-test). By analyzing the data, it is worth to note that the lower score of MCI users is due to the fact that people bought less items, with respect the given list. This should be taken into consideration in a future work, since it could be an indication of fatigue or annoyance. Similar considerations can be done for the payment task (Fig. 3(c)(d)): there is a significant difference between the two groups, MCI users obtained a lower payment score and needed more time to complete the task. In general, we can note that all the healthy users were able to complete the required tasks without any difficulty, whereas people affected by MCI found the task quite difficult. It is also worth noting that difficulties in using euro currency still remain among older people. Some could not understand how to play, e.g. where to click in order to add money. This is of great importance, since it put in evidence a lack of naturalness in the developed interface.

Fig. 3.
figure 3

The computed metrics for the “Supermarket” test for the “Healthy” group (light blue) and for the “MCI” group (dark blue). Average values and associated standard deviations are shown. (Color figure online)

Full size image

Finally, it can be noticed that the number of remembered items is consistently different between Healthy and MCI subjects (see Fig. 3(e)). Healthy people can remember an average of 8 bought items (over 10), whereas for MCI users we have \(3.60\pm 1.65\). This is especially true for those who obtained a low rating in the session of the GPCog test, which assesses memory.

Since one of the aim of our approach is to evaluate, whether it could be possible to substitute the pencil-paper tests with more effective evaluation methods, it is interesting to understand if they can measure and evaluate similar aspects of the cognitive status. To this aim, we have computed the Pearson correlation coefficient among the results of the Pfeiffer and GPCog tests and the parameters computed by the “Supermarket” test. Results are reported in Table 1, from which it can be noticed how no correlation is present with respect the results of the Pfeiffer test, whereas there is moderate (Shopping Score, Shopping Time and Payment time) or strong (Payment Time and number of Remembered Items) correlation withe the results of the GPCog test. This result suggests that the proposed approach might be used to evaluate the cognitive status of elderly people. However, we should take into account that there is no correlation with the Pfeiffer test, for this reason a further study is necessary, though our aim is not to replicate the tests but to obtain a more complex evaluation.

Table 1. Pearson correlation coefficients among the parameters computed by the “Supermarket” test and the Pfeiffer and GPCog tests. *\(p<0.05\); **\(p<0.001\)
Full size table

3.2 Shape-Sorter Immersive Virtual Environment

Results are divided by considering separately adult subjects (Adults) and younger ones (Children). Table 2 shows a comparison of the computed metrics for “Adults” and “Children” groups.

Table 2. Data collected for the “Shape-Sorter” test for “Adults” and “Children” groups.
Full size table

There are not statistically relevant differences, except for the “Completion time” metric, which is lower in the “Adults” group (\(p<0.05\), t-test). It is worth noting that “Completion time” takes into account both the time to put the single objects in the right position, and the time to plan the actions (i.e. to find the correspondence between a shape and a hole, to decide whether the hole is not occluded, and to decide which object grab first). This may be correlated with the cognitive capabilities of planning actions and distinguish shape and distance, thus it could be used as a further assessment of the cognitive status for older people. Nevertheless, a systematic analysis of such factors and a correlation with paper-pencil tests have been planned but not performed, yet.

A preliminary analysis of the trajectories shows that differences between children and adults are present, especially in the smoothness of the path. Figure 4 shows a comparison between the two groups, by looking at trajectories along the X-axis (the same is for the other axes). This suggests that a specific analysis of the 3D movements of the subject could be used in order to assess his/her cognitive capabilities. It is worth noting that the “Shape-Sorter” test can be implemented with configurations of stereoscopic displays, which allow the perception of depth, thus avoiding the use of HMDs [37, 38]. The possibility of performing the test without wearing a HMD could be a factor to be considered for assessing elderly people.

Fig. 4.
figure 4

Trajectories of the user dominant hand, along the X-axis for the “Adults” group (top) and for the “Children” group (bottom). Different colors represent the movements to place different objects in the corresponding holes. (Color figure online)

Full size image

4 Discussion and Conclusion

In this paper, we propose human-computer interaction based tests to assess the cognitive status of elderly people. In particular, we have developed two VR applications that take into consideration some cognitive functionalities: memory and attention by using non-immersive VR (i.e. the “Supermarket” test), and reasoning and visual perception by employing immersive VR (i.e. the “Shape-Sorter” test). We designed the two approaches in order to allow a natural interaction in VR (also in collaboration with the medical staff), which is a fundamental aspect for using them with elderly people. The results show that subjects can accomplish the proposed tasks: this is a good starting point for further development of VR approaches, though some works is still necessary to simplify the interaction and to improve the engagement of MCI patients.

The comparison of the results of our proposed VR tests with two standard paper-pencil tests, i.e. the Pfeier and the GPCog, shows a good correlation with the latter, only. This deserves further investigations in order to understand the evidenced difference.

During the tests performed by using the immersive VR application, the 3D trajectories of the participants have been recorded: a preliminary analysis shows that there are differences between the two analyzed groups. This suggests that also the behavioral data could be used to assess the cognitive capabilities. However, there is the need of further studies to assess these preliminary results and to apply the proposed approach to the assessment of elderly people.

The obtained results suggest that the proposed approaches can help doctors to assess the cognitive status of elderly people in a objective and automatic way. Moreover, in perspective, these VR approaches could be used both for a more complete assessment and for the cognitive training of elderly people.