Keywords

1 Introduction

Ageing becomes the highest priority for the public health policy for France, like in many countries in the world. Over the past decades, healthcare has experienced many changes. France has created specialized nursing home establishments for dependent elderly people. In this context, innovative solutions based on new technologies are proposed. For example, Virtual Reality (VR) techniques have been proposed to support traditional care. VR “is a scientific and technical domain that uses computer science and behavioral interfaces to simulate in a virtual world the behavior of 3D entities, which interact in real time with each other and with one or more users in pseudo-natural immersion via sensorimotor channels” [1]. Indeed, VR makes it possible to expose users to multisensory (often audio-visual only) situations which are controlled.

This paper introduces a new motion-based tool for occupation and monitoring of residents in nursing homes. In the next Sect. 2 (Research Context), we introduce the context of this work. More precisely, this section describes the health status of residents (Sect. 2.1), explains the difficulties to implement adapted activity programs (Sect. 2.2), proposes to implement therapeutic activities in order to support health management (Sect. 2.3) and focuses on the interest on and full-body interaction with virtual environments. The following Sect. 3 (System Description) describes needs and specifications at the basis of our work (Sect. 3.1). Then, we present the software (Sect. 3.2) and hardware (Sect. 3.3) used to develop our VR-based system. Furthermore, we describe the application functioning (Sect. 3.4) and expose the different functional modules (Sect. 3.5). Finally, this paper ends with a conclusion and proposes tracks for future work.

2 Research Context

2.1 Older and Highly Dependent Residents

Today, nursing homes must manage a growing number of residents, who enter in these institutions older [2, 3] and more dependent [4]. In French specialized nursing homes, about 54% of the residents are confined to their bed or chair and/or show a severe global cognitive deficit. Moderately dependent people account for 37% of the residents hey show intact or relatively intact cognitive abilities, but they need help to perform Activities of Daily Living (ADL, 17%), or they need help with transfer but move autonomously once they are standing (20%). They all often need help with washing and dressing and cooking. Fully independent or quasi-independent people account for 9% of the residents, but have about 6 pathologies (average per resident) [3, 5]. Overall, at least one-third of residents in the French nursing homes has an Alzheimer’s disease or a related dementia [3, 6].

Even nursing homes are first and foremost places to live, the residents’ health status calls for vigilance regarding the risks of death, life-threatening illness or a permanent functional deficit [7,8,9,10,11,12]. Thus, some institutions implement strengthening muscle exercises and balance and fall prevention workshops.

2.2 Difficulties to Implement Adapted Activity Programs

To maintain the quality of life as high as possible is an important goal of health care, especially in incurable pathologies. In very old and dependent people, the stress is made on social participation. The most popular activities in nursing homes are singing, memory game, cooking, board game, cultural outing and walk, intergenerational event, computer workshop, as well as soft gymnastics. Digitally-enabled activities are also taking off, notably thanks to WiiTM and KinectTM, which have democratized video games in the 2010s by making them more accessible and trans generational. Consoles which designed games for young as well as for the elderly were among the most appreciated new technologies in nursing homes.

The management of dependence proposes to cure diseases, to rehabilitate the individual health (e.g., psychotherapy, exercise programs), to modify task procedures, or to adapt the environment (e.g., prostheses, care programs). Various non-pharmacological approaches are proposed, preferentially focusing cognition, motor skills, mood, or social life. Non-pharmacological programs could maintain or improve functioning [13,14,15,16,17] and reduce the subjective burden of caregiving [16, 18]. Unfortunately, despite their benefit, activities for institutionalized people are still underdeveloped. In practice, the management of dependence in elderly relies mainly on the contribution of external help, from families, associations and professional caregivers. In nursing homes, it relies almost exclusively on the professional caregivers who do, in the place of the residents, the activities in which the latter have lost their autonomy. Cooking or gardening proposed to elderly institutionalized people are usually animations, focusing on pleasure, not on residents’ empowerment. These animations are not like therapeutic activities. In France, nearly half of nursing homes do not have procedures to prevent the decline of autonomy in ADL of their residents, and only 37% propose educational programs (i.e., information on diseases, medications, warning signs, non-drug therapies). Moreover, more than one-third of residents has indicated that they did not want or could not participate in the occupational activities [19]. Listening to the radio or watching television is the main activity for 49% of residents, while participating in group activities and participating in personal activities are the main activity for 25% and 28% of residents respectively. The most dependent are resting (38% of them) or “bored, look out the window…” (37% of them). A low level of activity has also been described in American studies. Thus, according to Harper Ice [20], 65% of people “do nothing” [21, see also 22,23,24]. Yet, healthcare institutions are constantly seeking to improve their services, so they can meet residents’ needs, without increasing the burden on professional caregivers.

In this context, the research has investigated the underlying causes of this low level of activity in nursing home residents. According literature, the proposed activities can be experienced as frustrating and uninteresting if they do not match individual skills [23, 25]. In addition, when activities are not relevant and meaningful, residents report preferring to watch television or do nothing rather than attend activity sessions [25]. According to data from a French survey [19], half of nursing home managers have difficulty in organizing activities that are adapted to the individual preferences and to the reduced abilities of residents. Consequently, diversification of supply and attention to individual skills, needs and interests in the selection of activities is essential to foster residents’ participation and strengthen their self-esteem, enjoyment and/or success [25,26,27]. Finally, the lack of activity in residents finds explanations in budgetary and human tensions which have been frequently identified [e.g., 28,29,30]. As a result, it is common to notice understaffed services and precarious employment (e.g., contracting, short-term, part-time). However, such organizational solutions affect the continuity of care and reduces opportunities for residents to benefit from the exercises. Professional caregivers lack of time and, sometimes, they also lack of materials and training, to incorporate, into their daily routine with residents, activities likely to maintain the residents functioning [31]. Consequently, medical and social management of elderly people may benefit from less using service providers to quickly execute the ADL for residents (e.g., dressing, toileting), and promoting therapeutic ADL achieved by residents as independently as possible with the adapted assistance of professional caregivers.

2.3 Improvement Levers: Strengthening Evaluation

Another interest of the therapeutic activity is the associated assessments of patients. In the French nursing home, a geriatric assessment is performed at the entrance of residents to assess their level autonomy and their need for assistances in seventeen basic and instrumental activities. This assessment is used to calculate the fees of managements and treatment in institutions and to calculate the eligibility for public subsidies and reimbursement of costs. In addition, a complementary geriatric assessment is recommended to evaluate sight and hearing, physical health (risk of falling, nutritional status, functional autonomy, depression, and global cognitive deficit. The purpose of this comprehensive assessment is to guide the design of a medical-social project which appropriated to each resident. Such standardized and archived assessments also improve quality of care in nursing homes. Indeed, objective standardized measures can compensate for the usual communication difficulties of residents [32, 33]. In addition, data archiving plays a crucial role in the continuity of care in this sector affected by turnover [34]. Finally, follow-up assessment can be used to measure the impact of an intervention. According to a French survey aiming to assess the quality of services in nursing homes [35], 90% of respondents considered that the different capacities and limits of the resident are regularly evaluated and more than 96% of respondents considered offering a personalized support, based on expectations and the needs collected directly from the resident. Moreover, in the same study, only 30% of responding institutions acknowledged that they did not perform an initial analysis of the resident’s needs and 40% did not update data about the residents’ needs. Some institutions also acknowledged that they did not have a procedure to evaluate their management of cognitive impairment (25%) or their management of mood and behavior disorders (31%). These data suggest that health professionals are satisfied with an appraising health status of residents, when the opportunity arises, without formal protocol nor quantitative scoring. The underused of a comprehensive assessment may be due to its time-cost and/or can a minor interest of the details of these many tests for their practice.

2.4 Interest of Virtual Reality and Full-Body Interaction

To enable this sector to cope with the constantly changing demands of healthcare and to benefit from it, the interest of new technologies is explored. Especially, VR techniques have been proposed to support traditional care. VR makes it possible to expose users to multisensory situations that are controlled. The scenario and tasks can be imaginary or realistic, but credible. The user is placed as the main actor of the simulation, which evolves according to her actions, made using interaction devices. Interactions with VR solicit various sensorial modalities (e.g., depending on the system: by means of his voice, her gestures). One of the many advantages of VR is that the patients do not need to be in a clinical setting as they can perform the exercises in their own home through tele rehabilitation systems. In addition, patient exercises can be recorded and the data can be used to evaluate patient rehabilitation performance [36, 37]. Another key point is that VR therapy is capable of motivating patients to a larger extent and thereby stimulating new motor and sensory abilities. VR systems can be classified in two categories, immersive systems which simulate a virtual environment, making the user feel that he is present in the virtual environment itself, and non-immersive systems, generally based on large 3D projected screens. While immersive VR systems involve complex interaction devices and head-mounted displays, non-immersive systems use simpler setup and low-cost interaction devices such as computer mice, joysticks, or gamepad [38,39,40,41]. A more recent approach is based on mobile devices, chosen because of its accessibility, cost and ease. For example, a study by Luis et al. [42] have used Unity 3D game engine along with Samsung Galaxy S7 and Samsung Gear VR.

Several studies focusing on upper limb motor functions have been done on immersive VR. For example, Cameirao et al. [43] have immersed fourteen participants with acute stroke in a VR rehabilitation gaming system. The interventions include hitting, grasping, and placing virtual objects. Stewart et al. [44] have used a desktop computer and 3D shutter glasses to provide a three-dimensional view of stimuli, and they have obtained successful results with movement and performance improvement. The VR games are also used to focus on arm and hand movement. Both studies have shown participants ‘improved functional ability after experiencing the treatment and this benefits their performance in ADL.

Non-immersive VR is gaining popularity as a technique to improve functioning in both motor and cognitive rehabilitation program. A study by Saposnik et al. [45] have compared the safety and efficacy of VR with recreational therapy on motor recovery. The Nintendo Wii gaming system was used. The result has shown that a motor rehabilitation is most effective when the exercise is intensive and specific, regardless of the type of exercise or task done. Thus, simple exercises can be as good as immersive VR based-rehabilitation. In another study, Cai et al. [46] have proposed a stimulation based assistance through iterative learning platform which applies electrical stimulation to two arm muscles. VR applied to health is a growing domain of interest [47, 48]. Body interaction is especially explored because it is known to facilitate user’s involvement in the tasks. In addition, pseudo-natural interaction allows dependent people to interact with the system without lengthy preliminary training.

3 System Description

3.1 Needs and Specifications

Elderly living in nursing homes are not stimulated enough, because of staff’s time constraints and lack of adapted materials. As a result, the VR system must be a motivating, useful and rewarding mediation tool for both residents and staff. In addition, it must be quick to set up and without requiring long preconfiguration and complex tuning. Finally, it must be inexpensive. In addition, residents of nursing homes have very diverse profiles and have little opportunity to carry out activities adapted to their abilities. Therefore, it is crucial to develop solutions likely to engage both motor components and cognitive components. Moreover, proposed activities must be accessible, meaningful, and enjoyable.

Therapists use to work on short sessions, during which they must evaluate and/or train residents. In addition, the technical part of care must be reduced as much as possible to favor the relational and educational aspects. Therefore, the system must be easy to use. It must propose to quickly launch of activities (e.g., exercise user’s settings). Advanced options have been identified, such as exercise program’s user’s configuration. In addition, resident’s data visualization may be useful.

Access to the detailed raw data of residents is not necessary. However, short reports of needs and warning messages when the system detects abnormal data can be useful for staff, especially managers. Variations in resident’s data could help testing the effectiveness of interventions and hence guide strategic planning and resource allocation.

A last element of constraint was to obtain a final cost of the system as small as possible for the institutions and to preserve our financial independence to remain free in the solutions of perpetuation of the project (licensing). Several tools – such as commercial solutions, middleware, libraries – could help us to develop our solution, but they are too expensive for a successful business model between nursing homes and academic research and development.

3.2 Development Tools

The software was developed under Unity3D (LTS 2017). The modeling was carried out using Autodesk® 3ds Max® (2019). The platform proposes to perform several activities, each configurable and achievable by means of a depth camera (Astra ProTM, Orbbec). The scripts were written in C # language.

3.3 Hardware

The 3D cameras from Astra series are manufactured by Orbbec in Shenzhen, China. They make it possible to follow the movements of the body of users in a 3D space. Their 3D cameras are small and inexpensive devices ($150 for Astra ProTM). They are compatible with the most popular operating systems and not require a high-performance PC. On Windows, a processor x86@1.8 GHz and 4 GB of RAM are enough. In addition, this sensor works well in most lighting conditions, from a lit environment to darkness. However, intense daylight is known to limit the performance of infrared sensors [49]. Contrary to systems like OptiTrackTM, a camera from Astraseries does not require any marker on users. It includes an infrared (IR) camera associated with a coded pattern projector and an RGB camera.

The depth sensor has a 58.4° horizontal and 45.5° vertical field of view. Based on the structured light technique, it provides a 16-bit image with a resolution of 640 × 480 pixels (at 30FPS). The color camera captures a field of 66.1° × 40.2° and can provide a 16-bit image, 1280 × 720@30fps. According to Orbbec tests [50], the sensor accuracy at about 1 m is ±1–3 mm. Orbbec provides a basic SDK based on OpenNI [51]. The middleware for Astra ProTM allows to follow up to 8 bodies placed at 60 cm to 8 m from the device, with an optimal detection between 60 cm and 5 m (2.5 m and 3 m according to our own tests). The data can then be sampled at 60° × 49.5° × 73° in height, width and depth respectively.

3.4 Application Functioning

Figure 1 presents the interaction between the different proposed functionalities.

Fig. 1.
figure 1

Schematic representation of the system architecture.

Full size image

Login Interface.

When the software launches, a dialog box displays, asking to enter a username and password and to accept the conditions of use. The dialog box also gives an access for registration (see Fig. 2). The first created account is a therapist-type account because it is not associated with any other user. To create a patient-type account, the user has to associate his account with a registered therapist. The main Scene of Menus is only available after logging in (see Fig. 1).

Fig. 2.
figure 2

Screenshot of the login interface.

Full size image

Scene of Menus.

The main Scene of Menus provides access to the Personal Data Management module, the System Diagnostic module and it launches on the Games Showcase module.

Personal Data Management.

The personal data management module allows each patient to delete his account and associated data. A part is also dedicated to the addition of personal data such as the age, the level of autonomy (GIR) or the level of activity of the resident. It is also possible to consult the performance data measured during the game (see also “Analysis module”).

System Diagnostic.

The diagnostic sub-menu allows you to check if the AstraTM acquisition device is working properly. Camera returns are displayed as well as different statistics are displayed. In case of malfunction, warning messages are displayed.

Games Showcase.

The Games Showcase sub-menu is generated at its launch, based on a simple XML file in which store different data, such as the game’s name as used in scripts, the game’s name as displayed for users, the description of the game. Our software currently offers three mini-games: (i) a sorting game; (ii) a search-object game; (iii) a ball game (Fig. 3). The sorting game is designed to train the aerobic abilities of the upper limbs and, depending on the settings, the lateral, as well as the cognitive abilities related to categorization. In a virtual bathroom, a colored, light or dark garment is instantiated at the center of the screen. The patient must then touch the laundry basket associated with it. When it’s done, another garment is instantiated. The number of clothing categories, the categories, and the positions of the corresponding baskets are configurable. In addition, this activity can be broken down into other scenes and categories of objects of everyday life to propose other levels of difficulty. In the bathroom environment, categorization is based on perceptual color criteria. In the kitchen, the categorization is based on criteria for locating items to be stored in a tempered closet or in the refrigerator. The object-finder game is more about cognitive stimulation. It takes place in a handyman’s workshop. An object is instantiated at the center of the screen. The patient must then find and touch the object in the scene. Another mode of the game proposed to search for a pair associated with the presented object. When the patient has pointed at an object in the virtual environment, another object is instantiated. This activity can be broken down into other scenes and categories of objects of everyday life. The ball game is inspired by soccer of handball. It is designed to stimulate lower limbs and balance skills. The scene is located by the sea, on a soccer field or in a meadow. In the center of the upper part of the screen is displayed a soccer goal, and a ball is instantiated at the bottom center of the screen. The patient must shoot in the ball to score a goal.

Fig. 3.
figure 3

Scene of Menus displaying the showcase of the proposed mini-games.

Full size image

3.5 Functional Modules

Apart from the graphic part, the system has 3 functional modules: (i) a game module that offers various fun and customizable activities; (ii) an interaction system, which allows the capture and tracking of the user’s body and harvesting information such as the orientation of the joints; (iii) a data analysis module.

Interaction Module.

User can control an avatar (humanoid or virtual body points) to interact in the game. The interaction module is composed of the Astra ProTM optoelectronic device that captures the RGB-D data about the environment. This data is sent to the motion tracking module, which extracts information such as the position and orientation of the user’s body joints. The data extraction module is used to generate an artificial skeleton used to animate the virtual avatar instantiated in the scene and to interact with the environment.

The interaction module makes it possible to project the 3D information on the screen and in the 3D virtual space. Of course, different interaction techniques are proposed depending on the mini-game considered. Basically, the techniques are based on the distance between one or more points of interest from the avatar and objects of interest from the virtual environment (e.g., between foot and ball, between the hand and a laundry basket). The selection is made when the user holds her hand over the object for a certain duration, defined in the Global Parameters sub-menu (Fig. 1). In some mini-games, such as soccer, the interaction module manages dynamic objects subject to physical laws (gravity), which can be moved by the user. Collisions of virtual objects of interest are transmitted to the Unity 3D physics engine which updates the scene information and generates the graphical output.

Game Module.

The game module manages the mini-games which are preconfigured but which can be parameterized by the therapist accompanying the resident. A mini-game is defined by several sets of parameters, those relating to the angle of the camera, the avatar, and the mini-game itself. The options for changing the position and rotation of the camera as well as its field of view are always available to obtain the best image according to the display device used. The position and rotation of the avatar can also be changed. Thus, the user can move the sensor (rotation and distance) to obtain the best possible image of the body. The avatar can be displayed as a mesh, in its skeleton version, or only as a set of interactive areas (e.g., a sphere at the hand). All mini-games include a stop parameter option: activities can stop after a certain amount of time and/or after several basic tasks have been completed. For the mini-games described, an elementary task corresponds to an instantiated and processed object. In addition, it is associated with a set of measures (e.g., objects found) and triggers effects (e.g., sound). When the stop condition is reached, the data is saved in the database.

Analysis Module.

The analysis module intervenes at the end of the mini-games to save in the database various performance measurements taken while the patient was performing the activities. The data include patient’s workspace, anthropometric data, joint angles, average movement speed, completion time and scoring. In addition, this module can display and filter data from the “Personal Data Management” section of the scene of Menus.

4 Conclusion

To support men and women institutionalized in long-term care is a challenge that is both ethical and technical. Professional caregivers work daily to make each residents’ day better. In this project, a participative method was used to design the system. Developments also strives to maintain our financial independence. The next step is to assess acceptability, accessibility and interest of our system. Nonetheless, perfect our health care system will stay difficult, without placing old-age as a public priority, and without helping institutions to promote the social and therapeutic roles of caregivers.