Keywords

1 Introduction

Growing ageing population in the United States is having significant implications on the current healthcare system as the elderly face neurodegenerative conditions which may reduce the level of independence and increase the risk of falls and injury. There are currently almost 40 million persons aged 65 years or older living in the US, while the number is expected to increase to 72.1 million by 2030 [1]. By improving the quality of independent living through increased physical activity these challenges can be partially mitigated [2]. Many elderly, however, lack access to exercise facilities, or the skills and motivation to perform exercise at home.

To improve the health behavior and to overcome the lack of motivation in the general population, various forms of computer-assisted coaching and “gamification” of activity monitoring have been investigated; initially in the academic space and later on, with the introduction of affordable motion sensing, also in the commercial space. The success of interactive exercise products, as pointed out by Sinclair et al. [3], by and large depends on two interrelated dimensions: (1) effectiveness, which relates to achieving exercise goals, and (2) attractiveness, which refers to the level of engagement for the user to retain required duration and level of exercise. Many of the commercial products have focused on the attractiveness aspect, while the academic field has tried to examine the effectiveness of these technologies in exercise training and rehabilitation. Many of the commercially available systems, however, are targeting different demographics, such as younger users, and are as such less applicable for most older adults as they do not offer appropriate type and level of exercise and fail to provide appropriate safety considerations. Furthermore, the feedback provided by such systems may be overly-engaging. In addition, the interaction modality may entail of complex user interfaces, which may not be easy to use for elderly with reduced sensory and cognitive functions [4]. Although several interactive systems for exercise in elderly have been presented in research (e.g., [5, 6]), majority of the works focused on short-term and controlled in-laboratory evaluations. A comprehensive review of the research literature on interactive exercise in older adults can be found in [7, 8].

2 Background

The goal of this research was to develop an interactive exercise coaching system for elderly that would be integrated with the semi-automated coaching framework at the Oregon Center for Aging & Technology (ORCATECH) Living Lab,Footnote 1 which is focused on exploring technologies to support independent living of elderly. The coaching platform comprises of unobtrusive sensing of participant’s behaviors in combination with artificial intelligence tools that aid the coach to send individualized messages to the participants [9]. Originally, the participants were encouraged to exercise alongside YouTube videos, however the system was not able to track individual’s exercise habits or provide feedback on the performance that could be used to close the loop of the health-coaching support. To achieve the interactive component for the health coaching, we considered several different solutions, including wearable devices and 2D cameras. After the release of Kinect for Xbox 360 (Microsoft, Redmond, WA) and accompanying Kinect SDK, we decided to use the Kinect as it offered unobtrusive, low-cost and relatively reliable way of measuring human motion kinematics. Although several commercial applications for exercise have been developed to date, one of the challenges is how this technology can be introduced in homes of elderly.

3 Methods

3.1 Design Process

The design of the Kinect-based exercise system architecture followed participatory design concepts by engaging the computer scientists and researchers with the health coaches and caregivers during the interactive software development process over the last three years. We also took into account user feedback at several stages of the project. The interactions among the members of the team included the following:

  • Identifying the requirements and objectives of the architecture;

  • Researching the needs and expectations of the targeted population;

  • Defining basic functionality of the exercise system;

  • Determining what data should be collected by the system;

  • Determining accuracy of the Kinect measurements;

  • Determining the conditions for home deployment;

  • Defining general user interaction flow with the system;

  • Selecting exercises appropriate for elderly users and the Kinect;

  • Recording exercise videos and defining movement features related to exercises;

  • Testing and modifying the prototype system at several stages;

  • Collecting and integrating user feedback;

  • Resolving various technical issues related to the deployment and maintenance;

  • Running in-home pilot studies with health-coaching support;

  • Discussing and evaluating various forms of data analysis.

We have approached the goals of this project in two stages. Our first prototype deployment was primarily focused on understanding better the user needs and technical challenges related to the exercise monitoring, user interfaces, and the use of Kinect technology in homes of elderly users. We therefore installed the prototype exercise system with 12 basic exercises into the homes of six independently-living elderly individuals for an informal evaluation study. The system and results are described in details in our prior publication [10]. The lessons learned from this study were then used to make considerable improvements to the exercise system and evaluate it in an 18-week long deployment in 7 homes of elderly users. In this paper, we thus focus on the second stage of the design and evaluation. For completeness, we briefly describe some of the findings from the first stage of the project while further details can be found in our referred publications [10, 11].

3.2 Design Objectives

The primary goal of this research was to integrate an automated exercise coaching with semi-automated health coaching of elderly in order to improve their fitness level in terms of standard measures of fitness, such as flexibility, strength, balance, and endurance [12]. Table 1 summarizes the design objectives for the development of the exercise system and provides brief overview of identified issues from Phase 1 (described in [10]) and how they were addressed in Phase 2.

3.3 Implementation

In this section we describe the design, implementation, and setup of the Kinect-based exercise system. For completeness we briefly refer to some of the findings and lessons learned from Phase 1 while providing more details on the final version of the system used in the last pilot study.

Software. The exercise software was implemented in C++ with support from open source 3D library Ogre (ogre3d.org) for graphics, MyGUI (mygui.info) for UI, Microsoft Kinect SDK for data acquisition, and MySQL for database management.

Kinect System. Our exercise system is based on Microsoft Kinect camera [13] which was originally developed for the gaming console Xbox 360. The Kinect is a depth-sensing system (combining RGB and infrared cameras) that provides 3D reconstruction of the scene and segmentation of human blobs with real-time estimation of the 3D location of 20 joints. The accuracy of the pose reconstruction depends on various factors including orientation of the body, self-occlusions, interference with other objects, etc. During the planning stages, we examined the accuracy of the Kinect tracking alongside a motion capture system to identify the exercises where the tracking was robust and to determine the accuracy of joint estimation [11]. One of the challenges of using the Kinect camera was its limited field of view which requires users to be positioned between 1.8 m and 4 m. This can be particular challenging in smaller and cluttered homes. In addition, the pose estimation becomes less reliable when users are seated or turned sideways. These limitations posed several constraints on the system setup and exercise selection.

Movement Analysis. The real-time movement analysis during the exercise was performed by first extracting measurement primitives from the skeletal data, such as joint angles, relative angles to the vertical/horizontal plane, distances, absolute positions, etc. These features were chosen manually based on the goals of specific exercise. The goals were defined in consultation with the health coach. The selected measurement primitives were then used to evaluate the performance of the exercise (e.g. how high person can reach), to support repetition counting, and to trigger feedback alerts. More details on the implementation can be found in [10].

Feedback and Visualization. During the development and testing phase, we examined different options on how to provide effective feedback during exercise. The possible feedback modalities included video, skeletal data, 3D graphics, 2D overlays, textual messages, and auditory feedback. During initial in-laboratory testing, we examined three different options as shown in Fig. 1. Informal usability assessment with coaches and several elderly users suggested a preference for full-screen video mode (Fig. 1a) which was further refined as shown in Fig. 2 (left). After the first pilot study [10], the feedback collected from six users indicated that although the participants liked the video guidance they were confused about what the camera sees and they were not always able to relate their movement to the movement of the coach. Therefore, we decided to include a mirrored human figure as captured by the Kinect depth sensor next to the video of the coach as shown in Fig. 2 (right). This element provided more intuitive way for a subject to relate their movement to the exercise performance. Additionally, we replaced all the videos with high definition recordings of the coach that were integrated into the 3D environment for more attractive overall appearance.

Fig. 1.
figure 1

Different in-exercise feedback options that were considered in early design stages.

Full size image
Fig. 2.
figure 2

Comparison of the in-exercise feedback screen between the initial [10] and the final version of the exercise software.

Full size image

The visual feedback in the exercise software also included several informational elements that were displayed as 2D overlaid graphics and text. In the first version, we included performance bars which indicated how well the user is performing a particular exercise based on the measurement primitives. The users, however, found the performance bars difficult to understand and map to their own movements. In the second version, we instead decided to report the performance in terms of the number of accomplished repetitions. The current repetition count was indicated by a large numerical counter and corresponding number of yellow stars on the bottom of the screen. When the user first started the exercise, gray stars were shown while their number corresponded to the number of repetitions of the previous session. This information was intended to encourage the user to try to reach or exceed previous performance. The number of yellow stars increased as user performed more repetitions.

The feedback also included auditory and textual messages triggered by the performance evaluation. For example, if the subject were to sit tall in a particular exercise, the system would trigger an alert whenever the user started slouching. In the first version of the system, the messages were shown under the performance bar measure (Fig. 2, left). In the second version, we tried to reduce the clutter on the screen and created a separate messaging panel to display feedback messages (Fig. 2, right). In addition to the corrective messages, the system also included several general encouraging messages (e.g., “Good job!”, “Keep up the good work!”) and exercise-specific messages that would remind the users for correct performance (e.g. in Leg Lifts exercise: “Kick one foot up, then the other.”). These messages were displayed randomly. The main exercise screen also included a countdown clock with a graphical display and a numerical counter. As opposed to the first version, where the exercise would finish after completing 10 repetitions, we limited the exercise duration to 45 s as recommended by the health coach.

User Interface (UI) and Navigation. Since the initial pilot study was primarily focused on testing the feasibility of collecting exercise data at home, the user interface was relatively simple. Although we considered several different modalities to control the software (e.g., speech, gestures, presenter remote), we decided to use a wireless mouse and keyboard since the participants were familiar with these devices. In general the users found the software to be easy to use, however some participants reported that they were not able to read the text on the screen from the distance and had difficulty controlling the application in Windows environment.

Our focus in the second phase was to improve the user experience, especially since the system was intended to be used over a much longer time period. As recommended by guidelines for design of software for elderly users [4], we implemented the following improvements to the original interface:

  • Large fonts for text messages and button labels;

  • Familiar icons on buttons (e.g., video controls used icons similar to VCR);

  • Consistent positioning of buttons with similar functions;

  • Simple graphical elements;

  • Color scheme with good contrast;

  • Improved text-to speech (offered by the new Windows 8 platform);

  • Minimal textual information on each screen;

  • Overall reduction of screen clutter; information organized into display panels;

  • Linear screen interaction flow;

Figure 3 shows several example screenshots from the updated software with the following interaction flow. From the main screen (Fig. 3a), the user is able to view help and safety videos, start a new session or complete unfinished session. Next, the user is presented with a survey of five questions about their general health and goals for the day (Fig. 3b). The user is able to skip a specific question if they prefer not to answer it. On the exercise selection screen users can select between three different exercise groups with various difficulty levels (Fig. 3c). Once the exercise group and level are selected, the user is prompted to perform optional warm-up which includes only the video playback of the coach without any feedback on the performance. Next, depending on selected preference, the full instructional video on benefits of the exercise or a short 10-second preview is displayed (Fig. 3d). Afterwards, the user performs the exercise for 45 s with the real-time feedback and accompanying video (Fig. 3e). At any time, the user can review instructions, skip to the next exercise, or exit the session. Once the exercise is completed, a bar chart showing current and past repetition counts is displayed. After completing all the exercises, the session summary screen is shown, displaying the summary statistics of the particular exercise session compared to the past performance (Fig. 3f). Demo video can be viewed at: http://tinyurl.com/KinectExercise.

Fig. 3.
figure 3

Software interaction flow: (a) home screen, (b) daily survey, (c) exercise selection, (d) exercise preview/instructions, (e) in-exercise feedback, (f) session summary.

Full size image

To simplify the navigation, we implemented support for a 3-button wireless PowerPoint remote with large buttons (Kensington, K72441AM). Two buttons on the remote were used to change the selection on the screen (which was highlighted) back and forth while the third button was used to confirm the current selection.

Exercises. The original system included 12 exercises (e.g., Heel Drags, Lateral Stepping, Leg Extensions, Cops and Robbers, Buddha’s Prayer, etc. [10]) focused on improving balance, flexibility, and strength. Based on the feedback collected from the first pilot study, we included several variations of these exercises to provide variety for users of different capabilities. The final system included about 40 exercises which were grouped into three groups (i.e., full body, upper body with core & lower body) and arranged into three difficulty levels, each containing between 6 to 15 exercises.

System Setup. In the first pilot study, we used all-in-one computer with large screen which provided easy setup and portability. We found, however, that in some homes it was difficult to find sufficient space; therefore, the physical setup often had to be improvised to achieve the required distance for the Kinect. For easy interaction, we configured the system as a ‘turn-key’ system. A small desktop PC running Windows 8.1 was connected to participant’s existing TV set using HDMI connection to transmit video and audio signals from the computer. To simplify the process of switching between the regular channels and PC input, we installed an HDMI switch. The PC was configured to be always on and to boot into the desktop without requiring users to log in. Remote connection to the PC was enabled via TeamViewer software (teamviewer.com) for administration of the system. For protection of privacy all the data saved on the PC were stored in temporary MySQL tables which were copied nightly to the external server. This process however turned out to cause occasional data loss as some of the homes experienced outages of network and power due to weather and construction.

4 Results

In this section we present results from 18-week study in the homes of 7 elderly individuals ranging from 77 to 96 years of age (mean age: 83.2). Baseline data on physical fitness (e.g., Berg Balance Test, Senior Fitness Test, etc.), general health, and physical activity were collected prior to the deployment of the Kinect system and subsequently every four weeks. As part of the Living Lab enrollment other quantitative data were collected during the study, such as sleep data, in-home motion sensors, cognitive games, etc. Subjects were also in contact weekly with a health coach who provided guidance on the exercise regimen and collected feedback on the system usage and any technical issues. The study protocols were approved by the Oregon Health and Science University IRB.

Figure 4 shows the exercise program adherence over the course of 18 weeks. The subjects were instructed to exercise 3–5 times a week. From the 7 subjects who were enrolled, four subjects performed the exercises somewhat regularly. Subject #1 got ill early on and never returned to the exercise. Subject #2 exercised regularly, however due to the internet connectivity issues, we were unable to recover the data of the second portion of the study. Subjects #3, #4, and #5 completed most of the exercise sessions. Subject #6 performed exercises intermittently but later on stopped using the system due to holidays and travel. Subject #7 initially used the system but found it was not as useful to him as he was already involved regularly in Tai Chi and riding exercise bike. Figure 4 also shows the type of exercise sessions the subjects performed. Most of the subjects performed full body exercise sessions. Subjects #2 and #4 were both able to increase the exercise level after a few weeks. Subject #3 was on the other hand alternating between the different exercise groups, which was also the general recommendation.

Fig. 4.
figure 4

Daily exercise adherence during the eighteen-week study. The color of the patches denotes the type and level of exercise (UB – Upper Body, LB – Lower Body, FB –Full Body).

Full size image

Figure 5 shows the results of the post-exercise survey compared between subjects #3 and #5 who had the most completed sessions. The subjects’ response data reflect their exercise habits. Subject #3 reported to be pain free most of the time and very motivated to exercise. On the other hand, subject #5 reported pain and low motivation, in particular in weeks 9 and 10 after which the subject took a break from the exercise. Daily survey responses could be in general used by the health coach or an automated system to provide appropriate intervention to increase the motivation of the user.

Fig. 5.
figure 5

Comparison of post-exercise survey replies for subjects #3 and #5. The charts show mean (*), standard deviation (Δ∇), and minimal/maximal response values per week.

Full size image

Figure 6 shows the raw Kinect skeleton output for six sample frames captured during the exercise Shallow Squats. The skeleton configurations are shown for every 30 frames corresponding to the time interval of 1 s. As mentioned previously, the skeletal data were used to extract the measurement primitives for repetition counting and feedback. Figure 7 shows the number of completed repetitions for the same exercise over the course of the study for subjects, #4 and #5, who had performed this exercise in majority of their sessions. Note that this exercise was included only in the Full Body - Level 1 and Lower Body – Level 2 exercise groups. For both subjects we can see the trend of an overall increase in the number of repetitions over time which is likely due to improved endurance.

Fig. 6.
figure 6

Raw skeleton sequence recorded for the exercise Shallow Squats (Subject #3).

Full size image
Fig. 7.
figure 7

Number of completed repetitions for the exercise Shallow Squats for subjects #4 and #5.

Full size image

5 Discussion and Conclusion

Based on the findings from our Phase 1 study we have successfully improved the exercise system to achieve the objectives for long-term use that were summarized in Table 1. Majority of the changes in the design, additional exercises, and overall system performance were well-accepted by the participants. There were several minor technical issues that were identified and corrected during the first two weeks after the installation. These included changes to the scripts that suppressed various system pop-ups and always put the exercise software in the foreground of the desktop. We also noticed that some users were either double clicking or holding the button on the remote for a longer time period which sometimes resulted in multiple confirmations. These navigation issues were resolved by a subsequent software update. Overall, the wireless remote was easy to use for the participants after the initial issues were resolved. Only one user experienced a failure of the remote during the course of the study.

Table 1. Design objectives for the Kinect-based exercise coaching system
Full size table

Since the users only had the wireless remote to control the system, we were not able to use the login mechanisms that would allow for the encryption of the hard drive in order to protect the privacy of the data in case the computer was stolen. Instead, all the data were stored in memory tables and subsequently copied to the remote server each night. This arrangement however created another technical challenge because some of the homes experienced internet and power outages that were not anticipated to happen at such frequency. Although the implementation worked for most users, several sessions from one of the users were lost in the process. For the future studies, we plan to investigate other mechanisms to ensure data privacy and security while providing a robust data collection regardless of the internet connectivity.

From the users’ perspective, the biggest challenge was switching the TV setting from their regular cable channel to the PC input. Initially the users were instructed how to do that via their TV remote; however, some were not able to remember the steps. To resolve this issue, we installed for some users a physical HDMI switch that would allow them to more easily switch the inputs at their convenience.

On the software side, one of the common issues reported by the participants was the lack of or incorrect repetition counting in some of the more complex exercises. As reported previously [10], the real-time analysis is sensitive to various factors, which include camera position, inclusion of other objects in the scene, orientation of the user with respect to the camera, type of chair, etc. We are currently working on more robust methods to perform the analysis and repetition counting while using the collected dataset for benchmarking.

The feedback collected from the interviews with the participants revealed that lack of exercising was primarily due to reasons unrelated to the system itself, such as illness, scheduling, low motivation, etc. The subjects did express hope that any technical issues would be resolved in the future, such as more reliable exercise recognition and issues with the TV setup. Overall, the subjects who did use the system on regular basis provided mostly positive impressions, such as:

  • “I was excited to exercise, but should stick with every other day, I did it 2 days in a row and was sore.”

  • “I don’t exercise that much, but try to complete the video 4x/week.”

  • “I exercise right before bed, I have seen that it helps me sleep better.”

  • “Coach very encouraging and that makes me want to do it.”

  • “Good program. Instructions well done; I like the bar chart, makes me feel better to exercise and helps me see what I need to work on… Feedback could be better, feels canned. Delays and technical issues would be great if not there.”

Due to the limited space, we have shown only a small subset of the results collected during the time the participants used the exercise system. Future analyses will include comparison of the exercise performance with the clinical measures that were collected before and during the study. Since the dataset also includes raw skeletal data, we are planning to further investigate how to quantify the exercise performance in terms of standard fitness measures, such as flexibility, balance, strength, and endurance [12]. Furthermore, we will analyze the strategies that the participants used to exercise by comparing their data to the data of the coach in the video. Such temporal analysis could quantify how closely the participants were following the movement of the coach or if they have developed their own strategy for each exercise. The results of the analysis will be important for implementing a more effective feedback in the future. Furthermore, we will investigate how the exercise system could be used in a closed-loop semi-automated coaching.