Development of IoT Robotic Devices for Elderly Care to Measure Daily Activities

Abstract

Various robotic devices for elderly care have been developed and commercialized in Japan. However, the introduction of such devices to the society has not been enough. One of the major reasons for this is that it is still not clear how we can best make use of them. In other words, we still do not have sufficient knowledge and evidences of such devices in the viewpoint of the benefit. Therefore, we started a project in which IoT (sensing and data communication) functions are embedded in the robotic devices to measure the activity of the users; how, when, where they are utilized by whom. It is also important to record the activities of caregivers. By utilizing developed devices, the activity data of elderly people in care facilities can be collected and analyzed in order to investigate the effective ways of device utilization. We are also using the receipt data of long-term care insurance to investigate the effect of welfare devices (including some of the robotic devices) covered by the insurance. Quantitative benefits of utilizing welfare devices in the viewpoint of the outcome in care will be shown.

1 Introduction

In Japan, more than 25% of the population is over the age of 65. The population of other advanced countries is also rapidly aging [1]. In order to solve such a social problem by improving the QoL (Quality of Life) of the elderly persons, and reducing the workload of the care givers, the robotic devices for elderly care have been intensively developed [2, 3]. However, these robotic devices have not become common in the care facilities yet. One of the major reasons is that the evaluation of the benefits insufficient and it is not clear how we can best use of the devices. To clarify the benefit of robotic devices, it is necessary to collect the usage data of devices and quantify its effect on the elderly care. Therefore, we started a project on “measurement, analysis and intervention technology of functioning of the elderly using care robots as probes” funded by NEDO (New Energy and Industrial Technology Development) in 2017.

2 Concept and Outline of the Project

2.1 Measurement of Daily Activities of the Elderly

In this project, IoT (sensing and data communication) functions are embedded in the robotic devices to measure the activity of the users [4]; how, when, where they are utilized by whom. This means that robotic devices work as assistive devices for elderly people and works as sensing probes for daily activities of the elderly people at the same time. The basic concept is shown in Figs. 1 and 2. We utilize off-the-shelf robotic devices for elderly care such as transfer aids, mobility aids which are commercialized from our previous project with companies. We additionally install sensing and data communication functions. Such as transfer aids, mobility aids in order for such devices to be able to collect activity data of daily living. The collected data can be analyzed and utilized for the feedback to realize better usage of the device for individuals, and to realize better function of the devices.

Fig. 1.

IoT robotic devices for elderly care

Fig. 2.

Concept of IoT robotic device

2.2 Measurement of Care Activities of Care Staffs

It is also important to know what kind of care activities are given by care staffs. We thus developed a wearable-type sensing device to measure the care activities of the caregivers working at care facilities. An “e-skin” by made by Xenoma in which multiple strain sensors are embedded was utilized to measure many actions. Then the deep neural network was utilized to build a function to estimate various postures and actions.

2.3 Analysis of Usage Data of Welfare Devices

As big data of device utilization of welfare devices such as wheelchairs and rollators, we are also collecting and analyzing Japanese long-term care insurance receipt data. We applied for the secondary use of long-term care nationwide receipt to the Ministry of Health, Labor and Welfare, and obtained the receipt data from FY2006 to FY2016. This data includes service utilization of long-term care insurance services together with attributes of the users such as region, age, gender, care level, with anonymized user ID. Each usage information on the rental services of welfare devices (including some of the robotic devices for elderly care) by a user per month corresponds to a receipt in the data. The nationwide macro trend such as distribution of the length of using welfare devices can be analyzed based on the data.

3 Development of IoT Robotic Devices

3.1 Development of IoT Assistive Walker

We added logger functions to the assistive walker, RT.1 and RT.2 (RT.Works co., ltd.). The log of walking will be sent to the cloud server system. We have constructed five RT.1 and five RT.2 with the logger function. The information which is sent to the cloud server system is as follows:

• Grip force at a handle (RT.1 only),

• Remaining battery capacity,

• Acceleration (front-back, right-left),

• Angular velocity,

• Velocity,

• Motor current,

• Position (longitude and latitude),

• Total moving distance,

• Number of steps user walked,

• Moving periods per minute.

In order to visualize the collected data from the robot, we developed walking map viewer which shows the recorded data during walking and the walking route. The screen capture image is shown in Fig. 3.

Fig. 3.

Walking map viewer

3.2 Development of IoT Rise Assisting Robot

Resyone Plus is the rise assisting robotic bed made by Panasonic corporation [5]. The half of the bed can transform into a wheelchair by pressing a button on a controller. By utilizing the device, one caregiver can let a person on a bed to depart from the room without making a transfer a wheelchair, even if the person is in rather severe condition. It was modified to have logging function to know how often the device was actually utilized. The operation logs (back rising, foot rising, height control and transformation into wheelchair) together with user’s activities (heartbeat, breath, body motion on the bed) and the location of the wheelchair (beacon location) can be measured.

3.3 Development of IoT Watch over Sensor

The silhouette watch-over sensor by KING TSUSHIN KOGYO CO., LTD. was modified to have logging function. It can record care receiver’s status data (such as lying on the bed, rinsing up, departure from the bed) and their message history and the silhouette image on the local server in the care facility. Then the local server can send the data periodically to the outside cloud server. An application software for data analysis running on a PC was developed to visualize the users’ life patterns such as wake-up time, bed time and sleeping hours from the collected data. By this additional function, the system was extended from a safety monitoring sensor to a lifelog system.

3.4 Development of IoT Rise Assisting Robot

The rise assisting robot, Hug T1-01 by Fuji Corporation, is a commercially available robotic device for assisting rising and transferring from a bed to a wheelchair etc. without manual lifting by the caregiver. We have added a logging system to Hug to record the operation log. The logging board is installed on the main control board of the original system, and the operation log is recorded on the SD card and sent to the cloud server.

3.5 Development of General-Purpose Logger for Robotic and Welfare Devices

In order to collect log data from robotic devices which cannot be modified to embed logging function inside, a universal data logger system was developed. The logger system consists of main control board and various sensor modules. The logger is attached to a robotic device externally, and measures various information depending on the purpose. It can also be utilized to collect log data from conventional welfare and assistive devices such as hoist and wheelchair. Various sensor modules are provided and can be selectively connected to the main board was shown in Fig. 4. It can upload the measured data to the cloud server via WiFi mobile router.

Fig. 4.

General-purpose logger.

The main board is RaspberryPi 3 and the Linux runs on the board. Each sensor module consists of a compact processor (nRF52 Bluefruit LE) and a set of sensors. We have developed five types of sensor modules as follows:

• A vital data measurement module catches vital signals from a human body,

• A motion data measurement module measures human body movement,

• A localization module measures position in outdoor environment,

• A chemical data measurement module measures chemical information related with excretion,

• A communication data measurement module measures communication information with social robots.

The outdoor localization module include a GPS board connected via UART to main module to measure data. Communication speed is 9600 bps. The data is sent every 1 s. GPS sensor outputs location data in NMEA format and the current position in longitude and latitude can be calculated from the data.

3.6 Development of Cloud Server System for IoT Robotic Devices

We have developed the recording system of IoT robotic devices for elderly care to the cloud server system. This function is extension of the care recording system for the care givers with smart phone which was also developed at AIST. The system application which involves the initial registration for the system operation and management of record on a personal computer is implemented as a web application and controlled via web browser on a personal computer. The IoT robotic devices which communicate with the cloud server has a special SIM card provided by SORACOM corporation. The data communication is made with the mobile phone line which is directory connected with AWS (Amazon Web Service), thus without using the internet. Both of the ID/PASSWORD and the special SIM card which consist of the destination and authentication information are required during the connection. In addition, the data communication between each robotic device and the server is encrypted by HTTPS protocol end to end. This safety process achieves secure data connection and wiretapping is impossible. Figure 5 shows the outline of the system for data collection.

Fig. 5.

Outline of the data collection scheme.

4 Experiment of Activity Recognition for the Elderly

4.1 Estimation of Walking Pattern from Log Data from Assistive Walker

We have conducted preliminary experiments to estimate and classify the body functions of the user, activities of the user, and the environment based on the data collected from robotic devices.

In order to investigate the method to estimate the body function of a user of the assistive walker, an experiment was performed using RT.2 by RT.works. Participants were healthy three persons in their 40’s and 50’s. They performed three walking patterns: Normal smooth walking, shuffle walking and limp walking. The measured sensor data collected in the experiment was acceleration from embedded sensor in RT.2. Figure 6 shows is the experimental environment. A slope (up and down) exists in the middle of the course.

Fig. 6.

Experimental environment for walking pattern recognition.

From the sensor data collected in the course, the walking motion was classified into three patterns. The sensor value was rather vibrational, thus it was integrated into velocity, and it was classified by multi class SVM. The result is shown in Fig. 7. The upper is normal smooth walking, middle is shuffle walking and the lower is limp walking. The horizontal axis shows the estimated result. Three bars in each graph corresponds to the normal waling (left), the shuffle waling (middle), and the limp walking (right) respectively. From the result, the recognition of normal walking was quite successful, while discrimination of shuffling and limping walking has rather lower success rate.

Fig. 7.

Result of classification of walking pattern by SVM (upper: smooth walking, middle: shuffling, lower: limping)

4.2 Estimation of Activities from General-Purpose Logger

Graphs in Fig. 8 show examples of measured data in preliminary experiment with the developed logger. The cloud server can receive the data and stores it in the SQL database. The data was sent in JSON format. In the experiment, a vital data measurement module, a motion data measurement module and a localization module are utilized, and the data was stored every 10 s.

Fig. 8.

Examples of measured data using general-purpose logger.

The subject in the experiment started walking from the 4th floor of our research center, walked down through stairs to the 1st floor and went outside. Then he took a rest for a few minutes and walked into a different entrance of the same building. Then he walked back to the initial room through the stairs. Upper left graph in Fig. 8 shows the heartbeat. Middle left graph shows the sweating (humidity). Lower left graph shows the angular velocity. Upper right graph shows the longitude. Middle right graph shows the latitude. Lower right shows the trajectory at vertical longitude and horizontal latitude. Between 200 s. and 500 s. the longitude and the latitude are taken. The participant seems to go out at that time. Between 300 s. and 400 s. the heartbeat and longitude and the latitude match well with each other. Then it can be seen from the graph that he took a rest. After 550 s. the angular velocity increases rapidly, which means that he climbed the staircase quickly. The heartbeat increased at the same time. As a result, it was shown that users’ motion and situation can be measured simultaneously with this logger, and that we can estimate the users activities with general-purpose while using the robotic devices.

5 Development of Measurement Method for Care Activities of Caregiver

5.1 Hardware System

The physical burden of caregivers is an important outcome index of care activities. To observe the caregivers’ behavior changes with the robotic devices, we developed a method based on a motion measurement suit (e-skin) made by Xenoma.

The e-skin is a cloth with strain sensors. Only thin and soft materials are used to construct the suit, therefore a caregiver will be able to perform daily task as usual. There are 14 strain sensors on the breast, back, shoulder, upper-arm and lower-arm as shown in Fig. 9. There is a hub device at the breast for transmitting measured data via Bluetooth to external device such as a smartphone. The hub includes 3-axis acceleration sensor and 3-axis angular velocity sensor.

Fig. 9.

Hardware configuration of e-skin sensor suit

The characteristic of strain sensors utilized in the sensor suit is non-linear and has large hysteresis. Figure 10 indicates the relationship between the sensor value and the joint angle. Therefore, simple estimation of the joint angles from the sensor data will have measurement errors of more than 10°. To solve this problem, we adopt convolutional neural network (CNN). The regression model is derived by the CNN between 14 sensors data and joint angle vector. Figure 11 indicates the proposed calculation model.

Fig. 10.

Characteristics of a strain sensor

Fig. 11.

Proposed method for estimating joint angles.

5.2 Algorithm for Pose Estimation

We performed the data collection and evaluation experiment to estimate joint angles using the proposed method. As the ground truth of the whole-body joint angles, joint angles measured by MVN motion capture system (Xsens Technologies B.V.) was utilized. The position of IMU sensor units on the sensor suite is shown in Fig. 12.

Fig. 12.

Position of IMU sensors of MVN motion capture system (left), and definition of joint angles for estimation (right)

The estimated joint angles are Roll(x), Pitch(z) and Yaw(y) of the shoulders and bending angles at elbow joints in both arms. Therefore, 8 degrees of freedom. Three adult males participated in the experiment. Measured data of two subjects (A, B) were utilized as teaching data. The third data taken from participant (C) was utilized as test data. We evaluate the estimated joint angles. Measured motions are elbow’s flexion and extension, arm up-down in the sagittal plane, arm up-down in the frontal plane, and random motion. With the MATLAB neural Network Toolbox, we derived CNN estimator to investigate the learning convergence.

Figure 13 show the learning curves with different parameters for data length and different layer number. The learning curve does not change for the data length over 50. Therefore, the length of the data utilized for the estimation was set as 50. In the same manner the number of layers was determined as 40. We construct the estimator with above parameters.

Fig. 13.

Learning curves with different data length (left) and different layer number (right)

The estimated joint angles are shown in Fig. 14. The blue lines indicate estimated values and the red lines indicate measured values. From these graphs, it was confirmed that it is possible to estimate the posture of the person wearing the sensor suit. The list of root-mean-square error of the estimated and measured joint angle shown in Table 1. Right side of the table shows average errors. The errors for X axis of the shoulder is the smallest and the errors for Z axis is the largest among estimated joint angles.

Fig. 14.

Estimated joint angles in elbow joint and shoulder joint. (Color figure online)

Table 1. Root-mean-square error of joint angle estimation.

5.3 Experiment of Motion Estimation of Caregiver

Figure 15 shows the result of the experimental result of the motion estimation of caregiving motion. The subject playing a role of a caregiver wearing the sensor suit performed the transfer motion of a care receiver from a bed to a wheelchair. The motion includes three tasks as follows: (1) raising the body of the care receiver on the bed, (2) setting the body of the care receiver at the side edge of the bed, and (3) lifting and transferring the body of the care receiver from the bed to the wheelchair. The estimated pose of the subject was visualized utilizing Unity graphics engine with Kyle model for human modeling.

Fig. 15.

Estimated joint angles and posed for transferring motion in care.

The estimated poses of the caregiver shown in Fig. 15 are not as accurate as the motion capture system, but we regard it to be sufficiently accurate for estimating and recognizing the care activities in the care domain. As the next step, we are planning to work on the automatic recognition of care activities from this information.

6 Analysis of Utilization of Assistive Device in Long-Term Care Insurance Receipt Data

6.1 Long-Term Care Insurance

In Japan, universal health coverage for long-term care was introduced in 2000 under the Long-Term Care Insurance (LTCI) system [6]. People aged ≥65 years old are entitled to receive long-term care services at home or in facilities, irrespective of income level and availability of family caregiving. Under the permission from the Ministry of Health, Labor and Welfare (MHLW) for the secondary use of the receipt data for research, we accessed the nationwide receipt data from FY2006 to FY2015. The data includes all care payment statement for preventive care service, daily life support service, and institutional care service. Each receipt corresponds to monthly payment of the care service utilized with the attribute information of the user such as anonymized ID, living area, age, gender, care level.

6.2 Analysis of Utilization of Assistive Devices Under Long-Term Care Insurance

The collected care insurance receipt data includes information about rental service for assistive devices such as care beds, wheelchairs. Recently, some of the robotic devices for elderly care have been covered by the long-term care are insurance. parts of the care equipment. With the receipt data, we are investigating the method to calculate the indices for the outcome in care domain related with the utilization of assistive devices such as the change of care level, the period of living at home, and cost for the care service.

The left graph in Fig. 16 indicates the change of the care level of rollator users in care level 2. Typically, an elderly person in care level 2 has difficulties in rising and gait, and partial or complete support is needed in toileting, bathing. The right graph corresponds to the elderly people who did not use any assistive device but use other services in long-term care insurance. From this figure, it can be noticed that long-term rollator users tend to keep their care level, and the difference becomes larger after years. Eight years after the first use of the device, the ratio of people getting worse in care level is 35% for rollator users, while that for non-device users is 70%. The ratio of getting better in care level is also higher for rollator users.

Fig. 16.

Effect of long-term rollator utilization on the change of care level.

We also have investigated about the duration of care device utilization as shown in Fig. 17. As a result, it was found that about the half of the hoist users kept using the device for more than five months, while more than half of automatic excretion device (for both of urine and feces) kept stopped utilization in two months. This means, the hoist is much more useful than automatic excretion device for keeping the elderly people to stay and live in their own home.

Fig. 17.

Period of use of care devices.

7 Conclusion

We have developed IoT robotic devices for elderly care which have additional function to measure the activity of the user and to send the data to cloud server. We have also developed a sensor suit to measure the activities of the caregivers. We are now collecting the activity data in care facilities. We have also collected “big data” of long-term care insurance receipt and analyzed the usage of various welfare devices (including some of the robotic devices) covered by the insurance. Quantitative benefits of utilizing the robotic devices and welfare devices in the viewpoint of the outcome in care will be shown in the future.