Virtual Reality-Based System for Hand Rehabilitation Using an Exoskeletal Orthosis

Abstract

This research shows an alternative device in the field of fine motor rehabilitation. This device is a flexible orthosis that helps the flexion and contraction of the hand fingers using a virtual interface environment. In addition, it has been designed with the ability to perform different assistive and resistive tasks, allowing its adaptation to the recovery status of the patient. The mechatronic prototype is controlled by algorithms based on fuzzy logic that compares data from the Unity3D graphics engine and flex sensors from the device. For the correct execution of the rehabilitation tasks the proposed fuzzy algorithms have been implemented using a Raspberry Pi. The proposed system is aimed to users with deficits in fine motor skills because of tendon injuries, achieving excellent control results for an efficient execution of tasks.

1 Introduction

The upper extremities are the essential axis to perform daily activities, however, when there is some type of injury in this area, the autonomy of the person is limited making physical rehabilitation the main choice when it comes to recover mobility of damaged limbs. Traditional rehabilitation has advanced on issues that prioritize the recovery of upper extremities functionality, despite this, the application of virtual systems, prosthetics, orthosis and control systems, are rarely taken into account either by economic or technical aspects.

That is why technological advances point to the development of rehabilitation systems focused on upper extremities, specifically on hands since these are the means to acquire data from the environment that surrounds us [1, 2]. In countries such as USA, Germany, Japan, have focused on the recovery of patients who have suffered sports injuries, traffic accidents and multiple sclerosis [3, 4].

In order to obtain significant support in the rehabilitation self-management, the introduction of the term virtual reality which allows to represent scenes, objects and actions that exist in the real world in controlled digital environments is essential [5]. The main feature that enhances the use of virtual reality in physical rehabilitation systems is the use of HMD (Head-Mounted Display) devices that provide a total immersion in a three-dimensional space. The relationship that exists between virtual reality and physical rehabilitation has technical and scientific foundations that are based on motor learning since any capacity of the human being can be improved by experience and practice. On the other hand, for virtual reality to be efficient, it is necessary to implement an orthosis or prosthesis according to the patient needs; this with the purpose of employing a control system that allows to develop movements approximations to produce exact results in the rehabilitation of the patient.

Because of all these reasons, through the development of activities and exercises in digitalized environments, it is possible to optimize the functioning of physical abilities as well as perception, attention, reasoning, abstraction, memory and orientation. The results obtained with virtual rehabilitation in relation to the conventional are justified, since the tasks generated are easier, less risky, with a high degree of personalization and entertainment. Nowadays, in the market there is a great diversity of robotic devices that assist the rehabilitation therapy of the hand, however, it is important to offer systems that enable a more complete rehabilitation, i.e., not only muscle strengthening or basic movement acquisition, but an increase of his cognitive abilities of oculo-motor coordination. In addition, this seeks to make patients feel included and able to perform normal activities of their daily life, as well as motivate them to develop their therapy with greater effectiveness [6, 7].

As described above, this article proposes the development of a control system based on fuzzy logic that allows fine motor rehabilitation of the hand. For a better control of patients’ progress, an active orthosis is presented as movement support, this generates a force feedback while allowing the patient to interact with environments RGS (Rehabilitation Gaming System). The orthosis makes use of servo motors as force actuators, flexibility sensors and the Leap Motion optoelectronic device that virtualize the human hand in the programmed 3D environment.

This article is divided into 8 sections including the Introduction. The Sect. 2 shows some Related Articles. In Sect. 3, the State of the Technology is presented. The Case Study is presented in Sect. 4. The Implementation Proposal of the system is explained in Sect. 5, while the Virtual Environment is detailed in Sect. 6. In Sect. 7 the obtained Results are presented. Finally, the Conclusions and Future research are developed in Sect. 8.

2 Related Articles

This section presents the methodology with which different authors have developed studies of control systems applied to the rehabilitation of upper limbs. Holmes [5], presents a study that is based on the generation of virtual reality interfaces for arm and hand rehabilitation, in which the use of optoelectronic sensors (Leap Motion, Microsoft Kinect V2, the bracelets Myo and Oculus Rift DK1) is necessary. These, allow to capture the movements of the user and translate them into digital environments which motivate the user to participate in entertaining rehabilitation sessions. It is settled that, when using these systems of virtual reality, the user needs frequent training. Gutiérrez [8], focuses on designing an exoskeleton prosthesis for the arm, which allows three degrees of freedom and is controlled by a diffuse system. The input data of this system are position errors that are turn into qualitative terms in which each of them corresponds to a numerical range. In addition, it obtains a Pulse Width Modulation (PWM) signal as the output of the system, which is responsible for triggering the actuator elements. He affirms that the control by diffuse logic requires several technical tests to effectively meet the robustness and adaptability requirements that the user requires.

Enríquez, y Narváez [9], on the other hand, disclose the design and structure of an exoskeletal prosthesis for rehabilitation of the hand, in which an anatomical study is carried out, showing angles of flexion, extension and the laterality that the different sections of the hand have. The study comes to the conclusion that it is required that the structure fulfill its function of rehabilitating, and that the prototype must be sketched in order to analyze degrees of freedom and consider economic-flexible materials.

Finally, Andaluz [10], seeks to perform exercises to improve fine motor skills and visualize them in digital environments at the same time that signals are sent to an On-Off controller. This study provides great information in virtual rehabilitation due to the force feedback; however, the control is not intuitive and hardly adapts to the progressive movements of rehabilitation.

Once the papers presented in this section have been analyzed, it is feasible to carry out the proposed study since there is no precedent for a diffuse control system applied to the virtual rehabilitation of the hand.

3 State of Technology

3.1 Virtual Rehabilitation

Virtual reality is defined as a set of techniques and methodologies that have the purpose of recovering a lost or reduced function through advanced interfaces that allow users to interact with computer-generated three-dimensional environments in real time. In addition, virtual rehabilitation offers standardization of training protocols, and undoubtedly presents a great functional, useful and motivating profile.

The process of developing virtual environments for rehabilitation is carried out through the execution of the following aspects:

• Select the rehabilitation protocol to be executed and analyze the motion kinematics that the patient must perform.

• Collect information on mechanisms that meet the protocol’s objectives.

• Construct geometric figures based on numerical descriptions that allow the mechanism to be shaped.

• Place objects in 3D space and define focus cameras.

• Convert the mathematical information of the figures and their characteristics into screen pixels.

3.2 Physical Rehabilitation Focused on the Mobilization of the Hand

It contains aspects of balance and movement stabilization recovery. It consists in the acquisition of minor movements by light flexions of the joints of the hand and the stabilization that involves muscle strengthening through activation exercises. In a general case of rehabilitation, the exercise that the patient must execute is the stretching and contraction of the fingers as shown in Fig. 1 [11, 12].

Fig. 1.

Stretching and contraction of the fingers

It is important that the flexion angles in this type of exercise change gradually, for example; initially the angles of the metacarpophalangeal joint (MCF) must be between 0º to 30º, later they should reach 50º and finally there must be a difference greater than 50º so that it could reach the appropriate mobility value of 90º [13].

3.3 Fuzzy Control System

Expressed better as a control by means of words that are interpreted with common logic and sentences, however, the processes are measured numerically. Due to this, the variables must be adapted before entering to the controller. This process of adaptation is called fuzzification and consists in giving a degree of membership within possible expressions, consequently each value of the variables will have a higher level of belonging in one expression than in the rest of expressions. After the fuzzification, variables of a linguistic form are created. In these variables the logical relationships (IF-THEN) are applied. Therefore, once the controller interprets relationships, these are translated from linguistic to numerical expressions [14].

3.4 Hand Orthosis

It is a therapeutic device that collaborates in the fulfillment of rehabilitation objectives, allowing a control of possible medical complications. The essential objectives are stabilization, limitation of amplitude, and suppression of pain. It is important that the orthosis covers most of the limb to be treated, however, when it comes to the hand, it is necessary that both the palmar areas of the fingers and the hand remain completely free. Regarding its constitution, it is recommended the use of flexible thermoplastic materials that provide comfort and ease of movement [15]. The Fig. 2 shows the orthosis made with flexible thermoplastic materials and an armrest.

Fig. 2.

Flexible material orthosis and armrest

4 Case Study

Within the field of medicine, virtual rehabilitation has taken up a large space in the last five years. Therefore, the use of technological tools with greater flexibility has been necessary, introducing devices with a high degree of immersion for their users. In the traditional rehabilitation, the patient makes use of his senses to have a feedback of the tasks he performs, so the virtual world must emulate these situations.

In fact, if a comparison is made between conventional physical rehabilitation and rehabilitation with the use of virtual reality, it can be seen that the second one not only emulates the progress that physical rehabilitation offers but surpasses it. This new therapy has been used to optimize learning processes or relearning movement patterns in people with cerebral stroke, perceptive-motor deformation, acquired brain injury, Parkinson’s disease, orthopedic rehabilitation, balance training and tele-rehabilitation. In Ecuador, there are 202,216 people who have some type of physical disability. 4,616 people [16] from this total are located in Tungurahua. The purpose of the case study is to use a flexible hand orthosis as a means of fine motor rehabilitation for people with musculoskeletal disorders in Tungurahua.

The Leap Motion sensor is used for detection and virtualization of hand movements, allowing the patient to interact with the virtual environment. The purpose of this control system is to integrate, through the use of fuzzy logic, personalized virtual systems with external devices that modify the functional-structural aspects of the neuro-musculoskeletal system.

4.1 Rehabilitation Protocol

The proposed activities to improve fine motor skills encompass a variety of options due to the different factors involved in rehabilitation. These can be classified by the type of injury or severity: (i) Individual fingers rehabilitation, the patient must execute alternating movements of the fingers at a certain angle depending on the size of the objects programmed in the virtual environment. Once the exercise is completed (5 min), it is important to have a rest period (2 min) in order to repeat 3 times, the rehabilitation process.

It is important that the patient begins his therapy with levels of difficulty in which virtual objects demand a reduced flexion of fingers, advancing progressively with levels in which the contact with the virtual environment requires a greater angle of flexion; (ii) Collective finger rehabilitation, the patient performs opening and closing exercises of the hand simulating the grip of existing objects in a person’s daily routine. By means of the orthosis, the patient performs the exercises in sessions (6 min) with breaks of (2 min), this type of routine presents objects with variable sizes, with which the orthosis tries to reactivate the nervous memories gradually returning the flexibility and movements of the injured hand.

5 Implementation Proposal

5.1 Proposed Architecture

The system allows feedback of hand strength in patients with no or little mobility in their upper extremities. This system allows a safe, adaptable and relatively economical rehabilitation.

The system is related to the user through a bilateral communication; (i) Initially, in the graphic interface designed using the UNITY framework, the patient is informed about the objectives of the rehabilitation exercise, as well as the position and angle reference of his injured limb (h_ref, ϴ_ref). (ii) The patient generates an initial movement in the fulfillment of the preset exercise, in this step the position of the hand is measured (h_d) as well as its angle (ϴ_d). (iii) Due to the poor motility of the user fingers, it is possible that in a certain instance he will be unable to complete the rehabilitation objective, in this step the flexion of the fingers is measured using a flexion sensor, in this way the angular position and flex error (F_e, ϴ_e) of the finger are calculated. These calculated values together with the size of the virtual object, allow the fuzzy controller to govern the orthosis to progressively fulfill the proposed exercise.

The orthosis plays an important role in the system, since the forces emitted by the servomotors must be controlled in magnitude and direction to contribute to the direct rehabilitation without any collateral damage. In Fig. 3. the interaction between the patient and the proposed system is described.

Fig. 3.

Block diagram of the rehabilitation system

5.2 Proposed Hardware Platform

This research presents the ergonomic design of an orthosis prototype printed on flexible material mainly aimed to exercise upper limbs. The structure is controlled by four actuators responsible for the generation of traction force in each finger, allowing a high degree of efficiency in a semi-assisted rehabilitation. Furthermore, in order to have a better performance of the system, an armrest has been developed that together with the orthosis provides support for the actuators. Both the orthosis and the actuators share fluorocarbon filaments, invisible and resistant, useful in the transmission of force.

The proposed hardware is subdivided into five stages. (i) Virtual Interface, essentially consists of a computer that allows the user to visualize the exercises generated in the powerful graphics engine Unity3D; (ii) Control System: it is composed of the Raspberry Pi 3 microcontroller, which is responsible for processing the information sent by Unity3D and the flex sensors. This in order to perform the comparison process with the fuzzy sets and thus send the appropriate control signal to the servomotors; (iii) Leap Motion: captures the movement of the hand in inertial reference to three coordinate axes X, Y, Z. It is an optoelectronic device that captures the reflection of light generated by infrared sensors by means of two integrated cameras, storing in matrices data of the digitized image; (iv) Actuators consist of a group of servomotors HS-311 that support a current of 180 mA and can generate a torque of up to 3.0 kg/cm, it also incorporates drivers that allow to manipulate the actuators without voltage drops that could compromise the controller; (v) Power Supply consists of an external source that converts 110[v] of alternate current (AC) to 5[v] direct current (DC) with the purpose to supply the necessary voltage and current for the control system.

For a better understanding of what was expressed previously, Fig. 4 shows the diagram that constitutes the hardware of the system.

Fig. 4.

System hardware diagram

5.3 Fuzzy Control

In this section the methodology to obtain the fuzzy control design which will be implemented and programmed in the RPI card is presented. The system control starts from obtaining the transfer function of the actuators. A unipolar rectangular signal is applied to the servomotor and recorded by means of a transducer. The information obtained by the acquisition system is interpreted by the Matlab software (Control System Toolbox) in order to publicize the third-degree transfer model (1).

$$ F_{ref} \left( s \right) = \frac{\theta \left( s \right)}{{V_{\alpha } (s)}} = \frac{19630}{{S^{3} + 198s^{2} + 6280s}} $$

(1)

Once the transfer function has been obtained (1), a Fuzzy Logic Controller (FLC) is designed for a higher order system. Prior to this, the Fuzzy Inference System (FIS) is built by means of Matlab’s Logic Fuzzy Design software in which readings of the response stability of the system are performed. The basic structure of an FLC consists of three stages: merger, fuzzy inference and defuzzification. Figure 5, shows the graph generated by the angular behavior present in the servomotor before the application of a rectangular stimulus.

Fig. 5.

Actuator response to a rectangular impulse

The fuzzification stage consists of assigning data with a certain degree of membership to fuzzy sets, for this it is essential to know the system’s membership functions: (i) Entry membership function “size”, generated by emitted values due to the interaction among virtual objects, it is activated when the digital image of the hand comes into contact with the programmed environment; (ii) Entry membership function “angle”, it is generated by the resultant values of the analog-digital conversion of the flex transducer which sets the angular position of the fingers; (iii) Output function “degrees” conformed by the values ​​included in the operating range of the servomotor from 0 to 200°.

The final stage occurs in the defuzzification or decision making of the system, these decisions are obtained by the centroid method that is simplified by solving the Eq. (2).

$$ Cog = \frac{{\mathop \smallint \nolimits_{a}^{b} F(x)xdx}}{{\mathop \smallint \nolimits_{a}^{b} F(x)x}} $$

(2)

By means of the Center of gravity (Cog) of the figures formed by the fuzzy sets, the system begins to make decisions in compliance with the control rules.

6 Virtual Environment

It is the work space where environments composed of collision regions in a real environment are operated. It is ideal for rehabilitation due to its innovation, interaction with the patient, as well as its flexibility and personalization. In the Unity3D environment, the movements obtained by the Leap Motion Sensor are displayed in real time. In the virtual context the system consists of three stages as shown in Fig. 6.

Fig. 6.

Virtual operation scheme

The proposed phases are: (i) Input Peripherals: allow control of the closed loop between the patient and the virtual environment, using the Leap Motion device to sense the vectorized position of the hand. In addition, the option of using HMD, Oculus Rift is proposed, this device will allow a total immersion of the patient; (ii) Interface module develops different and novel virtual environments in order that the patient can interact efficiently with their rehabilitation protocol, and (iii) Output Peripherals: require communication ports and advanced programming to generate a software link between the actuators of the orthosis and the virtual environment, as well as the graphic response obtained with the application of an HMD. The interaction with the virtual environment is carried out with the Leap Motion device which allows the digitization of strategic points of the hand. With the use of the orthosis the behavior of these points is verified when performing activities of opening-closing and individual movements of the fingers. Figure 7 shows the capture of the strategic points of the hand by the Leap Motion sensor in full execution of rehabilitation tasks.

Fig. 7.

Virtual rehabilitation exercises.

Through virtual environments RGS, personalized virtual training programs and rehabilitation protocols have been designed aiming the individual and collective recovery of the fingers. Figure 8 shows examples of the proposed RGS environments in this system.

Fig. 8.

RGS environments for individual and collective rehabilitation of the fingers.

7 Experimental Results

In patients with fine motor lesions, the prototype served as a significant support for the recovery of muscle memory. To verify the functioning of the system, the orthosis was applied without the help of the control system, this in order to visualize the mobility of a patient with slight injuries of the hand. In the graphic engine, when a small object is visualized, the patient needs to execute a 90˚ movement to complete the exercise, however, due to his low mobility, the task will not be completed, generating an error between the desired position and the ideal goal of rehabilitation as shown in Fig. 9.

Fig. 9.

Rehabilitation protocol without intervention of the control system.

Subsequently, the patient performs the same rehabilitation exercise, but this time supported by the control system. As depicted in Fig. 10, there is a point at which the patient fulfills his goal through the actuators implemented in the orthosis, i.e., the control system detects when the user is unable to move forward with the exercise, activating the actuators with the necessary force feedback to complete the established protocol.

Fig. 10.

Rehabilitation protocol with intervention of the control system

8 Conclusions and Future Work

The proposed system presents an efficient alternative for the rehabilitation of the hand through an orthosis that allows to execute a series of exercises that support the increase of performance in the active mobility of the hand. The flexible structural design of the orthosis was adapted to different patients in terms of the size of its limb. Fuzzy logic adapts excellently to systems with several inputs and several outputs, showing robustness and adaptability to the variation of parameters such as the angles of the fingers and the size of virtual objects. The use of virtual environments for rehabilitation with force feedback, introduces an innovative way to improve motivation, concentration and physical efficiency.

For future research, it is proposed to improve the visual interface with augmented reality implementation, as well as to modify the orthosis in order to apply the rehabilitation protocol to the wrist and arm of the patient.