Keywords

1 Introduction

Stroke is one of the major worldwide causes of human disability leading to sensory, cognitive and motor impairment as well as to difficulties in social involvement or in communication activities that imply social constraints. These directly affect the quality of life, making the creation of alternative, faster and more effective ways of rehabilitation mandatory. Stroke recovery though constitutes a complex, challenging problem of critical importance, demanding new approaches that will provide more effective rehabilitation programs, taking into account all the behavioral, cognitive and motor parameters. The main purpose of rehabilitation is to enhance experience-dependent neuroplasticity, which refers to the ability of the central nervous system to be reorganized and remodeled, particularly after central nervous system injury or a specific type of lesion, influence motor learning, which refers to a change in the capability of responding [21] and augment the functional muscles’ output. This reveals that recovery concerns a combination of instinctive and learning-dependent processes that constitute a cyclical process, involving assessment for identification of patient’s need, setting of realistic and feasible goals, intervention to serve the goals and reassessment to evaluate the efficiency of the selected program [13]. Although these principles have been identified to be able to ensure the fulfilment of patient’s requirements to the greatest possible extent, the need of therapy-based rehabilitation at home remains a challenging and difficult problem to cope with.

Taking these aspects into account, several novel therapies have been developed and tested, including Virtual Reality (VR) and novel robotic techniques. VR technology, with the multi-sensory training, that it can support, has the potential to meet the standards of the new rehabilitation methods’ need and provide an enhanced environment in which people with stroke can improve their abilities in order to facilitate their lives long-term. Moreover, the possibility of mirroring the non-affected limb, which allows the application of Mirror Therapy (MT), which is based on the illusion that one action is performed normally, no matter what the condition is [6], complements the above.

1.1 State-of-the-Art

VR technology, as an interactive computer gaming technology, works on the principle of providing the patients with an interactive user interface and implementing different task-oriented scenarios. Greenleaf and Tovar were the first who tested VR in the rehabilitation field concerning human motion and motor control [8]. Until now there are more than 40 studies involving more than 1000 patients referring to upper extremity impairment [4, 5, 16, 21], lower extremity impairment [11, 14] and gait and balance impairment [3, 10]. We need to mention that the majority of the studies concerning the integration of VR platforms in rehabilitation, combine the VR training with the conventional therapy as the aim is to complement and, if possible, to accelerate the rehabilitation process and not to replace the conventional methods. Studies that have tried to use only VR training have concluded that they didn’t receive the expected positive feedback and results from the patients [15]. However, the majority of studies concerns motor disabilities and the up-to-date literature has failed to provide sufficient information concerning possible effect sizes on cognitive processes [13]. There is a lack of therapeutic systems concerning cognitive deficits or the complete combination of movement and cognitive ones. Mirelman et al. [14], however, were the first who tried to use a VR system for retraining and improving both motor and cognitive functions in patients with Parkinson’s disease who appeared, at the end, to be positively affected by this kind of intervention. Thus, our platform will continue this effort by providing a possible movement and cognitive training.

Used tracking devices may vary according to the needs of the examined scenarios and the abilities of each of the patients. Up-to-date technology permits the choice among several of such devices that will allow the satisfaction of patients’ needs with the use of existing design-for-fun games or design-for-rehabilitation ones. These devices concern Nintendo, Sony and Microsoft products, such as Wii remotes [1], Sony Eye Toy [17], Xbox Kinect [18] and Microsoft Kinect [7]. We chose Leap Motion as it has not been tested for therapeutical reasons although it constitutes a low-cost, easily accessible tool, especially for rehabilitation at home. Very recently, its use was noticed for the first time by Vourvopoulos et al. [22], who combined the technology of VR with this of Motor Imagery for patients with stroke.

Another way, but not so usually used, to complement and even complete the rehabilitation process is the Music Therapy (MT). MT extends from music listening and musical instrument playing to Neurologic Music Therapy (NMT), which constitutes a new neuroscience-based model of music perception and production and examines the possible influence of music in brain structure and function as well as behavioral [20]. Music production, compared to music listening, is not so commonly used for rehabilitation purposes although music production activates regions like cerebellum and basal ganglia that are responsible for movement and accurate [23]. However, therapists have begun to trust some of the new techniques supported by the NMT, specialized mainly for arm rehabilitation like the Patterned Sensory Enhancement (PSE) and Therapeutic Instrumental Playing (TIMP). A research group innovatively published recently a study protocol with which they want to examine if TIMP technique would be indeed useful and efficient for stroke patients, being in the chronic phase of the stroke, who continue their rehabilitation program at home [19].

Research activity, based on the up-to-date bibliography, reveals the need for a complete rehabilitation program referring to behavior, cognitive and motor impairments of patients with stroke, in order to improve the overall quality of life both for patients and their families. This intervention will rely on VR technology consisting of well-designed gaming and evaluation trials and Brain-Computer Interfaces (BCIs) introducing a novel augmented human performance platform based on different kinds of feedback like musical, specific rhythmic-based sensory feedback. In order to take advantage of the benefits of neuroplasticity, this system will be able to be used as soon as the condition of the patient allows it. To the best of our knowledge, no such VR platform for rehabilitation, featuring the sense of Presence and the interactive VE combined with the benefits of music, exist up to date in the field of stroke patient rehabilitation. Thus, the primary aim of this research is to answer the following research question: Can NMT be more effective if provided through a VE? Are stroke rehabilitation and motor ability improvement more efficiently, achieved via a novel approach based on a VR-induced NMT methodology that can support the self-efficacy of a patient? This paper constitutes mainly a methodological paper, aiming to present the idea and the integration of the platform as well as the upcoming steps (Fig. 1).

Fig. 1.
figure 1

Overview of our VR musical platform

Full size image

The majority of the used VR rehabilitation platforms are based on commercial gaming (CG) and there are only a few studies examining the differences between VE and CG without being able to draw a substantial conclusion. This implementation though seems a bit problematic as most of such commercial games have been investigated as tools for learning motor skills or for improving cardiovascular fitness. Thus, we chose to develop a new, authentic scenario in order to be able to handle all the necessary parameters concerning the needs of patients, their close environment, and their rehabilitation team. An authentic VR system can offer clinicians the ability to control and grade tasks in order to challenge the user, ensuring an adequate interactive, motivating environment and incorporating multimodal feedback provided in real time and in an environment totally guided. According to our opinion, commercial movement games, require increased abilities and capacities from the patients, which means that they can be usually used more effectively at the late chronic stage of the condition, integrating patients into everyday activities. These games are considered inappropriate to be applied in the early stages of education, as they require well-coordinated, ready-to-answer and instant response, for participating actively in the game. Therefore, they cannot be compared to activities depicted in VR “games”, the main purpose of which is to guide and, therefore, to educate the perception (the sensory feeling of the motion) of motor activity. The aim of a well specifically designed VR system is to bring the complexity of the physical world into a controlled environment [12] where simple actions can be part of a larger functional activity that has an escalating difficulty for more complicated motor activity.

2 Our Method

Our aim is to provide an alternative and efficient way to improve the fine movements of the paretic hand in stroke, patients. We track simple hand movements, used in everyday life, adapting them to musical exercises by means of virtually playing musical instruments regardless of the actual abilities of each patient. The Virtual Environment (VE) allows patients to play instruments which their actual abilities would otherwise, in a Physical Environment (PE), restrict them from playing. Combining Presence Positive Technologies for Well-being and techniques of Neurologic Music Therapy, we have developed a VR game using an HMD Oculus Rift for securing the sense of presence and a Leap Motion to track fingers’ and hands’ motion. For the adequate movements that will constitute the proper exercises, a team of doctors of KAT hospital in Athens has advised us and aided us in our qualitative clinical study. KAT hospital is one of the biggest hospitals in Attica, oriented in orthopedics and trauma and considered as the hospital for accidents and fractures (Fig. 2).

Fig. 2.
figure 2

The equipment needed for our platform. We used the most accessible VR technology in order to create a low-cost platform, easily reachable by patients. They wear an Oculus Rift headset, have their hands tracked by a LeapMotion controller and they can hear the music through a set of desktop speakers.

Full size image

2.1 The Game

We have created a VR system with a very easy-to-read scenario of an extraterrestrial being, Orpheus, who approaches friendly the patient and transports him/her to his planet, Antiphon, in order to play music together. For the patient’s participation, simple and basic movements were selected, which constitute the background of the motor and functional pattern of the hand. Orpheus first performs these movements, in order to invite the patient to imitate them. This concept works like a stimulus-mirror of a feedback signal, facilitating the retraining of affected motor control of patients with stroke or other neurological lesions.

The environment and the setup of the game have been carried out in Unreal Engine 4.0 (UE4) due to the possibilities offered and its ease of use. In order to support the illusion of a different musical planet and the general space attribute, we used particles like musical notes that were coming out of the bushes and musical keys falling from the trees instead of leaves.

Our character, Orpheus, was designed, rigged and animated in Maya 2016. We chose a cartoon-based model in order to avoid any discomfort that a human-like one could cause. We modified also his voice with a cartoon-based effect to create a bigger familiarity with the patient (Fig. 3).

Fig. 3.
figure 3

Left: Orpheus explains the procedure to patients and gets ready to play the first instrument. Right: Panoramic shot of Antiphon: the environment where the game takes place

Full size image

The purpose of the game is the proper execution of four pre-selected exercises; one intro exercise, which presents the salutation between Orpheus and the patients, and three more exercises concerning the fine movement of their hand. The movements selected were focused on the extremity of the hand, wrist, and fingers, referring to basic and simple movements which constitute the background of the motor and functional pattern of the hand.

Minutely, there was a responding greeting to the appearance of the Orpheus, referring to the extent of the wrist and the fingers, which is followed by a gentle bending of only the phalanges of the fingers. This is the only movement that did not follow any music.

The second movement includes the flexion and extension of the wrist with the fingers in a neutral position, following the basic movement of the wrist. This movement requires the stabilization of the forearm and arm so that the movement of the wrist is accurately performed and excited in this particular context. The music that accompanied this movement was the 4/4 tango, written in 1935 by Carlos Gardel, “Por Una Cabeza”. Each correctly executed patient’s movement causes the production of percussion sound. Orpheus, as well as each patient, have in front of them one set of wooden bongos. The patient could recognize the right moment for him/her to play, as his musical instrument was changing color (it was turned into green) in order to give a proper feedback and guidance. At the end of each attempt, there was a “stick and carrot” type approach using a visual (green or red musical note) and audio (Orpheus giving reward or prompting the patient to try again) signal that verified the quality and efficiency of the movement’s execution. The musical pieces chosen for each exercise, have an intense and clear rhythm in order to support patients’ movement and their style was adequately selected to fit with the instrument’s choice. Moreover, these pieces were mainly well-known so that can make patients feel familiar with the process and enjoy it. Our second step is to use music personally chosen by each patient so that we can have a customized game able to maximize the familiarity factor and consequently trigger even more brain areas (Fig. 4).

Fig. 4.
figure 4

Exercise 1. Orpheus introduced patients to bongos playing. Patients have to use Orpheus’ movement as a mirror and repeat it. Patients also know when it is the right moment for them to play, as their musical instrument changes color (it is turned into green) in order to give a proper feedback and guidance. At the end of each attempt, there is a “stick and carrot” type approach using a visual (green note for the correct and red note for the false attempt) and audio (Orpheus giving reward or prompting the patient to try again) signals that verified the quality and efficiency of the execution of the movement. (Color figure online)

Full size image

The third movement includes the pronation and supination of the forearm with the movement concentrating on the area of the wrist and forearm, requiring also the stabilization of the shoulder and the elbow. The music that accompanies this movement is the 4/4 piece, written by Peter Green in 1968 and known by Santana, “Black Magic Woman”. The patient with his movements plays the cymbals.

The fourth and last movement concerns the opposition of the fingers with the thumb. It begins with the thumb-index, thumb-middle finger, thumb-ring finger and thumb-pinky finger opposition. All these are considered to be the most difficult movement of the hand, which plays an essential role in the functional performance, because it is related to its grasping ability, and of course it requires high hand stability. The music that accompanies this movement is the 5/4 jazz composition of Paul Desmond, which was recorded by David Brubeck Quartet in 1959, “Take Five”. The patients were encouraged to play the castanets.

During the gaming process, the motor actions follow a certain order, always the same one for everyone, while offering to the patients the opportunity to try to imitate the Orpheus motor inducement. Subsequently, the patient can choose from the intro menu which exercise wants to repeat and at which level, as each exercise has 3 level of difficulty. The sound signals produced by the music and the patient’s ability to be involved in music produced by himself (as a sensory information stimulus) offer a very important motivation and interest to complete the motor effort (Table 1).

Table 1. Description of exercises used
Full size table

The above movements were combined with specific neurologic music therapy techniques, such as Therapeutic Instrumental Playing (ΤΙΜP) and Patterned Sensory Enhancement (PSE). ΤΙΜP maps functional movements into musical instruments (mostly percussion, keyboard, or digital sound surfaces) and engages the patient in practicing motions repetitively and cyclically via musical instruments. PSE regulates functional movement by translating the temporal, spatial, and force-dynamic components of movement kinematics into sound patterns, and then plays them back to provide feedback and feedforward regulation for enhanced motor control.

We chose to implement these techniques through a VE in order to provide patients with opportunities that they may not be able to physically have otherwise. This enabled them to play the instruments we selected not in the traditional way but in a more simple and easy way, assuring that it is still interesting for them. We chose musical pieces with intense and clear rhythm in order to support their movement. Moreover, these pieces were mainly well-known so that can make patients feel familiar with the process and enjoy it.

For the hand motion detection, Leap Motion V3 ORION was preferred. We used getnamo’s plugin for the UE4 in order to integrate leap motion SDK with UE4. For the detection of each gesture, simple state machines were implemented. Each gesture was broken down into steps (states) that we wanted the user to complete in order and each step was defined by flexible rules which the user had to meet in order to advance. After a step was completed our gesture detection library was attempting to detect the next one. The above process had to be done in a certain time window, in case the user was not able to finish before the time runoff, the exercise was marked as a failure. Finally, when all the steps were completed or the time window was over, the user started again from the beginning until he completed a certain number of repetitions.

2.2 The Participants

After the completion of the platform, patients for assessing its efficacy were selected. Thus, at this first level of our preliminary study, we applied our platform to three patients (two females and one male), aged from 32 to 52 years old and suffered from neurological damage from a motor control lesion. None of the participants mentioned vision or other health problems. Moreover, none of them had more than basic computer skills and they reported a low frequency of computer use.

The first two cases refer to central nervous system damage, mainly to the brain, which presented with hemiplegia, where the patient is aware of what to do but cannot perform any action because the control, carried by the central structures for motion action to the periphery, has been affected. This means that there is no coordination and collaboration between responsible muscle groups. In this case, the “periphery” is intact and is expected to provide the appropriate information (feedback) signals to use them in the process of motor education and re-training. The main objective of this training is to use the VE as a main tool, providing a proper sensory stimulus to the process of motor control retraining. On the other hand, the third case is a clear damage of the executive mechanism (the peripheral nervous system and the muscles) where, while the motor control mechanism is not affected, the patient is unable to perform motor attacks due to muscle weakness. In this case, the justification of education through the virtual environment is based on the possible ability to enhance muscle effort through the sensory feedback she receives after every effort. It works like the technique of “biofeedback”, with the difference that the internal signal for the hidden motor activity is taken from the virtual-mirror motor action and patient’s transportation in a well-organized environment (Fig. 5).

Fig. 5.
figure 5

Patient A, woman 56 years old, suffering from right hemiplegia due to stroke, performs the program of VRehaM. This patient prefers to follow the program in standing position.

Full size image

All patients, however, had overtaken the acute stage of their condition and were in the stage of re-education and re-training. They did not have any other limiting factors that could affect their basic clinical picture or that could possibly have made it difficult for them to follow such programs of a mental nature. It should be mentioned also that all patients were minutely informed of the procedure.

2.3 Integration of VRehaM

The integration of our platform was performed in the Occupational Therapy department of the Physical and Rehabilitation Medicine (PRM) Department of KAT hospital of Athens, and it was performed within the framework of the daily therapeutic program of the particular patients. Patients tried the platform twice a week for two weeks. The duration of the intervention depended on the exercises that the patient wanted to repeat. The average duration was 15–20 min per session. The room was totally quiet, it was illuminated by daylight without intense brightness and there was no pictures or other evidence on the wall or in the environment that could draw the attention of the patient from the beginning. For this reason, also, there was no other person inside the treatment room, except the therapists and the program manager. Patients A and C were in-patients, in the clinic of PRM department, and patient B was discharged and taken his therapy at the out-patient basis.

At the end and at the beginning of the procedure, patients were persuaded to complete a questionnaire concerning stress and self-efficacy in order to assess the motivation provided by our platform as well as its efficiency. For the optimal validity of our results, we used also a Control Group for the questionnaire, consisting of 3 more patients that followed their conventional therapy without having the experience of VRehaM.

This preliminary study constitutes our platform’s first application in order to be tested and consequently, reassessed based on patients’ need. Our main purpose is to examine how patients would react to such a platform, firstly in terms of psychological factors that can ensure a higher motivation and therefore a bigger consistency in the exercises’ program. Moreover, though Leap’ motion capture we were able to create a database for each patient concerning parameters of each own attempts, such as timing, the range of movement and the percentage or correct and wrong attempts in order for us to proceed to our next complete step, which will be the short and long-term comparison.

The exercises have been put in order based on an escalating difficulty. Each exercise though has a three-level difficulty which means that in each level the range of movement has to be greater in order for the patient to hear the sound of the instrument.

3 Results

All 3 patients were familiar with the motor skills and motor patterns presented in VRehaM. The results though were not evolving in the same way for all them. There was a general difficulty in the stabilization of the hand, as our series of exercises concerns fine movements. It was obvious that, despite the initial difficulty in carrying out the movement, after a few attempts and sometimes with the therapist oral guidance, they began to follow the movement pattern, indicating that it began to break and disrupt the pattern of the hand that already prevailed. Although sometimes the movement was not achievable, we notice a small but certain range of motion which indicated that the patient clearly perceived the order of the exercise. Usually, after a few attempts, this range of motion was increased as a result of the influence of the sensory feedback-information signal received by the patient from the VR musical system.

In the welcome exercise, although there was a lack of music, patients were motivated by the new environment and they managed to complete the movement almost always successful.

The first exercise was the one with the greatest successes. The patients admitted that the instructions given by Orpheus were transparent and the rhythm they had to follow was clear.

The second exercise was the one with the fewer successful attempts. Given that it requires the cooperation of all muscle group of the hand, forming a synergistic movement pattern, presented an important enough difficulty, mainly for patient A and B who face the central nervous system lesion (stroke). Moreover, there was a clear predominance of the flexion pattern which created a difficulty at the beginning of each attempt. It was clear that patients were concentrated and were trying to execute the order and this persistence was our main priority.

Although the third and last exercise (fourth movement) was the most difficult one, demanding high coordination, the outcome was a surprise as we noticed a high efficacy in movement formation. It concerns the finest movement of the hand and as the finger opposition refers to the grasping capacity of the hand, it is its main function. We are optimistic that the rhythm of the selected piece of music also helped in this outcome as we chose a 5/4 piece allowing patients to follow the whole pattern for each movement/finger. This means that the execution of the movement was carried out in a slower rhythm.

At the end of the process, patients answered to some questions in order for us to be easier to assess some more factors of our platform. So, based on these testimonials, patients admitted that during the process they felt familiar with the new environment and with Orpheus and that they enjoyed the possibility provided to play instruments that physically weren’t able to do (see Fig. 6). So, during the last exercise, despite its great difficulty, patients were so motivated to continue trying playing music that results were remarkable and unexpectable, even for the occupational therapists.

Fig. 6.
figure 6

Statements from participants’ interviews

Full size image

We have to mention also the effort of patients to stay on the rhythm which was one of our main consideration. Being on the rhythm requires a higher perception ability as well as the ability to count stably and this gives us the opportunity to take cognitive functions into account. Patients also mentioned, that after being a while into the VE they forgot that they have to follow orders concerning movement execution and they felt like they had only to think the movement and follow the music. We can attribute this to the ability of VR to synchronize our brain activity in the alpha-wave state, as we noticed in one of our previous research [2]. After around one minute into a VE, our brain starts to present a predominance of alpha waves, ranging from 8 to 12 Hz, which can lead to a more concentrated and stressless state.

At the end and at the beginning of the process, patients were persuaded to complete a questionnaire concerning self-efficacy. In order to avoid biased answers, questions were not always the same. Patients had to answer 10 questions and the total score was counted out of 40. The clustered columns presented below concern the qualitative average score for each attempt before and after the application. The values have been computed as the average of the 3 patients. Although our number of patients is small, we noticed, for the four times of platform’s application, a stable increase in the score with the highest one ranking in 36 out of 40. We also observed that there is an increase in self-efficacy both before and after the integration. This means that patients felt more confident about the goal they could reach and that they had a bigger motivation to progress. In order to have a more valid assessment of this motivation, we gave this questionnaire to three more patients that they were following simply their conventional therapy, without having the experience of VRehaM. We can observe that for the Control group four days were needed in order to observe an increase in the index of self-efficacy compared to the VRehaM group where the increase is obvious from the first time and remains stable until the fourth. Patients also mentioned, at the interview given at the end of this preliminary study, that most of the times were looking forward to attending the session. We used no statistical inference due to the number of our patients. Our purpose was to present the idea of the platform and to test its usability, if it can be accepted, and consequently helpful, by both patients and therapists and to evaluate possible corrections needed for further research (Fig. 7).

Fig. 7.
figure 7

Comparison of the questionnaire’s result concerning self-efficacy. On the left, the clustered columns depicted the results from the VRehaM group showing a stable increase to each attempt. On the right, there are the results concerning the Control group, that follows only a conventional occipital therapy, presenting a small increase of the index of self-efficacy started mainly on the fourth attempt.

Full size image

4 Discussion and Future Work

Neurological disorders, as stroke, and their consequences are a leading cause of death and disability around the world the majority of survivors experience mainly chronic motor deficits associated with reduced quality of life. It is estimated that 55% to 75% of survivors continue to experience motor deficits and among them, 50% to 75% of individuals have persistent impairment of the affected upper limb (UL) [13]. Thus, there is a need to identify the best training strategy for the retraining of motor function of the UL.

In our intervention, a great deal of attention was paid to the selection of motor activities performed in the VE. Key motor tasks were selected, such as flexion and extension of wrist and fingers, pronation, and supination of the wrist as well as finger opposition. All these movements are essentially an indispensable part of all motor patterns of the hand being an essential part of all daily living activities. No activities were focused on the elbow or the shoulder because it would have greatly broadened the field without any significant functional advantage. The aim was to select the simple stationary movements to enhance the effort of improving the function of the hand and to give a further stimulus to the patient’s education and functional training. Thus, we developed a new low-cost VR rehabilitation game in order to encourage patients to continue their rehabilitation programs at home so that they can feel more familiar with the process and to feel the need to participate in social activities. In a recent review [9], based mainly on patients’ testimonials it was indicated that patients don’t prefer the home-based rehabilitation, even if they the possibility to follow one, as they feel that they don’t have the proper guidance and they are socially restricted. This is why we chose to use Orpheus for a proper guidance so that patients have a motivating interaction that will allow them not to feel isolated.

From observations made during the use of our platform, as well as from interviews received by the selected patients at the end of the procedure, the application did not cause any difficulty to them, nor any problem at all, there was no interruption and they all completed the exercises successfully. Throughout the procedure, it was observed that when the patient understood the motor mechanism of the movement, then it was gradually improved indicating that the VE provides the appropriate sensory feedback stimulus concerning motor activity. It seems that further improvements could be made to the program in order to additionally facilitate, mainly the patients, to understand the mechanism of the movement faster and easier. Different kind of music, according to patients’ preferences may be used also in order to examine if a further motivation can be produced. We intend to use different songs for each level of difficulty in each exercise in order to take into account not only the range of movement but also the timing based on which it is executed. Moreover, different instruments will represent each level of the difficulty.

The traditional Music Therapy procedure is focused on the specific activities while the VR approach takes advantage of more cognitive-sensory stimuli additionally to that given from the task-oriented approach. That is to say, the traditional approach is reinforced with an additional advantage, that of enriched information, that naturally gives greater motivation and interest to treatment. We chose though to use the most used techniques of NMT concerning arm rehabilitation in order to examine if VR, combined with this powerful neurological stimulus, can provide a bigger efficacy. Based on this preliminary study, we are very optimistic that this combination can give us a big advantage and if there is a better clinical and functional effect, our better designed prospective upcoming study will provide a further proof. This study will include also EEG recordings in order to have a holistic picture concerning the influence of VR on the brain compared to a PE of NMT and thus, we can conclude if the effect of virtual instruments and the consequent motivation constitutes a placebo effect or they actually affect brain plasticity that can facilitate the process of rehabilitation.