Keywords

1 Introduction

Every year, millions of people are affected by the consequences of having a stroke. According to the Observatorio Nacional de Salud, in 2014, the prevalence of stroke in Colombia was of 62582 cases [1]. A report made by the American Heart Association in 2018 [2] states that 795000 people experience a stroke every year in the United States. The report also shows that in 2015, worldwide prevalence of cerebrovascular disease was 42.4 million, affecting 0.58% of the world population. These concerning statistics demonstrate the need to find new ways to improve the quality of life of patients affected by stroke.

In the last few years, research in motor rehabilitation has focused on the use of new technologies. Johnson et al. [3] designed a Functional Magnetic Resonance Imaging (fMRI)-based Brain Computer Interface (BCI) in which stroke patients were given feedback in the means of repetitive Transcranial Magnetic Stimulation (rTMS), in order to enhance neural plasticity. The research exhibited benefits of the combination between BCIs and rTMS, as patients showed significant improvements over time. As well, Frolov et al. [4] designed a Motor Imagery-based BCI (MI-BCI) in which patients performed exoskeleton-driven movements of the hand; one group of patients controlled the movement with help of the MI-BCI and the other group of patients did not control the exoskeleton at all. While both groups showed important motor improvements, the results of the group that controlled the exoskeleton were significantly better.

A systematic review about BCIs for motor rehabilitation of stroke was performed in early 2018. The inclusion criteria were the following: papers written in English in the areas of engineering, computer science, medicine or neuroscience that were published from 2013 onwards. Overall, many types of BCI-based motor therapies have been subject of research. An option for improving brain plasticity in stroke patients is through the use of MI-BCI [5, 6], as it has shown satisfactory results, and in some cases, not even requiring a training session [7]. Efficiency of different ways to give feedback has also been studied. In different projects, robotic aid was given to the patients whenever the systems detected movement intention [8], under the assumption of a superiority of somatosensory feedback over visual feedback [9]. While most researchers use subjective tests such as the Fugl-Meter Assessment (FMA) or the Modified Ashworth Scale (MAS) to detect motor improvements in patients, some others have opted for other methods, such as the analysis of functional connectivity of the brain [10] or Magnetoencephalography (MEG) for sensorimotor rhythms [11]. Whereas many projects regarding the use of BCIs for motor rehabilitation of stroke have been made, to the best of our knowledge, there is no research in this area related to the use of video-games as a means of upper limber motor rehabilitation. In addition, video-games have been used to improve some cognitive tasks showing promising results [16,17,18,19].

This work aimed to perform a feasibility study on healthy subjects of a system that provides entertainment to the patients as they carry out their motor rehabilitation therapies. The system is based on previous work of our research group [12], in which subjects achieved brain modulation while controlling a movement intention-based video-game, defining movement intention as both: motor imagery and movement. Through MI-BCI, the video-game is controlled by the subjects.

2 Methods

The proposed system is composed by a Brain-Computer Interface, a forearm motion tracking system and a postural tracking system. All of these modules allow the user to interact with two designed video-games.

2.1 Brain-Computer Interface

Through EEG signal analysis, an algorithm is capable of recognizing certain patterns. A g.Nautilus g.LADYBird with 32 active channels is used in this research to acquire the EEG signals. Electrodes were located near the motor cortex (Fz, FC1, FC2, C3, Cz, C4, CP1, CP2 and Pz), in order to acquire signals related to the motor imagery of the patient; this way, subjects receive online feedback regarding their movement intent. Signals were sampled at 250 sps, with a resolution of 16 bits. From each of the 9 channels, 13 features were extracted, obtaining a total of 117 features. The selected features were the following: Mean, variance, skewness, kurtosis, root mean square, relative power by frequency bands (Delta, Theta, Alpha, Beta, Gamma, Mu), Hjorth’s mobility and Hjorth’s complexity. These signals were split into windows of two seconds, in order to compute the selected features in each window. However, each window had an overlap of 75%; this way, signals were processed every 500 ms, making it an acceptable response time in order to give appropriate feedback to the users [13].

Finally, Support Vector Machine (SVM) models were used to classify the signals between motor imagery and standby state. For each session, two SVM models are generated: a quadratic SVM and a cubic SVM. These models were chosen in a way in which they are robust enough to classify these signals but simple enough for real-time processing. On the one hand, because of its lower complexity, the quadratic SVM does not achieve exceptional accuracy in individual trials but it allows for better generalization. On the other hand, the cubic SVM is able to reach higher accuracy values, but it is more prone to present over-fitting [14]. In order to decide which was the best model in each session, the cross-validation loss of the models was calculated. Particularly, the model with the lowest mean squared error during the training session was chosen.

2.2 Forearm Motion Tracking Module

In order to stimulate and quantify the motor progress during a rehabilitation process, we recorded the upper limb movement. Participants wear two Myo Armbands, one on each forearm to obtain movement data. Each device has a accelerometer and a gyroscope which are used to calculate, starting from a reference point, the positions of an forearms. The algorithm to estimate the position of the forearm is obtained from an open source code created by Thalmic Labs, developers of Myo Armband. By using this algorithm, the user can control an avatar in the video-game by performing arm movements, while the device records the signals of the user, for comparison between sessions and between both arms. Also, the device can use different vibration patterns as an haptic feedback for the video-game.

2.3 Postural Tracking Module

Patients with affected mobility in the upper limb tend to overcompensate their movements; for this reason, postural tracking can be useful during the rehabilitation sessions [15]. This is why, during the experiment, the subject is asked to use a special t-shirt with markers which a video acquisition and processing system use to track posture during the whole test. The position of the markers is analyzed in two different reference points: shoulders and spine. Whenever the system detects unnatural movements, the user is notified through different types of haptic feedback (provided by the Myo Armbands), depending on the axis of the overcompensation. In the first case, if the user performs shoulder overcompensation, the armbands have a strong vibration every 2 s. In the second case, if the user performs spine overcompensation, the armbands have a weak vibration every 0.5 s.

2.4 Experimental Protocol

Three stages were implemented during the tests:

  • Preparation: During this stage, the acquisition devices are placed on the subject. The user is asked to put on the special t-shirt that allows the system to perform postural tracking. Additionally, the impedance of the electrodes is measured in order to assure quality of the signal.

  • Training: For this stage, records of healthy subjects during both relaxation and movement imagery are acquired, as the module aimed to generate a decision model capable of classifying both states.

  • Execution: The subjects play the designed video-game, which is composed of two scenarios or mini-games. For the first scenario, subjects are asked to move their forearms in a specific manner, while all the systems record their corresponding signals and help to offer feedback. For the second game, the subject controls an avatar through forearm movement. Since the system was developed for stroke patients and their movement may be limited, the system decides when can the users control it via forearm movement and when can they control it via BCI control. This way, the subjects control the avatar even if they can not move their compromised forearm.

Fig. 1.
figure 1

Clay Game. The top of the figure shows user interface. The bottom shows the user movements. The lights in the top are equivalent to the arrows in the bottom. A. Selection of dominant hand. B. Description of the required action. C. Reference point from where movements are performed. D. Button to close the game.

Full size image

2.5 Video-Game

The created video-game is named MindSense and it is composed by 2 mini-games, which were designed specially for this study in the game development platform Unity.

The first mini-game, the Clay Game, consists on two lights controlled by the user, one light per arm. Subjects are asked to move their dominant arm as far as possible while pulling from a piece of clay placed in the center of a table. The mini-game uses an algorithm which detects whenever the forearm of the user stays still; once the algorithm detects that the arm stops, the subject is asked to imitate the movement with the other arm in the opposite direction. Depending on the symmetry of the paths, the user receives a score as feedback. A screenshot of the Clay Game is presented in Fig. 1, in which the center of the table is marked in a red circle, indicating that the user must perform the movements by taking this point as a reference.

Fig. 2.
figure 2

Food Game. The user controls both arms through the Myo Armband or the BCI. The goal is to move the hand with the fork, from the food to the mouth.

Full size image

The second mini-game, the Food Game, consists on controlling an avatar by moving both arms. The goal of the game is to feed the avatar, as it will imitate the forearm movements performed by the user. This game uses a movement recognition algorithm: initially, the Myo Armband provides signals related to forearm movement. Whenever the user stays still, the control of the avatar is transferred to the BCI; this way, the subject is able to control the avatar by using motor imagery. There is a plate of food in front of the avatar and the user must control it in order to feed it. However, if the BCI controls the system, the arm of the avatar follows a default path towards the mouth. A screenshot of the Food Game is presented in Fig. 2: the user must take a piece of food with the fork before eating. The scores of this game are a measurement of how fast can the user eat the food; both by moving the arm and by using motor imagery.

Both mini-games make use of the same algorithm in order to detect arm movement. First, the difference in position of the arm between two frames is recorded in a 2-second sliding window; if it surpasses certain threshold, it is registered as a frame in which the subject moved the arm, otherwise, it is registered as a frame in which the subject stayed still. If the subject moved during 20% of the 2-second window, it is considered that there is a general movement of the arm. Both the movement threshold and the percentage needed for the arm to be considered to be moving were defined empirically.

The experimental protocol is presented in Fig. 3.

Fig. 3.
figure 3

Experimental protocol. First, subjects were selected. During each experiment, subjects were asked to sign up or log into the game (depending on the session number). After placing the devices and training the BCI, the subjects are asked to complete the mini-games multiple times.

Full size image

3 Results

For each session, movement intention accuracy of the system was obtained. Results for each session and mean performance per subject are shown in Fig. 4. In the figure, the first three bars of each subject show accuracy of movement intention detection, while the last bar, shows the mean of these values. Despite the instability of the obtained accuracy, in most of the sessions an accuracy higher than 68% was reached, with atypical sessions with less than 45% of accuracy. Additionally, except for two subjects (51.01% for subject 3 and 56.06% for subject 6), all of the mean accuracy values are above 65%.

Fig. 4.
figure 4

Accuracy of the system during training stage.

Full size image

Results of the Clay Game are shown in Fig. 5. Each session, the subjects were asked to complete the exercise during five trials. The full path in each trial is divided in 10 equally distributed steps. A score is calculated for each step, with a total of 50 scores per session and 150 per subject. Each score was calculated depending on how symmetrical the motion signal of the dominant forearm was compared to the other forearm in a particular step of the path. In Fig. 5, scores of the subject 1 show a pattern in which the score decreased as the steps increased. This pattern is repeated in the sessions of most of the subjects, as can be seen in the mean scores. None of the subjects had a score below 60 in any step nor session.

For the Food Game, the results are quite different, since the game does not show scores directly to the user. Instead, it gives feedback to the users each time they complete the exercise. In this case, the scores of the game are represented as the interval between the moment in which the users grab food and the moment in which they feed the avatar. As mentioned before, the subjects were asked to do the exercise during 30 s. Then, during the next 30 s, motor imagery is used, and the process starts all over again. This way, intervals corresponding to motor imagery can be separated from intervals controlled by the movement of the forearm. The mean elapsed time for both motor imagery and forearm movement for all the subjects during the 3 sessions are shown on Table 1. It is clear that motor imagery intervals have higher values compared to intervals from forearm movement. Particularly, the elapsed time for motor imagery was \(22.22\pm 11.37\) s, whereas the elapsed time for forearm movement was \(1.47\pm 0.62\) s. As well, the number of repetitions for motor imagery was \(5.75\pm 1.75\), while the number of repetitions for forearm movement was \(20.33\pm 6.75\). In all the metrics, the coefficient of variation (standard deviation by mean) was less than 1, which signifies a low variation, indicating a relatively good stability. Furthermore, a tendency of changes in scores was not found between sessions, as scores decreased in some subjects while they increased in others.

Table 1. Mean of elapsed time for repetitions in-game during motor imagery (MI) and forearm movement (FM). S: Session, ET: Elapsed time in seconds, R: Repetitions, M ± STD: Mean ± Standard deviation
Full size table
Fig. 5.
figure 5

Scores of subject 1 and average scores through subjects in Clay Game (blue: first session, red: second session, yellow: third session). (Color figure online)

Full size image

4 Analysis

As noted before, the movement detection accuracy results were not as promising as hoped before for some sessions. One possible explanation is that subjects may have not been focused in the task at hand in some of the sessions. It is important to remark that, in systems where brain-wave modulation is key, the user must be compromised with the tasks for the movement intention detection to be adequate. If this is the case, results may improve in a stroke rehabilitation therapy, due to the compromise of the patients with the rehabilitation processes. Another possible explanation for the low results in some of the sessions may be that not enough machine learning models were considered. It is possible for different models to achieve a better performance.

Furthermore, an enhancement in the performance of the subjects was not expected. Instead, the aim of this study was to assess the feasibility of the system and find specific features that can be improved. However, there were important quantitative results.

For the Clay Game, it is clear that participants had struggle maintaining the symmetry between forearm movements as the distance between them increased. This tendency could have a relation with the eye-hand coordination of the participants, as it may be easier for them to imitate this movement when they are observing both hands. With respect to the scores between sessions, some subjects improved their scores, while others worsened them. The fact that none of the participants got scores below 0.6 is useful for a characterization process in order to monitor the rehabilitation course of stroke patients.

For the Food Game, intervals on the motor imagery sections were higher than those on the forearm movement sections. This is because it is easier to perform movement than imagining doing so (in a way in which the system detects it). For these same reasons, the amount of repetitions was lower on the motor imagery section. Furthermore, the found intervals are useful in order to start a characterization process for analyzing stroke patients performance on rehabilitation processes with this system. The same happens with repetitions, as its values tend to be at least three times higher on forearm movement sections for healthy subjects. Taking all results into account, the system was suitable for healthy subjects.

While no enhancement in performance of the subjects was expected, results of the subjects through the sessions point out that the system is usable, since all the subjects surpassed a score threshold on each mini-game. In the case of the Clay Game, the minimum score was 0.6, whereas for the Food Game, during the movement intention step, all the subjects completed at least 3 repetitions. Also, it is remarkable that all the subjects completed at least 3 repetitions during the first session. This means that all the subjects were capable of using the system since the first session.

5 Conclusions

For this project, a HCI system was designed and assessed. The system includes EEG, postural analysis, motion analysis and various types of feedback, on healthy subjects. The detection of movement intention is key for the development of the system, as, for stroke patients, it allows for improved brain plasticity, leading to motor rehabilitation. Concerning this, it is essential to find ways to improve this aspect of the system. An approach to this may be to enhance the experimental protocol in order to secure the total involvement of the subject in the tasks at hand. For instance, it is possible to suppress distractions by developing an adequate room in which external sound is nullified and by forbidding the entrance of electronic devices to the experimental area. Another approach to improving the detection of movement intention is to evaluate the use of more machine learning algorithms, taking into account that it is crucial to maintain a low processing time, in order for the system to work online.

Regarding the Clay Game, the participants seemed to be comfortable while using the system, since all the users were capable of reaching an acceptable result (more than 60 points). The minimum score reached for all the healthy subjects could be considered a beginning in order to characterize the used metric for future studies on stroke patients. Also, subjects tended to present a better performance during the first steps of the path. According to this, for stroke patients, the HCI should include a new score system in which a higher value is given to the first steps of the path. Another important result for future works are the intervals obtained in the Food Game, as the elapsed time and quantity of repetitions are relatively stable, making them possible metrics to characterize the rehabilitation processes for stroke patients. As a general outcome, the obtained results are essential to improve the designed system and apply it on motor rehabilitation processes of stroke patients.