1 Introduction

Biofeedback is increasingly used within treatment programs for various health conditions and disorders. Data sourced from different bodily functions is, more than ever, readily available to map to media feedback through recent technical system advances. In line with this, audio-biofeedback systems have been investigated in respect of benefit for regaining postural control [1, 2]. The objective of this study was to investigate to what extent audio-visual biofeedback affected physical balance. Patients were provided with audio and visual cues to perform certain physical exercises as part of their rehabilitation process. It was argued that the main challenge when designing a multimodal system is being aware of feedback relationships [3]. Using multiple biofeedback parameters may result in a perceptive and cognitive overload, which also might influence testers at a psychological level. Different studies compared biofeedback-induced therapy with regular therapy and discovered there was no significant difference in the improvement rate [4, 5]. However, biofeedback may add more variability to the training sessions.

For this study a comparison between two different exposure situations: visual biofeedback and audio-visual biofeedback - indicated that the type of biofeedback used represented a difference for stroke patients. In order to investigate the objective of the study a bespoke balance board and a music rhythm game, named “Balance Hero” were utilized. The balance board functioned as a controller for the game by measuring the pressure exerted on it. Test subjects were stroke patients, five males and two females, between 33–70 years-of-age, involved in a local rehabilitation program in Denmark. Patients sat on the balance board and shifted their weight by leaning to each side in order to control a music rhythm game. The aim of the game was to simulate guitar playing by controlling a 3D virtual fret using visual or/and audio cues. The cues were 3D note objects that scrolled on screen according to the rhythm of the song selected.

1.1 Balance Impairment, Sensory Systems and the Auditory Impact on Human Motor Behavior

1.1.1 Balance Impairment in Stroke Patients

Physical balance is defined as a harmonic arrangement that uses all the information sent by the sensory systems to place the body in connection with the gravity and earth. A balance disorder is a condition which increases the risk of falling by making an individual physically unstable. It can be caused by certain health conditions, medications, and a dysfunction of the inner ear or brain damage [6].

Research indicated that there are several medical conditions associated with gait and balance disorders: affective disorders and psychiatric conditions, cardiovascular disease, infectious and metabolic disease, musculoskeletal disorders, neurological disorders, sensory abnormalities and others. One of the neurological disorders that cause balance impairment is stroke [7]. This study focused on stroke patients with balance impairment. In order to determine how stroke affects postural stability it is necessary to investigate the implication of stroke; it is defined as a disturbance of cerebral function with apparently no other cause than vascular origins [8]. Commonly, it results from the blockage of a major artery in the brain and it leads to the death of all cells in the affected area [9]. Stroke causes symptoms such as loss of strength, sensation, vision or other neurological functions [10]. As an effect of a stroke, patients may need to adjust to the new functionality of the sensory system. In consequence, physical therapy has rehabilitation purposes, to restore movement, balance and coordination [11]. In order to regain postural stability certain physical exercises are practiced by patients on a daily basis. The system used to test our hypothesis simulated a training session.

1.1.2 Sensory System

To determine the functionality of physical balance it is essential to explore the mechanisms of sensory systems. Maintaining postural control is a multisensory process that uses the information received from different sensory systems.

There are three major sensory systems involved in balance and posture: visual, vestibular and somatosensory [12]. Balance generally decreases in the absence of any visual input or in experimental conditions that affect the quality of visual input [13].

Spatial hearing cues provide moderately accurate abilities to localize sound sources. A connection can be established between spatial hearing and postural control [14]. Moreover, a single fixed sound source can provide sufficient spatial cues for the central nervous system to an improved control of postural stability [14]. It has also been outlined that the compensation effect received in the vestibular system from the auditory cues is more reduced compared to the one from visual cues. In our study the veracity of the aforementioned idea would be further observed through the design of the testing sessions.

1.1.3 Auditory Impact on Human Motor Behavior

Concerning the psychological reaction to audio input, in relation with physical movements, studies established that rhythm enhanced human motor behavior [15]. Moreover, by synchronizing movements to the rhythm, individuals indicated gratification, as it replicated natural movement forms.

The idea was further supported in a similar study, confirming that young adults endured longer periods of physical exercise training when their body movement was synchronized to the rhythm of music, compared to moving asynchronously [16]. Additionally, the increase in endurance is caused by an emotional response from music [15]. Moreover, in research with acquired brain injured patients (stroke) and audio and multimedia feedback of near-range and far-range balance and proprioception, positive outcomes were indicated in [17].

2 Methods

In order to investigate how the audio component in audio-visual biofeedback affects postural control, a system was developed, to observe, measure, and analyze physical balance. The system consists of: (a) a self-created balance board and (b) a music rhythm game. The bespoke components are detailed below.

Three stages of a user centered design were approached to satisfy the necessities of patients and stakeholders; early focus on users and tasks, empirical measurement and iterative design [18]. Different aspects of the system were discussed and further implemented in order to achieve an efficient and suitable system: (a) balance board – functionality and synchronization and (b) the music rhythm game: speed, music, appearance and functionality.

2.1 Balance Board

A balance board was created to measure and record the pressure exerted on it. It consisted of: a Spider 8 data logger, 4 S-type load cells, 3 connection cables for the load cells, 8 bolts to secure the load cells, 2 wood countertops, and 4 wood support blocks (Fig. 1). The load cells register the variables made by the bending of metal foil inside the cell, transforming the mechanical deformation into an electrical output signal. To mount the load cells, two pieces of wood countertop were used. The Spider 8 data logger was connected to a PC through a USB/RS232 converter cable in order to communicate with the PC. The load cells were calibrated before each test, to register and control the game.

Fig. 1.
figure 1

Setup of the balance board (Width: 60 cm/Length: 70 cm/Height: 13 cm)

Full size image

For this study, a program was created in Visual Basic 6 to receive the data from the four load cells and send it through a local User Datagram Protocol (UDP) connection. Lastly, this was mapped to the Unity game engine within which the graphical music rhythm game was programmed.

2.2 Graphics and Sound

A music rhythm game was developed to stimulate transitions in postural position and immerse the patients in a fun and engaging interactive environment. The aim was to simulate guitar playing by controlling a 3D virtual fret using visual or/and audio cues. The cues were 3D note objects that scroll on screen according to the rhythm of the song selected (Fig. 1). The player had to control a diamond-shaped object to hit colored note objects by leaning left or right. Unity was the main development platform, with imported models from Google SketchUp Pro 8.

The audio stimuli was represented by the song “Viva la Vida” by Coldplay (2008) which was suggested by the physiotherapist as preferred for the target group. The physiotherapist argued that the patients were familiar with the song and that it was appropriate for the age range. Different aspects were considered, such as musical style, tempo and rhythm. Additionally, extra-musical associations evoked by the music were examined in line with that they are considered as primary factors in stimulating emotional responses in aiding motor activity [19].

2.3 Programming

The game was developed in Unity, a game engine utilizing several different programming languages. The programming language used to develop the game was C#. An essential factor in the game was to design the in-game movement to be controlled through the readings of the balance board. The 3D virtual fret board was programmed to have five lanes where the user controlled the diamond-shaped object, each corresponding to a certain pressure value on the balance board. This value was calculated as the difference between the two left sensor values and two right sensor values on the balance board. In consequence, when the participant distributes their weight evenly on both sides of the board, the in-game character will remain in the middle lane.

In order to gain an overview of the patients’ balance distribution while playing the game, the values gained from the balance board were written to a comma separated value (CSV) file every 20th of a second.

The synchronization between the note objects and the rhythm of the song playing were programmed to correspond during the entire gameplay. The beats per minute (BPM) of the song were used in the formula below to calculate note intervals:

$$ \frac{\frac{60}{BPM}}{{N_{value} }} = N_{interval} ; {\text{where }}N_{value} \in \left\{ {0.25, 0.5, 1, 2, 4} \right\} $$

Here Nvalue is equivalent to the different note values from a whole note to a sixteenth part note. The result, Ninterval is the note interval in seconds used to synchronize the note objects to the rhythm of the song. Moreover the speed at which the note objects are moving is proportional to the BPM. This formula allows a change of pitch with a speed modifier by utilizing the modified BPM value.

The adjustment of the music was necessary in order to obtain coordination between the physical exercise proposed by the physiotherapist and the song. The notes were placed according to the training exercise suggested. In accordance, the music was changed for a better quality of sound that fitted the requirements mentioned above.

2.4 Data Analysis

The study was conducted on seven balance-impaired stroke patients between 33–70 years-of-age. The unit of analysis was group-based; therefore the procedure was grouped into: (1) a control group with two testers who performed a balance training exercise without any technological input, (2) a visual biofeedback group, with two testers playing the game via visual input, and (3) an audio-visual biofeedback group, with three testers playing the game via audio and visual input.

Following each test, an evaluation session consisting of semi-structured interviews was conducted and supervised by a physiotherapist, to collect user data and feedback. The physiotherapist suggested what type of physical exercise will improve the rehabilitation process, to be implemented in the game. Furthermore, the physiotherapist tried the system and indicated a possible usefulness for the rehabilitation process.

2.4.1 Control Group

Two balance-impaired stroke patients performed a balance training exercise, by sitting on the balance board. The exercise was developed to simulate the game without technological input. The patients had to perform a physical exercise by following post-it notes numbered from one to five, to simulate the game without technological input. The lanes in the game were represented by numbers. The first lane (numbered one) suggested a full leaning to the left and the last lane (numbered five) a full leaning to the right.

2.4.2 Visual and Audio-Visual Biofeedback Group

The visual group was formed of two balance-impaired stroke patients, who played the game with visual input. The audio-visual group was represented by three patients performing the full system test, which included both visual and audio biofeedback.

To analyze the data, graphs were created from CSV files and had two points of reference: the x-axis, representing the time interval for each participant, and the y-axis, showing the pressure exerted on the balance board. A computer-generated line was added to the existing graph, representing the flawless game-play. The aspects evaluated were values and line fluctuations.

The points obtained by the patients playing the game were calculated into a mean value and an improvement rate (%) for each patient and subsequently for both groups. The formula utilized for the improvement rate was the following:

$$ I_{rate } = \frac{{V_{1} + V_{2} }}{2} $$
$$ V_{1,2} = \left( {S_{n} - S_{n - 1} } \right)* \frac{100}{{ S_{n - 1} }} $$

Where V = Value, Irate = Improvement Rate and Sn = Score corresponding to a test session (e.g. S2 – score obtained in the second session of the test). In order for the formula to be applicable, it is necessary to mention that the “n” in “Sn” can be equal only to 2 or 3, representing the second, respectively third test session. The individual mean and improvement rate indicated the progress of each patient throughout the testing session. The groups’ mean and improvement rates facilitated a comparison between the visual and audio-visual biofeedback groups. The patients’ feedback was collected via semi-structured interviews with a focus on gameplay, sound and motivational aspects. In order to validate the study, a physiotherapist specialized in rehabilitating stroke patients supported the experiment by providing feedback throughout the entire process. The procedure was recorded with a video camera, in order to validate the results and enable the possibility of a post-analysis of the sessions for further assessment.

3 Results

The data collected during the test sessions, based on the semi-structured interviews revealed that five of the users were males and two were females, between 33–70 years-of-age. Five of them suffered from stroke in the left side of the brain, one from stroke in the right side and one from stroke in the brain stem.

3.1 Balance Board Data Readings

3.1.1 Control Group

The fluctuations of the graph lines decreased for both patients from the first to the second test.

3.1.2 Visual Biofeedback Group

The fluctuations of the graph lines decreased for both patients, throughout testing sessions. In consequence, the points obtained in the game by both participants increased, indicating an improvement rate of 2.98 % and 8.44 %. The individual scores, group scores and improvement rates can be seen in Table 1.

Table 1. Visual representation of Patients’ scores, mean values and improvement rates
Full size table

3.1.3 Audio-Visual Biofeedback Group

The fluctuations of the graph lines presented a minor decrease from the first to the third test for two patients, while for the third patient the fluctuations increased. In consequence, the improvement rates for the first two patients were 3.55 % and 4.07 %, while the third patient presented an improvement rate of -3.38 %. The individual scores, group scores and improvement rates can be seen in Table 1.

Figures 1, 2 and 3 illustrate the graphs for one of the patients (Ki.) through all three tests. The improvement is visible, as the fluctuations decrease throughout the test sessions (Fig. 4).

Fig. 2.
figure 2

Screenshot of the music rhythm game – Balance Hero

Full size image
Fig. 3.
figure 3

Ki.’s performance in the first test

Full size image
Fig. 4.
figure 4

Ki.’s performance in the second test

Full size image

3.2 Semi-structured Interviews

The results from the first interview indicated that three out of five patients considered the game easy to understand, while the other two experienced difficulties in understanding and playing the game. Three out of five patients stated that the speed of the game was too fast. Regarding motivation, two patients affirmed that they would use the setup at home. The second interview indicated that four out of five patients considered the game easy to understand. Concerning the speed, all patients considered it suitable. Four patients indicated that they considered the game motivating and would use it for further training. In relation with the in-game music, the patients indicated enjoyment and recognition when hearing the song selected (Fig. 5).

Fig. 5.
figure 5

Ki.’s performance in the third test

Full size image

The results retrieved from comparisons between the data sets (2) and (3) suggested a superior in-game improvement between test sessions for (2) based on the higher scores obtained in the game. Another analysis criterion was fluctuations in the graphs’ line (Fig. 3). The data set (3) indicated an inferior in-game improvement compared to (2). Moreover, one of the testers from (3) had the lowest score in the entire procedure. Regarding the data set (1) testers were less motivated to perform the training exercises although their performance was superior to (2) and (3).

4 Discussion

Results from the control group test revealed that the performance level of the patients increased throughout the testing session. The two patients presented fewer fluctuations of the graph lines than the patients in the other groups; however, the comparison is invalid as the conditions are different. The patients performed the training exercise using the post-it notes, numbered from one to five; therefore they had no clear point of reference to indicate the correct amount of leaning.

The results retrieved from comparisons between (2) and (3) suggested a superior in-game improvement between test sessions for (2). A better in-game improvement rate, higher scores and less fluctuations of the graph lines were observed in (2). This indicates that the patients (2) had better postural stability than (3). The better in-game performance by (2) can be a consequence of several factors, including: cognitive overload, individual level of balance and sample size.

4.1 Cognitive Overload

Using multiple biofeedback parameters may influence users on a psychological level, according to [3]. However, a related study suggests that sensory overloading might not be of great concern when designing for augmentation [20]. As a consequence, users had smoother movements when provided with both audio and visual information. This argument is supported by additional research stating that rhythm positively influences motor behavior [15].

4.2 Individual Level of Balance

The testing sessions were conducted over a period of only two weeks, and each patient played the game for approximately 25 min in total. Since the amount of training with the setup was limited, it cannot be established how their individual balance level affected the result.

4.3 Sample Size

The number of available patients was limited, thus the result cannot be generalized. Based on the semi-structured interviews and the feedback from the physiotherapist, the patients from the Visual and Audio-Visual biofeedback groups considered the game motivating, as it follows the concept of “Gameflow” [21]. Along with this, it was discovered that music improves motivation for exercising [19]. In [22] certain users were more receptive to audio stimuli while others to visual stimuli, however when exposed to both types of stimuli simultaneously the degree of immersion increased significantly. The feedback from the physiotherapist who supervised the testing session corresponds with these points.

5 Conclusion

The main objective of this study was to investigate how the audio component in audio-visual biofeedback affects the rehabilitation process of balance-impaired stroke patients. It can be concluded that the Audio-Visual biofeedback group indicated less postural stability when compared to the Visual biofeedback group. However, the Audio-Visual biofeedback group performed smoother moves during the test sessions, which according to the physiotherapist are beneficial for the rehabilitation process. Both groups considered the game motivating.

A limitation of the study was the size of the sample and the testing time. A larger sample should be tested over a longer time period to achieve more conclusive results. Moreover, there is a need to establish a baseline level of balance for each patient at initiation to draw an accurate comparison of intervention. Nonetheless, whilst there is a need for extra validation of the system, the physiotherapist concluded that it could still be useful as an additional and recreational training tool for regaining postural stability.