Keywords

1 Introduction

Over the last three decades a large body of work has focused on developing assistive technology devices for people having various severe motor disabilities (e.g. locked-in-syndrome, amyotrophic lateral sclerosis, spinal cord injury, quadriplegia, muscular dystrophy, cerebral palsy) to allow them to regain motor functions for recovering some degree of interaction capabilities with their environment. Numerous studies have proposed and assessed these types of assistive technologies which apply the general underlying principles which are based on human-machine interfaces that record and transform the available biosignals (e.g., muscle, brain activity; eye, head, tongue movements) into control signals to guide an external device (e.g., [1–9]).

A particular class of assistive systems is based on human-machine interfaces which relate body motions with their sensory consequences through an external device. Such human body-machine interfaces allow the users to implement full or shared control of the assistive device by employing motion signals recorded from the user’s body (e.g., eye, head, shoulder, tongue movements) [10–12]. Independently of the interface and of the type of control signals considered, the user must learn a mapping between the movements of the body segments that remain functional into the corresponding sensory consequences such as the spatial displacement of an external device that often takes the form of a cursor displayed on a computer screen. However, learning the control of an external device is generally a challenging task since the user employs body segments or portions of body segments that are still functional but that would not necessarily be used to perform the same task under normal conditions [10–12].

A particular instance of this class of assistive systems are the devices controlled through head motion to regain navigation (e.g., wheelchairs) or reaching (e.g., computer cursors) capabilities (e.g., [4, 5, 13–15]). In the case of head controlled devices, the execution of reaching tasks which would be usually performed with the upper extremity are now executed with limited head motions which is rather unnatural [16]. Thus, becoming skillful with head reaching controlled devices depends on the learning of sensorimotor mapping between relatively limited head movements and spatial displacements of the external device to control [10–12]. The challenge associated with these types of motor learning and performance tasks can result in an increase in a user’s fatigue, motivation, frustration and cognitive workload which can affect the motor performance and/or the capability to perform other tasks [17–20].

However, this particular body of work generally has primarily focused on developing reliable, easy-to-use and low cost head-controlled devices while conducting a relatively limited assessment of the cognitive-motor performance and in particular the motor learning processes underlying the reaching skill with such devices [4, 5, 8, 15, 21–26]. First, although it was previously reported that the mastery of head-controlled pointing systems generally requires practice, the assessment of motor performance throughout the practice period with those devices is relatively limited [8, 15, 23–25, 27–33]. Among the few studies that examined practice of head-controlled pointing devices, analyses are generally limited (e.g., mainly use movement time) and the results are relatively heterogeneous. Such heterogeneity may be attributed to the differences between the various types of head-controlled systems [30]. However, although heterogeneous, as a whole these findings revealed that to reach a certain degree of skillfulness, a practice period is needed and can range from about 60 to almost 800 trials, depending on the device itself [25, 28, 30–32]. Next, the assessment of motor performance after the practice period was generally not assessed by requiring the individual to perform the task under different conditions to inform the state of the acquired mapping (e.g., transfer, reduced visual feedback). Lastly, besides the motor aspect of the performance, the consideration of a user’s cognitive states such as cognitive workload was generally overlooked by previous studies [15, 23–25, 28–32]. Moreover, it is important to assess cognitive workload while individuals perform during/after practice since it can: (i) inform about the relationship between the attentional and motor processes (e.g., [18–20, 34]); (ii) impact the capability of the user to perform other tasks and/or to be unable to handle unexpected event (e.g., [18–20, 34]); and (iii) contribute to develop better assistive systems [35, 36]. For instance, it was shown that when individuals perform a motor task under various conditions of challenges and/or while learning a new mapping while executing arm reaching movements, the performer’s cognitive workload and allocation of attentional resources can be impacted (e.g., [18–20, 34, 37]). Thus, our aim was to examine the effects of practice on cognitive-motor performance while users control a virtual robotic arm with limited head movements by employing various control modalities (e.g., various directions, velocity control). Theoretically, we posit that work in this area can provide information about the cognitive-motor processes during the learning of sensorimotor maps during performance of reaching tasks while mitigating biases due to previous motor experiences [10–12]. From an applied perspective, our work could contribute to the enhancement of the design, training and assessment of head-controlled reaching devices.

Therefore, as an initial step, by considering two simple directional control modalities through which individuals employed limited head motion to learn control of a virtual robotic arm, we examined their reaching performance: (i) throughout the practice period while subsequently assessing the corresponding cognitive workload and (ii) after the practice period by assessing transfer to untrained targets and reliance on visual feedback while performing the newly acquired reaching skills.

2 Materials and Methods

2.1 The Human-Robot Interface

Signals from two infrared sensors placed on the forehead and chin of the participants were acquired with a motion capture camera-based system (Optotrak™) and were sent to the human body-machine interface to track a user’s rotation movements of the head. Then, this information was used to compute the desired directional displacements of a virtual robotic arm in a two dimensional (2D) workspace. A chin sensor was employed for selecting/confirming the acquisition of the target by opening the mouth. Once the target was selected, the participant moved his/her head to reach this target with the robotic arm.

2.2 Participants and Reaching Task

Twelve healthy individuals (N = 12), with normal or corrected-to-normal vision, participated in this study. The participants were randomly assigned into two groups either a four directional (up, down, left and right) or eight directional (up, down, left, right and the four diagonals) limited head motions to move the end-effector of the robotic arm. The head controlled device worked in a similar way to a regular joystick. Namely, in the resting position, the sensor position placed on the forehead was in the neutral region (or deadband) and thus the robotic arm stayed motionless. However, when a movement of head rotation in a particular direction was executed, the forehead sensor moved out from the deadband and thus resulted in a spatial displacement of the arm in the selected direction. As such, participants were able to stop and drive the robotic arm at a constant velocity throughout the 2D workspace until the end-effector reached a given target. The participants were also able to select the presented target to reach by opening their mouth beyond a certain threshold. For consistency, both groups followed exactly the same experimental protocol, the only difference being the number of directions in which they could move the robotic arm to learn a very simple sensorimotor map between head movements and the spatial displacements of the virtual robotic end-effector.

To familiarize participants of both groups with the interface of the head-controlled system, they performed head movements in any of the four (first group) or eight (second group) directions, which once decoded, were displayed in the form of a corresponding label (e.g., “Up”; “Up and left”) on a blank display to allow participants to understand the range of motion required to control the device while minimizing any learning processes. The participants performed this familiarization stage until they felt comfortable to generate appropriate head movements for all the directions their interface allowed.

Next, the experiment was divided into two main stages which were the practice and post-practice periods. During both periods, the target (red diamond) to be reached was displayed on the computer screen to the participants who had to select the target (which turned green when selected) and then reach this target with the robotic arm. Once the target was touched by the robotic end-effector, the next target was presented and the information related to the previous trial was erased.

During the practice period, participants of both groups performed reaching movements toward targets that were randomly presented and spatially distributed to cover the entire workspace (for consistency, the random sequence was the same for every participant).

After completion of the practice period, a rest (>20 min) was provided to the participant during which time they completed two surveys that included a visual analog scale (VAS) and the NASA TLX [38] questionnaires to assess their cognitive workload as well as the level of task difficulty and effort of the participants who just performed the reaching task. Then, after the rest period, two post-practice tests were presented to the participants to assess immediate retention as well as transfer of motor performance to targets that were not experienced before with full and partial visual feedback. The first and second post-practice tests allowed for the performance of reaching movements with full and partial visual feedback where half of the trials (n = 10) included known targets whereas the other half (n = 10) included new targets. Partial visual feedback was defined as the user was unable to see the entire forearm during the spatial region defined as the distance between the robotic end-effector and the target to reach between 1/3 and 2/3 of its radius. The order of the presentation of the two post-practice tests was counterbalanced to avoid any order effects and the presentation of the targets followed a random sequence (but the same random sequence for all participants).

Motor performance was quantified during the practice period as well as the two post-practice periods by computing the number of control signals (CS, defined as the number of changes in direction) sent to the robotic arm, movement time (MT), movement length (ML), and normalized jerk (NJ) for each participant and each trial [29, 35, 39, 40]. Since the distance between the random targets were different, each metric computed for a particular movement executed between two successive targets was normalized to the corresponding Euclidian distance. Then, for the practice period, these normalized metrics were tested using 2 Group (four, eight directions) × 20 Blocks repeated measures ANOVAs with repeated measures on the last factor where one practice block included 10 trials. In addition, the early (the four first trial blocks), middle (trial blocks 9, 10, 11, 12) and late (the four last trial blocks) practice periods were analyzed by employing 2 Group (four, eight directions) × 3 Period (early, middle, late) repeated measures ANOVAs, with repeated measures on the last factor. For the post-practice periods, these metrics were tested using 2 Group (four, eight directions) × 2 Modalities (full, partial visual feedback) × 2 Targets (trained, untrained) repeated measures ANOVAs with repeated measures on the last two factors. A Greenhouse-Geisser correction was applied when sphericity was violated for all repeated measures analyses. Tukey-Kramer multiple comparison tests were used to assess significant interactions and Blocks and Phases main effects. The scores from the VAS and NASA TLX questionnaires were contrasted using either an independent samples t-test or Mann-Whitney test depending on whether the assumption of normality was met or not. A significance criterion of α = 0.05 was used for all tests.

3 Results

3.1 Reaching Performance

Generally, the findings revealed that both groups: (i) performed similarly during practice as well as during the post-test sessions; (ii) were able to transfer their reaching skills to targets that were not experienced before but also exhibited reduced performance with partial visual feedback while still able to reach the targets.

Specifically, the practice period revealed a main effect of the trial blocks for CS, MT and ML (F(1,19) = 5.05; p < 0.001; all conditions considered). The post hoc analysis revealed that the values of CS, MT and ML obtained for the blocks 2, 3 and 7 were significantly higher compared to those for the block 11, 13, 15 and 19 (p < 0.05, all comparisons considered). Moreover, compared to block 17, the values of CS and MT were significantly larger than those obtained for the block 7 while ML were significantly higher for the blocks 2 and 3 (p < 0.05, all comparisons considered) (Fig. 1). Also, the values of CS and ML for block 14 were larger than those of the blocks 13 and 19 (p < 0.05). No effect was observed for NJ (p > 0.05) (Figs. 2 and 3).

Fig. 1.
figure 1figure 1

Performance across the practice blocks considering the number of control signals provided to the robotic arm (A), the movement time (B), movement length (C) and normalized jerk (D) during reaching movements (mean ± SD). AU: Arbitrary unit; Bi: Trial block i. The forks indicate the statistical difference between the blocks 2, 3, 7 (thick ticks) versus the blocks 11, 13, 15, 17 and 19 (thin ticks). The black diamond and gray square represent the first (i.e., using the control modality that includes four directions) and second (i.e., using the control modality that includes eight directions) group, respectively. *:p < 0.05; **: p < 0.01; ***: p < 0.001.

Full size image
Fig. 2.
figure 2figure 2

Reaching performances summarized throughout various practice periods (A-D) and during the post-practice sessions (E-H) (mean ± SD). (A-D) Kinematic performance indexed by the CS, MT, ML and NJ during early, middle and late practice periods. (E-H) Kinematic performance indexed by the same four metrics during post-practice for previously experienced and novel targets with full and partial visual feedback. fFT: Previously experienced targets with full visual feedback; fFN: Novel targets with full visual feedback; pFT: Previously experienced targets with partial visual feedback; fFN: Novel targets with partial visual feedback. The black and gray bars represent the first (i.e., using the control modality that includes four directions) and second (i.e., using the control modality that includes eight directions) group, respectively. The forks represent the comparison between the early (thick lines) and middle/late (thin lines) period. *: p < 0.05; **: p < 0.01; ***: p < 0.001.

Full size image
Fig. 3.
figure 3figure 3

Questionnaires results to assess the cognitive load, task difficulty and effort during of the performed reaching task. Left panel: results for the visual analog scale for both groups. Q1: “how hard it was to perform the reaching task?”; Q2: “how much did I have to concentrate to perform the reaching task?”; Q3: “how mentally loaded did I feel while performing the reaching task?”; Q4: “how effortful it was to perform the reaching task?”; Q5: “how tired was I after the reaching task?”; Q6: “how do you feel the arm velocity was” (the anchors were too slow, appropriate, too fast). Right panel: result for the NASA TLX for both groups. Qi: score reported for the question #i. The black and gray bars represent the first (i.e., using the control modality that includes four directions) and second (i.e., using the control modality that includes eight directions) group, respectively.

Full size image

When contrasting the value of CS, MT, ML and NJ during the early, middle and late practice periods, the results revealed a main effect for the factor Period for all four metrics (F(2,20) = 6.03, p < 0.009; all conditions considered). The post hoc analysis revealed that the values for MT and ML were significantly higher during the early compared to the middle and late practice p < 0.04; all conditions considered). The same comparison for the values for the CS and NJ during early compared to the middle practice revealed a significant difference (p = 0.05) and an absence of significant effect, respectively. However, the values of these two metrics revealed that they were significantly higher during the early compared to late practice period (p < 0.01; all conditions considered). No difference between middle and late period was observed (p > 0.05).

Finally, when analyzing the values for the post-test sessions, the results revealed a main effect of the feedback modality (F(1,10) = 16.51, p < 0.003; all conditions considered). The post hoc analysis, revealed that the CS, MT, ML and NJ were significantly larger for the performance with full compared to partial visual feedback (p < 0.003). No main or interaction effects for the factor groups and targets were observed (p > 0.05; all conditions considered). No main or interaction effects for the factor groups and targets reached significance (p > 0.05; all conditions considered).

3.2 Questionnaires

For both groups none of the components of the VAS and NASA TLX revealed any statistical differences (p > 0.19, all questions considered). It must be noted that the mental demand dimension assessed with both the VAS and NASA TLX provided convergent results.

4 Discussion and Conclusion

Overall, the findings revealed that individuals from both groups learned to relate their head movements to the spatial displacements of the robotic end-effector resulting in similar improvements of their reaching skills. Individuals of both groups were able to attain a similar degree of generalization of their reaching skills to targets not experienced before while showing a performance decrement with partial visual feedback (albeit the targets were reached). Similar motor performance for both groups was accompanied with a comparable perception of the degree of cognitive workload and more generally task difficulty and effort. At this time, it is critical to keep in mind that our findings were obtained with a limited number of subjects and as such the points discussed below should be taken with caution since further analyses with a larger sample and increased statistical power must be conducted.

First, it was observed that, compared to the four directional group, individuals of the eight directional group presented a similar cognitive-motor performance (cognitive workload, CS, MT, ML and NJ) whereas the latter had the possibility to employ the four diagonals as additional control signals to move the robotic end-effector. Thus, participants of the eight directional group do not seem to take advantage of these additional control signals to perform better considering the practice period allotted in this study. This finding expands and confirms those from previous studies that proposed and assessed head controlled pointing device with diagonal control directions. Namely, it was reported that some users used the diagonal directions only rarely or that the single-direction movements were more efficient compared to diagonal movements [4, 8, 33].

This is interesting since it could be expected that additional degrees of freedom in the control of the robotic arm to perform this reaching task could have (i) resulted in higher performance (e.g., straighter and faster reaching movements) and/or (ii) have an effect on the cognitive workload since a different system of commands may have modified the degree of challenge and thus affected information processing. However, it is also possible that while the performance metrics analyzed here are similar for both groups, additional long term training and/or specific training conditions for the individuals of the eight directional group may allow them to take advantage of the additional control signals enhancing thus their performance [8]. Further analyses could clarify this point.

In addition, a possible reason for similar (and moderate) levels of cognitive workload for both groups would be that the velocity at which the robotic end-effector was operated was perceived by individuals of both groups as appropriate (see VAS, question 6). The velocity at which the arm moves is important since any substantial augmentation of the velocity could result in an increase of the challenge and thus would likely increase the cognitive workload as shown on other tasks [18, 19]. Thus, the use of an appropriate end-effector velocity to perform the task may possibly place the individuals in the neighborhood of a reasonable level of challenge to learn the reaching movements. Although likely not optimal for some individuals, a reasonable level of challenge would place participants of both groups in a relatively suitable functional task difficulty which would not overwhelm their information processing system resulting in similar (and rather moderate) levels of cognitive workload [41].

The use of an appropriate end-effector velocity may also have contributed to enhanced transfer of performance to novel targets since it was previously suggested that the closer the level of challenge is to optimal, the greater learning benefit as transfer performance improves [41, 42]. Also, the relatively positive transfer of reaching performance to novel targets may also have been a result of how the targets were presented (i.e., which spatially covered the entire workspace). From a computational standpoint, to reach the novel targets, the human motor control system would perform neural computations that may be functionally similar to interpolations. It must be noted that in our case the practice paradigm required the participants to explore the entire workspace while executing reaching movements with the robotic arm. This type of practice method is more ecologically valid in comparison to restraining the individuals to reach specific and limited regions of the workspace using fewer targets as it is the case in classic paradigms that employ center-out reaching movements or standardized ISO (International Organization for Standardization) 9241-9 tasks. Although these laboratory tasks may provide a direct way to assess motor performance and learning processes, they have limited ecological validity.

While the targets were reached in the presence of limited visual feedback of the upper arm, the reduced performance indicates that, after practice, the participants still rely on visual inputs to control the robotic arm. A possible explanation about this reliance on visual feedback may be due to the nature of the control which results into some reduced association between sensorimotor signals related to head commands and the end-effector displacements which are discrete and continuous, respectively [43, 44]. Further analyses with a larger pool of participants should help to further inform this particular aspect of sensorimotor feedback and control.

Future work will include other conditions to examine how the manipulation of the end-effector velocity can influence the cognitive-motor performance of the participants. The long term goal of this work is to inform: (i) the human cognitive-motor processes underlying learning and performance during arm reaching movements executed through human-machine interface when humans perform alone or when teaming with an adaptive system, as well as (ii) develop intelligent human-robotic interfaces and rehabilitation systems for individuals with severe motor disabilities.