1 Introduction

Brain-computer interface (BCI) systems attempt to augment or expand a user’s control capabilities whereby the user controls a computer directly with his or her thoughts [1]. Direct use of captured brain signals allows a BCI to bypass the neuromuscular system, providing a promising research avenue for restoring communication or movement in patients suffering from injury or neuromuscular diseases [1]. This could take the form of a robotic wheelchair [2], a remotely-controlled assistive robot that navigates a building [3], or even a prosthetic limb that responds to neural signals like a biological limb [4]. BCIs could also prove useful to fully-abled users for teleoperating a robot in a remote location, potentially providing faster and more intuitive control either alone or as an enhancement to traditional interfaces such as joysticks, voice control, or typing commands into a computer terminal.

The ideal, field-deployable BCI system should be safe, intuitive, and practical to use. Functional near-infrared spectroscopy (fNIRS) is an emerging optical neuroimaging technique that can be applied as wearable and battery-operated miniaturized devices [5]. Imagined body movements (motor imagery) are frequently used in BCI and are regarded as an intuitive control method. This work outlines the development of a four-class, motor-imagery-based optical BCI to control a small bipedal humanoid robot, the DARwIn-OP.

Four mental tasks are mapped to navigation commands: turn right, turn left, move forwards, and move backwards. Visual feedback is provided via a first-person view of the room through either a virtual representation or a camera in the robot’s head presented on a computer monitor. The fNIRS-BCI-robot system developed here serves as a prototype for a telepresence or assistance robot navigating in remote and/or inaccessible locations. To our knowledge, this is the first report of a four-class motor-imagery-based fNIRS BCI-robot system.

2 Background

2.1 Motor Imagery, Movement, and the Motor Cortex

Motor movement (or motor execution) is the process of physically moving an area of the body, such as finger tapping. Motor imagery is an imagined movement of the body during which the muscles remain inactive. It is a conscious utilization of the unconscious motor preparation performed before a motor movement [6]. Because of this close association between motor imagery and naturally produced movement preparation, motor imagery could provide an intuitive mapping for BCI commands.

The primary motor cortex (M1) of the brain is the main control area where a one-to-one relationship between motor movement and brain activation exists, and is subdivided into sections responsible for control of the different areas of the body (the cortical homunculus) [7]. Some studies have shown motor imagery to have a similar activation pattern to motor execution, although the signals are significantly weaker [8, 9].

The brain’s hemodynamic response is the rapid delivery of oxygen-rich blood via the hemoglobin protein in red blood cells in response to the immediate need of the brain tissue and is correlated with functional activity such as cognitive tasks and motor activity [10, 11]. Oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) levels in cortical tissue can be measured to track the flow of blood throughout the brain. Both motor imagery and motor movements cause a rise in HbO and decrease in HbR, although the motor imagery response is slower in time and smaller in scale [12, 13].

2.2 Functional Near-Infrared Spectroscopy (fNIRS)

Functional near-infrared spectroscopy (fNIRS) is a noninvasive optical brain imaging technique that has shown promise for BCI applications [14–17], including the detection of motor movements [18, 19] and motor imagery [20, 21]. It uses near infrared light to measure changes in HbO and HbR levels due to the hemodynamic response. In the common configuration, light sources and detectors are placed on the scalp and two wavelengths of light are transmitted through the top layer of the cerebral cortex. Light at wavelengths between approximately 700–900 nm can pass through skin, bone, and water, but is absorbed mainly by HbO and HbR and at different rates [22]. The relative change in HbO and HbR, and therefore the oxygenation of the tissue, can be calculated from changes in the reflected light using the modified Beer-Lambert law [23].

Many fNIRS devices are relatively low-cost, portable, and potentially wireless [5]. This allows them to be used in more natural settings, such as sitting at a desk, rather than in restricted and artificial lab environments. Despite the time delay due to the slow hemodynamic response, fNIRS provides a unique trade-off between time and spatial resolution and is free from most artifacts, such as muscle activity and eye blinks. It can also be used with other measurements, such as physiological signals [24] and electroencephalography (EEG) [25–27].

2.3 Motor Imagery and Robot-Control BCIs

Overall, preliminary studies show promise for fNIRS-based BCIs. fNIRS has higher spatial resolution than EEG and higher time resolution than fMRI, which provides a balanced trade-off. Several EEG- or fMRI-based studies have used motor imagery as the sole input method with up to four different tasks [28, 29]. It may also be possible to distinguish between simple and compound motor imagery (e.g. moving right hand vs. right hand and foot together) [30]. Many fNIRS motor imagery experiments focus on detecting imagery from a single hand vs. resting state [20], left hand vs. right hand [21, 31], or three motor imagery tasks and rest [32].

Robot control is a natural application of BCIs, and previous BCIs have controlled flying [33], wheeled [2, 34], and humanoid [3, 35–37] robots. Incorporating robot control into a BCI provides visual feedback and can increase subject motivation during use. Improved motivation and feedback, both visual and auditory, have demonstrated promise for reducing subject training time and improving BCI accuracy [38, 39]. While many studies have focused on EEG or fMRI as a control method, fNIRS has also been used for robot control, although previous studies have focused on motor movement or other mental tasks [40–43].

3 Methods

3.1 Participants

Eleven healthy participants volunteered in the experiment. Subjects were aged 18–35, right-handed, English speaking, and with vision correctable to 20/20. No subjects reported any physical or neurological disorders, or were on medication. The experiment was approved by the Drexel University institutional review board, and subjects were informed of the experimental procedure and provided written consent prior to participating.

3.2 Data Acquisition and Protocol

Motor imagery and motor movement data were recorded in three one-hour-long sessions over the course of three days. The first two sessions are training days, used to collect initial data to train a classifier, and the third day is used to test the BCI by navigating a robot within a room containing a goal location and an obstacle to be avoided. A total of 80 motor imagery and 32 motor movement trials were collected for each subject over the course of the two training days, with an additional 40 motor imagery trials collected during the robot-control day. The four tasks were the (actual or imagined) tapping of the right hand, left hand, right foot, and left foot. Preliminary results for three subjects using the two training days have been reported previously [44].

Twenty-four optodes (physical locations) over the motor cortex were recorded using a Hitachi ETG-4000 optical topography system, as shown in Fig. 1(A). HbO and HbR levels were recorded at each location at a sampling rate of 10 Hz.

Fig. 1.
figure 1

(A) Sensor layout showing light sources (red squares) and detectors (blue squares). Optodes are numbered 1–24. Adjacent sources and detectors are 3 cm apart. (B) Trial timing diagrams for training days 1 & 2 (top) and robot control (bottom). The variable n represents the length of time taken to record participant’s intended task. (Color figure online)

Full size image

The timings for training and robot-control days are shown in Fig. 1(B). Each trial began with 6 s of rest followed by a cue to indicate the upcoming task. For the two training sessions this text indicated a specific task (e.g. left foot), while during the robot-control task it read “free choice”, indicating the subject should choose the task corresponding to the desired action of the robot. Subjects then performed the designated task for 30 s. During the robot control session, the task was followed by a pause so that the subject could indicate which task they had performed. All trials ended with a 15-s period indicating the task was over. On the robot control day, the time during the results stage was used by a support vector machine (SVM) which had been trained on data from the two training sessions to predict the task being performed. The corresponding command was sent to the robot, which took the corresponding action.

Each day was split into two runs, as shown in Fig. 2. There was a brief pause between the runs until the participant indicated they were ready to continue. The two training days began with 16 motor movement trials, followed by 40 motor imagery trials. Motor movement was performed before motor imagery in order to improve the subject’s ability to imagine performing the task. Each training session run had an equal number of the four tasks (motor execution or motor imagery of the right hand, left hand, right foot, and left foot) in a randomized order.

Fig. 2.
figure 2

Experiment protocol and trial organization

Full size image

For the third day (robot control), the two runs were identical, except that in the first run they were controlling a virtual robot and in the second they controlled an actual robot. Subjects selected a motor imagery task corresponding to the desired action of the (virtual or physical) robot. The task-to-command mappings are: left foot/walk forward, left hand/turn left 90°, right hand/turn right 90°, and right foot/walk backward. These four tasks were chosen to emulate a common arrow-pad setup, and so that each action had a corresponding opposite action that could undo a movement.

Robot-Control Task. The robot-control session had two parts, beginning with control of a virtual robot and followed by control of an actual robot. During the first run, a second monitor displayed a first-person view of a virtual maze using the Maze Suite program [45], as shown in Fig. 3(A). During the second run, this monitor displayed a first person view recorded from a camera located in the head of the DARwIn-OP, a small humanoid robot [46], shown in the experiment setup in Fig. 3(B). DARwIn-OP is 0.455 m tall, has 20 degrees of freedom, and walks on two legs in a similar manner to humans. The objective in both runs was to use the BCI to navigate through a series of three room designs, in which there was a single goal location (a green cube) and an obstacle (a red cube). The room layouts for the virtual and physical robots were designed to be as similar as possible. This allowed the subjects to acquaint themselves with the new robot-control paradigm before adding the complexities inherent in using a real robot.

Fig. 3.
figure 3

View of (A) virtual room and task screen and (B) the physical robot setup. (Color figure online)

Full size image

A room was successfully completed if the user navigated to the green cube, and it was failed if the user touched the red cube. After completion or failure of a room, the subject would advance to the next room. The run ended if the subject completed (or failed) all three rooms or reached the maximum of 20 trials.

3.3 Data Analysis

Data from all three days (two training days and a robot-control day) were analyzed offline in order to perform an initial evaluation of the system. The approach described here could also be applied for a real-time BCI for robot control. The raw optical data was converted to HbO and HbR data, and then low-pass filtered using a 20th order FIR filter with a cutoff of 0.1 Hz. Artifacts and optodes with poor signal quality in the two training sessions were noted by an expert and contaminated sections were removed. Four subjects’ data were excluded entirely due to extremely poor signal quality. No optode cleaning was applied to the robot-control data, in order to more accurately simulate real-time field conditions. Common average referencing (CAR) based spatial filtering was used to enhance the signal quality. CAR is a simple method, commonly used in EEG, in which the average value of all optodes at each time point is used as a common reference (i.e. that value is subtracted from each optode at that time point). This enhances changes in small sets of optodes while removing global spatial trends from the data. The data for all optodes in each task period were then baseline corrected by taking the average of the first 2 s of the task period, then subtracting that value from all of the data in that task for that optode.

Features were calculated on the total hemoglobin (HbT) at each optode, as this outperformed HbO, HbR, and Oxy (the difference between HbO and HbR), on average across subjects in a preliminary analysis on the first two days [44]. The average value for each optode was calculated for each task period from 9–15 s after the start of the task as this period is the expected peak activation.

The features were classified using a linear support vector machine (SVM) using the scikit-learn package [47]. Results are reported as accuracy (average number of correct classifications), precision (positive prediction value), recall (sensitivity or true positive rate), and F-score (the balance between precision and recall). The F-score is calculated as shown in Eq. 1.

$$ {\text{ F-score}} = 2 \times \frac{{{\text{precision}} \times {\text{recall}}}}{{{\text{precision}} + {\text{recall}}}} $$
(1)

4 Results

Table 1 shows the offline results of the robot-control data (Day 3) using a classifier trained using the training sessions (Day 1 and 2). To reduce the effect of running the experiment across multiple days, half of the robot-control data (randomly selected, evenly distributed across tasks) was added to the training data from the first two days to calibrate the classifier to the new cap position. Figure 4 and Table 2 show the results after 10 repetitions of randomly assigning the robot-control data between training and test sets. The results shown in Tables 1 and 2 were compared to test the significance of adding same-day calibration trials to the training data. Four paired, one-sided t-tests indicated significant improvement as reported in Table 3.

Table 1. Results for robot control using day 1 & day 2 as training data
Full size table
Fig. 4.
figure 4

Accuracy using days 1, 2, and half of the robot control data for training. Error bars show the standard deviation across 10 repetitions, with the error bar for average showing the standard deviation of the accuracy across subjects.

Full size image
Table 2. Results using days 1, 2, and half of the robot control data for training
Full size table
Table 3. Results (p-values and 95 % confidence intervals) for one-sided paired t-tests comparing accuracy measures with and without same-day calibration data included in the training data
Full size table

5 Discussion

This is the first study to our knowledge that attempts to create a four-class motor imagery fNIRS BCI for robot control. The preliminary results show that seven subjects achieved an average accuracy of 32.5 % on a four-class fNIRS BCI robot control task with minimal preprocessing to reflect field conditions. Including a portion of the data from day 3 (robot control) into the classifier significantly improved the accuracy for several subjects, increasing the average accuracy to 43.6 %. There was a great deal of variability in performance between subjects, with subject S0 achieving 60 % or higher accuracy using both methods, and subject S5 achieving over 60 % accuracy with the addition of same-day calibration data. Many factors could account for this variability, including differences in participants’ ability to perform motor imagery as well as physiological differences (e.g. hair thickness) that can interfere with fNIRS recordings.

Table 3 shows that the improvement on all four metrics was statistically significant (using α = 0.05 and false discovery rate correction). This indicates that the inclusion of a calibration session prior to using a BCI improves the accuracy by adding examples to the training set that are more similar to those present in the testing set. These new examples would have the same cap position as the test set data, while the training data from previous days may have a slightly different cap placement. Additionally, brain functionality in individuals varies over time, potentially causing changes in the recorded data across days. Inclusion of calibration tasks immediately before the BCI could provide more relevant examples to the classifier.

A potential improvement to the protocol is to shorten the task length from 30 to 15 s. Preliminary analyses show that using all 30 s of the task did not improve results, and several participants self-reported mental strain caused by sustaining the same imagery for 30 s. The shorter time window would also allow for nearly twice the amount of viable data taken within the same session time period.

Brain-computer interfaces using fNIRS show great promise for both the neuromuscularly disabled as well as the healthy user. They have far-reaching applications in the fields of computing, telepresence robotics, and prosthetic control, and are becoming increasingly common. Our pilot study demonstrates the feasibility of using a four-class motor imagery-based fNIRS BCI to control a humanoid robot. Additionally, the use of more data and a calibration session immediately prior to the use of the BCI may improve accuracy.