Brain-Controlled Selection of Objects Combined with Autonomous Robotic Grasping

Abstract

A Brain–Computer Interface (BCI) could help to restore mobility of severely paralyzed patients, for instance by prosthesis control. However, the currently achievable information transfer rate of noninvasive BCIs is insufficient to control complex prostheses continuously in many degrees of freedom. In this paper we present an autonomous system for grasping natural objects that compensates the low information flow from noninvasive BCIs. Using this system, one out of several objects can be grasped without any muscle activity. Rather, the grasp is initiated by decoded voluntary brain wave modulations. Object selection and grasping are performed in a virtual reality environment. A universal grasp planning algorithm calculates the trajectory of a gripper online. The system can be controlled after less than 10 min of training. We found that decoding accuracy increases over time and that an increased sense of agency achieved by permitting free selections renders the system to work most reliably.

Appendix

In Sect. 2.4 we stated the rasterizing of the object and gripper surfaces with virtual point poles. Here we describe the algorithm in more detail.

Our grasp planning algorithm is organized by simulating the action of forces between target object and manipulator in consecutive time frames. While the object poles P ^O are defined as positive, the manipulator poles P ^M are defined as negative. In accordance with Khatib [27], we assume that opposite poles attract each other while like poles do not interact. The magnitude of the force between two poles P ^O _i and P ^M _j we calculated as

$$ \overrightarrow{F}\left({P}_i^O,{P}_j^M\right)={e}^{-\overrightarrow{P_i^O{P}_j^M}} $$

(1)

where $\overrightarrow{P_i^O{P}_j^M}$ is the distance between the poles, and the unit of F is arbitrary. The exponential function limits F to a maximum of 1 unit. This avoids infinite forces at collision scenarios and provides a suitable scaling to instantiate both propulsive forces between manipulator and object and repulsive forces to reject manipulator poles that penetrate the object’s boundary.

The total propulsive force $\overrightarrow{F}\left({P}_i^M\right)$ affecting one point pole P ^M _i on the manipulator is calculated from a set of object point poles A _O where

$$ {A}_O\left({P}_i^M\right):=\left\{{P}_j^O\Big|{P}_j^O\in {P}^O \wedge {\overrightarrow{n}}_j^O\cdot {\overrightarrow{n}}_i^M<0\right\} $$

(2)

which indicates that only pairwise point poles with an angle between the surface normal ${\overrightarrow{n}}_i^M$ and ${\overrightarrow{n}}_j^O$ greater than π/4 are involved. We included this constraint to restrict interactions to opposing surface force vectors. The force $\overrightarrow{F}\left({P}_i^M\right)$ that moves the manipulator is then calculated as

$$ \overrightarrow{F}\left({P}_i^M\right)={\displaystyle \sum_{P_j^O\in {A}_O\left({P}_i^M\right)}\overrightarrow{F}\left({P}_i^O,{P}_j^M\right)} $$

(3)

The manipulator’s effective joint torque $\overrightarrow{\tau}$ can be calculated by means of the Jacobian J generated from the joint angles $\overrightarrow{q}$ and the point poles P ^M [28] by

$$ \overrightarrow{\tau}={\displaystyle \sum_iJ{\left({P}_i^M,\overrightarrow{q}\right)}^T\left[\begin{array}{c}\hfill \overrightarrow{F}\left({P}_i^M\right)\hfill \\ {}\hfill \overrightarrow{M}\hfill \end{array}\right]} $$

(4)

where external moments are considered $\overrightarrow{M}=\overrightarrow{0}$. In order to simulate the manipulator movement, we calculated the new joint angle q _k (t) of an axis k by solving the equation system

$$ {\dot{q}}_k(t)={\dot{q}}_k\left(t-\Delta t\right)+\Delta t*\frac{\tau_k}{{\overrightarrow{a_k}}^TI\left(\overrightarrow{q}\right)\overrightarrow{a_k}} $$

(5)

$$ {q}_k(t)={q}_k\left(t-\Delta t\right)+\Delta t*{\dot{q}}_k(t) $$

(6)

where $I\left(\overrightarrow{q}\right)$ is the inertia tensor of the robot’s solid elements and $\overrightarrow{a_k}$ defines one of the manipulator axes. We chose a heuristically dynamic calculation of the time frame length Δt which is proportional to the mean distance between the set of point poles P ^M and P ^O.

Collision detection was performed for the new posture before a new time frame was assigned to be valid and the position update was sent to the manipulator. We used standard techniques [29] to detect surface intersections. If intersections were detected, repulsive forces were calculated for the affected point poles directing to their position of the last valid time frame and satisfying Eq. (1). If no intersections were detected, the robot moved to the new coordinates. This procedure was repeated until the force closure condition [30] was satisfied.