1 Introduction

Humanoid robots are inherently unstable due to their small footprint compared to the height of their center of mass (CoM) [24]. This is a problem widely encountered in locomotion generation for robots such as humanoid walking and kicking. Even though there are many improvements in fall prevention and controlled walking [7, 26] robots are still at risk to fall due to obstacles, impacts, or uneven/soft ground.

Fig. 1.
figure 1

Stepping motion to prevent a fall using the SSIP model. (Color figure online)

Full size image

In order to prevent a humanoid robot from falling, two steps are necessary: First, the fall has to be detected. Fall detection must be fast in order to have sufficient time left to perform a counter-action. Furthermore it has to be robust to false-positive detections to prevent the robot from initiating a counter action when it is not necessary. To this time, fall detection is mostly achieved using a form of sensor classification. In contrast to other existing methods, the approach presented in this paper does not only classify sensor data to detect an ongoing fall, but it rather determines whether the sensor data can lead to a fall in the near future. A physical model is used to simulate the effect of the forces acting on the robot. The model can be used to calculate whether the robot will be in a stable or an unstable state. In case of an unstable state, the model calculates how the robot will fall. This can be used to prevent the robot from falling over.

The main contribution of this paper is the development of a physical model for a humanoid robot capable of describing the robot during a fall. We extend the commonly used inverted pendulum model by a stand space and body dynamics of the robot. Analyzing the common approach to use linearization for the non-linear inverted pendulum shows that it is not feasible to describe a falling motion. Therefore, we use an iterative model to simulate the dynamics of the robot. This model is used to detect imminent falls and perform a counter motion. We demonstrate that a fall can be detected significantly earlier in comparison to traditional sensor classification methods. Falls due to small to medium impacts to the robot can mostly be prevented using a lunge.

The paper is organized as follows: In Sect. 2, we analyze related work, followed by the development of the physical model and functions in order to detect and prevent a fall in Sect. 3. In Sect. 4, the evaluation of the developed approach is described. A comprehensive discussion of the results is given in Sect. 5, followed by the conclusion in Sect. 6.

2 Related Work

Fall detection is reliant on sensors such as inertia sensors and pressure sensors. A classification of the sensor data into stable states and unstable states is a common approach used in several works [16, 18]. Such approaches are very robust but detect falls rather late. A related approach by Höhn et al. uses pattern recognition on the sensor data to detect specific patterns only present during a fall [8]. Other approaches use a model to predict the expected sensor values. The deviation between the expected sensor values and the real sensor values is used for fall detection. Renner and Behnke model the expected sensor data using sinusoidal functions [15]. In order to detect falls during layered motions, Tay et al. [20] use interpolated sensor models based on body angles. Also machine learning can be utilized for training a sensor model [10]. Latest approaches use two-level classification neural networks in combination with regression to achieve high levels of accuracy in fall detection and posture monitoring [1].

A general concept to evaluate the stability of humanoid robots is the Zero-Moment-Point (ZMP) [21]. The ZMP is widely used to stabilize humanoid walking [7] and kicking [22]. However, Renner and Behnke [15] as well as Höhn et al. [8] evaluated the ZMP as not being suited for fall detection. Also, the Foot-Rotation Indicator (FRI) [6], was not applied to fall detection yet.

A common approach for the fall prevention is the integration into the generation of the walking motion [2, 7]. Dynamic adaptations of the gait phases leading to different step sizes and step durations are able to compensate small impacts and irregularities such as an uneven floor. An extension to this approach is the calculation of a specific point for the foot placement. The Capture Point (CP) [14] and the Foot-Placement-Estimator (FPE) [23] both rely on this approach. Most of these approaches use an inverted pendulum model as a basis. Zhao et al. [27] argue that this model does not model important dynamics of the hip and the ankle as well as the ground reaction force. They developed a hip-ankle and bent-knee strategy to deal with instabilities [11]. A combination of both strategies with machine-learned weights is used by Yi et al. [25].

Mao et al. argue that the Capture Point approach is not feasible for practical use. They developed a continuous step controller [13] that counters the instability with a series of steps. In order to deal with uneven ground, Lee and Goswami [12] developed a moment-based controller that uses the ground reaction force and the center of pressure minimizing the torque on the foot. An extension is the reactive stepping controller [26] modeling the robot as a turning wheel. In contrast to the CP and FPE, a specific target point for the foot can be calculated and it is not necessary to continuously update the target point to a position further away from the origin. This is especially helpful performing a controlled motion on a pre-calculated trajectory.

3 Approach

In order to prevent a humanoid robot from falling, a fall detection method is required, capable of robustly detecting a fall as early as possible. This is necessary to have enough time left to perform a counter-motion. All related work uses some kind of classification of sensor data in order to detect specific patterns only present during a fall. These approaches only detect a fall when it is already in progress. Analyzing a falling motion of a humanoid robot, multiple stages can be identified. An event that brings the robot into instability accelerates the upper body of the robot. The stand space (convex hull of the foot/feet on the ground) generates a counteracting force reducing the acceleration of the upper body. In this first stage, it is not clear whether the robot will only stagger or fall down. If the upper body acceleration is too strong, the robot will fall. Otherwise it will only stagger, which should not be detected as a fall. Therefore, all other approaches do not classify this stage as falling, because it is not clear yet whether a fall will happen or not.

To detect a fall as early as possible, it is already necessary to evaluate whether a fall will happen later on in this first stage. In order to make a statement whether the measured sensor data will lead to a fall in the future, it is necessary to calculate how they will effect the robot. This task is done using a physical model of the robot applying the measured accelerations and velocities. The Linear Inverted Pendulum model (LIP) models the movement of the robot’s CoM as a 3-dimensional linear inverted pendulum [9], see Fig. 2. An inverted pendulum is suited to describe a falling robot as falling is a rotational motion around the edge of the foot. However, the simplified LIP model is not suited to model the robot during a fall for two reasons: First, the stand space of the foot, which generates a counteracting force to the fall, is not considered. In addition, all forces generated by the joints of the robot are also not integrated into this model. Second, the non-linear pendulum equations cannot be calculated directly. Therefore, they are linearized in this model. Linearization is done by the assumption that either \(sin(\alpha ) \approx \alpha \) for small values of \(\alpha \) or the height of the pendulum mass from the ground is constant [7]. When generating upright motions such as walking, these assumptions can be made. In the case of modeling a fall however, they are not valid. As linearization is not an option and precise modeling is necessary, we develop an iterative model using a numerical approximation of the non-linear differential equation with the Euler integration.

Fig. 2.
figure 2

Schematics for the inverted pendulum model (left) and the extended Stand Space Inverted Pendulum model (right).

Full size image

3.1 Stand Space Inverted Pendulum Model (SSIP)

In order to design an iterative inverted pendulum model, we develop a function for the angular acceleration \(\ddot{\theta } \in \mathbb {R}^{3}\) of the CoM. Such a function can be defined using the torque \(\tau \in \mathbb {R}^{3}\) acting on the base of the pendulum. It uses the vector \(r \in \mathbb {R}^{3}\) of the lever arm of force, the linear force \(F \in \mathbb {R}^{3}\) produced by the mass m, and the gravity \(g \in \mathbb {R}^{3}\) as well as the inertia tensor \(I \in \mathbb {R}^{3x3}\).

$$\begin{aligned} \tau&= r \times F \end{aligned}$$
(1)
$$\begin{aligned} \tau&= I \ \ddot{\theta } \end{aligned}$$
(2)
$$\begin{aligned} I \ \ddot{\theta }&= r \times (m \ g \ sin(\theta )) \end{aligned}$$
(3)
$$\begin{aligned} \ddot{\theta }&= (r \times (m \ g \ sin(\theta ))) \cdot I^{-1} \end{aligned}$$
(4)

Only the tangential part of the gravity is acting to rotate the mass, which is dependent on the deflection \(\theta \in \mathbb {R}^{3}\) of the pendulum, see Fig. 2. An iterative term for the angular velocity \(\dot{\theta } \in \mathbb {R}^{3}\) and the deflection can be defined using Euler integration.

$$\begin{aligned} \dot{\theta }_{n + 1}&= \dot{\theta }_{n} + \triangle t \cdot (r \times (m \ g \ sin(\theta ))) \cdot I^{-1} \end{aligned}$$
(5)
$$\begin{aligned} \theta _{n + 1}&= \theta _{n} + \triangle t \cdot \dot{\theta }_{n + 1} \end{aligned}$$
(6)

Iterating Eqs. 5 and 6 with a small \(\triangle t\) give a precise approximation of the rotational motion of the pendulum. However, this model cannot assert whether the robot will fall anyway. Therefore, we integrate the stand space of the robot given by its feet. The pendulum has a plate attached to the bottom and the point of rotation moves to the edge of this plate, see Fig. 2. The plate generates a stabilizing torque counteracting the tangential part of the gravity which is responsible for the angular motion of the pendulum and the respective fall of the robot. We can define this counteracting torque \(\tau _{stand}\) using the vector from the origin of the foot to the point of rotation on the edge of the foot \(l \in \mathbb {R}^{3}\) and the gravity. But the lever arm of force l is dependent on the deflection of the pendulum.

$$\begin{aligned} \tau _{stand}&= (l_{G} - l_{1}) \times g \end{aligned}$$
(7)
$$\begin{aligned} \tau _{net}&= \sum _{i} \tau _{i} \end{aligned}$$
(8)
$$\begin{aligned} \ddot{\theta }&= (\tau _{fall} + \tau _{stand}) \cdot I^{-1} \end{aligned}$$
(9)

The length between the projection of the gravitational force into the stand space and the edge of the stand space defines the acting part of the the stabilizing torque’s lever arm of force, \(l_{2}\). Using the length r and the deflection of the pendulum, we can calculate the inactive part \(l_{1}\). By subtracting \(l_{1}\) from the whole length of the stand space in the direction of the fall, we can obtain the acting part \(l_{2}\), see Eq. 7.

Utilizing the fact that the net torque can be defined as the sum of all individual torques acting around an axis, see Eq. 8, we can sum up the stabilizing torque and the one responsible for the fall and integrate them into Eq. 9.

This extended model could already give an indication whether the robot will fall, when initialized with the measured sensor data and then iterated. But the robot might perform a motion while getting into instability. Such motion generates an additional torque that can amplify or counteract the falling motion. Therefore the torque generated by the motion of all joints of the robot in respect to the rotation point at the edge of the foot has to be calculated.

The inverse rigid body dynamics [5] describes the problem of calculating the torques based on the kinematics (motion) of a body and the body’s inertial properties. A model of the robot’s rigid body given by the mass and moment of inertia of all body parts and how they are connected is needed. In addition with the generalized angular positions (q), velocities (\(\dot{q}\)), and accelerations (\(\ddot{q}\)) of all joints, the inverse dynamics can be defined as \(\tau = ID(model, \, q, \, \dot{q}, \, \ddot{q})\). In order to calculate the inverse dynamics, the recursive Newton-Euler algorithm is used in this work.

$$\begin{aligned} \ddot{\theta }&= (\tau _{fall} + \tau _{stand} + \sum _{i \in N} \tau _{i}) \cdot I^{-1}\end{aligned}$$
(10)
$$\begin{aligned} \dot{\theta }_{n + 1}&= \dot{\theta }_{n} + \triangle t \cdot (\tau _{fall} + \tau _{stand} + \sum _{i \in N} \tau _{i}) \cdot I^{-1} \end{aligned}$$
(11)

The obtained torques for each joint i around the axis orthogonal to the fall direction can be summed up and added to Eq. 9, resulting in Eq. 10. The sum of torques around the fall axis can be used to calculate the complete rotational acceleration acting on the robot. This can be used to form the final iterative model (Eq. 11) using Eqs. 5 and 6.

Initializing this model with the measured angle and angular velocity of the robot and iterating it will either provide an upright stabilizing pendulum motion or a rotational pendulum motion. We can decide whether the robot will fall directly based on this simulation. When the Stand Space Inverted Pendulum model will fall, also the robot will fall down approximately following the simulated motion.

3.2 Fall Prevention

In order to prevent the robot from falling, one or more steps are necessary [19]. Orienting towards a human model, a dedicated long-range step is most promising. Performing such a step requires one foot to keep contact to the ground, and one to perform the stepping motion. Deciding which foot performs which task is dependent on the direction of the fall and the current position of the feet. If one foot is already in mid-air, it is used to perform the step. If both feet are on or close to the ground, the foot in the direction of the fall has to keep contact with the ground as the point of rotation is at its edge.

Such a step should meet two requirements in order to prevent the fall. The rotational acceleration as well as the rotational velocity have to be compensated to stop the rotational motion of the fall. Regarding the first requirement, we can refer to the inverted pendulum model, see Fig. 2. The rotational acceleration is caused by the tangential part of gravity, which depends on the deflection of the pendulum. It is no longer accelerated if the base is positioned directly under the CoM. This can be achieved by placing the stepping foot directly under the CoM, which will then become the base of the pendulum.

Calculating the target position for the foot can be done using the motion of the robot’s CoM over time calculated by the model. For convenience, we can define the iterative SSIP model as a closed function \(\theta _{t} = SSIP(\theta _{0}, \dot{\theta _{0}}, t)\). This can be used to calculate the position of the CoM \(\mathcal {P}\) at time t using the Rodrigues formula Rot() [3] and the CoM in upright position \(CoM_{0}\), see Eq. 12.

$$\begin{aligned} \mathcal {P}(t)&= Rot(SSIP(\theta _{0}, \dot{\theta _{0}}, t)) \cdot CoM_{0}\end{aligned}$$
(12)
$$\begin{aligned} \mathcal {T}(t)&= T_{POR \rightarrow Robot} \cdot \begin{bmatrix}\mathcal {P}(t)_{x} \\ \mathcal {P}(t)_{y} \\ footheight \end{bmatrix} \end{aligned}$$
(13)

Equation 12 gives us the position of the CoM at time t with respect to the point of rotation (POR). As we know the point of rotation at the edge of the foot, we can transform the function into the robot coordinate system in order to actuate the joints using inverse kinematics [16]. As the foot should be positioned on the ground with its sole, the height of the last joint from the sole footheight is used as the z-coordinate.

In addition to the rotational acceleration, the rotational velocity has to be compensated as well. We could actuate a joint in the opposite direction using the acceleration to compensate the velocity. But as motors often cannot be actuated precisely, this could cause another instability. Therefore we propose a different strategy: If the CoM has a deflection in the opposite direction of the fall when the stepping foot makes contact with the ground, the acceleration caused by the deflected pendulum and the rotational velocity of the fall can cancel each other out. This leaves the question of how much the pendulum has to be deflected in order to compensate the rotational velocity completely and coming to a stop in the upright position.

Making the simplifying assumption that the falling motion started in an almost upright position of the CoM, we already know which deflection is necessary to compensate the velocity. It is the current state of the pendulum in the opposing direction. Such a deflection could be achieved by doubling the distance of the target position of the foot from the POR. But when the stepping foot makes contact with the ground, its stand space will also compensate a part of the velocity. We do not have to calculate how much rotational velocity it will compensate, but rather how much we can reduce the deflection while still being in a stable state. As the projection of the gravitational force acting on the CoM has to be within the stand space, see Sect. 3.1 and Fig. 2, in order to still be stable, we can calculate the deflection using \(tan^{-1}\frac{l_{G}}{r}\).

$$\begin{aligned} \mathcal {D}(t)&= Rot(SSIP(\theta _{0}, \dot{\theta _{0}}, t) - tan^{-1}\frac{l_{G}}{r}) \cdot CoM_{0} \end{aligned}$$
(14)
$$\begin{aligned} \mathcal {T}(t)&= T_{POR \rightarrow Robot} \cdot \begin{bmatrix} \mathcal {D}(t)_{x} + \mathcal {P}(t)_{x} \\ \mathcal {D}(t)_{y} + \mathcal {P}(t)_{y} \\ footheight \end{bmatrix} \end{aligned}$$
(15)

Instead of doubling the distance between the POR and the target position, we can define a second function for the target position integrating the deflection that will be compensated by the stand space, see Eq. 14. By adding up both target positions before transforming them into the robot’s coordinate system, we can achieve a target position for the foot that will compensate both rotational acceleration and velocity, see Eq. 15.

Fig. 3.
figure 3

Experimental setup

Full size image

A trajectory for the stepping foot can be computed using the function \(\mathcal {T}(t)\) from Eq. 15. Figure 1 visualizes the trajectory with the reachable poses in green and non-reachable parts in red. In order to keep the robot stable on the foot after the step, an additional closed-loop PID controller for the hip and knee joints is used. A controlled transition into the stand is used to bring the robot back into a standard operation state.

4 Evaluation

A series of experiments was performed to evaluate the method developed. The algorithms described were implemented for the humanoid robot NAO [17] using the B-Human code release [16]. All experiments used this robot and shared the same setup (cf. Fig. 3). A controlled impact to the upper body of the robot was generated using a padded weight of 800 g attached to a cord of 1m, which was fixated above the robot. The weight was deflected to different angles to produce impacts with varying strength. For each deflection set, impacts from four sides (left, right, front, and back side of the robot) were tested with five repetitions each. For the evaluation of the fall detection, two types of experiments were performed: A general functionality test (a) evaluating the correct detection of the state the robot is in. The second experiment (b) compared a traditional sensor classification method used by the team B-Human [16] with the method presented in this paper. For the fall prevention, an additional functionality test (c) was done. The results of all experiments can be reviewed in Tables 1, 2 and 3. All experiments were performed with an impact of different strength indicated by the displacement of the pendulum and from the different directions front (f), back (b), left (l), and right (r).

Table 1. Results of the fall detection functionality experiment (5 trials each).
Full size table
Table 2. Results of the fall detection comparative experiment (5 trials each)
Full size table
Table 3. Results of the fall prevention functionality experiment (5 trials each)
Full size table

For the functionality test of the fall detection (cf. Table 1), four different impacts with increasing strength (two led to staggering, two led to a fall) were tested in addition to the trivial case of normal walk without an impact. 80 passes were performed in total. In the cases of trivial walking and staggering due to small impacts, the state was correctly classified as stable. Also the cases that led to a fall due to strong and very strong impacts were correctly classified as a fall. Only in the case of strong staggering due to a medium impact, the detection rate dropped to a value between 60% and 0%.

The comparative experiment for the fall detection was performed with two types of impacts (\(45^{\circ }\) and \(60^{\circ }\) weight deflection), both led to a fall. The time between the impact and the detection of a fall was measured. The results are presented in Table 2. For all eight types of impacts (four sides times two different strengths, 40 passes in total), the method presented in this paper was able to detect the fall significantly earlier (t-Test with \(p < 0.001\)). This resulted in earlier detection between 408 ms to 527 ms for the strong impacts and between 223 ms to 266 ms for very strong impacts.

The evaluation of the fall prevention was performed with three different strengths of impact (\(45^{\circ }\), \(55^{\circ }\) and \(65^{\circ }\) weight deflection). All impacts would have led to a fall without a counter-motion. Table 3 shows the results for all 60 passes of the experiment with the rate of successfully achieved fall preventions. For a weight deflection of \(45^{\circ }\), the fall could be prevented for almost all passes (18 of 20). With increasing strength of impacts, the successful fall preventions dropped to 7 of 20 for the deflection of \(55^{\circ }\) and to 3 of 20 for the deflection of \(65^{\circ }\). It is to note that impacts from the front, which led to a backwards fall of the robot, could be generally prevented much less than from the other sides whereas impacts from the sides could be prevented better.

5 Discussion

The approach developed in this paper uses a physical model of the humanoid robot in order to calculate the effects of sensor data towards the future motion of the robot and if this motion represents a fall. We argued that this approach should be able to detect a fall earlier than traditional classification of sensor data. The results of the comparative study (b) confirm this assumption, detecting a fall between a quarter to half a second earlier than the method used by the team B-Human in the RoboCup. It is to note that this is a comparison to a single specific approach and the findings cannot be generalized without further evaluation. However, Renner and Behnke [15] define the forewarning time as a criterion in their evaluation. The forewarning time is the duration between fall detection and the robot’s body reaching 25\(^\circ \) deviation from the upright posture. This deviation is also the criterion detecting forward and backward falling in the B-Human system. Therefore, the mean difference values from Table 2 are comparable to the forewarning times reported by Renner and Behnke. The times their method achieved is 500–600 ms, but under the assumption that the robot is executing a regular walk movement, which is exploited by their approach. In addition, most of their attempts would not have brought the robot to a fall even without a fall prevention, so they were definitely not comparable to the very strong impact category here, where the advantage of the new approach over the old one is smaller.

The functionality evaluation (a) revealed that when the NAO is heavily staggering (medium impact) but not falling, this approach falsely detects a fall. These false positive detections most likely originate from deviations between the model and the NAO robot. Hardware limitations of the NAO such as imprecise inertia and joint angle sensors and strong joint play in the leg joints are likely to produce a deviation from the model. Furthermore, the NAO does only provide joint angle measurements, but not the joint velocities and accelerations needed to calculate the inverse dynamics used in our model. Joint velocities and accelerations are therefore calculated by a motor model [4] and only give approximated values. These reasons indicate why false-positive measurements are likely.

The results from the evaluation of the fall prevention (c) show a distinct drop in the rate of successful fall preventions between \(45^{\circ }\) and \(55^{\circ }\) weight deflection. In the first case almost all falls could be prevented whereas in the latter less than half of all passes were successful. This can be explained by the fact that the leg motors of the NAO are too slow to reach the position necessary to prevent the fall. During the experiments, we observed that the leg which should be at least positioned under the CoM was not able to reach this position before the fall was no longer preventable. We argue that the rate of successful fall prevention could be improved for strong pushes using a hardware platform with superior motors for the leg joints. In general, pushes from the front could be prevented worse than others. This relates to the fact that the actuation limits of the NAO’s legs are more restricted to the back than to the front. Overall, falls to the side could be prevented better, which affiliates to the moment of inertia of the robot. Rotation to the front and back are more accelerated than to the sides.

Concluding our findings, the presented approach is capable of detecting fall earlier than the method we compared to and prevent falls in limitation to the capabilities of the hardware. The main limiting factor is the hardware of the NAO that was used for the evaluation. In future work, we would like to generalize our findings using a superior hardware platform. In addition, we could model the robot in more detail if sufficient computation power is accessible to iterate the model, potentially further improving the results.

6 Conclusion

This paper presents a novel approach for fall detection and prevention for humanoid robots. We developed an iterative model based on the inverted pendulum and extended it with a stand space and the body dynamics of the robot. The model is used to simulate the effects of acting forces on the robot measured by sensors. If the simulated motion represents a fall, a counter-motion is performed. The trajectory for this motion is also based on the future states of the robot’s CoM calculated by the model. We demonstrate that this method can detect a fall significantly earlier than traditional methods. The practical use is limited by the hardware of the robot, but this approach is promising for robots with accurate sensors and precise joints. With improved computational power, a more precise model can be used to potentially obtain even better results.