Development of Humanoid Robot Locomotion Based on Biological Approach in EEPIS Robot Soccer (EROS)

Abstract

In this paper we propose the development of EROS locomotion by using neural oscillator. We investigated muscular structure of human body for designing the neuron structure. Two motoric neurons, extensor neuron and flexor neuron, represent one structure of joint that generating the angle of joint. Sensoric neuron connection also designed for adapting the environment. Three kinds of sensor such as ground reaction sensor, tilt sensor, and angular velocity sensor are utilized for validate the proposed method. Evolutionary algorithm was used for optimizing synapse weight among motoric neuron, while recurrent neural network was used for the dynamical condition learning. The locomotion system of this research was shown using Open Dynamic Engine (ODE). The proposed method can generate locomotion pattern and its stability learning system improves the stability of locomotion. The proposed approach formed the walking locomotion that potentially can be developed to become adaptive locomotion.

1 Introduction

Many researchers develop the locomotion of humanoid robot soccer. Various methods have been used to support their research. In RoboCup humanoid soccer league, researchers compete to develop humanoid robot soccer capable to play with human soccer player plan to be held in 2050. The rules and the structure of robot are made to be similar with soccer rules and human structure. For humanoid robot structure, as the height of robot became higher, soles of feet size become smaller than before until similar as human size. Robot should be capable to walk, run, kick, and move during playing soccer on an unstable surface of grass. Many methods were used for locomotion in humanoid robot soccer. The conventional methods based on ZMP approach were discussed for locomotion approach [1–3]. Kajita et al. introduced a new method of a biped walking pattern generation by using a preview control of the zero-moment point (ZMP) [1]. Kim et al. also used ZMP as the feedback response to realize the dynamic locomotion HUBO robot [3]. In the previous works, the pattern equation based on ZMP approach was used to form the trajectory pattern for robot movement [4, 5]. This approach was implemented in E-1205, EROS-I, EROS-II, and EROS-III as depicted in Fig. 1.

Fig. 1.

EROS family. (a) E1205. (b) EROS-I. (c) EROS-II. (d). EROS-III

In this paper, the locomotion system was realized using biological structure approach. We adapted the human mechanism to generate the locomotion. We investigate the coupled muscle in human to acquire the inter-connection networking in neurons structure. We create the model neuron structure that composed from motoric and sensoric neuron. In order to acquire the effect value of synapse weight among motoric neuron, we used multi objective evolutionary algorithm. Recurrent neural network was used for learning the synapse weight between sensoric and joint neuron. To implement this locomotion system, we used computer simulation ODE. This paper is organized as follows. Section 2 explains the related work in this research. Section 3 talks about motion generator based on biological approach. Section 4 explains the stability system, Sect. 5 shows several experimental results and Sect. 6 concludes the paper.

2 Related Work

Locomotion model based on neural oscillator is a kind of biological system characterized by their behavior pattern with complexity of large degrees of freedom that can be stable and also flexible depending on the environment condition. Few researchers used central pattern generation with has basis in neurophysiological studies. Neural oscillator is a type of neural network for the generation of rhythmic motion [7–14]. Taga et al. have designed the coupled neural oscillator implemented for human locomotion. Global limit cycle was used where it was generated by a global entrainment between rhythmic activities of a nervous system to realize the stable and flexible locomotion [7]. Moreover, the mechanical formula was also created in order to acquire the feedback calculation. Vitor Matos et al. presented Central Pattern Generation (CPG) approach based on phase oscillators to bipedal locomotion [8]. This method can realize the transition from walking to running that can be adept for humanoid robot soccer that has high mobility.

Baydin also implemented CPG in 2012, in order to control humanoid biped walking mechanism. In his research evolutionary algorithm were implemented in order to determine the weight value and oscillatory parameter of CPG network [9]. Shan et al. used multi-objective GA for optimizing weight value [10]. While other researchers implemented reinforcement learning method for controlling CPG based locomotion [11].

In 2014, John Nassour presented an extended mathematical model of CPG for the locomotion [13]. His idea was to design the multi layers neuron connection in order to control the locomotion in various model of walking and to adapt the environmental condition. In order to support the stabilization, He et al. combine ZMP to CPG system [14]. In the proposed method the combination between the evolutionary algorithm for optimizing the pattern locomotion and recurrent neural network for increase the stabilization were used to build the locomotion system.

3 Locomotion Model

In this system, we used six joints in each leg. We adapted the mechanical structure of human as shown in Fig. 2. We investigated the musculoskeletal system that composed of the connection among rigid link. In this system, robot was installed with the ground reaction sensor with four point sensors in each sole of feet. Moreover, the robot was also equipped with tilt, accelerometer, and gyroscope sensor.

Fig. 2.

Joint structure

3.1 Neural Oscillator Based Locomotion Generator

In biological approach, neural oscillator was used as the basic element of the locomotion generator. We used neural oscillator generated by mutual inhibition between certain neurons with adaptation signal input as shown in Fig. 3. The neural oscillator model generated oscillatory signal activity, which consisted of two tonically excited neurons with self-inhibition effect linked reciprocally via inhibitory connections. We used neural oscillator model proposed by Matsuoka [15, 16].

In Eqs. (1), (2), and (3), $x_i$, $y_i$, $v_i$, are the inner state, the output of the ith neuron, and a variable representing the degree of self-inhibition effect of the ith neuron, respectively. $S_i$ is an external input with a constant rate; Time constant of the inner state and the adaptation effect are notated by $\tau$ and $\tau '$. $w_{ij}$ represents the strength of the inhibitory connection between the neurons. $\sum _{j=1}^{n} w_{ij} y_j$ represents the total signal input from the other neurons that have mutual connection.

$$\begin{aligned} \tau \dot{x}_i + x_i = x_0 - \sum _{j=1}^{n} w_{ij} y_j - bv_i + S_i \end{aligned}$$

(1)

$$\begin{aligned} \tau '\dot{v}_i + v_i = y_i \end{aligned}$$

(2)

$$\begin{aligned} y_i = \max (x_i, 0) \end{aligned}$$

(3)

$$\begin{aligned} \varTheta _n = y_{2n} - y_{(2n+1)} \end{aligned}$$

(4)

Neuron motoric system received two different types of sensory information: proprioceptive and exteroceptive information. In Eqs. (1) and (2), we compute by using Runge-Kutta gill method. In neuron oscillator based locomotion, there are two neurons (flexor and extensor) represented as the union of joint system are computed using Eq. (4). $\varTheta _n$ represents nth joint angle that resulted by coupled neuron process.

Fig. 3.

Coupled neuron

The robot has 18 degree of freedom; with 2 legs and 2 hands, each leg has 6 joints and each hand has 3 joints. There are 3 joints in hip: hip-x joint which has rotate axis in x-axis; hip-y joint which has rotate in y-axis; hip-z joint which has rotate in x-axis, 1 joint in knee, and 2 joints in ankle: ankle-x joint which has rotate in x-axis and ankle-y joint which has rotate in y-axis. The neuron structure for this locomotion system is shown in Fig. 4. We make limitation angle in each joint. Hip-x and ankle-x joints limitation are represented by $-\pi /2 < \theta _n < \pi /2$ hip-y and ankle-y limitation are represented by $-\pi /3 < \theta _n < \pi /3$, and knee joint limitation is represented by $-\pi /2 < \theta _n < 0$. The neuron structure is divided into server neuron and client neuron: neuron $E_1$, $E_2$, $E_3$, and $E_4$ act as the server neurons, which generate main signal to client neuron. These neurons are located in hip-x and knee joint. $E_n$ implies an extensor neuron in nth joint. In order to optimize and summarize the weight of synapse, we separated the weight into seven parts: S1–S7. The weight values in each part are the same, while the detail of inter-connection among neuron can be shown in Fig. 4.

Fig. 4.

Interconnection neuron diagram

3.2 Synapse Weight Optimization

In this research, the optimization process was conducted to find the value of effect between neurons. There are many methods for parameters optimization. In this research, multi-objective optimization is used since we have three objective optimization problems. For solving multi-objective optimization problems, there are two main methods: Pareto front and weighting factor. One of the main drawbacks of these methods is choosing the most appropriate value of weighting factors. The development of locomotion based on neural oscillator implies the optimization of the oscillation of body tilt $\bar{\sigma }$ (minimization) in pitch and roll direction, the change rate of walking direction (minimization) for maintaining the go straight movement, and the velocity of movement $\bar{\upsilon }$ (maximization). In order to acquire the minimization value represent by $\bar{\upsilon }$, we used the remaining parameter of velocity, which can not be reached by robot walking in certain time. In the optimization case, we calculated the absolute average value of body tilt in pitch and roll direction, which was computed using Eqs. (7) and (8) and the velocity of robot walking was computed using Eq. (9). The weight among neuron should be determined to acquire the desired velocity with minimum oscillation of body tilt. In order to acquire the fitness evaluation computed by Eq. (10), we run the robot in ODE and analyze the output of sensor.

$$\begin{aligned} \sigma (t , w_{ij}):|\mathrm R| \end{aligned}$$

(5)

$$\begin{aligned} \mathrm L(t,w_{ij}), \mathrm x(t), \mathrm y(t):\mathrm R \end{aligned}$$

(6)

In Eq. (5), the body tilt ($\sigma$) is normalized in absolute value. And in Eq. (6), $\mathrm L$, $\mathrm x$, $\mathrm y$ are the real number represented the length of movement in both x-axis, and y-axis respectively. These parameters were acquired in a time sampling during simulation process in ODE.

$$\begin{aligned} \bar{\sigma }_{pitch}(w_{ij})=\big ({}^1/_{T}\big ) {\sum \nolimits _{t=0}^{T}} \sigma _{pitch}(t,w_{ij} ) \end{aligned}$$

(7)

$$\begin{aligned} \bar{\sigma }_{roll}(w_{ij})=\big ({}^1/_{T}\big ) {\sum \nolimits _{t=0}^{T}} \sigma _{roll}(t,w_{ij} ) \end{aligned}$$

(8)

$$\begin{aligned} \bar{\upsilon }(w_{ij})=\big ({}^1/_{T}\big ) {\sum \nolimits _{t=0}^{T}} \frac{\mathrm d}{\mathrm {dt}} \mathrm L\big (t, w_{ij}, \mathrm x(t), \mathrm y(t)\big ) \end{aligned}$$

(9)

$$\begin{aligned} f= \mathop {\text {arg min}}\limits _{w_{ij}} (\bar{\sigma }_{pitch}\varepsilon _1 + \bar{\sigma }_{roll}\varepsilon _2 + \bar{\upsilon }\varepsilon _3) \end{aligned}$$

(10)

We implemented steady state genetic algorithm (SSGA) for optimizing the value of weight among neuron ($w_{ij}$). In SSGA, there is one individual that inserted to the new population in one generation. We used weighting factor in order to optimize multi-objective problems. $\varepsilon _1$, $\varepsilon _2$, $\varepsilon _3$ are the weight coefficients for tilt oscillation in pitch direction, roll direction, and for walking velocity respectively. In SSGA process, first we select the parent from initial generation; second we create the new individual by using mutation and crossover process. After that we evaluate the new individual by using fitness calculation computed in Eq. (10).

Fig. 5.

Chromosome of an individual

In SSGA, the chromosome of the individual is composed of the weight parameters among neuron. Each parameter in represented by one gene. Since we have seven parts of weights, thus one individual has seven genes represented in Fig. 5. Each gene has minimum ($h_{min}$) and maximum value ($h_{max}$). The detail of mutation and crossover process was explained in our previous paper [17].

4 Stability System

The stability system is built to have responsive ability to the disturber comes from environmental condition such as normal force resulted depending on surface condition. We designed the connection between sensoric neuron and joint neuron system that explained in Table 1. Local effect approach in this model was used to summarize the structure neuron. In these cases, we have three balancing systems: ground reaction, pose control, and hand reaction. Each of them has different connection structure.

Fig. 6.

Sensor connection structure in joint angle level

4.1 Synapse Structure

In the proposed model, We implemented three different kind of sensors as mentioned in previous section. The synapse connection structure is illustrated in Fig. 6 and explained in Table 1. In ground reaction, there are four sensoric neurons located in the four corners of foot, $Sp_1$–$Sp_4$ for left foot and $Sp_5$–$Sp_8$ for right foot. These sensoric neurons are connected to ankle joint neurons: ankle-x joint and ankle-y joint. When front sensoric neurons ($Sp_1$, $Sp_2$, $Sp_5$, $Sp_6$) detect the ground, they send the positive impulse to ankle-x joint. When the right side sensoric neurons ($Sp_2$, $Sp_3$, $Sp_6$, $Sp_7$) detect the ground, positive impulse will be sent to joint ankle-y. These sensoric neurons (ground sensor) also influence knee joint and hip-x joint. When sensoric neuron detects the ground, it sends impulse to knee and hip-x joint. In tilt and angular velocity sensors, there are two degrees of freedom: pitch ($St_2$ and $Sa_2$) and roll ($St_1$ and $Sa_1$). $St_1$ and $Sa_1$ are connected to hand-x joint, while $St_2$ and $Sa_2$ are connected to ankle-y, hip-y, hand-y joint. The output sensors are resulted during the simulation process.

$$\begin{aligned} \theta _n = \varTheta _n + {\sum \nolimits _{i=1}^{m}} Sa_i q_{3,i} \alpha _{3,i}^{n} + {\sum \nolimits _{i=1}^{l}} St_i q_{2,i} \alpha _{2,i}^{n} + {\sum \nolimits _{i=1}^{k}} Sp_i q_{1,i} \alpha _{1,i}^{n} \end{aligned}$$

(11)

Table 1. Connection structure

In Eq. (11), m, l, k are number of tilt, angular velocity, and ground sensor, respectively. The values of synapse $q_{1,i}$, $q_{2,i}$, and $q_{3,i}$ are controlled by using recurrent neural network (RNN) for acquiring the best stabilization parameter. This system will be detailed explained in the next section. $\theta _n$ represents the angle joint in nth joint. $\alpha _{3,i}^{n}$, $\alpha _{2,i}^{n}$, $\alpha _{1,i}^{n}$ represent the impuls effect of angular velocity sensor, tilt sensor, and ground sensor in n-th joint explained in Table 1, Since 0 implies there is no effect, 1 and $-1$ imply positive and negative effect.

Fig. 7.

Recurrent neural network structure

4.2 Recurrent Neural Network

In order to optimize the synapse weight, we utilized the recurrent neural network back propagation through time (BPTT) as depicted in Fig. 9. This system updates the correction synapse weight from sensoric neuron to joint neuron in real time. In our recurrent neural network system p represents the state number of current condition of Robot. In this system we assumed the number of state condition is one. In Eq. (12), $x_i^p(t)$ is the input value in t time sampling; $s_h^p(t-1)$ is output value of hidden neuron in $t-1$ time sampling; $b_j$ is bias input for hidden layer; $x_y^p(t)$ is output value of hidden neuron in t time sampling (Fig. 7).

$$\begin{aligned} s_y^p(t)=f(ns_y^p(t))=f\left( \sum _{i}^{l} x_i^p(t)\mathrm V_{ij}^p(t) + \sum _{h}^{m} s_h^p(t-1)\mathrm U_{h}^p + b_j\right) \end{aligned}$$

(12)

$$\begin{aligned} y_k^p(t)=g(O_k^p(t))=g\left( \sum _{j}^{m} s_j^p(t)W_{kj}^p(t) + b_k \right) \end{aligned}$$

(13)

After forward process is done and resulted the output neuron value $y_k^p$ computed in Eq. (13), waiting time is required to acquire the sensor value $\mathrm S(k, y_k^p(t))$ and analyze the weight value. In Eqs. (14) and (15), $\delta _k^p$ is error propagation in output layer and is error propagation in hidden layer. We used hand actuator as the response output $y_k^p(t)$. The number of neurons in the input layer, in the hidden layer, and in the output layer are denoted by l, m, and n, respectively.

$$\begin{aligned} \delta _k^p = (d_k^p - \mathrm S(k, y_k^p(t)))g'(O_k^p(t)) \end{aligned}$$

(14)

$$\begin{aligned} \delta _j^p = \sum _k^n \delta _k^p \mathrm W_{kj}^p(t) f'(ns_j^p(t)) \end{aligned}$$

(15)

$$\begin{aligned} \mathbf W ^p(t+1) = \mathbf W ^p(t) + \eta _w\mathbf s ^p(t)\delta _k^p \end{aligned}$$

(16)

$$\begin{aligned} \mathbf V ^p(t+1) = \mathbf V ^p(t) + \eta _v\mathbf x ^p(t)\delta _j^p \end{aligned}$$

(17)

$$\begin{aligned} \mathbf U ^p(t+1) = \mathbf U ^p(t) + \eta _u\mathbf s ^p(t-1)\delta _j^p \end{aligned}$$

(18)

The activation function for hidden layer f(x) used sigmoid function. $d_k^p$ is the desire output and the output function for are sigmoid function. In this case, the desire output is the condition where robot maintains the angular velocity of robot becomes zero. We have 12 neurons in the input layer resulted from sensors ($Sp_k$, $St_l$, $Sa_m$), four neurons in the hidden layer, and 12 neurons in the output layer. The configuration of input neuron is explained in Table 2. We need to train the parameters $\mathbf V$, $\mathbf U$, and $\mathbf W$ as the weight among neuron. $\eta _v$, $\eta _u$, and $\eta _w$ are the learning rate for $\mathbf V$, $\mathbf U$, $\mathbf W$ parameter recursively. In BPTT, the error propagation is done recursively. As depicted in Eqs. (16), (17), and (18), the weight of synapse between sensoric and joint neuron can be regenerated.

Table 2. Configuration of input and output neuron

Fig. 8.

Fitness evolution

5 Experimental Result

We designed the robot by using computer simulation ODE adapting the rules of RoboCup 2015 [18]. Parameter properties from EROS robot were inserted in simulation robot as shown in Table 3. We conducted the experiment gradually as follow; first, optimizing the weight among neuron ($w_{ij}$) to acquire the walking locomotion pattern; second, optimization for increasing the stability in locomotion. In order to find the pattern of trajectory, we observed the signals generated from many combinations of interconnection in server neuron introduced by Matsuoka [16]. After the best pattern of locomotion was determined, we optimize the synapse weight among neurons. SSGA algorithm was applied that use speed of walking ($\bar{\upsilon }$) and tilt sensor ($\bar{\sigma }$) as the objective function. The parameters used for weight among neuron optimization was tabulated in Table 4.

Fig. 9.

Length of walking comparison

In this experiment, we acquire the fitness diagram from SSGA process that shown in Fig. 8 resulted from two objective values (Speed and Stabilization). The fitness total decreased significantly until 50-th generation. Stability fitness and speed fitness also decreased, which implied the oscillation data in pitch and roll direction to become smaller and the speed become higher. The good pattern locomotion were reach in 50-th generation. We finished the generation until 500-th generation. The locomotion system takes one individual (S1–S7 = 1.546, 1.230, 1.901, 1,676, 2.158, 1.390, 1.120) as the best individual in SSGA, and then become the parameter in robot. Next, we analyzed the signal resulted from coupled motoric neuron.

Table 3. Robot specification

Table 4. Parameters of SSGA

By using evolutionary algorithm, walking pattern based on neural oscillator can be formed. However, since the stability level resulted from evolutionary computation was not strong enough to cover outside disturbances, the stability system is required to solve this problem.

Fig. 10.

Comparison angular velocity data between locomotion system without SLS and with SLS (a) Roll direction (b) Pitch direction

Fig. 11.

Comparison angular velocity data between locomotion system without SLS and with SLS (a) Roll direction (b) Pitch direction

Fig. 12.

Signal output of coupled neuron

In the second experiment, we continued to implement the stability system that used recurrent neural network model to support the balancing system. We defined $\eta _v=0.02$, $\eta _u=0.02$, and $\eta _w=0.01$ as the learning rate. We analyzed the signal oscillation without stability learning system (SLS) and with SLS to compare the difference among them. Figures 10 and 11 shows the angular velocity oscillation and tilt oscillation comparison in both of locomotion system. The locomotion with SLS has smaller tilt and angular velocity oscillation, which imply the locomotion with SLS more stable than locomotion without SLS. Beside that, the SLS also increase the speed of walking in robot locomotion as shown in Fig. 9. The locomotion with SLS can reach the length of movement longer because the SLS can effectively reduce the disturbance that hamper the robot walking. The weight between sensoric neuron and joint neuron is changing every time sampling depending on the robot condition. SLS is effective for balancing system of humanoid robot locomotion. Beside that, the values of time sampling configuration are important. We have captured the running process simulation as seen in Fig. 13. This model is expected forming the human-like locomotion system.

Fig. 13.

Running process in ODE simulation

The output of coupled neuron representing the signal of joints in robot when walking on flat surface was recorded as shown in Fig. 12. The hip-x, knee, and ankle-x joints have phase difference 180 degrees, while hip-y and ankle-y joints have the same phase. These signals have been influence with ground reaction. Therefore, these signals have different pattern depending on the environment condition. In this paper, we only consider the disturbances in the flat surface. We proved that the stability system can reduce the oscillation of robot walking and also improve the speed of robot walking. This biological approach based robot locomotion are needed improvement, therefore the robot can walk in different direction, walk in uneven surface, and reduce the external disturbance.

6 Conclusion

In this research, we have designed the humanoid biped robot locomotion based on neural oscillator. The locomotion pattern in humanoid robot can be acquired by controlling the weight of synapse by using evolutionary algorithm. The design of neural structure between sensoric neuron and joint neuron could change the stabilization approach by using ZMP method. The stabilization of locomotion can be increased by learning the weight of synapse using recurrent neural network. Signal pattern resulted from the coupled neuron is suitable for the ground reaction. We have designed the synapse from sensoric to joint neuron and the walking movement without turning left or right. As the future research, we are designing to add a neuron systems adapted as brain neuron systems that process the information from sensoric neuron system and generate signal to motoric neuron system. Brain neuron systems also control the timing impulse for controlling the signal generated by motoric neuron.