Spatio-temporal SRU with global context-aware attention for 3D human action recognition

Abstract

3D action recognition has attracted much attention in machine learning fields in recent years, and recurrent neural networks (RNNs) have been widely used for 3D action recognition due to their efficiency in processing sequential data. However, in order to achieve good performance, traditional RNN architectures are usually time-consuming for the training and inference process. To address the problem, a global context-aware attention spatio-temporal SRU (GCA-ST-SRU) method is proposed and applied for 3D action recognition in this paper, through extending the original simple recurrent unit (SRU) algorithm to joint spatio-temporal domain with an attention mechanism. First, deep neural networks were employed to learn the features of skeleton joints at each frame, and then these new high-level feature sequences were classified using the GCA-ST-SRU method which can learn the spatio-temporal dependence between different joints in the same frame and pay more attention to informative joints. Extensive experiments were conducted on the UT-Kinect and SBU-Kinect Interaction datasets to evaluate the effectiveness of the proposed method. Compared with several existing algorithms including SRU, long short-term memory (LSTM), spatio-temporal LSTM (ST-LSTM) and global context-aware attention LSTM (GCA-LSTM), our method has exhibited better performance in classification accuracy and computational efficiency. The experimental results demonstrate the effectiveness and practicability of our algorithm. Compared to the methods with similar performance, our algorithms can reduce training time and improve the inference speed, and thus it achieves a balance between speed and accuracy.

Appendices

Appendix 1 . Long short-term memory network (LSTM)

The LSTM network is a popular model for processing sequence data which has good performance on modeling long-term temporal dependencies. According to the length of the sequence data, the compute process of the LSTM network is divided into multiple steps, and the computation process on each step can be formulated as below:

$$ \left(\begin{array}{c}{i}_t\\ {}{f}_t\\ {}{o}_t\\ {}{u}_t\end{array}\right)=\left(\begin{array}{c}\sigma \\ {}\sigma \\ {}\sigma \\ {}\tanh \end{array}\right)\left(M\left(\begin{array}{c}{x}_t\\ {}{h}_{t-1}\end{array}\right)\right), $$

(18)

$$ {c}_t={i}_t\odot {u}_t+{f}_t\odot {c}_{t-1}, $$

(19)

$$ {h}_t={o}_t\odot \tanh \left({c}_t\right), $$

(20)

where M is a model parameter matrix, x_t is the input at t^th step, h_t − 1 is the output state at (t − 1)^th step, σ denotes the sigmoid activation function. The detailed process is given as follows: The input gate i_t, forget gate f_t, output gate o_t and input information u_t are firstly calculated according to Eq. (18), and the elements of these matrices range from 0 to 1. Then, the internal state c_t is updated according to Eq. (19), where ⊙ denotes the element-wise product, i_t determines the weight of u_t, and f_t determines the weight of the previous internal state c_t − 1. Finally, the output state h_t is obtained by Eq. (20).

Appendix 2. Spatio-temporal long short-term memory network (ST-LSTM)

The ST-LSTM is a spatial extension of LSTM, which can capture the dependence of joints in temporal and spatial domains simultaneously. The ST-STLM transition equations are formulated as below:

$$ \left(\begin{array}{c}{i}_{j,t}\\ {}{f}_{j,t}^S\\ {}{f}_{j,t}^T\\ {}{o}_{j,t}\\ {}{u}_{j,t}\end{array}\right)=\left(\begin{array}{c}\sigma \\ {}\sigma \\ {}\sigma \\ {}\sigma \\ {}\tanh \end{array}\right)\left(M\left(\begin{array}{c}{x}_{j,t}\\ {}{h}_{j-1,t}\\ {}{h}_{j,t-1}\end{array}\right)\right), $$

(21)

$$ {c}_{j,t}={i}_{j,t}\odot {u}_{j,t}+{f}_{j,t}^S\odot {c}_{j-1,t}+{f}_{j,t}^T\odot {c}_{j,t-1}, $$

(22)

$$ {h}_{j,t}={o}_{j,t}\odot \tanh \left({c}_{j,t}\right), $$

(23)

where x_{j, t} is the input at the spatio-temporal step (j, t) corresponds to the t^th frame and j^thjoint, Eq. (21) extends f_t in Eq. (18) to ${f}_{j,t}^S$ and ${f}_{j,t}^T$, which correspond to the forget gate in spatial and temporal domain, respectively. Similarly, In Eq. (22), the computation of internal state c_{j, t} depends on c_{j − 1, t} at previous temporal step and c_{j, t − 1} at previous spatial step.

Appendix 3. simple recurrent unit (SRU)

The SRU is a recurrent network like LSTM and GRU, but the majority of computation for each step is independent of the recurrence and can be easily parallelized [13]. The transition equations are formulated as below:

$$ {\tilde{x}}_t=W{x}_t, $$

(24)

$$ {f}_t=\mathrm{sigmoid}\left({W}_f{x}_t+{b}_f\right), $$

(25)

$$ {r}_t=\mathrm{sigmoid}\left({W}_r{x}_t+{b}_r\right), $$

(26)

$$ {c}_t={f}_t\odot {c}_{t-1}+\left(1-{f}_t\right)\odot {\tilde{x}}_t, $$

(27)

$$ {h}_t={r}_t\odot \tanh \left({c}_t\right)+\left(1-{r}_t\right)\odot {x}_t, $$

(28)

where x_t denotes input at time t, ${\tilde{x}}_t$ is a linear transformation of x_t, f_t is the forget state, r_t is the reset gate, c_t denotes internal state and h_t denotes output state. Different from LSTM, the computation of ${\tilde{x}}_t$, f_t and r_t of SRU does not depend on the internal state c_t − 1 at previous time step, and the inference speed is promoted by parallelizing these computations.

The difference between the computation process of LSTM and SRU is shown in Fig. 6. For LSTM, the symbols in each rectangle represent the variables that need to be computed in each time step t = 1, 2, ..., n. It shows that the computation of h_t depends on completing the previous time step, and the sequential dependencies limit the inference speed of the LSTM. For SRU, the computations of ${\tilde{x}}_t$, f_t and r_t depend only on the input x_t at each time step, so they are independent of the recurrence as shown in the dotted-line box, the recurrent computations of c_t and h_t involves only element-wise product implementations that are relatively lightweight, and thus they are fast. Parallelizing the majority of computation for each step can significantly improve the inference speed of SRU.

Fig. 6

Illustration of the compute process of LSTM and SRU. a Recurrent computation of LSTM, b Recurrent computation of SRU