Overview of Long Short-Term Memory Neural Networks

Abstract

Long Short-term Memory was designed to avoid vanishing and exploding gradient problems in recurrent neural networks. Over the last twenty years, various modifications of an original LSTM cell were proposed. This chapter gives an overview of basic LSTM cell structures and demonstrates forward and backward propagation within the most widely used configuration called traditional LSTM cell. Besides, LSTM neural network configurations are described.

Chapter Highlights

 

• Long short-term memory (LSTM) is a special type of recurrent neural network (RNN).

• LSTM unit has a memory and multiple weighted gates. Therefore it does not suffer from vanishing or exploding gradient problems of RNN and can process sequences of arbitrary length.

• An original LSTM unit has no forget gate (NFG).

• Traditional LSTM configuration:

  $$ \begin{pmatrix} g_{t} \\ i_{t} \\ f_{t} \\ o_{t} \\ \end{pmatrix} = \begin{pmatrix} \text {tanh}\\ \sigma \\ \sigma \\ \sigma \\ \end{pmatrix} \cdot \begin{pmatrix} W^{(g)} &{}U^{(g)} \\ W^{(i)} &{}U^{(i)} \\ W^{(f)} &{}U^{(f)} \\ W^{(o)} &{}U^{(o)} \\ \end{pmatrix} \cdot \begin{pmatrix} x_{t} \\ h_{t-1} \end{pmatrix};$$
  $$C_{t}= f_{t} \bigodot C_{t-1}+i_{t}\bigodot g_t{}; \qquad h_{t}=o_{t} \bigodot \text {tanh}(C_{t}).$$

• Traditional LSTM with peephole connections is distinguished for precise timing and often referred as ‘Vanilla’ LSTM.

• ConvLSTM are effective in spatiotemporal sequential problems.

• Updates in Phased LSTM occur at irregularly sampled time points $t_{j}$ which can be controlled.

• Depending on input and output sequnces’ length, following LSTM models are differentiated: ‘One-to-One’, ‘One-to-Many’, ‘Many-to-One’, ‘Many-to-Many’.

• LSTM architecture can have different directionality, dimensionality and combination of them.