Dropout for Recurrent Neural Networks

Abstract

Neural networks are computational structures which can be trained to perform tasks based on training examples or patterns. Recurrent neural networks are a type of network designed to process time-series data. Dropout is a neural network regularization technique. The literature advises that Dropout should not be directly applied to recurrent neural networks as its effects are too dramatic when applied recurrently. This direct approach is described as naive. Instead, there are two specialised recurrent neural network Dropout algorithms proposed by different authors. However, these specialised Dropout algorithms have not been tested against one another and the naive algorithm under identical experimental conditions. This paper compares all of these algorithms and finds that the naive approach performed as well as or better than the specialised Dropout algorithms.