Regularized Stacked Auto-Encoder Based Pre-training for Generalization of Multi-layer Perceptron

Abstract

Generalization capability of multi-layer perceptron (MLP) depends on the initialization of its weights. If the weights of an MLP are not initialized properly, it may fail to achieve good generalization. In this article, we propose a weight initialization technique for MLP to improve its generalization. This is achieved by a regularized stacked auto-encoder based pre-training method. During pre-training, the weights between each adjacent layers of an MLP, upto the penultimate layer, are trained layer wise by an auto-encoder. To train the auto-encoder, we use weighted sum of two terms: (i) mean squared error (MSE) and (ii) sum of squares of the first order derivatives of the outputs with respect to inputs. Here, the second term acts as a regularizer. It is used to penalize the training of auto-encoder during pre-training to generate better initial values of the weights for each successive layers of MLP. To compare the proposed initialization technique with random weight initialization, we have considered ten standard classification data sets. Empirical results show that the proposed initialization technique improves the generalization of MLP.