Excessive, Selective and Collective Information Processing to Improve and Interpret Multi-layered Neural Networks

Abstract

This paper aims to propose a new type of learning method to train multi-layered neural networks. In deep learning, pre-training by unsupervised learning such as auto-encoders and Boltzmann machines is used to produce initial connection weights to be used in the main training or fine-tuning. It has been observed that the connection weights are not necessarily effective for training supervised learning, because the objectives of unsupervised and supervised learning are naturally different. However, without the appropriate pre-training, multi-layered neural networks have difficulty in training, because information on input patterns and also the errors decrease naturally by going through many hidden layers. To overcome this problem of vanishing information, particularly, on input patterns, we try to produce redundant and excessive information in terms of the activation of output neurons before training multi-layered neural networks. Then, the excessive information can be reduced by the vanishing information property of multi-layered neural networks. It can be expected that appropriate connection weights can be found among many candidates created in a process of producing the excessive information. Finally, all of the connection weights are collectively treated for better interpretation. The method was applied to two data sets: artificial and symmetric data set and the real snack food selection data set. In both experimental results, redundant and excessive information generation was observed in terms of connection weights and improved generalization performance was observed.