Part 2: Multilayer perceptron and natural gradient learning
Since the perceptron was developed for learning to classify input patterns, there have been plenty of studies on simple perceptrons and multilayer perceptrons. Despite wide and active studies in theory and applications, multilayer perceptrons still have many unsettled problems such as slow learning speed and overfitting. To find a thorough solution to these problems, it is necessary to consolidate previous studies, and find new directions for uplifting the practical power of multilayer perceptrons. As a first step toward the new stage of studies on multilayer perceptrons, we give short reviews on two interesting and important approaches; one is stochastic approach and the other is geometric approach. We also explain an efficient learning algorithm developed from the statistical and geometrical studies, which is now well known as the natural gradient learning method.
KeywordsMultilayer Perceptrons Gradient Decent Learning Backpropagation Learning Natural Gradient Singularity Stochastic Neural Networks Neuromanifold
Unable to display preview. Download preview PDF.
- 3).Amari, S. and Nagaoka, H.,Methods of Information Geometry, AMS New York and Oxford University Press, 2000.Google Scholar
- 5).Amari, S., Park, H. and Ozeki, T., “Singularities Affect Dyanmics of Learning in Neuromanifold,”Neural Computation, 17, in press, 2005.Google Scholar
- 6).Bishop, C.M.,Neural Networks, for Pattern Recognition, Clarendon Press, Oxford, 1995.Google Scholar
- 7).Bottou, L. “Online Algorithms and Stochastic Approximations,”Online Learning and Neural Networks (Saad, D. ed.), Cambridge University Press, Cambridge, UK, 1998.Google Scholar
- 19).Rosenblatt, F.,Principles of Neurodynamics, Spartan New York, 1961.Google Scholar
- 20).Rumelhart, D., Hinton, G. E. and Williams, R. J., “Learning Internal Representations by Error Backpropagation,” inParallel Distributed Processing: Explorations in the Microstructure of Cognition, 1, Foundations, MIT Press, Cambridge, MA, 1986.Google Scholar