Theoretical Analysis of Function of Derivative Term in On-Line Gradient Descent Learning
In on-line gradient descent learning, the local property of the derivative term of the output can slow convergence. Improving the derivative term, such as by using the natural gradient, has been proposed for speeding up the convergence. Beside this sophisticated method, ”simple method” that replace the derivative term with a constant has proposed and showed that this greatly increases convergence speed. Although this phenomenon has been analyzed empirically, however, theoretical analysis is required to show its generality. In this paper, we theoretically analyze the effect of using the simple method. Our results show that, with the simple method, the generalization error decreases faster than with the true gradient descent method when the learning step is smaller than optimum value η opt . When it is larger than η opt , it decreases slower with the simple method, and the residual error is larger than with the true gradient descent method. Moreover, when there is output noise, η opt is no longer optimum; thus, the simple method is not robust in noisy circumstances.
Unable to display preview. Download preview PDF.
- 1.Krogh, A., Hertz, J., Palmer, R.G.: Introduction to the Theory of Neural Computation. Addison-Wesley, Redwood City (1991)Google Scholar
- 4.Hara, K., Katahira, K., Okanoya, K., Okada, M.: Statistical Mechanics of On-Line Node-perturbation Learning. Information Processing Society of Japan, Transactions on Mathematical Modeling and Its Applications 4(1), 72–81 (2011)Google Scholar
- 6.Rattray, M., Saad, D.: Incorporating Curvature Information into On-line learning. In: Saad, D. (ed.) On-line Learning in Neural Networks, pp. 183–207. Cambridge University Press, Cambridge (1998)Google Scholar
- 8.Fahlman, S.E.: An Empirical Study of Learning Speed in Back-Propagation Networks, CMU-CS-88-162 (1988)Google Scholar