An Optimized Second Order Stochastic Learning Algorithm for Neural Network Training
The performance of a neural network depends critically on its model structure and the corresponding learning algorithm. This paper proposes bounded stochastic diagonal Levenberg-Marquardt (B-SDLM), an improved second order stochastic learning algorithm for supervised neural network training. The algorithm consists of a single hyperparameter only and requires negligible additional computations compared to conventional stochastic gradient descent (SGD) method while ensuring better learning stability. The experiments have shown very fast convergence and better generalization ability achieved by our proposed algorithm, outperforming several other learning algorithms.
KeywordsSecond order method Fast convergence Stochastic diagonal Levenberg-Marquardt Convolutional neural network
This work is supported by Universiti Teknologi Malaysia (UTM) and the Ministry of Science, Technology and Innovation of Malaysia (MOSTI) under the ScienceFund Grant No. 4S116.
- 1.Becker, S., Le Cun, Y.: Improving the convergence of back-propagation learning with second order methods. In: Proceedings of the connectionist models summer school, pp. 29–37 (1988)Google Scholar
- 3.Chen, X.-W., Member, S., Lin, X.: Big data deep learning : challenges and perspectives. IEEE Access 2 (2014)Google Scholar
- 5.Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), vol. 9, pp. 249–256 (2010)Google Scholar
- 6.Igel, C., Hüsken, M.: Improving the Rprop learning algorithm. In: Proceedings of the Second International Symposium on Neural Computation (Nc), pp. 115–121 (2000)Google Scholar
- 9.Liew, S., Khalil-Hani, M., Syafeeza, A., Bakhteri, R.: Gender classification: a convolutional neural network approach. Turk. J. Elec. Engin. http://journals.tubitak.gov.tr/elektrik/accepted.htm
- 10.Milakov, M.: Convolutional Neural Networks in Galaxy Zoo Challenge, pp. 1–7 (2014)Google Scholar
- 13.Zeiler, M.D.: ADADELTA: an adaptive learning rate method. CoRR abs/1212.5701 (2012)Google Scholar