Two-Point Step Size Gradient Method for Solving a Deep Learning Problem
- 14 Downloads
This paper is devoted to an analysis of the rate of deep belief learning by multilayer neural networks. In designing neural networks, many authors have applied the mean field approximation (MFA) to establish that the state of neurons in hidden layers is active. To study the convergence of the MFAs, we transform the original problem to a minimization one. The object of investigation is the Barzilai–Borwein method for solving the obtained optimization problem. The essence of the two-point step size gradient method is its variable steplength. The appropriate steplength depends on the objective functional. Original steplengths are obtained and compared with the classical steplength. Sufficient conditions for existence and uniqueness of the weak solution are established. A rigorous proof of the convergence theorem is presented. Various tests with different kinds of weight matrices are discussed.
KeywordsDeep Boltzmann machine mean field approximation gradient iterative methods
Unable to display preview. Download preview PDF.
- 2.H. K. Jabbar and R. Z. Khan, “Methods to avoid over-fitting and under-fitting in supervised machine learning (comparative study),” in: Computer Science, Communication & Instrumentation Devices, Editors: J. Stephen, H. Rohil, and S. Vasavi, (2015), pp. 163–172.Google Scholar
- 3.R. Salakhutdinov and G. E. Hinton, “Deep Boltzmann Machines,” Proc. Conf. Artif. Intel. Stat. (AISTATS 2009), 448–455 (2009).Google Scholar
- 5.R. Salakhutdinov, “Learning Deep Boltzmann Machines using adaptive MCMC,” Proc. 27th Int. Conf. Mach. Lear., Haifa, Israel, 943–950 (2010).Google Scholar
- 6.R. Salakhutdinov and H. Larochelle, “Efficient learning of Deep Boltzmann Machines,” J. Mach. Learn. Res., 9, 693–700 (2010).Google Scholar
- 8.K. Cho, T. Raiko, A. Ilin, and J. Karhunen, “A two-stage pretraining algorithm for Deep Boltzmann Machines,” Artif. Neural Netw. Mach. Learn. (ICANN), 8131, 106-113 (2013).Google Scholar
- 9.K. Cho, T. Raiko, and A. Ilin, “Gaussian–Bernoulli Deep Boltzmann Machine,” IEEE Int. Joint Conf. Neural Netw., Dallas, Texas, USA, 1–7 (2013).Google Scholar
- 18.A. Zhang, J. Zhu, and B. Zhang, “Max-margin infinite hidden Markov models,” Proc. 31st Int. Conf. Mach. Learn. (PMLR), 32, 1, 315–323 (2014).Google Scholar
- 19.G. S. Tsanev, “Deep multiconnected Boltzmann machine for classification,” Amer. J. Eng. Res., 6, 5, 186–194 (2017).Google Scholar