Abstract
Restricted Boltzmann Machines are generative models which can be used as standalone feature extractors, or as a parameter initialization for deeper models. Typically, these models are trained using Contrastive Divergence algorithm, an approximation of the stochastic gradient descent method. In this paper, we aim at speeding up the convergence of the learning procedure by applying the momentum method and the Nesterov’s accelerated gradient technique. We evaluate these two techniques empirically using the image dataset MNIST.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
In both cases the number of hidden units was equal 900.
References
Bengio, Y.: Learning deep architectures for AI. Foundations and Trends in Machine Learning 2(1) (2009) 1–127
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786) (2006) 504–507
Taylor, G.W., Hinton, G.E., Roweis, S.T.: Modeling human motion using binary latent variables. In Schölkopf, B., Platt, J.C., Hoffman, T., eds.: NIPS, MIT Press (2006) 1345–1352
Mohamed, A.R., Hinton, G.E.: Phone recognition using restricted boltzmann machines. In: ICASSP, IEEE (2010) 4354–4357
Salakhutdinov, R., Mnih, A., Hinton, G.E.: Restricted boltzmann machines for collaborative filtering. In Ghahramani, Z., ed.: ICML. Volume 227 of ACM International Conference Proceeding Series., ACM (2007) 791–798
Salakhutdinov, R., Hinton, G.E.: Replicated softmax: an undirected topic model. In Bengio, Y., Schuurmans, D., Lafferty, J.D., Williams, C.K.I., Culotta, A., eds.: NIPS, Curran Associates, Inc. (2009) 1607–1614
Neapolitan, R.E.: Probabilistic reasoning in expert systems - theory and algorithms. Wiley (1990)
Pearl, J.: Probabilistic reasoning in intelligent systems - networks of plausible inference. Morgan Kaufmann series in representation and reasoning. Morgan Kaufmann (1989)
Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences of the United States of America 79(8) (1982) 2554–2558
Hopfield, J.J.: The effectiveness of neural computing. In: IFIP Congress. (1989) 503–507
Ackley, D.H., Hinton, G.E., Sejnowski, T.J.: A learning algorithm for Boltzmann Machines. Cognitive Science 9(1) (1985) 147–169
Larochelle, H., Bengio, Y.: Classification using discriminative restricted boltzmann machines. In Cohen, W.W., McCallum, A., Roweis, S.T., eds.: ICML. Volume 307 of ACM International Conference Proceeding Series., ACM (2008) 536–543
Hinton, G.E.: Training products of experts by minimizing contrastive divergence. Neural Computation 14(8) (2002) 1771–1800
Fischer, A., Igel, C.: An introduction to Restricted Boltzmann Machines. In Álvarez, L., Mejail, M., Déniz, L.G., Jacobo, J.C., eds.: CIARP. Volume 7441 of Lecture Notes in Computer Science., Springer (2012) 14–36
Hinton, G.E.: A practical guide to training restricted boltzmann machines. In: Neural Networks: Tricks of the Trade (2nd ed.). (2012) 599–619
Swersky, K., Chen, B., Marlin, B.M., de Freitas, N.: A tutorial on stochastic approximation algorithms for training restricted boltzmann machines and deep belief nets. In: ITA, IEEE (2010) 80–89
Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Improving neural networks by preventing co-adaptation of feature detectors. CoRR abs/1207.0580 (2012)
Wager, S., Wang, S., Liang, P.: Dropout training as adaptive regularization. CoRR abs/1307.1493 (2013)
Wan, L., Zeiler, M.D., Zhang, S., LeCun, Y., Fergus, R.: Regularization of neural networks using dropconnect. In: ICML (3). (2013) 1058–1066
Wang, S., Manning, C.D.: Fast dropout training. In: ICML (2). (2013) 118–126
Sutskever, I., Martens, J., Dahl, G.E., Hinton, G.E.: On the importance of initialization and momentum in deep learning. In: ICML (3). Volume 28 of JMLR Proceedings., JMLR.org (2013) 1139–1147
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Zaręba, S., Gonczarek, A., Tomczak, J.M., Świątek, J. (2015). Accelerated learning for Restricted Boltzmann Machine with momentum term. In: Selvaraj, H., Zydek, D., Chmaj, G. (eds) Progress in Systems Engineering. Advances in Intelligent Systems and Computing, vol 366. Springer, Cham. https://doi.org/10.1007/978-3-319-08422-0_28
Download citation
DOI: https://doi.org/10.1007/978-3-319-08422-0_28
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-08421-3
Online ISBN: 978-3-319-08422-0
eBook Packages: EngineeringEngineering (R0)