Improving Convergence of Restricted Boltzmann Machines via a Learning Adaptive Step Size

  • Noel Lopes
  • Bernardete Ribeiro
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7441)


Restricted Boltzmann Machines (RBMs) have recently received much attention due to their potential to integrate more complex and deeper architectures. Despite their success, in many applications, training an RBM remains a tricky task. In this paper we present a learning adaptive step size method which accelerates its convergence. The results for the MNIST database demonstrate that the proposed method can drastically reduce the time necessary to achieve a good RBM reconstruction error. Moreover, the technique excels the fixed learning rate configurations, regardless of the momentum term used.


Restricted Boltzmann Machines Deep Belief Networks Deep learning Adaptive step size 


  1. 1.
    Almeida, L.B.: C1.2 Multilayer perceptrons. In: Handbook of Neural Computation, pp. C1.2:1–C1.2:30. IOP Publishing Ltd. and Oxford University Press (1997)Google Scholar
  2. 2.
    Bengio, Y.: Learning deep architectures for AI. Foundations and Trends in Machine Learning 2(1), 1–127 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Hinton, G.E.: Training products of experts by minimizing contrastive divergence. Neural Comput. 14, 1771–1800 (2002)zbMATHCrossRefGoogle Scholar
  4. 4.
    Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  5. 5.
    Hinton, G.: A practical guide to training restricted boltzmann machines. Tech. rep., Dep. of Computer Science, University of Toronto (2010)Google Scholar
  6. 6.
    Larochelle, H., Erhan, D., Courville, A., Bergstra, J., Bengio, Y.: An empirical evaluation of deep architectures on problems with many factors of variation. In: Proc. of the 24th Intl. Conference on Machine Learning, pp. 473–480. ACM (2007)Google Scholar
  7. 7.
    Le Roux, N., Bengio, Y.: Representational power of restricted boltzmann machines and deep belief networks. Neural Comput. 20, 1631–1649 (2008)MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Lee, H., Grosse, R., Ranganath, R., Ng, A.Y.: Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In: Proc. of the 26th Annual Intl. Conference on Machine Learning, pp. 609–616. ACM (2009)Google Scholar
  9. 9.
    Lopes, N., Ribeiro, B.: Restricted boltzmann machines and deep belief networks on multi-core processors. In: IEEE World Congress on Computational Intelligence (2012)Google Scholar
  10. 10.
    Ranzato, M., Boureau, Y., LeCun, Y.: Sparse feature learning for deep belief networks. In: Advances in Neural Information Processing Systems, vol. 20 (2007)Google Scholar
  11. 11.
    Roux, N.L., Bengio, Y.: Deep belief networks are compact universal approximators. Neural Computation 22(8), 2192–2207 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Schulz, H., Müller, A., Behnke, S.: Investigating convergence of restricted boltzmann machine learning. In: NIPS 2010 Workshop on Deep Learning and Unsupervised Feature Learning, Whistler, Canada (2010)Google Scholar
  13. 13.
    Silva, F.M., Almeida, L.B.: Acceleration Techniques for the Backpropagation Algorithm. In: Almeida, L.B., Wellekens, C. (eds.) EURASIP 1990. LNCS, vol. 412, pp. 110–119. Springer, Heidelberg (1990)CrossRefGoogle Scholar
  14. 14.
    Swersky, K., Chen, B., Marlin, B., de Freitas, N.: A tutorial on stochastic approximation algorithms for training restricted boltzmann machines and deep belief nets. In: Information Theory and Applications Workshop (2010)Google Scholar
  15. 15.
    Zainuddin, Z., Mahat, N., Hassan, Y.A.: Improving the convergence of the backpropagation algorithm using local adaptive techniques. Intl. Journal of Computational Intelligence, 172–175 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Noel Lopes
    • 1
    • 2
  • Bernardete Ribeiro
    • 1
    • 3
  1. 1.CISUC - Center for Informatics and Systems of University of CoimbraPortugal
  2. 2.Polytechnic Institute of GuardaUDI/IPG - Research UnitPortugal
  3. 3.Department of Informatics EngineeringUniversity of CoimbraPortugal

Personalised recommendations