Skip to main content
Book cover

Neural Networks and Deep Learning pp 235–270Cite as

Restricted Boltzmann Machines

  • 417k Accesses

Abstract

The restricted Boltzmann machine (RBM) is a fundamentally different model from the feed-forward network. Conventional neural networks are input-output mapping networks where a set of inputs is mapped to a set of outputs. On the other hand, RBMs are networks in which the probabilistic states of a network are learned for a set of inputs, which is useful for unsupervised modeling.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-94463-0_6
  • Chapter length: 36 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   59.99
Price excludes VAT (USA)
  • ISBN: 978-3-319-94463-0
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   79.99
Price excludes VAT (USA)
Hardcover Book
USD   69.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Figure 6.1
Figure 6.2
Figure 6.3
Figure 6.4
Figure 6.5
Figure 6.6
Figure 6.7
Figure 6.8
Figure 6.9
Figure 6.10

Notes

  1. 1.

    This example is tricky in terms of semantic interpretability for the case in which no trucks are selected. Even in that case, the probabilities of various ice-creams turn out to be non-zero depending on the bias. One can explain such cases by adding a dummy truck that is always selected.

Bibliography

  1. D. Ackley, G. Hinton, and T. Sejnowski. A learning algorithm for Boltzmann machines. Cognitive Science, 9(1), pp. 147–169, 1985.

    CrossRef  Google Scholar 

  2. C. Aggarwal. Machine learning for text. Springer, 2018.

    Google Scholar 

  3. Y. Bengio and O. Delalleau. Justifying and generalizing contrastive divergence. Neural Computation, 21(6), pp. 1601–1621, 2009.

    MathSciNet  CrossRef  Google Scholar 

  4. M. Carreira-Perpinan and G. Hinton. On Contrastive Divergence Learning. AISTATS, 10, pp. 33–40, 2005.

    Google Scholar 

  5. G. Dahl, R. Adams, and H. Larochelle. Training restricted Boltzmann machines on word observations. arXiv:1202.5695, 2012.https://arxiv.org/abs/1202.5695

  6. A. Fischer and C. Igel. An introduction to restricted Boltzmann machines. Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, pp. 14–36, 2012.

    Google Scholar 

  7. Y. Freund and D. Haussler. Unsupervised learning of distributions on binary vectors using two layer networks. Technical report, Santa Cruz, CA, USA, 1994

    Google Scholar 

  8. P. Gehler, A. Holub, and M. Welling. The Rate Adapting Poisson (RAP) model for information retrieval and object recognition. ICML Confererence, 2006.

    Google Scholar 

  9. W. Gilks, S. Richardson, and D. Spiegelhalter. Markov chain Monte Carlo in practice.CRC Press, 1995.

    Google Scholar 

  10. G. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8), pp. 1771–1800, 2002.

    CrossRef  Google Scholar 

  11. G. Hinton. A practical guide to training restricted Boltzmann machines. Momentum, 9(1), 926, 2010.

    Google Scholar 

  12. G. Hinton, P. Dayan, B. Frey, and R. Neal. The wake–sleep algorithm for unsupervised neural networks. Science, 268(5214), pp. 1158–1162, 1995.

    CrossRef  Google Scholar 

  13. G. Hinton, S. Osindero, and Y. Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18(7), pp. 1527–1554, 2006.

    MathSciNet  CrossRef  Google Scholar 

  14. G. Hinton and T. Sejnowski. Learning and relearning in Boltzmann machines. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, MIT Press, 1986.

    Google Scholar 

  15. G. Hinton and R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313, (5766), pp. 504–507, 2006.

    Google Scholar 

  16. G. Hinton and R. Salakhutdinov. Replicated softmax: an undirected topic model. NIPS Conference, pp. 1607–1614, 2009.

    Google Scholar 

  17. G. Hinton and R. Salakhutdinov. A better way to pretrain deep Boltzmann machines. NIPS Conference, pp. 2447–2455, 2012.

    Google Scholar 

  18. T. Hofmann. Probabilistic latent semantic indexing. ACM SIGIR Conference, pp. 50–57, 1999.

    Google Scholar 

  19. J. J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. National Academy of Sciences of the USA, 79(8), pp. 2554–2558, 1982.

    MathSciNet  CrossRef  Google Scholar 

  20. D. Koller and N. Friedman. Probabilistic graphical models: principles and techniques. MIT Press, 2009.

    Google Scholar 

  21. H. Larochelle and Y. Bengio. Classification using discriminative restricted Boltzmann machines. ICML Conference, pp. 536–543, 2008.

    Google Scholar 

  22. H. Larochelle, M. Mandel, R. Pascanu, and Y. Bengio. Learning algorithms for the classification restricted Boltzmann machine. Journal of Machine Learning Research, 13, pp. 643–669, 2012.

    MathSciNet  MATH  Google Scholar 

  23. H. Larochelle and I. Murray. The neural autoregressive distribution estimator. International Conference on Artificial Intelligence and Statistics, pp. 29–37, 2011.

    Google Scholar 

  24. Y. LeCun, S. Chopra, R. M. Hadsell, M. A. Ranzato, and F.-J. Huang. A tutorial on energy-based learning. Predicting Structured Data, MIT Press, pp. 191–246,, 2006.

    Google Scholar 

  25. G. Montufar and N. Ay. Refinements of universal approximation results for deep belief networks and restricted Boltzmann machines. Neural Computation, 23(5), pp. 1306–1319, 2011.

    MathSciNet  CrossRef  Google Scholar 

  26. V. Nair and G. Hinton. Rectified linear units improve restricted Boltzmann machines. ICML Conference, pp. 807–814, 2010.

    Google Scholar 

  27. R. M. Neal. Connectionist learning of belief networks. Artificial intelligence, 1992.

    Google Scholar 

  28. R. M. Neal. Probabilistic inference using Markov chain Monte Carlo methods. Technical Report CRG-TR-93-1, 1993.

    Google Scholar 

  29. R. M. Neal. Annealed importance sampling. Statistics and Computing, 11(2), pp. 125–139, 2001.

    MathSciNet  CrossRef  Google Scholar 

  30. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Ng. Multimodal deep learning. ICML Conference, pp. 689–696, 2011.

    Google Scholar 

  31. C. Peterson and J. Anderson. A mean field theory learning algorithm for neural networks. Complex Systems, 1(5), pp. 995–1019, 1987.

    MATH  Google Scholar 

  32. S. Rendle. Factorization machines. IEEE ICDM Conference, pp. 995–100, 2010.

    Google Scholar 

  33. R. Salakhutdinov, A. Mnih, and G. Hinton. Restricted Boltzmann machines for collaborative filtering. ICML Confererence, pp. 791–798, 2007.

    Google Scholar 

  34. R. Salakhutdinov and G. Hinton. Semantic Hashing. SIGIR workshop on Information Retrieval and applications of Graphical Models, 2007.

    Google Scholar 

  35. R. Salakhutdinov and G. Hinton. Deep Boltzmann machines. Artificial Intelligence and Statistics, pp. 448–455, 2009.

    Google Scholar 

  36. R. Salakhutdinov and H. Larochelle. Efficient Learning of Deep Boltzmann Machines. AISTATs, pp. 693–700, 2010.

    Google Scholar 

  37. T. J. Sejnowski. Higher-order Boltzmann machines. AIP Conference Proceedings, 15(1), pp. 298–403, 1986.

    MathSciNet  Google Scholar 

  38. P. Smolensky. Information processing in dynamical systems: Foundations of harmony theory. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Volume 1: Foundations. pp. 194–281, 1986.

    Google Scholar 

  39. N. Srivastava and R. Salakhutdinov. Multimodal learning with deep Boltzmann machines. NIPS Conference, pp. 2222–2230, 2012.

    Google Scholar 

  40. N. Srivastava, R. Salakhutdinov, and G. Hinton. Modeling documents with deep Boltzmann machines. Uncertainty in Artificial Intelligence, 2013.

    Google Scholar 

  41. A. Storkey. Increasing the capacity of a Hopfield network without sacrificing functionality. Artificial Neural Networks, pp. 451–456, 1997.

    Google Scholar 

  42. I. Sutskever and T. Tieleman. On the convergence properties of contrastive divergence. International Conference on Artificial Intelligence and Statistics, pp. 789–795, 2010.

    Google Scholar 

  43. T. Tieleman. Training restricted Boltzmann machines using approximations to the likelihood gradient. ICML Conference, pp. 1064–1071, 2008.

    Google Scholar 

  44. Y. Teh and G. Hinton. Rate-coded restricted Boltzmann machines for face recognition. NIPS Conference, 2001.

    Google Scholar 

  45. M. Welling, M. Rosen-Zvi, and G. Hinton. Exponential family harmoniums with an application to information retrieval. NIPS Conference, pp. 1481–1488, 2005.

    Google Scholar 

  46. E. Xing, R. Yan, and A. Hauptmann. Mining associated text and images with dual-wing harmoniums. Uncertainty in Artificial Intelligence, 2005.

    Google Scholar 

  47. http://www.netflixprize.com/community/topic_1537.html

Download references

Author information

Authors and Affiliations

  1. IBM T. J. Watson Research Center, International Business Machines, Yorktown Heights, NY, USA

    Charu C. Aggarwal

Authors
  1. Charu C. Aggarwal
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and Permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this chapter

Verify currency and authenticity via CrossMark

Cite this chapter

Aggarwal, C.C. (2018). Restricted Boltzmann Machines. In: Neural Networks and Deep Learning. Springer, Cham. https://doi.org/10.1007/978-3-319-94463-0_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-94463-0_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-94462-3

  • Online ISBN: 978-3-319-94463-0

  • eBook Packages: Computer ScienceComputer Science (R0)