Variational EM Learning of DSBNs with Conditional Deep Boltzmann Machines

  • Xing Zhang
  • Siwei Lyu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8681)


Variational EM (VEM) is an efficient parameter learning scheme for sigmoid belief networks with many layers of latent variables. The choice of the inference model that forms the variational lower bound of the log likelihood is critical in VEM learning. The mean field approximations and wake-sleep algorithm use simple models that are computationally efficient, but may be poor approximations to the true posterior densities when the latent variables have strong mutual dependencies. In this paper, we describe a variational EM learning method of DSBNs with a new inference model known as the conditional deep Boltzmann machine (cDBM), which is an undirected graphical model capable of representing complex dependencies among latent variables. We show that this algorithm does not require the computation of the intractable partition function in the undirected cDBM model, and can be accelerated with contrastive learning. Performances of the proposed method are evaluated and compared on handwritten digit data.


Latent Variable Inference Model Latent Variable Model Complex Dependency Visible Variable 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bengio, Y., LeCun, Y.: Scaling learning algorithms towards ai. In: Bottou, L., Chapelle, O., DeCoste, D., Weston, J. (eds.) Large-Scale Kernel Machines. MIT Press (2007)Google Scholar
  2. 2.
    Cover, T., Thomas, J.: Elements of Information Theory, 2nd edn. Wiley-Interscience (2006)Google Scholar
  3. 3.
    Dayan, P., Hinton, G.E.: Varieties of helmholtz machines. Neural Networks 9, 1385–1403 (1996)CrossRefzbMATHGoogle Scholar
  4. 4.
    Dempster, A., Laird, N., Rubin, D.: Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society, Series B 39, 1–38 (1977)zbMATHMathSciNetGoogle Scholar
  5. 5.
    Hinton, G.E.: Training products of experts by minimizing contrastive divergence. Neural Computation 14, 1771–1800 (2002)CrossRefzbMATHGoogle Scholar
  6. 6.
    Hinton, G.E., Dayan, P., Frey, B.J., Neal, R.: The wake-sleep algorithm for unsupervised neural networks. Science 268, 1158–1161 (1995)CrossRefGoogle Scholar
  7. 7.
    Hinton, G.E., Osindero, S., Teh, Y.: A fast learning algorithm for deep belief nets. Neural Computation 18(10), 1527–1554 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    Jaakkola, T., Jordan, M.: Improving the mean field approximation via the use of mixture distributions. In: Jordan, M.I. (ed.) Learning in Graphical Models. MIT Press (1998)Google Scholar
  9. 9.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  10. 10.
    Mnih, A., Gregor, K.: Neural variational inference and learning in belief networks. arXiv:1402.0030v1 (cs.LG) (January 2014)Google Scholar
  11. 11.
    Neal, R.M.: Connectionist learning of belief networks. Artificial Intelligence 56, 71–113 (1992)CrossRefzbMATHMathSciNetGoogle Scholar
  12. 12.
    Neal, R.M., Hinton, G.E.: A view of the em algorithm that justifies incremental, sparse, and other variants. In: Learning in Graphical Models, pp. 355–368. Kluwer Academic Publishers (1998)Google Scholar
  13. 13.
    Pearl, J.: Probabilistic Reasoning in Intelligent Systems. Morgan-Kaufmann (1988)Google Scholar
  14. 14.
    Salakhutdinov, R., Hinton, G.E.: Deep boltzmann machines. In: AISTATS (2009)Google Scholar
  15. 15.
    Saul, L.K., Jaakkola, T., Jordan, M.I.: Mean field theory for sigmoid belief networks. Journal of Artificial Intelligence Research 4, 61–76 (1996)zbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Xing Zhang
    • 1
  • Siwei Lyu
    • 1
  1. 1.Computer Science DepartmentState University of New YorkUSA

Personalised recommendations