Abstract
Neuroscience studies inspire that structures are needed in the hidden space of deep learning models. In this paper, we propose a capsule restricted Boltzmann machine and a capsule Helmholtz machine by replacing individual hidden variables with encapsulated groups of hidden variables. Our preliminary experiments show that capsule activities in both models can be dynamically determined in context, and these activity spectra exhibit between-class patterns and within-class variations. Our models offer a novel approach to visualizing and understanding the hidden states.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C.: A neural probabilistic language model. J. Mach. Learn. Res. 2, 1137–1155 (2003)
Bornschein, J., Bengio, Y.: Reweighted wake-sleep. In: International Conference on Learning Representations (2015)
Courville, A., Bergstra, J., Bengio, Y.: A spike and slab restricted Boltzmann machine. In: International Conference on Artificial Intelligence and Statistics, pp. 233–241 (2011)
Dayan, P., Hinton, G., Neal, R., Zemel, R.: The Helmholtz machine. Neural Comput. 7, 1022–1037 (1995). https://doi.org/10.1162/neco.1995.7.5.889
Fodor, J., Pylyshyn, Z.: Connectionism and cognitive architecture: a critical analysis. Cognition 28(1–2), 3–71 (1988). https://doi.org/10.1016/0010-0277(88)90031-5
Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)
Hinton, G.: Aetherial symbols. In: AAAI Spring Symposium on Knowledge Representation and Reasoning: Integrating Symbolic and Neural Approaches (2015)
Hinton, G., et al.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82–97 (2012). https://doi.org/10.1109/MSP.2012.2205597
Hinton, G., Krizhevsky, A., Wang, S.: Transforming auto-encoder. In: International Conference on Artificial Neural Networks, pp. 44–51 (2011). https://doi.org/10.1007/978-3-642-21735-7_6
Hinton, G., McClelland, J., Rumelhart, D.: Distributed representations. In: Rumelhart, D., McClelland, J. (eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, pp. 77–109. MIT Press, Cambridge (1986)
Hinton, G., Osindero, S., Teh, Y.: A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006). https://doi.org/10.1162/neco.2006.18.7.1527
Hinton, G., Sabour, S., Frosst, N.: Matrix capsules with EM routing. In: International Conference on Learning Representations (2018)
Kingma, D., Welling, M.: Auto-encoding variational Bayes. In: International Conference on Learning Representations (2014)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
LeCun, Y., et al.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989). https://doi.org/10.1162/neco.1989.1.4.541
Li, Y., Zhu, X.: Exploring Helmholtz machine and deep belief net in the exponential family perspective. In: ICML 2018 Workshop on Theoretical Foundations and Applications of Deep Generative Models (2018)
Li, Y., Zhu, X.: Exponential family restricted Boltzmann machines and annealed importance sampling. In: International Joint Conference on Neural Networks, pp. 39–48 (2018). https://doi.org/10.1109/IJCNN.2018.8489413
Mnih, A., Gregor, K.: Neural variational inference and learning in belief networks. In: International Conference on Machine Learning, pp. II-1791–II-1799 (2014)
Rezende, D.J., Mohamed, S., Wierstra, D.: Stochastic backpropagation and approximate inference in deep generative models. In: International Conference on Machine Learning, pp. II-1278–II-1286 (2014)
Sabour, S., Frosst, N., Hinton, G.: Dynamic routing between capsules. In: Neural Information Processing Systems, pp. 3856–3866 (2017)
Salakhutdinov, R.: Learning and evaluating Boltzmann machines. Department of Computer Science, University of Toronto, Toronto, Canada, Technical report (2008)
Tieleman, T.: Training restricted Boltzmann machines using approximations to the likelihood gradient. In: International Conference on Machine Learning, pp. 1064–1071 (2008). https://doi.org/10.1145/1390156.1390290
Welling, M., Rosen-Zvi, M., Hinton, G.: Exponential family harmoniums with an application to information retrieval. In: Advances in Neural Information Processing Systems, pp. 1481–1488 (2005)
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. ArXiv p. arXiv:1708.07747v2 (2017)
Zhao, S., Song, J., Ermon, S.: Towards a deeper understanding of variational autoencoding models. arXiv p. arXiv:1702.08658 (2017)
Acknowledgement
This project was supported by the NRC New Beginnings Ideation Fund. The authors would like to thank the anonymous reviewers for valuable comments.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Crown
About this paper
Cite this paper
Li, Y., Zhu, X. (2019). Capsule Generative Models. In: Tetko, I., Kůrková, V., Karpov, P., Theis, F. (eds) Artificial Neural Networks and Machine Learning – ICANN 2019: Theoretical Neural Computation. ICANN 2019. Lecture Notes in Computer Science(), vol 11727. Springer, Cham. https://doi.org/10.1007/978-3-030-30487-4_22
Download citation
DOI: https://doi.org/10.1007/978-3-030-30487-4_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-30486-7
Online ISBN: 978-3-030-30487-4
eBook Packages: Computer ScienceComputer Science (R0)