A Structural Learning Method of Restricted Boltzmann Machine by Neuron Generation and Annihilation Algorithm

  • Shin KamadaEmail author
  • Takumi Ichimura
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9950)


Restricted Boltzmann Machine (RBM) is a generative stochastic energy-based model of artificial neural network for unsupervised learning. The adaptive learning method that can discover the optimal number of hidden neurons according to the input space is important method in terms of the stability of energy as well as the computational cost although a traditional RBM model cannot change its network structure during learning phase. Moreover, we should consider the regularities in the sparse of network to extract explicit knowledge from the network because the trained network is often a black box. In this paper, we propose the combination method of adaptive and structural learning method of RBM with Forgetting that can discover the regularities in the trained network. We evaluated our proposed model on MNIST and CIFAR-10 datasets.


Restricted Boltzmann Machine Neuron generation and annihilation Structural learning Regularity of RBM Network 


  1. 1.
    Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. Arch. 2(1), 1–127 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Hinton, G.E.: A practical guide to training restricted boltzmann machines. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade, 2nd edn. LNCS, vol. 7700, pp. 599–619. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  3. 3.
    Kamada, S., Ichimura, T.: A learning method of adaptive deep belief network by using neuron generation and annihilation algorithm. In: Proceedings of 17th Annual Meeting of Self-Organizing Maps in Japanese, pp. 12.1–12.6 (2016)Google Scholar
  4. 4.
    Ranzato, M., Boureau, Y., LeCun, Y.: Sparse feature learning for deep belief networks. In: Advances in Neural Information Processing Systems 20 (NIPS 2007), pp. 1185–1192 (2007)Google Scholar
  5. 5.
    Ishikawa, M.: Structural learning with forgetting. Neural Netw. 9(3), 509–521 (1996)CrossRefGoogle Scholar
  6. 6.
    Kamada, S., Fujii, Y., Ichimura, T.: Structural learning method of restricted Boltzmann machine with forgetting. In: Proceedings of 17th Annual Meeting of Self-Organizing Maps in Japanese, pp. 13.1–13.6 (2016)Google Scholar
  7. 7.
    Hinton, G.E.: Training products of experts by minimizing contrastive divergence. Neural Comput. 14, 1771–1800 (2002)CrossRefzbMATHGoogle Scholar
  8. 8.
    Carlson, D., Cevher, V., Carin, L.: Stochastic spectral descent for restricted Boltzmann machines. In: Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, pp. 111–119 (2015)Google Scholar
  9. 9.
    Kamada, S., Ichimura, T., Fujii, Y.: A consideration of convergence of energy function in restricted Boltzmann machine by Lipschitz continuity. In: Proceedings of IEEE SMC Hiroshima Chapter YRW 2015 in Japanese, pp. 53–56 (2015)Google Scholar
  10. 10.
    Ichimura, T., Yoshida, K. (eds.): Knowledge-Based Intelligent Systems for Health Care. Advanced Knowledge International (ISBN 0-9751004-4-0) (2004)Google Scholar
  11. 11.
    LeCun, Y., et al.: THE MNIST DATABASE of handwritten digits (2015).
  12. 12.
    Krizhevsky, A.: Learning multiple layers of features from tiny images. Master of thesis, University of Toronto (2009)Google Scholar
  13. 13.
    Dieleman, S., Schrauwen, B.: Accelerating sparse restricted Boltzmann machine training using non-Gaussianity measures. In: Deep Learning and Unsupervised Feature Learning (NIPS-2012) (2012)Google Scholar
  14. 14.
    Hinton, G.E., Osindero, S., Teh, Y.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Kamada, S., Ichimura, T.: An adaptive learning method with forgetting in deep belief network. In: Proceedings of 9th SICE Symposium on Computational Intelligence, pp. 92–96 (2016)Google Scholar
  16. 16.
    Kamada, S., Ichimura, T.: An adaptive learning method of restricted Boltzmann machine by neuron generation and annihilation algorithm. In: Proceedings of 2016 IEEE SMC (SMC 2016) (accepted)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Graduate School of Information SciencesHiroshima City UniversityHiroshimaJapan
  2. 2.Faculty of Management and Information SystemsPrefectural University of HiroshimaHiroshimaJapan

Personalised recommendations