Natural Computing

, Volume 18, Issue 4, pp 721–733 | Cite as

A regularization-reinforced DBN for digital recognition

  • Junfei QiaoEmail author
  • Guangyuan Pan
  • Honggui Han


The problem of over fitting in DBN is extensively focused on since different networks may respond differently to an unknown input. In this study, a regularization-reinforced deep belief network (RrDBN) is proposed to improve generalization ability. In RrDBN, a special regularization-reinforced term is developed to make the weights in the unsupervised training process to attain a minimum magnitude. Then, the non-contributing weights are reduced and the resultant network can represent the inter-relations of the input–output characteristics. Therefore, the optimization process is able to obtain the minimum-magnitude weights of RrDBN. Moreover, contrastive divergence is introduced to increase RrDBN’s convergence speed. Finally, RrDBN is applied to hand-written numbers classification and water quality prediction. The results of the experiments show that RrDBN can improve the recognition performance with less recognition errors than other existing methods.


Generalization Regularization Recognition Deep belief net 



This work is supported by the National Science Foundation of China under Grants 61203099 and 61225016, Beijing Science and Technology Project under Grants Z141100001414005 and Z141101004414058, Hong Kong Scholar Program under Grant XJ2013018, Beijing Nova Program under Grant Z131104000413007, Beijing Municipal Education Commission Foundation under Grants km201410005001 and KZ201410005002, China Postdoctoral Science Foundation under Grants 2014M550017, Ph.D. Program Foundation from Ministry of Chinese Education under Grants 20121103120020 and 20131103110016. Major part of this research in this paper is carried out while the author is visiting the University of Victoria. He expresses his sincere thanks to China Scholarship Council for financial support and to the Department of Electrical and Computer Engineering at the University of Victoria for the kind hospitality.


  1. Altineay H, Demirelder M (2003) Undesirable effects of out-put normalization in multiple classifier systems. Pattern Recognit Lett 24:1163–1170CrossRefGoogle Scholar
  2. Areerachakul S, Sanguansintukul S (2009) Water quality classification using neural networks: case study of canals in Bangkok. In: ICITST 2009, pp 1–5Google Scholar
  3. Bengio Y (2009) Learning deep architectures for AI. Found Trends Mach Learn 2(1):1–127MathSciNetCrossRefGoogle Scholar
  4. Bengio Y, Lamlin P, Popovici D, Larochelle H (2007) Greedy layer-wise training of deep networks. In: Proceedings of the advances in neural information processing systems (NIPS2006), vol 19, pp 1–8Google Scholar
  5. Bouamar M, Ladjal M (2007) Evaluation of the performances of ANN and SVM techniques used in water quality classification. In: ICECS 2007, pp 1047–1050Google Scholar
  6. Chen H, Murray AF (2003) A continuous restricted Boltzmann machine with an implementable training algorithm. IEEE Proc Vis Image Signal Process 3(150):153–158CrossRefGoogle Scholar
  7. Dahl GE, Dong Y, Li D (2011) Large vocabulary continuous speech recognition with context-dependent DBN–HMMS. In: IEEE international conference on acoustics, speech and signal processing, pp 4688–4691Google Scholar
  8. Deselaers T, Hasan S, Bender O (2009) A deep learning approach to machine transliteration. In: The 4th workshop on statistical machine translation, pp 233–241Google Scholar
  9. Fasel I, Berry J (2010) Deep belief networks for real-time extraction of tongue contours from ultrasound during speech. In: The 20th international conference on pattern recognition. Association for Computational Linguistics, Stroudsburg, PA, USA, pp 1493–1496Google Scholar
  10. Forsee FD, Hagan FD (1997) Gauss–Newton approximation to Bayesian regularization. In: IEEE international joint conference on neural networks, vol 6, pp 1930–1935Google Scholar
  11. Han H, Qiao J (2013) A structure optimization algorithm for feed forward neural network construction. Neurocomputing 99(1):347–357CrossRefGoogle Scholar
  12. Hinton GE (2002) Training products of experts by minimizing contrastive divergence. Neural Comput 14:1771–1800CrossRefGoogle Scholar
  13. Hinton G (2006) Deep learning. [Online].
  14. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507MathSciNetCrossRefGoogle Scholar
  15. Hinton GE, Osindero S, The YW (2006a) A faster learning algorithm for deep belief nets. Neural Comput 1(7):1527–1544CrossRefGoogle Scholar
  16. Hinton GE, Osindero S, Welling M (2006b) Unsupervised discovery of non-linear structure using contrastive backpropagation. Cogn Sci 30(4):725–731CrossRefGoogle Scholar
  17. Hinton GE, Srivastava N, Krizhevsky A (2012) Improving neural networks by preventing co-adaptation of feature detectors. University of Toronto, Toronto. arXiv preprint arXiv:1207.0580
  18. Itamar A, Derek C, Rose T (2010) Deep machine learning—a new frontier in artificial intelligence research. IEEE Comput Intell Mag 11:13–18Google Scholar
  19. Jurgen S (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117CrossRefGoogle Scholar
  20. Kunisch K, Zou J (1998) Iterative choices of regularization parameter in linear in verse problems. Inverse Probl 14:1247–1264CrossRefGoogle Scholar
  21. Kuremoto T, Kimura S, Kobayashi K (2014) Time series forecasting using a deep belief network with restricted Boltzmann machines. Neurocomputing 137:47–56CrossRefGoogle Scholar
  22. LeCun Y, Cortes C, Burges C (1998) The MNIST database of handwritten digits. [Online].
  23. Lee H, Ekanadham C, Ng AY (2008) Sparse deep belief net model for visual area V2. In: Advances in neural information processing systems 20 (NIPS’ 07), Vancouver, Canada, pp 873–880Google Scholar
  24. Li D, Seltzer ML, Dong Y (2010) Binary coding of speech spectrograms using a deep auto-encoder. In: The 11th annual conference of International Speech Communication Association, pp 1692–1695Google Scholar
  25. Li J, Wang X, Xu B (2013) Understanding the dropout strategy and analyzing its effectiveness on LVCSR. In: ICASSP 2013, pp 7614–7618Google Scholar
  26. Mohamed A, Dahl GE, Hinton GE (2012) Acoustic modeling using deep belief networks. IEEE Trans Audio Speech Lang Process 20(1):14–22CrossRefGoogle Scholar
  27. Najah A, Elshafie A, Karim OA, Jaffar O (2009) Prediction of Johor river water quality parameter using artificial neural networks. J Sci Res 1:422–435CrossRefGoogle Scholar
  28. Patan K (2007) Stability analysis and the stabilization of a class of discrete-time dynamic neural networks. IEEE Trans Neural Netw 3(18):660–673CrossRefGoogle Scholar
  29. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representation by back-propagating errors. Nature 232:533–536CrossRefGoogle Scholar
  30. Tieleman T. (2008) Training restricted Boltzmann machines using approximations to the likelihood gradient. In; Proceedings of the 25th international conference on machine learning, Helsinki, Finland, pp 1064–1071Google Scholar
  31. Tieleman T, Hinton GE (2009) Using fast weights to improve persistent contrastive divergence. In: Proceedings of the 26th international conference on machine learning, Montreal, Canada, pp 1033–1040Google Scholar
  32. Yu C, Zheng D, Zhao T (2012) Chinese relation extraction based on deep belief nets. J Softw 23(10):2572–2585CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2017

Authors and Affiliations

  1. 1.Beijing Key Laboratory of Computational Intelligence and Intelligent SystemBeijing University of TechnologyBeijingChina

Personalised recommendations