Abstract
Neural networks have drawn much attention in modern machine learning community as they have achieved many successful applications, such as image recognition, speech recognition and system identification. According to the principle of parsimony, simpler neural models are preferable to more complex ones if they have similar generalization performance. However, when building a neural networks model, the neuron number is often determined randomly or by trial-and-error. These methods can often lead to the over-complex networks with many redundant neurons and therefore may result in over-fitting problems. In this paper, a new approach is proposed for obtaining a simplified neural networks with fewer neurons but still keeping a good performance comparing to the initial fully networks. More specifically, the initial neural model with a fixed model size is built using Matlab toolbox. Then, the orthogonal matching pursuit method is employed to select important neurons and drop out redundant neurons, leading to a more compact model with reduced size. Two simulation examples are used to demonstrate the effectiveness of the proposed method.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Lecun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)
Cai, T., Wang, L.: Orthogonal matching pursuit for sparse signal recovery with noise. IEEE Trans. Inf. Theory 57, 4680–4688 (2011)
Zhang, L., Li, K., Bai, E.W., Irwin, G.W.: Two-stage orthogonal least squares methods for neural network construction. IEEE Trans. Neural Netw. Learn. Syst. 26, 1608–1621 (2014)
Pan, W., Dong, H., Guo, Y.: DropNeuron: simplifying the structure of deep neural networks, arXiv preprint arXiv:1606.07326 (2016)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)
Chen, S., Billings, S.A., Luo, W.: Orthogonal least squares methods and their application to non-linear system identification. Int. J. Control 50, 1873–1986 (1989)
Wang, J., Kwon, S., Shim, B.: Generalized orthogonal matching pursuit. IEEE Trans. Sig. Process. 60, 6202–6216 (2012)
Pati, Y., Rezaiifar, R., Krishnapresad, P.: Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. In: Proceeding of 27th Asilomar Conference on Signals, Systems and Computers, pp. 1–5 (1993)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Tang, X., Wang, X., Zhang, L. (2017). Orthogonal Matching Pursuit for Multilayer Perceptions Neural Networks Model Reduction. In: Yue, D., Peng, C., Du, D., Zhang, T., Zheng, M., Han, Q. (eds) Intelligent Computing, Networked Control, and Their Engineering Applications. ICSEE LSMS 2017 2017. Communications in Computer and Information Science, vol 762. Springer, Singapore. https://doi.org/10.1007/978-981-10-6373-2_6
Download citation
DOI: https://doi.org/10.1007/978-981-10-6373-2_6
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-6372-5
Online ISBN: 978-981-10-6373-2
eBook Packages: Computer ScienceComputer Science (R0)