Abstract
The objective of our study is to find out how a sparse structure affects the performance of a recurrent neural network (RNN). Only a few existing studies have dealt with the sparse structure of RNN with learning like Back Propagation Through Time (BPTT). In this paper, we propose a RNN with sparse connection and BPTT called Multiple time scale RNN (MTRNN). Then, we investigated how sparse connection affects generalization performance and noise robustness. In the experiments using data composed of alphabetic sequences, the MTRNN showed the best generalization performance when the connection rate was 40%. We also measured sparseness of neural activity and found out that sparseness of neural activity corresponds to generalization performance. These results means that sparse connection improved learning performance and sparseness of neural activity would be used as metrics of generalization performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Olshausen, B.A., Field, D.J.: Sparse coding of sensory inputs. Current Opinion in Neurobiology 14(4), 481–487 (2004)
Waydo, S., Kraskov, A., Quian Quiroga, R., Fried, I., Koch, C.: Sparse representation in the human medial temporal lobe. J. Neurosci. 26(40), 10232–10234 (2006)
Smith, E., Lewicki, M.: Efficient auditory coding. Nature 439(7079), 978–982 (2006)
Vinje, W.E., Gallant, J.L.: Sparse Coding and Decorrelation in Primary Visual Cortex During Natural Vision. Science 287(5456), 1273–1276 (2000)
Kanerva, P.: Sparse distributed memory and related models, pp. 50–76. Oxford University Press, Inc. (1993)
Palm, G., Sommer, F.: Associative data storage and retrieval in neural networks. Models of Neural Networks III, 79–118 (1996)
Jaeger, H., Haas, H.: Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication. Science 304(5667), 78–80 (2004)
Andreea, L., Pipa Gordon, T.J.: SORN: a self-organizing recurrent neural network. Front. Comput. Neurosci. 3(23) (2009), doi:10.3389/neuro.10.023
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation, pp. 318–362. MIT Press (1986)
Yamashita, Y., Tani, J.: Emergence of functional hierarchy in a multiple timescale neural network model: A humanoid robot experiment. PLoS Comput. Biol. 4(11), e1000220 (2008)
Hinoshita, W., Arie, H., Tani, J., Okuno, H.G., Ogata, T.: Emergence of hierarchical structure mirroring linguistic composition in a recurrent neural network. Neural Networks (in Press, 2011), doi:10.1016/j.neunet.2010.12.006
The ‘independent components’ of natural scenes are edge filters. Vision Research 37, 3327–3338 (1997)
Willmore, B., Tolhurst, D.: Characterizing the sparseness of neural codes. Network: Computation in Neural Systems 12(3), 255–270 (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Awano, H. et al. (2011). Use of a Sparse Structure to Improve Learning Performance of Recurrent Neural Networks. In: Lu, BL., Zhang, L., Kwok, J. (eds) Neural Information Processing. ICONIP 2011. Lecture Notes in Computer Science, vol 7064. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-24965-5_36
Download citation
DOI: https://doi.org/10.1007/978-3-642-24965-5_36
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-24964-8
Online ISBN: 978-3-642-24965-5
eBook Packages: Computer ScienceComputer Science (R0)