Abstract
Deep Convolutional Neural Networks, like most connectionist models, suffers from catastrophic forgetting while training for a new, unknown task. One of the simplest solutions to this issue is adding samples of previous data, with the drawback of increasingly having to store training data; or generating patterns that evoke similar responses of the previous task.
We propose a model using a Recurrent Neural Network-based image generator in order to provide a Deep Convolutional Network a limited number of samples for new training data. Simulation results shows that our proposal is able to retain previous knowledge whenever some few pseudo-samples of previously recorded patterns are generated.
Despite having lower performance than giving the network samples of the real dataset, this model is more biologically plausible and might help to reduce the need of storing previously trained data on bigger-scale classification classification models.
D. Mellado—The authors acknowledge the support by Chilean Grants “Proyecto estudiantes de los convenios de desempeño UVA 1315, UVA 1401 and UVA 1402” from Universidad de Valparaíso, and CONICYT + PAI/CONCURSO NACIONAL INSERCIÓN EN LA ACADEMIA, CONVOCATORIA 2014 + Folio (79140057). The work of R. Salas was partially funded by project grant FONDEF IDEA ID16I10322.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dhar, V.: The scope and challenges for deep learning. Big Data 3(3), 127–129 (2015)
French, R.M.: Catastrophic forgetting in connectionist networks. Trends Cogn. Sci. 3(4), 128–135 (1999)
Crossberg, S.: Studies of Mind and Brain: Neural Principles of Learning, Perception Development, Cognition and Motor Control. D. Reidel Publishing, Dordrecht (1982)
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., Hadsell, R.: Overcoming catastrophic forgetting in neural networks. Proc. Nat. Acad. Sci. 114(13), 3521–3526 (2017)
Salas, R., Moreno, S., Allende, H., Moraga, C.: A robust and flexible model of hierarchical self-organizing maps for non-stationary environments. Neurocomputing 70(16), 2744–2757 (2007)
Mellado, D., Salas, R., Chabert, S., Veloz, A., Saavedra, C.: Incremental ensemble of deep learning with pseudorehearsal. In: II Latin American Conference on Statistical Computing, Valparaiso, Chile, March 2017
Ratcliff, R.: Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychol. Rev. 97(2), 285–308 (1990)
McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: the sequential learning problem. In: Bower, G.H. (ed.) Psychology of Learning and Motivation, vol. 24, pp. 109–165. Academic Press, New York (1989)
French, R.M.: Pseudo-recurrent connectionist networks: an approach to the ‘sensitivity-stability’ dilemma. Connect. Sci. 9(4), 353–380 (1997)
French, R.M.: Semi-distributed representations and catastrophic forgetting in connectionist networks. Connect. Sci. 4(3–4), 365–377 (1992)
Robins, A., McCallum, S.: Catastrophic forgetting and the pseudorehearsal solution in hopfield-type networks. Connect. Sci. 10(2), 121–135 (1998)
Robins, A.: Catastrophic forgetting, rehearsal and pseudorehearsal. Connect. Sci. 7(2), 123–146 (1995)
Hattori, M., Hosaka, T.: Applying pseudorehearsal to multilayer neural networks trained by no-prop algorithm. In: Advances in Computational Intelligence, November 2015
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Gregor, K., Danihelka, I., Graves, A., Rezende, D.J., Wierstra, D.: DRAW: a recurrent neural network for image generation, February 2015. arXiv: 1502.04623 [cs]
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of 13th International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)
Chollet, F., et al.: Keras (2015). https://github.com/fchollet/keras
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Mellado, D., Saavedra, C., Chabert, S., Salas, R. (2017). Pseudorehearsal Approach for Incremental Learning of Deep Convolutional Neural Networks. In: Barone, D., Teles, E., Brackmann, C. (eds) Computational Neuroscience. LAWCN 2017. Communications in Computer and Information Science, vol 720. Springer, Cham. https://doi.org/10.1007/978-3-319-71011-2_10
Download citation
DOI: https://doi.org/10.1007/978-3-319-71011-2_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-71010-5
Online ISBN: 978-3-319-71011-2
eBook Packages: Computer ScienceComputer Science (R0)