Overcoming Catastrophic Interference with Bayesian Learning and Stochastic Langevin Dynamics

  • Mikhail Leontev
  • Alexander Mikheev
  • Kirill Sviatov
  • Sergey SukhovEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11554)


Neural networks encounter serious catastrophic forgetting when information is learned sequentially. Although simply replaying all previous data alleviates the problem, it may require large memory to store all previous training examples. Even with enough memory, joint training can be infeasible if access to past data is limited. We developed generative methods for preventing catastrophic forgetting that do not require the presence of previously used data. Developed methods are based on activation maximization of output neurons and on sampling of posterior probability of data distribution. The methods can work for regular feedforward networks. The proof of concept experiments were performed on publicly available datasets.


Catastrophic interference Feedforward neural network Generative replay 


  1. 1.
    McCloskey, M., Cohen, N.-J.: Catastrophic interference in connectionist networks: the sequential learning problem. In: Bower, G.H. (ed.) Psychology of Learning and Motivation, vol. 24, pp. 109–165. Academic Press, San Diego (1989)Google Scholar
  2. 2.
    Liu, B.: Lifelong machine learning: a paradigm for continuous learning. Front. Comput. Sci. 11(3), 359–361 (2017)Google Scholar
  3. 3.
    Silver, D.-L., Yang, Q., Li, L.: Lifelong machine learning systems: beyond learning algorithms. In: AAAI Spring Symposium Lifelong Machine Learning, p. 5. AAAI, Stanford (2013)Google Scholar
  4. 4.
    Choy, M.-C., Srinivasan, D., Cheu, R.-L.: Neural networks for continuous online learning and control. IEEE Trans. Neural Netw. 17(6), 1511–1531 (2006)Google Scholar
  5. 5.
    Goodfellow, I.-J., Mirza, M., Xiao, D., Courville, A., Bengio, Y.: An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211 (2013)
  6. 6.
    Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. U.S.A. 114(13), 3521–3526 (2017)Google Scholar
  7. 7.
    Zenke, F., Poole, B., Ganguli S.: Continual learning through synaptic intelligence. In: Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 3987–3995. PMLR, Sydney (2017)Google Scholar
  8. 8.
    Lee, S.-W., Kim, J.-H., Jun, J., Ha, J.-W., Zhang, B.-T.: Overcoming catastrophic forgetting by incremental moment matching. In: Advances in Neural Information Processing Systems, pp. 4652–4662. Curran Associates Inc., Long Beach (2017)Google Scholar
  9. 9.
    Robins, A.: Catastrophic forgetting, rehearsal and pseudorehearsal. Connect. Sci. 7(2), 123–146 (1995)Google Scholar
  10. 10.
    French, R.-M.: Pseudo-recurrent connectionist networks: an approach to the ‘sensitivity-stability’ dilemma. Connect. Sci. 9(4), 353–380 (1997)Google Scholar
  11. 11.
    Ans, B., Rousset, S.: Avoiding catastrophic forgetting by coupling two reverberating neural networks. Comptes Rendus de l’Académie des Sciences-Series III-Sciences de la Vie 320(12), 989–997 (1997)Google Scholar
  12. 12.
    Mocanu, D.-C., Vega, M.-T., Eaton, E., Stone, P., Liotta, A.: Online contrastive divergence with generative replay: Experience replay without storing data. arXiv preprint arXiv:1610.05555 (2016)
  13. 13.
    Shin, H., Lee, J.-K., Kim, J., Kim, J.: Continual learning with deep generative replay. In: Advances in Neural Information Processing Systems, pp. 2990–2999. Curran Associates Inc., Long Beach (2017)Google Scholar
  14. 14.
    van de Ven, G.-M., Tolias, A.-S.: Generative replay with feedback connections as a general strategy for continual learning. arXiv preprint arXiv:1809.10635 (2018)
  15. 15.
    Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013)
  16. 16.
    Erhan, D., Bengio, Y., Courville, A., Vincent, P.: Visualizing higher-layer features of a deep network. Univ. Mont. 1341(3), 1 (2009)Google Scholar
  17. 17.
    Yosinski, J., Clune, J., Fuchs, T., Lipson, H.: Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579 (2015)
  18. 18.
    Lewis, J.: Creation by refinement: a creativity paradigm for gradient descent learning networks. In: IEEE 1988 International Conference on Neural Networks, vol. 2, pp. 229–233. IEEE, San Diego (1988)Google Scholar
  19. 19.
    Welling, M., Teh, Y.-W.: Bayesian learning via stochastic gradient Langevin dynamics. In: Proceedings of the 28th International Conference on Machine Learning (ICML 2011), pp. 681–688. Omnipress, Bellevue (2011)Google Scholar
  20. 20.
    Douglass, K.-M., Sukhov, S., Dogariu, A.: Superdiffusion in optically controlled active media. Nat. Photon. 6(12), 834–837 (2012)Google Scholar
  21. 21.
    Romero, A.-H., Sancho, J.-M.: Brownian motion in short range random potentials. Phys. Rev. E 58(3), 2833 (1998)Google Scholar
  22. 22.
    Kamaruzaman, A.-F., Zain, A.-M., Yusuf, S.-M., Udin, A.: Levy flight algorithm for optimization problems-a literature review. Appl. Mech. Mater. 421, 496–501 (2013)Google Scholar
  23. 23.
    Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Kotel’nikov Institute of Radio Engineering and Electronics of Russian Academy of Sciences (Ulyanovsk branch)UlyanovskRussia
  2. 2.S.P. Kapitsa Technological Research InstituteUlyanovsk State UniversityUlyanovskRussia
  3. 3.Ulyanovsk State Technical UniversityUlyanovskRussia

Personalised recommendations