An Incremental Scheme with Weight Pruning to Train Deep Neural Network

  • Haonan GuoEmail author
  • Zhicong Yan
  • Jichao Yang
  • Shenghong Li
Conference paper
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 517)


Deep neural networks have present state-of-the-art results in many different machine learning tasks. In the traditional machine learning task, we train models on the formerly prepared dataset. However, in the real-world scenarios, training data are always collected in an incremental manner, in which new samples and new classes will be added to the training data gradually. Since the traditional training method with stochastic gradient descent will suffer from catastrophic forgetting problem when training on the new data set, in this paper, we proposed a new scheme to train deep neural networks incrementally. We first train the deep model on the original dataset with a weight-pruning manner, then on the newly added training data, we train the former pruned weights while remaining the former trained core-part weights unchanged. Experiments on MNIST demonstrated that our method is efficient and can even get better performance than training from scratch on the whole dataset in the traditional manner.


Deep neural network Catastrophic forgetting Weight pruning Incremental scheme 



This research work is funded by the National Key Research and Development Project of China (2016YFB0801003)


  1. 1.
    He, K., Zhang, X., Ren, S., et al.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification 1026–1034 (2015)Google Scholar
  2. 2.
    Sermanet, P., Eigen, D., Zhang, X., et al.: OverFeat: integrated recognition, localization and detection using convolutional networks. Eprint Arxiv (2013)Google Scholar
  3. 3.
    Hinton, G., Deng, L., Yu, D., et al.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82–97 (2012)CrossRefGoogle Scholar
  4. 4.
    Mikolov, T., Deoras, A., Povey, D., et al.: Strategies for training large scale neural network language models. In: Automatic Speech Recognition and Understanding. IEEE, pp. 196–201 (2011)Google Scholar
  5. 5.
    Hubel, D.H., Wiesel, T.N.: Receptive fields and functional architecture of monkey striate cortex. J. Physiol. 195(1), 215–243 (1968)CrossRefGoogle Scholar
  6. 6.
    Lin, M., Chen, Q., Yan, S.: Network in network (2013). arXiv preprint arXiv:1312.4400
  7. 7.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. Comput. Sci. (2014)Google Scholar
  8. 8.
    Szegedy, C., Liu, W., Jia, Y., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)Google Scholar
  9. 9.
    Moe-Helgesen, O.M., Stranden, H.: Catastophic forgetting in neural networks. Dept. Comput. Inf. Sci. Nor. Univ. Sci. Technol. (NTNU) Trondheim Nor. Tech. Rep. 1, 22 (2005)Google Scholar
  10. 10.
    Xiao, T., Zhang, J., Yang, K., et al.: Error-driven incremental learning in deep convolutional neural network for large-scale image classification. In: Proceedings of the 22nd ACM International Conference on Multimedia. ACM, pp. 177–186 (2014)Google Scholar
  11. 11.
    Han, S., Pool, J., Tran, J., et al.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems, pp. 1135–1143 (2015)Google Scholar
  12. 12.
    Srinivas, S., Babu, R.V.: Data-free parameter pruning for deep neural networks (2015). arXiv preprint arXiv:1507.06149
  13. 13.
    Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding (2015). arXiv preprint arXiv:1510.00149

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  • Haonan Guo
    • 1
    Email author
  • Zhicong Yan
    • 1
  • Jichao Yang
    • 2
  • Shenghong Li
    • 1
  1. 1.Shanghai Jiao Tong UniversityShanghaiChina
  2. 2.Shanghai Starriver Bilingual SchoolShanghaiChina

Personalised recommendations