Advertisement

Reconstruction Error Aware Pruning for Accelerating Neural Networks

  • Koji KammaEmail author
  • Toshikazu Wada
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11844)

Abstract

This paper presents a pruning method, Reconstruction Error Aware Pruning (REAP), to reduce the redundancy of convolutional neural network models for accelerating the inference. REAP is an extension of one of the state-of-the-art channel pruning methods. Our method takes 3 steps, (1) evaluating the importance of each channel based on the reconstruction error of the outputs in each convolutional layer, (2) pruning less important channels, (3) updating the remaining weights by the least squares method so as to reconstruct the outputs. By pruning with REAP, one can produce a fast and accurate model out of a large pretrained model. Besides, REAP saves us lots of time and efforts required for retraining the pruned model. As our method requires a large computational cost, we have developed an algorithm based on biorthogonal system to conduct the computation efficiently. In the experiments, we show that REAP can conduct pruning with smaller sacrifice of the model performances than several existing state-of-the-art methods such as CP [9], ThiNet [17], DCP [25], and so on.

Keywords

Neural network Pruning Biorthogonal system 

Notes

Acknowledgment

This work was supported by JSPS KAKENHI Grant Number 19K12020 and the Environment Research and Technology Development Fund (3-1905) of the Environmental Restoration and Conservation Agency of Japan.

References

  1. 1.
    Aghasi, A., Abdi, A., Nguyen, N., Romberg, J.: Net-Trim: convex pruning of deep neural networks with performance guarantee. In: Advances in Neural Information Processing Systems, vol. 30, pp. 3177–3186. Curran Associates Inc. (2017)Google Scholar
  2. 2.
    akamaster. Proper implementation of resnet-s for cifar10/100 in pytorch that matches description of the original paper (2019)Google Scholar
  3. 3.
    Courbariaux, M., Bengio, Y., David, J.-P.: BinaryConnect: training deep neural networks with binary weights during propagations. In: Advances in Neural Information Processing Systems, vol. 28, pp. 3123–3131. Curran Associates Inc. (2015)Google Scholar
  4. 4.
    Deng, J., Dong, W., Socher, R., Li, J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR (2009)Google Scholar
  5. 5.
    Dong, X., Chen, S., Pan, S.: Learning to prune deep neural networks via layer-wise optimal brain surgeon. In: Advances in Neural Information Processing Systems, vol. 30, pp. 4857–4867. Curran Associates Inc. (2017)Google Scholar
  6. 6.
    Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. In: Proceedings of International Conference on Learning Representations, pp. 1–14 (2016)Google Scholar
  7. 7.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition, pp. 770–778 (2016)Google Scholar
  8. 8.
    He, T., Fan, Y., Qian, Y., Tan, T., Yu, K.: Reshaping deep neural network for fast decoding by node-pruning, pp. 245–249 (2014)Google Scholar
  9. 9.
    He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of International Conference on Computer Vision (2017)Google Scholar
  10. 10.
    He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., Han, S.: AMC: AutoML for model compression and acceleration on mobile devices. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 815–832. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01234-2_48CrossRefGoogle Scholar
  11. 11.
    Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks, pp. 2261–2269 (2017)Google Scholar
  12. 12.
    Khosla, A., Jayadevaprakash, N., Yao, B., Fei-Fei, L.: Novel dataset for fine-grained image categorization. In: First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011Google Scholar
  13. 13.
    Krizhevsky, A., Nair, V., Hinton, G.: Cifar-10 (Canadian institute for advanced research)Google Scholar
  14. 14.
    LeCun, Y., Denker, J.S., Solla, S.A.: Optimal brain damage. In: Advances in Neural Information Processing Systems, vol. 2, pp. 598–605. Morgan-Kaufmann (1990)Google Scholar
  15. 15.
    Liu, B., Wang, M., Foroosh, H., Tappen, M.F., Pensky, M.: Sparse convolutional neural networks, pp. 806–814 (2015)Google Scholar
  16. 16.
    Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: Proceedings of International Conference on Computer Vision (2017)Google Scholar
  17. 17.
    Luo, J.-H., Wu, J., Lin, W.: ThiNet: a filter level pruning method for deep neural network compression. In: Proceedings of International Conference on Computer Vision (2017)Google Scholar
  18. 18.
    Simonyan, K., Zisserman, A.: Very deep convoolutional networks for large-scale image recognition. In: Proceedings of International Conference on Learning Representations, pp. 1–14 (2015)Google Scholar
  19. 19.
    Wang, H., Zhang, Q., Wang, Y., Hu, H.: Structured probabilistic pruning for convolutional neural network acceleration. In: Proceedings of British Machine Vision Conference (2018)Google Scholar
  20. 20.
    Xie, G., Wang, J., Zhang, T., Lai, J., Hong, R., Qi, G.-J.: Interleaved structured sparse convolutional neural networks. In: Proceedings of Computer Vision and Pattern Recognition (2018)Google Scholar
  21. 21.
    Xue, J., Li, J., Gong, Y.: Restructuring of deep neural network acoustic models with singular value decomposition. In: INTERSPEECH (2013)Google Scholar
  22. 22.
    Ye, J., et al.: Learning compact recurrent neural networks with block-term tensor decomposition. In: Proceedings of Computer Vision and Pattern Recognition (2018)Google Scholar
  23. 23.
    Yu, X., Liu, T., Wang, X., Tao, D.: On compressing deep models by low rank and sparse decomposition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017Google Scholar
  24. 24.
    Zhou, A., Yao, A., Wang, K., Chen, Y.: Explicit loss-error-aware quantization for low-bit deep neural networks. In: Proceedings of Computer Vision and Pattern Recognition (2018)Google Scholar
  25. 25.
    Zhuang, Z., et al.: Discrimination-aware channel pruning for deep neural networks. In: Proceedings of Advances in Neural Information Processing Systems (2018)Google Scholar
  26. 26.
    Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., Shelhamer, E.: cuDNN: efficient Primitives for Deep Learning. Technical report (2011)Google Scholar
  27. 27.
    Zhao, Q., et al.: M2Det: a single-shot object detector based on multi-level feature pyramid network. In: Proceedings of AAAI Conference on Artificial Intelligence (AAAI) (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Wakayama UniversityWakayama-shiJapan

Personalised recommendations