Layer Removal for Transfer Learning with Deep Convolutional Neural Networks

  • Weiming ZhiEmail author
  • Zhenghao Chen
  • Henry Wing Fung Yueng
  • Zhicheng Lu
  • Seid Miad Zandavi
  • Yuk Ying Chung
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10635)


It is usually difficult to find datasets of sufficient size to train Deep Convolutional Neural Networks (DCNNs) from scratch. In practice, a neural network is often pre-trained on a very large source dataset. Then, a target dataset is transferred onto the neural network. This approach is a form of transfer learning, and allows very deep networks to achieve outstanding performance even when a small target dataset is available. It is thought that the bottom layers of the pre-trained network contain general information, which are applicable to different datasets and tasks, while the upper layers of the pre-trained network contain abstract information relevant to a specific dataset and task. While studies have been conducted on the fine-tuning of these layers, the removal of these layers have not yet been considered. This paper explores the effect of removing the upper convolutional layers of a pre-trained network. We empirically investigated whether removing upper layers of a deep pre-trained network can improve performance for transfer learning. We found that removing upper pre-trained layers gives a significant boost in performance, but the ideal number of layers to remove depends on the dataset. We suggest removing pre-trained convolutional layers when applying transfer learning on off-the-shelf pre-trained DCNNs. The ideal number of layers to remove will depend on the dataset, and remain as a parameter to be tuned.


Convolutional neural networks Transfer learning Deep learning 


  1. 1.
    Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report (2009)Google Scholar
  2. 2.
    Bengio, Y.: Deep learning of representations for unsupervised and transfer learning. In: JMLR W&CP: Proceedings of Unsupervised and Transfer Learning (2011)Google Scholar
  3. 3.
    Bengio, Y., Bastien, F., Bergeron, A., Boulanger-Lew, N., Breuel, T., Chherawala, Y., Cisse, M., Côté, M., Erhan, D., Eustache, J., Glorot, X., Muller, X., Lebeuf, S.P., Pascanu, R., Rifai, S., Savard, F., Sicard, G.: Deep learners benefit more from out-of-distribution examples. In: JMLR W&CP: Proceedings of AISTATS 2011 (2011)Google Scholar
  4. 4.
    Caruana, R.: Learning many related tasks at the same time with backpropagation. In: Advances in Neural Information Processing Systems, pp. 657–664. Morgan Kaufmann (1995)Google Scholar
  5. 5.
    Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Computer Vision and Pattern Recognition. IEEE (2009)Google Scholar
  6. 6.
    Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., Darrell, T.: DeCAF: A deep convolutional activation feature for generic visual recognition. CoRR abs/1310.1531 (2013)Google Scholar
  7. 7.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Computer Vision and Pattern Recognition. IEEE (2014)Google Scholar
  8. 8.
    Graham, B.: Fractional max-pooling. CoRR abs/1412.6071 (2015)Google Scholar
  9. 9.
    Hu, F., Xia, G.S., Hu, J., Zhang, L.: Transferring deep convolutional neural networks for the scene classification of high-resolution remote sensing imagery. Remote Sens. 7(11), 14680–14707 (2015)CrossRefGoogle Scholar
  10. 10.
    Mehdipour Ghazi, M., Yanikoglu, B., Aptoula, E.: Plant identification using deep neural networks via optimization of transfer learning parameters. Neurocomputing 235, 228–235 (2017)CrossRefGoogle Scholar
  11. 11.
    Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014. IEEE (2014)Google Scholar
  12. 12.
    Razavian, A.S., Azizpour, H., Sullivan, J., Carlsson, S.: CNN features off-the-shelf: an astounding baseline for recognition. In: Computer Vision and Pattern Recognition Workshops, CVPRW 2014. IEEE (2014)Google Scholar
  13. 13.
    Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., Lecun, Y.: OverFeat: integrated recognition, localization and detection using convolutional networks. In: International Conference on Learning Representations (2014)Google Scholar
  14. 14.
    Shin, H., et al.: Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging 35, 1285–1298 (2016)CrossRefGoogle Scholar
  15. 15.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)Google Scholar
  16. 16.
    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)zbMATHMathSciNetGoogle Scholar
  17. 17.
    Szegedy, C., et al.: Going deeper with convolutions. In: Computer Vision and Pattern Recognition. IEEE (2015)Google Scholar
  18. 18.
    Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks?. In: Advances in Neural Information Processing Systems (2014)Google Scholar
  19. 19.
    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). doi: 10.1007/978-3-319-10590-1_53 Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Weiming Zhi
    • 1
    Email author
  • Zhenghao Chen
    • 2
  • Henry Wing Fung Yueng
    • 2
  • Zhicheng Lu
    • 2
  • Seid Miad Zandavi
    • 2
  • Yuk Ying Chung
    • 2
  1. 1.Department of Engineering ScienceUniversity of AucklandAucklandNew Zealand
  2. 2.School of Information TechnologiesUniversity of SydneySydneyAustralia

Personalised recommendations