Cluster Computing

, Volume 22, Supplement 3, pp 7359–7368 | Cite as

No projection in the residual network

  • Huanglu WenEmail author
  • Liejun Wang


Convolution networks continue to create state-of-the-art results in computer vision, and the Residual Network is an important milestone. In the original residual network, 1 \(\times \) 1 convolution with stride 2 is used as the projection to do the linear transformation between feature maps of different sizes and different number of channels. This projection structure does not satisfy the concept of residual learning and is not able to use all of the input information. We propose a method which will make the Residual Network completely free of this structure and realize what shortcut connections should be. Compared with the original Residual Network, our models achieve higher learning efficiency and higher average performance with fewer parameters and lower computational complexity on CIFAR-10/100.


Convolution networks Residual network Shortcut connection CReLU ELU 



This work is supported by Chinese National Natural Science Foundation (Program No. 61471311 & No. 61771416).


  1. 1.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)CrossRefGoogle Scholar
  2. 2.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)Google Scholar
  3. 3.
    Szegedy, C., Liu, W., Jia, Y.Q., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2015)Google Scholar
  4. 4.
    He, K.M., Zhang, X.Y., Ren, S.Q., Sun, J.: Deep residual learning for image recognition. In: Proceedings of 29th IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2016)Google Scholar
  5. 5.
    Zagoruyko, S., Komodakis, N.: Wide residual networks. In: Proceedings of British Machine Vision Conference (2016)Google Scholar
  6. 6.
    He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: Proceedings of European Conference on Computer Vision (ECCV) (2016)Google Scholar
  7. 7.
    Veit, A., Wilber, M., Belongie, S.: Residual networks behave like ensembles of relatively shallow networks. In: Proceedings of 30th Annual Conference on Neural Information Processing Systems (NIPS) (2016)Google Scholar
  8. 8.
    Shang, W., Sohn, K., Almeida, D., Lee, H.: Understanding and improving convolutional neural networks via concatenated rectified linear units. In: Proceedings of 33rd International Conference on Machine Learning (ICML) (2016)Google Scholar
  9. 9.
    Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: Proceedings of 14th International Conference on Artificial Intelligence and Statistics (AISTATS) (2011)Google Scholar
  10. 10.
    Clevert, D.A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (ELUs). In: ICLR (2016)Google Scholar
  11. 11.
    Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proceedings of 32nd International Conference on Machine Learning (ICML) (2015)Google Scholar
  12. 12.
    Nesterov, Y.: Gradient methods for minimizing composite functions. Math. Program. 140(1), 125–161 (2013)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. Comput. Sci. 3(4), 212–223 (2012)Google Scholar
  14. 14.
    Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Tech Report (2009)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2017

Authors and Affiliations

  1. 1.School of Information Science and EngineeringXinjiang UniversityÜrümchiChina
  2. 2.School of Software EngineeringXinjiang UniversityÜrümchiChina

Personalised recommendations