Abstract
In recent years, excellent convolutional neural networks (CNN) have been used to solve a variety of visual tasks, and the image classification is an important role for the most visual task. To improve the performance of classification networks, several recent methods have shown the benefit of extracting deeper features by increasing depth of networks and fusing features by linear connection. But deep features only describe the high-level semantic features and loses the shallow features such as edge contour. And features fusion don’t describe the impact factor of features for results. In this paper, we study shallow and deep features fusion and propose a new architectural unit, which we call the “Self-adaptive Weight Fusion” (SFW) method. SFW adaptively recalibrates the features relation by determining impact factors of shallow and deep features, which is used for classify objects. Experimental results demonstrate that SFW can be transplanted to different networks. In particular, we find that this mechanism can produce significant performance improvements for existing state-of-the-art deep architectures with minimal additional computational cost. In addition, results show that shallow and deep features fusion is beneficial to learn more general and robust features for a network.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Ma, L., Lu, J., Feng, J., Zhou, J.: Multiple feature fusion via weighted entropy for visual tracking. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3128–3136 (2015)
Al-Wassai, F.A., Kalyankar, N.V., Al-Zaky, A.A.: Multisensor images fusion based on feature-level. arXiv preprint arXiv:1108.4098 (2011)
Zhou, W., Newsam, S., Li, C., Shao, Z.: PatternNet: a benchmark dataset for performance evaluation of remote sensing image retrieval. arXiv preprint arXiv:1706.03424 (2017)
Kang, L., Hu, B., Wu, X., Chen, Q., He, Y.: A short texts matching method using shallow features and deep features. In: Zong, C., Nie, J.Y., Zhao, D., Feng, Y. (eds.) Natural Language Processing and Chinese Computing. CCIS, vol. 496, pp. 150–159. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-45924-9_14
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. arXiv preprint arXiv:1709.01507 (2017)
Huang, G., Liu, Z., Weinberger, K.Q., van der Maaten, L.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 1(2), p. 3, July 2017
Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5987–5995. IEEE, July 2017
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Rabinovich, A.: Going deeper with convolutions. In: CVPR, June 2015
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Bodla, N., Zheng, J., Xu, H., Chen, J.C., Castillo, C., Chellappa, R.: Deep heterogeneous feature fusion for template-based face recognition. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 586–595. IEEE, March 2017
Russakovsky, O., et al.: ImageNet Large Scale Visual Recognition Challenge. arXiv:1409.0575 (2014)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998)
Shen, L., Sun, G., Huang, Q., Wang, S., Lin, Z., Wu, E.: Multi-level discriminative dictionary learning with application to large scale image classification. IEEE Trans. Image Process. 24(10), 3109–3123 (2015)
Levine, S., Finn, C., Darrell, T., Abbeel, P.: End-to-end training of deep visuomotor policies. J. Mach. Learn. Res. 17(1), 1334–1373 (2016)
Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
Shen, L., Lin, Z., Huang, Q.: Relay backpropagation for effective learning of deep convolutional neural networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 467–482. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_29
Feichtenhofer, C., Pinz, A., Zisserman, A.: Convolutional two-stream network fusion for video action recognition. In: CVPR 2016, pp. 1933–1941. IEEE Computer Society, Las Vegas (2016)
Wei-bin, L., Zzhi-yuan, Z., Wei-wei, X.: Feature fusion methods in pattern classification. J. Beijing Univ. Posts Telecommun. 40(4), 1–8 (2017)
Feichtenhofer, C., Pinz, A., Zisserman, A.P.: Convolutional two-stream network fusion for video action recognition (2016)
Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Advances in Neural Information Processing Systems, pp. 568–576 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Tian, Q., Wan, S., Jin, P., Xu, J., Zou, C., Li, X. (2018). A Novel Feature Fusion with Self-adaptive Weight Method Based on Deep Learning for Image Classification. In: Hong, R., Cheng, WH., Yamasaki, T., Wang, M., Ngo, CW. (eds) Advances in Multimedia Information Processing – PCM 2018. PCM 2018. Lecture Notes in Computer Science(), vol 11164. Springer, Cham. https://doi.org/10.1007/978-3-030-00776-8_39
Download citation
DOI: https://doi.org/10.1007/978-3-030-00776-8_39
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00775-1
Online ISBN: 978-3-030-00776-8
eBook Packages: Computer ScienceComputer Science (R0)