Advertisement

Global Features of Fused Frame Relationships Help Video Classification

  • MengYao KongEmail author
  • Pin Lv
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1074)

Abstract

ECO [20] is a newly proposed efficient video classification network that combines 2D and 3D networks. On this basis, we try to extract the global features composed of inter-frame relationships and fuse the global features with the initial frame feature, which can enhance the learning of video features to achieve better classification results. In this way, the new frame feature not only contains the information of itself but also contains the global feature which is related to other frames., so the module can more easily learn about distinguishing features. The main contribution of this paper is proposing a module for extracting global features to enhance video feature learning. It is very lightweight and basically does not add computation to the model. At the same time, it is flexible and easy to generalize to various network structures. Finally, based on the baseline the classification accuracy is increased by 0.8% on the HMDB51 dataset.

Keywords

Global feature Inter-frame relation Action recognition 

References

  1. 1.
    Carreira, J., Zisserman, A.: Quo vadis, action recognition? A new model and the kinetics dataset. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4724–4733. IEEE (2017)Google Scholar
  2. 2.
    Chen, Y., Kalantidis, Y., Li, J., Yan, S., Feng, J.: Multi-fiber networks for video recognition. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 352–367 (2018)Google Scholar
  3. 3.
    Hu, H., Gu, J., Zhang, Z., Dai, J., Wei, Y.: Relation networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3588–3597 (2018)Google Scholar
  4. 4.
    Hara, K., Kataoka, H., Satoh, Y.: Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and ImageNet? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6546–6555 (2018)Google Scholar
  5. 5.
    Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T.: HMDB: a large video database for human motion recognition. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 2556–2563. IEEE (2011)Google Scholar
  6. 6.
    Kim, J., Ricci, M., Serre, T.: Not-So-CLEVR: visual relations strain feedforward neural networks (2018)Google Scholar
  7. 7.
    Lin, J., Gan, C., Han, S.: Temporal shift module for efficient video understanding. arXiv preprint arXiv:1811.08383 (2018)
  8. 8.
    Qiu, Z., Yao, T., Mei, T.: Learning spatio-temporal representation with pseudo-3D residual networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5534–5542. IEEE, October 2017Google Scholar
  9. 9.
    Santoro, A., Raposo, D., Barrett, D.G., Malinowski, M., Pascanu, R., Battaglia, P., Lillicrap, T.: A simple neural network module for relational reasoning. In: Advances in Neural Information Processing Systems, pp. 4967–4976 (2017)Google Scholar
  10. 10.
    Su, B., Zhou, J., Ding, X., Wang, H., Wu, Y.: Hierarchical dynamic parsing and encoding for action recognition. In: European Conference on Computer Vision (ECCV), pp. 202–217 (2016)Google Scholar
  11. 11.
    Su, B., Zhou, J., Ding, X., Wu, Y.: Unsupervised hierarchical dynamic parsing and encoding for action recognition. IEEE Trans. Image Process. 26, 5784–5799 (2017)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Su, B., Hua, G.: Order-preserving Wasserstein distance for sequence matching. In: Computer Vision and Pattern Recognition (CVPR), pp. 1049–1057 (2017)Google Scholar
  13. 13.
    Su, B., Hua, G.: Order-preserving optimal transport for distances between sequences. IEEE Trans. Pattern Anal. Mach. Intell. (2018) Google Scholar
  14. 14.
    Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4489–4497 (2015)Google Scholar
  15. 15.
    Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6450–6459 (2018)Google Scholar
  16. 16.
    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)Google Scholar
  17. 17.
    Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., Van Gool, L.: Temporal segment networks: towards good practices for deep action recognition. In: European Conference on Computer Vision, pp. 20–36. Springer, Cham (2016)Google Scholar
  18. 18.
    Xie, S., Sun, C., Huang, J., Tu, Z., Murphy, K.: Rethinking spatiotemporal feature learning: speed-accuracy trade-offs in video classification. In: European Conference on Computer Vision, pp. 318–335. Springer, Cham (2018)Google Scholar
  19. 19.
    Zhou, B., Andonian, A., Oliva, A., Torralba, A.: Temporal relational reasoning in videos. In: European Conference on Computer Vision, pp. 831–846. Springer, Cham (2018)Google Scholar
  20. 20.
    Zolfaghari, M., Singh, K., Brox, T.: ECO: efficient convolutional network for online video understanding. arXiv preprint arXiv:1804.09066 (2018)

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.University of Chinese Academy of Sciences, UCASBeijingPeople’s Republic of China
  2. 2.Institute of Software, Chinese Academy of Sciences, ISCASBeijingPeople’s Republic of China

Personalised recommendations