Recurrent RetinaNet: A Video Object Detection Model Based on Focal Loss

  • Xiaobo Li
  • Haohua Zhao
  • Liqing ZhangEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11304)


Object detection in still images has been extensively investigated recent years. But object detection in videos is still a challenging research topic. Directly using those methods for still images in videos would suffer from blurring and low resolution in video images. Some methods utilized the temporal information to boost the detection accuracy, but they are usually expensive in time in estimating optical flow. In this paper, we propose Recurrent RetinaNet, a flexible end-to-end approach for object detection in videos. In this work, a backbone network is leveraged to generate several feature maps, then a feature pyramid network extracts pyramid features from the feature maps. Detection boxes are generated according to the shapes of pyramid features. Two subnets with convolutional layers and Convolutional LSTM layers, are added on the top for box regression and classification. Note that the boxes are generated regardless of the content of images, there may be an extreme foreground-background imbalance. Thus, focal loss, which has been shown effective in object detection on images, is employed as the loss function for the classification subnet. Experiments show that the approach improves the detection accuracy and avoids the detection loss of some objects in some cases compared to RetinaNet and the model complexity is still good enough for real-time applications.


Object detection Focal loss Convolutional LSTM 



The work was supported by the National Basic Research Program of China (Grant No. 2015CB856004) and the Key Basic Research Program of Shanghai Municipality, China (15JC1400103,16JC1402800).


  1. 1.
    Abadi, M., et al.: TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems (2015). Software available from
  2. 2.
    Chollet, F., et al.: Keras (2015).
  3. 3.
    Deng, J., et al.: ImageNet: a large-scale hierarchical image database. In: CVPR 2009, IEEE (2009)Google Scholar
  4. 4.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR 2014 (2014)Google Scholar
  5. 5.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. Cornell University Library (2017)Google Scholar
  6. 6.
    Hetang, C., Qin, H., Liu, S., Yan, J.: Impression network for video object detection. arXiv preprint arXiv:1712.05896 (2017)
  7. 7.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  8. 8.
    Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
  9. 9.
    Kang, K., et al.: Object detection in videos with tubelet proposal networks. In: Proceedings of CVPR, vol. 2, p. 7 (2017)Google Scholar
  10. 10.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  11. 11.
    Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: CVPR, vol. 1, p. 4 (2017)Google Scholar
  12. 12.
    Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. arXiv preprint arXiv:1708.02002 (2017)
  13. 13.
    Xingjian, S., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.K., Woo, W.C.: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: NIPS 2015 (2015)Google Scholar
  14. 14.
    Xue, W., Zhao, H., Zhang, L.: Encoding multi-resolution two-stream CNNs for action recognition. In: Hirose, A., Ozawa, S., Doya, K., Ikeda, K., Lee, M., Liu, D. (eds.) ICONIP 2016. LNCS, vol. 9949, pp. 564–571. Springer, Cham (2016). Scholar
  15. 15.
    Zhang, K., Zhang, L.: Extracting hierarchical spatial and temporal features for human action recognition. Multimedia Tools Appl. 77(13), 16053–16068 (2018). SpringerCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Department of Computer Science and EngineeringShanghai Jiao Tong UniversityShanghaiChina

Personalised recommendations