Real-Time Detection and Tracking Using Hybrid DNNs and Space-Aware Color Feature: From Algorithm to System

  • Liang FengEmail author
  • Hiroaki Igarashi
  • Seiya Shibata
  • Yuki Kobayashi
  • Takashi Takenaka
  • Wei Zhang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12046)


Object detection and tracking are vital for video analysis. As the development of Deep Neural Network (DNN), multiple object tracking is recently performed on the detection results from DNN. However, DNN-based detection is computation-intensive. In order to accelerate multiple object detection and tracking for real-time application, we present a framework to import the tracking knowledge into detection to allow a less accurate but faster DNN for detection and recover the accuracy loss. By combining different DNNs with accuracy-speed trade-offs using space-aware color information, our framework achieves significant speedup (6.8\(\times \)) and maintains high accuracy. Targeting NVIDIA Xavier, we further optimize the implementation from system and platform level.


DNN Object detection Tracking GPU 


  1. 1.
    Abdelali, H.A., et al.: Fast and robust object tracking via accept-reject color histogram-based method. J. Vis. Commun. Image Represent. 34, 219–229 (2016)CrossRefGoogle Scholar
  2. 2.
    Bernardin, K., Stiefelhagen, R.: Evaluating multiple object tracking performance: the CLEAR MOT metrics. J. Image Video Process. 2008, 1 (2008)CrossRefGoogle Scholar
  3. 3.
    Bewley, A., et al.: Alextrac: affinity learning by exploring temporal reinforcement within association chains. In: ICRA, pp. 2212–2218. IEEE (2016)Google Scholar
  4. 4.
    Bewley, A., et al.: Simple online and realtime tracking. In: ICIP, pp. 3464–3468. IEEE (2016)Google Scholar
  5. 5.
    Bochinski, E., et al.: High-speed tracking-by-detection without using image information. In: AVSS, pp. 1–6. IEEE (2017)Google Scholar
  6. 6.
    Bochinski, E., et al.: Extending IOU based multi-object tracking by visual information. In: AVSS, pp. 1–6. IEEE (2018)Google Scholar
  7. 7.
    Danelljan, M., et al.: Adaptive color attributes for real-time visual tracking. In: CVPR, pp. 1090–1097 (2014)Google Scholar
  8. 8.
    Dollár, P., et al.: Fast feature pyramids for object detection. TPAMI 36(8), 1532–1545 (2014)CrossRefGoogle Scholar
  9. 9.
    Hamid Rezatofighi, S., et al.: Joint probabilistic data association revisited. In: ICCV, pp. 3047–3055 (2015)Google Scholar
  10. 10.
    Han, S., et al.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems, pp. 1135–1143 (2015)Google Scholar
  11. 11.
    He, K., et al.: Mask R-CNN. In: ICCV, pp. 2961–2969 (2017)Google Scholar
  12. 12.
    Hubara, I., et al.: Quantized neural networks: training neural networks with low precision weights and activations. J. Mach. Learn. Res. 18(1), 6869–6898 (2017)MathSciNetGoogle Scholar
  13. 13.
    Kim, C., et al.: Multiple hypothesis tracking revisited. In: ICCV, pp. 4696–4704 (2015)Google Scholar
  14. 14.
    Leal-Taixé, L., et al.: Motchallenge 2015: towards a benchmark for multi-target tracking. arXiv:1504.01942 (2015)
  15. 15.
    Liu, W., et al.: SSD: single shot MultiBox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). Scholar
  16. 16.
    Possegger, H., et al.: In defense of color-based model-free tracking. In: CVPR, pp. 2113–2120 (2015)Google Scholar
  17. 17.
    Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement. arXiv:1804.02767 (2018)
  18. 18.
    Ren, S., et al.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)Google Scholar
  19. 19.
    Van De Weijer, J., et al.: Learning color names for real-world applications. TIP 18(7), 1512–1523 (2009)MathSciNetzbMATHGoogle Scholar
  20. 20.
    Wojke, N., et al.: Simple online and realtime tracking with a deep association metric. In: ICIP, pp. 3645–3649. IEEE (2017)Google Scholar
  21. 21.
    Womg, A., et al.: Tiny SSD: a tiny single-shot detection deep convolutional neural network for real-time embedded object detection. In: CRV, pp. 95–101. IEEE (2018)Google Scholar
  22. 22.
    Xiang, Y., et al.: Learning to track: Online multi-object tracking by decision making. In: ICCV, pp. 4705–4713 (2015)Google Scholar
  23. 23.
    Yang, B., Nevatia, R.: An online learned CRF model for multi-target tracking. In: CVPR, pp. 2034–2041. IEEE (2012)Google Scholar
  24. 24.
    Yu, F., Li, W., Li, Q., Liu, Y., Shi, X., Yan, J.: POI: multiple object tracking with high performance detection and appearance feature. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9914, pp. 36–42. Springer, Cham (2016). Scholar
  25. 25.
    Zhang, L., et al.: Global data association for multi-object tracking using network flows. In: CVPR, pp. 1–8. IEEE (2008)Google Scholar
  26. 26.
    Zhu, G., et al.: MC-HOG correlation tracking with saliency proposal. In: 30th AAAI (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Liang Feng
    • 1
    Email author
  • Hiroaki Igarashi
    • 2
  • Seiya Shibata
    • 2
  • Yuki Kobayashi
    • 2
  • Takashi Takenaka
    • 2
  • Wei Zhang
    • 1
  1. 1.Hong Kong University of Science and TechnologyKowloonHong Kong
  2. 2.NEC CorporationKawasakiJapan

Personalised recommendations