Advertisement

International Journal of Computer Vision

, Volume 122, Issue 2, pp 313–333 | Cite as

Multi-Camera Multi-Target Tracking with Space-Time-View Hyper-graph

  • Longyin Wen
  • Zhen Lei
  • Ming-Ching Chang
  • Honggang Qi
  • Siwei Lyu
Article

Abstract

Incorporating multiple cameras is an effective solution to improve the performance and robustness of multi-target tracking to occlusion and appearance ambiguities. In this paper, we propose a new multi-camera multi-target tracking method based on a space-time-view hyper-graph that encodes higher-order constraints (i.e., beyond pairwise relations) on 3D geometry, appearance, motion continuity, and trajectory smoothness among 2D tracklets within and across different camera views. We solve tracking in each single view and reconstruction of tracked trajectories in 3D environment simultaneously by formulating the problem as an efficient search of dense sub-hypergraphs on the space-time-view hyper-graph using a sampling based approach. Experimental results on the PETS 2009 dataset and MOTChallenge 2015 3D benchmark demonstrate that our method performs favorably against the state-of-the-art methods in both single-camera and multi-camera multi-target tracking, while achieving close to real-time running efficiency. We also provide experimental analysis of the influence of various aspects of our method to the final tracking performance.

Keywords

Multi-camera multi-target tracking Single-camera multi-target tracking Space-time-view hyper-graph Dense sub-hypergraph search 

Notes

Acknowledgments

We would like to thank Dawei Du for a number of suggestions that considerably improved the quality of this paper. Longyin Wen and Siwei Lyu were supported by US National Science Foundation Research Grant (CCF-1319800). Zhen Lei was supported by the National Key Research and Development Plan (Grant No. 2016 YFC0801002), the Chinese National Natural Science Foundation Projects #61375037, #61473291. Honggang Qi was supported by National Nature Science Foundation of China #61472388.

Supplementary material

Supplementary material 1 (avi 11348 KB)

References

  1. Andriyenko, A., & Schindler, K. (2011). Multi-target tracking by continuous energy minimization. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1265–1272).Google Scholar
  2. Andriyenko, A., Schindler, K., & Roth, S. (2012). Discrete-continuous optimization for multi-target tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1926–1933).Google Scholar
  3. Attanasi, A., Cavagna, A., Castello, L. D., Giardina, I., Jelic, A., Melillo, S., et al. (2015). GReTA—a novel global and recursive tracking algorithm in three dimensions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(1), 1.CrossRefGoogle Scholar
  4. Berclaz, J., Fleuret, F., & Fua, P. (2009). Multiple object tracking using flow linear programming. In Winter-PETS (pp. 1–8). Snowbird: IEEE.Google Scholar
  5. Berclaz, J., Fleuret, F., Türetken, E., & Fua, P. (2011). Multiple object tracking using k-shortest paths optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(9), 1806–1819.CrossRefGoogle Scholar
  6. Breitenstein, M. D., Reichlin, F., Leibe, B., Koller-Meier, E., & Gool, L. J. V. (2011). Online multi-person tracking-by-detection from a single, uncalibrated camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(9), 1820–1833.CrossRefGoogle Scholar
  7. Brendel, W., Amer, M. R., & Todorovic, S. (2011). Multiobject tracking as maximum weight independent set. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1273–1280).Google Scholar
  8. Chari, V., Lacoste-Julien, S., Laptev, I., & Sivic, J. (2015). On pairwise costs for network flow multi-object tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 5537–5545).Google Scholar
  9. Dehghan, A., Tian, Y., Torr, P. H. S., & Shah, M. (2015). Target identity-aware network flow for online multiple target tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1146–1154).Google Scholar
  10. Dollár, P., Wojek, C., Schiele, B., & Perona, P. (2012). Pedestrian detection: An evaluation of the state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(4), 743–761.CrossRefGoogle Scholar
  11. Felzenszwalb, P. F., McAllester, D. A., & Ramanan, D. (2008). A discriminatively trained, multiscale, deformable part model. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–8).Google Scholar
  12. Ferryman, J. M., & Shahrokni, A. (2009). PETS2009: Dataset and challenge. In Winter-PETS (pp. 1–6).Google Scholar
  13. Fleuret, F., Berclaz, J., Lengagne, R., & Fua, P. (2008). Multicamera people tracking with a probabilistic occupancy map. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2), 267–282.CrossRefGoogle Scholar
  14. Hofmann, M., Wolf, D., & Rigoll, G. (2013). Hypergraphs for joint multi-view reconstruction and multi-object tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 3650–3657).Google Scholar
  15. Hong, L., & Cui, N. (2000). An interacting multipattern joint probabilistic data association (imp-jpda) algorithm for multitarget tracking. Signal Processing, 80(8), 1561–1575.CrossRefGoogle Scholar
  16. Huang, C., Li, Y., & Nevatia, R. (2013). Multiple target tracking by learning-based hierarchical association of detection responses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(4), 898–910.CrossRefGoogle Scholar
  17. Isard, M., & Blake, A. (1998). Condensation—conditional density propagation for visual tracking. International Journal of Computer Vision, 29(1), 5–28.CrossRefGoogle Scholar
  18. Izadinia, H., Saleemi, I., Li, W., & Shah, M. (2012) (MP)\(^2\)T: Multiple people multiple parts tracker. In Proceedings of European Conference on Computer Vision (pp. 100–114).Google Scholar
  19. Jiang, H., Fels, S., & Little, J. J. (2007). A linear programming approach for multiple object tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–8).Google Scholar
  20. Khan, Z., Balch, T. R., & Dellaert, F. (2005). MCMC-based particle filtering for tracking a variable number of interacting targets. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(11), 1805–1918.CrossRefGoogle Scholar
  21. Kim, J., Dai, Y., Li, H., Du, X., & Kim, J. (2013). Multi-view 3D reconstruction from uncalibrated radially-symmetric cameras. In Proceedings of IEEE International Conference on Computer Vision (pp. 1896–1903).Google Scholar
  22. Klinger, T., Rottensteiner, F., & Heipke, C. (2015). Probabilistic multi-person tracking using dynamic bayes networks. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, II–3/W5, 435–442.CrossRefGoogle Scholar
  23. Kostrikov, I., Horbert, E., & Leibe, B. (2014). Probabilistic labeling cost for high-accuracy multi-view reconstruction. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1534–1541).Google Scholar
  24. Kuhn, W., & Tucker, A. (1951) Nonlinear programming. In Proceedings of 2nd Berkeley Symposium (pp. 481–492).Google Scholar
  25. Kuo, C. H., & Nevatia, R. (2011). How does person identity recognition help multi-person tracking? In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1217–1224).Google Scholar
  26. Leal-Taixé, L., Milan, A., Reid, I.D., Roth, S., & Schindler, K. (2015). Motchallenge 2015: towards a benchmark for multi-target tracking. CoRR abs/1504.01942.Google Scholar
  27. Leal-Taixé, L., Pons-Moll, G., & Rosenhahn, B. (2011). Everybody needs somebody: modeling social and grouping behavior on a linear programming multiple people tracker. In Workshops in Conjunction with IEEE International Conference on Computer Vision (pp. 120–127).Google Scholar
  28. Leal-Taixé, L., Pons-Moll, G., & Rosenhahn, B. (2012) Branch-and-price global optimization for multi-view multi-object tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1987–1994).Google Scholar
  29. Leven, W. F., & Lanterman, A. D. (2009). Unscented kalman filters for multiple target tracking with symmetric measurement equations. IEEE Transaction on Automatic Control, 54(2), 370–375.MathSciNetCrossRefGoogle Scholar
  30. Liu, H., & Yan, S. (2012). Efficient structure detection via random consensus graph. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 574–581).Google Scholar
  31. Liu, H., Yang, X., Latecki, L. J., & Yan, S. (2012). Dense neighborhoods on affinity graph. IJCV, 98(1), 65–82.MathSciNetCrossRefMATHGoogle Scholar
  32. Liu, Y., Li, H., & Chen, Y. Q. (2012). Automatic tracking of a large number of moving targets in 3d. In Proceedings of European Conference on Computer Vision (pp. 730–742).Google Scholar
  33. Marchesotti, L., Marcenaro, L., Ferrari, G., & Regazzoni, C. S. (2002) Multiple object tracking under heavy occlusions by using kalman filters based on shape matching. In Proceedings of IEEE International Conference on Image Processing (pp. 341–344).Google Scholar
  34. Milan, A. (2011) Continuous energy minimization tracker. http://www.milanton.de/contracking/index.html.
  35. Milan, A., Leal-Taixé, L., Schindler, K., Roth, S., & Reid, I.D. (2015). Multiple object tracking benchmark: 3d mot. https://motchallenge.net/results/3D_MOT_2015/.
  36. Milan, A., Roth, S., & Schindler, K. (2014). Continuous energy minimization for multitarget tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(1), 58–72.CrossRefGoogle Scholar
  37. Ojala, T., Pietikäinen, M., & Mäenpää, T. (2000). Gray scale and rotation invariant texture classification with local binary patterns. In Proceedings of European Conference on Computer Vision (pp. 404–420).Google Scholar
  38. Pellegrini, S., Ess, A., Schindler, K., & Gool, L. J. V. (2009). You’ll never walk alone: modeling social behavior for multi-target tracking. In Proceedings of IEEE International Conference on Computer Vision (pp. 261–268).Google Scholar
  39. Pirsiavash, H., Ramanan, D., & Fowlkes, C. C. (2011). Globally-optimal greedy algorithms for tracking a variable number of objects. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1201–1208).Google Scholar
  40. Reid, D. B. (1979). An algorithm for tracking multiple targets. IEEE Transactions on Automatic Control, 24, 843–854.CrossRefGoogle Scholar
  41. Shi, X., Ling, H., Hu, W., Yuan, C., & Xing, J. (2014). Multi-target tracking with motion context in tensor power iteration. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 3518–3525).Google Scholar
  42. Shu, G., Dehghan, A., Oreifej, O., Hand, E., & Shah, M. (2012). Part-based multiple-person tracking with partial occlusion handling. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1815–1821).Google Scholar
  43. Smith, K., Gatica-Perez, D., & Odobez, J. M. (2005). Using particles to track varying numbers of interacting people. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 962–969).Google Scholar
  44. Stiefelhagen, R., Bernardin, K., Bowers, R., Garofolo, J. S., Mostefa, D., & Soundararajan, P. (2006). The CLEAR 2006 evaluation. CLEAR (pp. 1–44). Berlin: Springer.Google Scholar
  45. Wen, L., Li, W., Yan, J., Lei, Z., Yi, D., & Li, S. Z. (2014) Multiple target tracking based on undirected hierarchical relation hypergraph. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (pp. 3457–3464).Google Scholar
  46. Wu, Z., Hristov, N.I., Kunz, T. H., & Betke, M. (2009). Tracking-reconstruction or reconstruction-tracking? Comparison of two multiple hypothesis tracking approaches to interpret 3D object motion from several camera views. In Proceedings of the IEEE Workshop on Motion and Video Computing (pp. 1–8).Google Scholar
  47. Wu, Z., Kunz, T. H., & Betke, M. (2011). Efficient track linking methods for track graphs using network-flow and set-cover techniques. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1185–1192).Google Scholar
  48. Yang, B., & Nevatia, R. (2012). Multi-target tracking by online learning of non-linear motion patterns and robust appearance models. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1918–1925).Google Scholar
  49. Yang, B., & Nevatia, R. (2012). An online learned CRF model for multi-target tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 2034–2041).Google Scholar
  50. Yang, M., Liu, Y., Wen, L., You, Z., & Li, S. Z. (2014). A probabilistic framework for multitarget tracking with mutual occlusions. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–8).Google Scholar
  51. Yu, Q., & Medioni, G. G. (2009). Multiple-target tracking by spatiotemporal monte carlo markov chain data association. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(12), 2196–2210.CrossRefGoogle Scholar
  52. Zhang, L., Li, Y., & Nevatia, R. (2008). Global data association for multi-object tracking using network flows. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–8).Google Scholar
  53. Zhou, D., Huang, J., & Schölkopf, B. (2006). Learning with hypergraphs: Clustering, classification, and embedding. Advances in Neural Information Processing Systems (pp. 1601–1608). Cambridge: MIT Press.Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Longyin Wen
    • 1
  • Zhen Lei
    • 2
  • Ming-Ching Chang
    • 3
  • Honggang Qi
    • 4
  • Siwei Lyu
    • 1
  1. 1.Computer Science DepartmentUniversity at Albany, State University of New YorkAlbanyUSA
  2. 2.National Laboratory of Pattern Recognition, Institute of AutomationChinese Academy of SciencesBeijingChina
  3. 3.Computer Engineering Department, University at AlbanyState University of New YorkAlbanyUSA
  4. 4.School of Computer and Control EngineeringUniversity of the Chinese Academy of SciencesBeijingChina

Personalised recommendations