Adaptive Resource Management for Sensor Fusion in Visual Tracking

Part of the KAIST Research Series book series (KAISTRS)


Sensor fusion for visual tracking is attractive since the integration of multiple sensors and/or features with different characteristics has potential to improve tracking performance. However, there exist several critical limitations to sensor fusion techniques: (1) the measurement cost increases typically as many times as the number of sensors, (2) it is not straightforward to quantify the confidence of each source and give each sensor a proper weight for state estimation, and (3) there is no principled algorithm for dynamic resource allocation to achieve better performance. We describe a method to combine information from multiple sensors and estimate the current tracker state by using a mixture of sequential Bayesian filters (e.g., particle filter)—one filter for each sensor, where each filter makes a different level of contribution to estimate the combined posterior in a reliable manner. In this framework, multiple sensors interact to determine an appropriate sensor for each particle dynamically; each particle is allocated to only one of the sensors for measurement and a different number of particles may be assigned to each sensor as a result. The level of the contribution of each sensor changes dynamically based on its prior information and relative measurement confidence. We apply this technique to visual tracking problems with multiple cameras or multiple features, and demonstrate its effectiveness through tracking results in real videos.


Visual tracking Resource allocation Sensor fusion Multiple cameras Multiple features Kernel-based bayesian filtering Mixture model 



This work is supported by the Center for Integrated Smart Sensors funded by the Ministry of Science, ICT & Future Planning as the Global Frontier Project.


  1. 1.
    Adlers M (2000) Topics in sparse least squares problems. Ph.D. thesis, Linköpings Universitet, Sweden.
  2. 2.
    Azoz Y, Devi L, Sharma R (1998) Reliable tracking of human arm dynamics by multiple cue integration and constraint fusion. In: Proceedings IEEE conference on computer vision and pattern recognition. Santa Barbara, CaliforniaGoogle Scholar
  3. 3.
    Birchfield S (1998) Elliptical head tracking using intensity gradients and color histograms. In: Proceedings IEEE conference on computer vision and pattern recognition. Santa Barbara, CA, pp 232–237Google Scholar
  4. 4.
    Black J, Ellis T (2006) Multi camera image tracking. Image Vision Comput J 24(11):1256–1267CrossRefGoogle Scholar
  5. 5.
    Blom H, Bar-Shalom Y (1988) The interacting multiple model algorithm for systems with markovianswitching coefficients. IEEE Trans Autom Control 33(8):780–783CrossRefGoogle Scholar
  6. 6.
    Cantarella J, Piatek M (2004) tsnnls: a solver for large sparse least squares problem with non-negative variables. (preprint)
  7. 7.
    Chen R, Liu J (2000) Mixture kalman filters. J Roy Statist Soc B 62(3):493–508CrossRefGoogle Scholar
  8. 8.
    Chen Y, Rui Y (2004) Real-time speaker tracking using particle filter sensor fusion. Proc IEEE 92(3):485–494CrossRefGoogle Scholar
  9. 9.
    Dockstader S, Tekalp A (2001) Multiple camera tracking of interacting and occluded human motion. Proc IEEE 89(10):1441–1455CrossRefGoogle Scholar
  10. 10.
    Doucet A, de Freitas N, Gordon N (2001) Sequential Monte Carlo methods in practice. Springer, BerlinGoogle Scholar
  11. 11.
    Han B, Comaniciu D, Zhu Y, Davis L (2008) Sequential kernel density approximation and its application to real-time visual tracking. IEEE Trans Pattern Anal Mach Intell 30(7):1186–1197CrossRefGoogle Scholar
  12. 12.
    Han B, Joo SW, Davis L (2007) Probabilistic fusion tracking using mixture kernel-based bayesian filtering. In: Proceedings 11th international conference on computer vision. Rio de Janeiro, BrazilGoogle Scholar
  13. 13.
    Han B, Joo SW, Davis LS (2011) Multi-camera tracking with adaptive resource allocation. Int J Comput Vision 91(1):45–58CrossRefGoogle Scholar
  14. 14.
    Han B, Zhu Y, Comaniciu D, Davis L (2009) Visual tracking by continuous density propagation in sequential bayesian filtering framework. IEEE Trans Pattern Anal Mach Intell 31(5):919–930CrossRefGoogle Scholar
  15. 15.
    Hoffmann C, Dang T (2009) Cheap joint probabilistic data association filters in an interacting multiple model design. Rob Auton Syst 57(3):268–278CrossRefGoogle Scholar
  16. 16.
    Isard M, Blake A (1998) ICONDENSATION: unifying low-level and high-level tracking in a stochastic framework. In: Proceedings European conference on computer vision. Freiburg, Germany, pp 893–908Google Scholar
  17. 17.
    Julier S, Uhlmann J (1997) A new extension of the Kalman filter to nonlinear systems. Proc SPIE 3068:182–193CrossRefGoogle Scholar
  18. 18.
    Kembhavi A, Schwartz W, Davis L (2008) Resource allocation for tracking multiple targets using particle filters. In: Worshop on visual surveillance, in conjunction with ECCVGoogle Scholar
  19. 19.
    Khan S, Shah M (2006) A multiview approach to tracking people in crowded scenes using a planar homography constraint. In: Proceedings European conference on computer vision, part IV. Graz, Austria, pp 133–146Google Scholar
  20. 20.
    Kim K, Davis L (2006) Multi-camera tracking and segmentation of occluded people on ground plane using search-guided particle filtering. In: Proceedings European conference on computer vision, part III. Graz, Austria, pp 98–109Google Scholar
  21. 21.
    Lawson CL, Hanson BJ (1974) Solving least squares problems. Prentice-Hall, New JerseyGoogle Scholar
  22. 22.
    Leichter I, Lindenbaum M, Rivlin E (2004) A probabilistic framework for combining tracking algorithms. In: Proceedings IEEE conference on computer vision and pattern recognition. Washington DC, pp 445–451Google Scholar
  23. 23.
    Mazor E, Averbuch A, Bar-Shalom Y, Dayan J (1998) Interacting multiple model methods in target tracking: a survey. IEEE Trans Aerosp Electron Syst 34(1):103–123CrossRefGoogle Scholar
  24. 24.
    Mccane B, Galvin B, Novins K (2002) Algorithmic fusion for more robust feature tracking. Int J Comput Vision 49(1):79–89CrossRefGoogle Scholar
  25. 25.
    von der Merwe R, Doucet A, de Freitas N, Wan E (2000) The unscented particle filter. Technical report CUED/F-INFENG/TR 380. Cambridge University Engineering Department, CambridgeGoogle Scholar
  26. 26.
    Mittal A, Davis LS (2003) M2tracker: a multi-view approach to segmenting and tracking people in a cluttered scene. Int J Comput Vision 51(3):189–203CrossRefGoogle Scholar
  27. 27.
    Musicki D (2008) Bearings only multi-sensor maneuvering target tracking. Syst Control Lett 57(3):216–221MathSciNetCrossRefGoogle Scholar
  28. 28.
    Musicki D, Scala BL (2008) Multi-target tracking in clutter without measurement assignment. IEEE Trans Aerosp Electron Syst 44(3):877–896CrossRefGoogle Scholar
  29. 29.
    Okuma K, Taleghani A, de Freitas N, Little J, Lowe D (2004) A boosted particle filter: multitarget detection and tracking. In: Proceedings European conference on computer vision. Prague, Czech RepublicGoogle Scholar
  30. 30.
    Pan P, Schonfeld D (2008) Adaptive resource allocation in particle filtering for articulated object tracking. In: Proceedings of the IEEE international conference on acoustics, speech, and signal processing. Las Vegas, Nevada, USA, pp 729–732Google Scholar
  31. 31.
    Perez P, Vermaak J, Blake A (2004) Data fusion for visual tracking with particles. Proc IEEE 92(3):495–513CrossRefGoogle Scholar
  32. 32.
    Rui Y, Chen Y (2001) Better proposal distributions: object tracking using unscented particle filter. In: Proceedings IEEE conference on computer vision and pattern recognition, vol. II. Kauai, Hawaii, pp 786–793Google Scholar
  33. 33.
    Salmond D (1990) Mixture reduction algorithms for target tracking in clutter. In: SPIE signal and data processing of small targets, vol 1305. Orlando, Florida pp 434–445Google Scholar
  34. 34.
    Sherrah J, Gong S (2001) Continuous global evidence-based bayesian modality fusion for simulataneous tracking of multiple objects. In: Proceedings 8th international conference on computer vision. Vancouver, CanadaGoogle Scholar
  35. 35.
    Siebel NT, Maybank SJ (2002) Fusion of multiple tracking algorithms for robust people tracking. In: Proceedings European conference on computer vision, Copenhagen, vol IV. Denmark, pp 373–387Google Scholar
  36. 36.
    Song C, Son J, Kwak S, Han B (2011) Dynamic resource allocation by ranking SVM for particle filter tracking. In: British machine vision conference. Dundee, UKGoogle Scholar
  37. 37.
    Spengler M, Schiele B (2003) Towards robust multi-cue integration for visual tracking. Mach Vis Appl 14(1):50–58CrossRefGoogle Scholar
  38. 38.
    Tran S, Davis L (2007) Object tracking at multiple levels of spatial resolutions. In: Proceedings of international conference on image analysis and processing, pp 149–154Google Scholar
  39. 39.
    Triesch J, von der Malsburg C (2001) Democratic integration: self-organized integration of adaptive cues. Neural Comput 13(9):2049–2074CrossRefGoogle Scholar
  40. 40.
    Vermaak J, Doucet A, Perez P (2003) Maintaining multi-modality through mixture tracking. In: Proceedings 9th internatinal conference on computer vision, vol II. Nice, FranceGoogle Scholar
  41. 41.
    Vermaak J, Gangnet M, Blake A, Perez P (2001) Sequential monte carlo fusion of sound and vision for speaker tracking. In: Proceedings 8th international conference on computer vision, vol I. Vancouver, Canada, pp 741–746Google Scholar
  42. 42.
    Williams J, Maybeck P (2003) Cost-function-based gaussian mixture reduction for target tracking. In: International conference on information fusion, vol 2, pp 1047–1054Google Scholar
  43. 43.
    Wu Y, Huang T (2001) A co-inference approach to robust visual tracking. In: Proceedings 8th international conference on computer vision, vol II. Vancouver, Canada, pp 26–33Google Scholar
  44. 44.
    Yang C, Duraiswami R, Davis L (2005) Fast multiple object tracking via a hierarchical particle filter. In: Proceedings 10th international conference on computer vision, vol. I. Beijing, China, pp 212–219Google Scholar
  45. 45.
    Zhong X, Xue J, Zheng N (2006) Graphical model based cue integration strategy for head tracking. In: Proceedings British machine vision conference. Edinburgh, UKGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2016

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringPOSTECHPohangKorea
  2. 2.Google Inc.Mountain ViewUSA
  3. 3.Department of Computer ScienceUniversity of MarylandCollege ParkUSA

Personalised recommendations