Advertisement

Incremental Tensor Subspace Learning and Its Applications to Foreground Segmentation and Tracking

Abstract

Appearance modeling is very important for background modeling and object tracking. Subspace learning-based algorithms have been used to model the appearances of objects or scenes. Current vector subspace-based algorithms cannot effectively represent spatial correlations between pixel values. Current tensor subspace-based algorithms construct an offline representation of image ensembles, and current online tensor subspace learning algorithms cannot be applied to background modeling and object tracking. In this paper, we propose an online tensor subspace learning algorithm which models appearance changes by incrementally learning a tensor subspace representation through adaptively updating the sample mean and an eigenbasis for each unfolding matrix of the tensor. The proposed incremental tensor subspace learning algorithm is applied to foreground segmentation and object tracking for grayscale and color image sequences. The new background models capture the intrinsic spatiotemporal characteristics of scenes. The new tracking algorithm captures the appearance characteristics of an object during tracking and uses a particle filter to estimate the optimal object state. Experimental evaluations against state-of-the-art algorithms demonstrate the promise and effectiveness of the proposed incremental tensor subspace learning algorithm, and its applications to foreground segmentation and object tracking.

References

  1. Black, M. J., & Jepson, A. D. (1998). EigenTracking: robust matching and tracking of articulated objects using a view-based representation. International Journal of Computer Vision, 26(1), 63–84.

  2. Black, M. J., Fleet, D. J., & Yacoob, Y. (1998). A framework for modeling appearance change in image sequence. In Proc. of IEEE international conference on computer vision (pp. 660–667), Jan. 1998.

  3. Chen, D., & Yang, J. (2007). Robust object tracking via online dynamic spatial bias appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(12), 2157–2169.

  4. Elgammal, A., Duraiswami, R., Harwood, M., & Davis, L. S. (2002). Background and foreground modeling using nonparametric kernel density estimation for visual surveillance. Proceedings of the IEEE, 99(7), 1151–1163.

  5. Gall, J., Rosenhahn, B., & Seidel, H.-P. (2008). Drift-free tracking of rigid and articulated objects. In Proc. of IEEE conference on computer vision and pattern recognition (pp. 1–8), June 2008.

  6. Golub, G. H., & Van Loan, C. F. (1996). Matrix computations. Baltimore: Johns Hopkins University Press.

  7. Grabner, H., Leistner, C., & Bischof, H. (2008). Semi-supervised on-line boosting for robust tracking. In Proc. of European conference on computer vision (pp. 234–247).

  8. Gu, M., & Eisenstat, S. C. (1993). A stable and fast algorithm for updating the singular value decomposition. Research report YALEU/DCS/RR-966, Department of Computer Science, Yale University, New Haven, June 1993.

  9. Gu, M., & Eisenstat, S. C. (1995). Downdating the singular value decomposition. SIAM Journal on Matrix Analysis and Applications, 16(3), 793–810.

  10. Hager, G. D., & Belhumeur, P. N. (1996). Real-time tracking of image regions with changes in geometry and illumination. In Proc. of IEEE conference on computer vision and pattern recognition (pp. 403–410), June 1996.

  11. Han, B., Zhu, Y., Comaniciu, D., & Davis, L. S. (2009). Visual tracking by continuous density propagation in sequential bayesian filtering framework. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(5), 919–930.

  12. Haritaoglu, I., Harwood, D., & Davis, L. S. (2000). W4: real-time surveillance of people and their activities. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8), 809–830.

  13. He, X., Cai, D., & Niyogi, P. (2005). Tensor subspace analysis. In Proc. of annual conference on neural information processing systems Dec. 2005 Cambridge: MIT Press.

  14. Ho, J., Lee, K., Yang, M., & Kriegman, D. (2004). Visual tracking using learned linear subspaces. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 782–789).

  15. Isard, M., & Blake, A. (1996). Contour tracking by stochastic propagation of conditional density. In Proc. of European conference on computer vision (vol. 2, pp. 343–356).

  16. Jacques, J. C. S. Jr., Jung, C. R., & Musse, S. R. (2006). A background subtraction model adapted to illumination changes. In Proc. of IEEE international conference on image processing (pp. 1817–1820), Oct. 2006.

  17. Jepson, A. D., Fleet, D. J., & El-Maraghi, T. F. (2003). Robust online appearance models for visual tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(10), 1296–1311.

  18. Kwon, J., & Lee, K. M. (2009). Tracking of a non-rigid object via patch-based dynamic appearance modeling and adaptive basin hopping Monte Carlo sampling. In Proc. of IEEE conference on computer vision and pattern recognition workshops (pp. 1208–1215), June 2009.

  19. Lathauwer, L. D., Moor, B. D., & Vandewalle, J. (2000). On the best Rank-1 and Rank-(R1,R2,…,Rn) approximation of higher-order tensors. SIAM Journal of Matrix Analysis and Applications, 21(4), 1324–1342.

  20. Lee, K., & Kriegman, D. (2005). Online learning of probabilistic appearance manifolds for video-based recognition and tracking. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 852–859).

  21. Levy, A., & Lindenbaum, M. (2000). Sequential Karhunen-Loeve basis extraction and its application to images. IEEE Transactions on Image Processing, 9, 1371–1374.

  22. Li, Y. (2004). On incremental and robust subspace learning. Pattern Recognition, 37(7), 1509–1518.

  23. Li, J., Zhou, S. K., & Chellappa, R. (2005). Appearance modeling under geometric context. In Proc. of IEEE international conference on computer vision (vol. 2, pp. 1252–1259).

  24. Li, X., Hu, W. M., Zhang, Z. F., Zhang, X. Q., & Luo, G. (2007). Robust visual tracking based on incremental tensor subspace learning. In Proc. of IEEE international conference on computer vision (pp. 1–8), Oct. 2007.

  25. Lim, H., Morariu, V. I., Camps, O. I., & Sznaier, M. (2006). Dynamic appearance modeling for human tracking. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 751–757).

  26. Mahadevan, V., & Vasconcelos, N. (2009). Saliency-based discriminant tracking. In Proc. of IEEE conference on computer vision and pattern recognition workshops (pp. 1007–1013), June 2009.

  27. Matthews, I., Ishikawa, T., & Baker, S. (2004). The template update problem. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(4), 810–815.

  28. Nickel, K., & Stiefelhagen, R. (2008). Dynamic integration of generalized cues for person tracking. In Proc. of European conference on computer vision, Part IV. Lecture notes in computer science (vol. 5305, pp. 514–526), Oct. 2008.

  29. Nummiaroa, K., Koller-Meierb, E., & Gool, I. V. (2003). An adaptive color-based particle filter. Image and Vision Computing, 21(1), 99–110.

  30. Papadimitriou, S., Sun, J., & Faloutsos, C. (2005). Streaming pattern discovery in multiple timeseries. In Proc. of international conference on very large data bases (pp. 697–708).

  31. Patwardhan, K., Morellas, V., & Sapiro, G. (2008). Robust foreground detection in video using pixel layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(4), 746–751.

  32. Perez, P., Hue, C., Vermaak, J., & Gangnet, M. (2002). Color-based probabilistic tracking. In Proc. of European conference on computer vision, Part I. Lecture notes in computer science (vol. 2350, pp. 661–675).

  33. Porikli, F., Tuzel, O., & Meer, P. (2006). Covariance tracking using model update based on Lie algebra. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 728–735).

  34. Ramanan, D., Forsyth, D. A., & Zisserman, A. (2007). Tracking people by learning their appearance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(1), 65–81.

  35. Ross, D. A., Lim, J., Lin, R.-S., & Yang, M.-H. (2008). Incremental learning for robust visual tracking. International Journal of Computer Vision, 77(2), 125–141.

  36. Sheikh, Y., & Shah, M. (2005). Bayesian object detection in dynamic scenes. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 74–79).

  37. Skocaj, D., & Leonardis, A. (2003). Weighted and robust incremental method for subspace learning. In Proc. of IEEE international conference on computer vision (vol. 2, pp. 1494–1501), Oct. 2003.

  38. Stauffer, C., & Grimson, W. E. L. (1999). Adaptive background mixture models for real-time tracking. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 2, pp. 246–252).

  39. Sun, J., Tao, D., & Faloutsos, C. (2006a). Beyond streams and graphs: dynamic tensor analysis. In Proc. of ACM SIGKDD international conference on knowledge discovery and data mining (pp. 374–383), Aug. 2006.

  40. Sun, J., Papadimitriou, S., & Yu, P. S. (2006b). Window-based tensor analysis on high-dimensional and multi-aspect streams. In Proc. of international conference on data mining (pp. 1076–1080), Dec. 2006.

  41. Sun, J., Tao, D., Papadimitriou, S., Yu, P. S., & Faloutsos, C. (2008). Incremental tensor analysis: theory and applications. ACM Transactions on Knowledge Discovery from Data, 2(3), 1–37.

  42. Tian, Y., Lu, M., & Hampapur, A. (2005). Robust and efficient foreground analysis for real-time video surveillance. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 1182–1187).

  43. Vasilescu, M. A. O., & Terzopoulos, D. (2002). Multilinear subspace analysis of image ensembles: TensorFaces. In Proc. of European conference on computer vision (pp. 447–460), May 2002.

  44. Vasilescu, M. A. O., & Terzopoulos, D. (2003). Multilinear subspace analysis of image ensembles. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 2, pp. 93–99), June 2003.

  45. Wang, H., & Ahuja, N. (2005). Rank-R approximation of tensors using image-as-matrix representation. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 2, pp. 346–353).

  46. Wang, H., & Ahuja, N. (2008). A tensor approximation approach to dimensionality reduction. International Journal of Computer Vision, 76(3), 217–229.

  47. Wang, Y., Tan, T., Loe, K. F., & Wu, J. K. (2005). A probabilistic approach for foreground and shadow segmentation in monocular image sequences. Pattern Recognition, 38(11), 1937–1946.

  48. Wang, Y., Loe, K., & Wu, J. (2006). A dynamic conditional random field model for foreground and shadow segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(2), 279–289.

  49. Wang, H., Yan, S., Huang, T., & Tang, X. (2007). A convergent solution to tensor subspace learning. In Proc. of international joint conference on artificial intelligence (pp. 629–634).

  50. Wong, S., Wong, K. K., & Cipolla, R. (2006). Robust appearance-based tracking using a sparse bayesian classifier. In Proc. of international conference on pattern recognition (vol. 3, pp. 47–50).

  51. Wu, Y., & Huang, T. S. (2004). Robust visual tracking by integrating multiple cues based on co-inference learning. International Journal of Computer Vision, 58(1), 55–71.

  52. Yan, S., Xu, D., Yang, Q., Zhang, L., Tang, X., & Zhang, H. (2005). Discriminant analysis with tensor representation. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 526–532), June 2005.

  53. Yan, S., Shan, S., Chen, X., Gao, W., & Chen, J. (2007). Matrix-structural learning (MSL) of cascaded classifier from enormous training set. In Proc. of IEEE conference on computer vision and pattern recognition (pp. 1–7), June 2007.

  54. Yang, J., Zhang, D., Frangi, A. F., & Yang, J. (2004). Two-dimensional PCA a new approach to appearance-based face representation and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(1), 131–137.

  55. Yang, C., Duraiswami, R., & Davis, L. S. (2005). Efficient mean-shift tracking via a new similarity measure. In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 176–183), June 2005.

  56. Yang, M., Fan, Z., Fan, J., & Wu, Y. (2009). Tracking nonstationary visual appearances by data-driven adaptation. IEEE Transactions on Image Processing, 18(7), 1633–1644.

  57. Ye, J. (2005). Generalized low rank approximations of matrices. Machine Learning, 61(1–3), 167–191.

  58. Ye, J., Janardan, R., & Li, Q. (2004a). Two-dimensional linear discriminant nalysis. In Proc. of neural information processing systems conference (pp. 1569–1576). Cambridge: MIT Press.

  59. Ye, J., Janardan, R., & Li, Q. (2004b). GPCA: an efficient dimension reduction scheme for image compression and retrieval. In Proc. of ACM SIGKDD international conference on knowledge discovery and data mining (pp. 354–363), Aug. 2004.

  60. Yu, T., & Wu, Y. (2006). Differential tracking based on spatial-appearance model (SAM). In Proc. of IEEE conference on computer vision and pattern recognition (vol. 1, pp. 720–727), June 2006.

  61. Zhou, S. K., Chellappa, R., & Moghaddam, B. (2004). Visual tracking and recognition using appearance-adaptive models in particle filters. IEEE Transactions on Image Processing, 13(11), 1491–1506.

  62. Zhou, X., Hu, W. M., Chen, Y., & Hu, W. (2007). Markov random field modeled level sets method for object tracking with moving cameras. In Proc. of Asian conference on computer vision, Part I (pp. 832–842).

Download references

Author information

Correspondence to Weiming Hu.

Electronic Supplementary Material

Results of the DTA-based algorithm for Example 1 (4.717 kb)

Results of the DTA-based algorithm for Example 2 (1.366 kb)

Results of the DTA-based algorithm for Example 3 (3.251 kb)

Results of the DTA-based algorithm for Example 5 (2.661 kb)

Results of the Riemannian metric-based algorithm for Example 2 (3.568 kb)

Results of the Riemannian metric-based algorithm for Example 3 (3.768 kb)

Results of the Riemannian metric-based algorithm for Example 5 (3.013 kb)

Results of the DTA-based algorithm for Example 1 (4.717 kb)

Results of the DTA-based algorithm for Example 2 (1.366 kb)

Results of the DTA-based algorithm for Example 3 (3.251 kb)

Results of the DTA-based algorithm for Example 4 (4.246 kb)

Results of the DTA-based algorithm for Example 5 (2.661 kb)

Results of the Riemannian metric-based algorithm for Example 2 (3.568 kb)

Results of the Riemannian metric-based algorithm for Example 3 (3.768 kb)

Results of the Riemannian metric-based algorithm for Example 4 (3.916 kb)

Results of the Riemannian metric-based algorithm for Example 5 (3.013 kb)

Rights and permissions

Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Reprints and Permissions

About this article

Cite this article

Hu, W., Li, X., Zhang, X. et al. Incremental Tensor Subspace Learning and Its Applications to Foreground Segmentation and Tracking. Int J Comput Vis 91, 303–327 (2011). https://doi.org/10.1007/s11263-010-0399-6

Download citation

Keywords

  • Incremental learning
  • Tensor subspace
  • Foreground segmentation
  • Tracking