Background subtraction with Kronecker-basis-representation based tensor sparsity and \(l_{1,1,2}\) norm


Background subtraction is an important and fundamental step in video analysis and is a challenging task due to the dynamic background, bad weather, complex moving behaviors and huge amount of data in real-life applications. To address this issue, we proposed a new decomposition model based on the tensor robust principal component analysis, which makes full use of the continuity of time and the correlation of space. The Kronecker-basis-representation based tensor sparsity was introduced into the proposed model to constrain the spatio-temporal correlation of the video background and to enhance the consideration of low-rank characteristics, thus effectively reducing the interference of the dynamic background. The \({l_{1,1,2}}\) norm was used to constrain the sparseness of the spatio-temporal structure of the foreground, which strengthens the spatio-temporal continuity and tube sparsity of the video foreground and improves the accuracy of moving objects extraction. Experiments demonstrate that, in most cases, the proposed algorithm achieves superior performance in terms of F-measure scores and visual effect of separating the foreground and background of the video compared with state-of-the-art methods.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4


  1. Bouwmans, T., Sobral, A., & Javed, S. (2017). Decomposition into low-rank plus additive matrices for background/foreground separation: A review for a comparative evaluation with a large-scale dataset. Computer Science Review, 23, 1–71.

    Article  Google Scholar 

  2. Bouwmans, T., & Zahzah, E. H. (2014). Robust PCA via principal component pursuit: A review for a comparative evaluation in video surveillance. Computer Vision and Image Understanding, 122, 22–34.

    Article  Google Scholar 

  3. Candes, E. J., Wakin, M. B., & Boyd, S. P. (2008). Enhancing sparsity by reweighted \(l_1\) minimization. Journal of Fourier analysis and applications, 14(5–6), 877–905.

    MathSciNet  Article  Google Scholar 

  4. Cao, W., Wang, Y., & Sun, J. (2016). Total variation regularized tensor RPCA for background subtraction from compressive measurements. IEEE Transactions on Image Processing, 25(9), 4075–4090.

    MathSciNet  Article  Google Scholar 

  5. Chen, C., He, B., & Ye, Y. (2016). The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent. Mathematical Programming, 155(1–2), 57–79.

    MathSciNet  Article  Google Scholar 

  6. Chen, M., Wei, X., & Yang, Q. (2018). Spatiotemporal GMM for background subtraction with superpixel hierarchy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6), 1518–1525.

    Article  Google Scholar 

  7. Cherapanamjeri, Y., Gupta, K., & Jain, P. (2017). Nearly optimal robust matrix completion. In The 34th international conference on machine learning (pp. 797–805). JMLR. org.

  8. Elgammal, A., Harwood, D., & Davis, L. (2000) Non-parametric model for background subtraction. In European conference on computer vision (pp. 751–767). Springer, Berlin

  9. Gillis, N., & Vavasis, S. A. (2018). On the complexity of robust PCA and \(l_{1}\)-norm low-rank matrix approximation. Mathematics of Operations Research, 43, 1074–1082.

    Article  Google Scholar 

  10. Goldfarb, D., & Qin, Z. (2014). Robust low-rank tensor recovery: Models and algorithms. SIAM Journal on Matrix Analysis and Applications, 35(1), 225–253.

    MathSciNet  Article  Google Scholar 

  11. Gong, P., Zhang, C., & Lu, Z. (2013). A general iterative shrinkage and thresholding algorithm for non-convex regularized optimization problems. In 30th international conference on machine learning (pp. 37–45). IEEE.

  12. Javed, S., Mahmood, A., & Al-Maadeed, S. (2018a). Moving object detection in complex scene using spatiotemporal structured-sparse RPCA. IEEE Transactions on Image Processing, 1–1, 1007–1022.

    MathSciNet  MATH  Google Scholar 

  13. Javed, S., Mahmood, A., Bouwmans, T., & Jung, S. K. (2018b). Spatiotemporal low-rank modeling for complex scene background initialization. IEEE Transactions on Circuits and Systems for Video Technology, 28(6), 1315–1329.

    Article  Google Scholar 

  14. Kumar, S., & Yadav, J. S. (2016). Segmentation of moving objects using background subtraction method in complex environments. Radio Engineering, 25(2), 399–408.

    Google Scholar 

  15. Lcandes, E. J., Li, X., & Ma, Y. (2011). Robust principal component analysis? Journal of the ACM (JACM), 58(3), 11.

    MathSciNet  Google Scholar 

  16. Li, L., Huang, W., & Gu, I. Y. H. (2004). Statistical modeling of complex backgrounds for foreground object detection. IEEE Transactions on Image Processing, 13(11), 1459–1472.

    Article  Google Scholar 

  17. Liu, G., Lin, Z., & Yan, S. (2013). Robust recovery of subspace structures by low-rank representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1), 171–184.

    Article  Google Scholar 

  18. Lu, C., Feng, J., & Chen, Y. (2018). Tensor robust principal component analysis with a new tensor nuclear norm. arXiv preprint arXiv:1804.03728.

  19. Mahadevan, V., & Vasconcelos, N. (2010). Spatiotemporal saliency in dynamic scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(1), 171–177.

    Article  Google Scholar 

  20. Rezaei, B., & Ostadabbas, S. (2017). Background subtraction via fast robust matrix completion. In 2017 IEEE international conference on computer vision (ICCV) (pp. 1871–1879). IEEE.

  21. Rezaei, B., & Ostadabbas, S. (2018). Moving object detection through robust matrix completion augmented with objectness. IEEE Journal of Selected Topics in Signal Processing, 12(6), 1313–1323.

    Article  Google Scholar 

  22. Wan, M., Gu, G., & Qian, W. (2018). Total variation regularization term-based low-rank and sparse matrix representation model for infrared moving target tracking. Remote Sensing, 10(4), 510.

    Article  Google Scholar 

  23. Wang, S., Wang, Y., & Chen, Y. (2018). Robust PCA using matrix factorization for background/foreground separation. IEEE Access, 6, 18945–18953.

    Article  Google Scholar 

  24. Wang, Y., Jodoin, P. M., & Porikli, F. (2014). CDnet 2014: An expanded change detection benchmark dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 387–394). IEEE Computer Society.

  25. Wang, Y., Yin, W., & Zeng, J. (2019). Global convergence of ADMM in nonconvex nonsmooth optimization. Journal of Scientific Computing, 78, 29–63.

    MathSciNet  Article  Google Scholar 

  26. Xie, Q., Zhao, Q., & Meng, D. (2018). Kronecker-basis-representation based tensor sparsity and its applications to tensor recovery. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(8), 1888–1902.

    Article  Google Scholar 

  27. Xu, Y., Wu, Z., & Chanussot, J. (2018). Joint reconstruction and anomaly detection from compressive hyperspectral images using Mahalanobis distance-regularized tensor RPCA. IEEE Transactions on Geoscience and Remote Sensing, 56(5), 2919–2930.

    Article  Google Scholar 

  28. Yang, H., Qu, S., & Javed, S. (2017). Real-time vehicle detection and counting in complex traffic scenes using background subtraction model with low-rank decomposition. IET Intelligent Transport Systems, 12(1), 75–85.

    Article  Google Scholar 

  29. Yong, H., Meng, D., & Zuo, W. (2018). Robust online matrix factorization for dynamic background subtraction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(7), 1726–1740.

    Article  Google Scholar 

  30. Yuan, M., & Lin, Y. (2006). Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1), 49–67.

    MathSciNet  Article  Google Scholar 

  31. Zhang, Z., Ely, G., & Aeron, S. (2014). Novel methods for multilinear data completion and de-noising based on tensor-SVD. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3842–3849). IEEE Computer Society.

Download references


This project is partially supported by the National Natural Science Foundation of China (11961010, 61661017, 61967005, 61362021), Guangxi Natural Science Foundation (2018GXNSFAA138169, 2017GXNSFBA198212), Guangxi Colleges and Universities Key Laboratory project of Data Analysis and Computation (LDAC201704).

Author information



Corresponding author

Correspondence to Xuewen Wang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chen, L., Liu, J. & Wang, X. Background subtraction with Kronecker-basis-representation based tensor sparsity and \(l_{1,1,2}\) norm. Multidim Syst Sign Process (2020).

Download citation


  • Background subtraction
  • Tensor robust principal component analysis
  • \(l_{1 , 1 , 2}\) norm
  • Alternating direction multiplier method