Sparse Gradient Pursuit for Robust Visual Analysis

  • Jiangxin Dong
  • Risheng LiuEmail author
  • Kewei Tang
  • Yiyang Wang
  • Xindong Zhang
  • Zhixun Su
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10111)


Many high-dimensional data analysis problems, such as clustering and classification, usually involve the minimization of a Laplacian regularization, which is equivalent to minimize square errors of the gradient on a graph, i.e., the disparity among the adjacent nodes in a data graph. However, the Laplacian criterion usually preserves the locally homogeneous data structure but suppresses the discrimination among samples across clusters, which accordingly leads to undesirable confusion among similar observations belonging to different clusters. In this paper, we propose a novel criterion, named Sparse Gradient Pursuit (SGP), to simultaneously preserve the within-class homogeneity and the between-class discrimination for unsupervised data clustering. In addition, we show that the proposed SGP criterion is generic and can be extended to handle semi-supervised learning problems by incorporating the label information into the data graph. Though this unified semi-supervised learning model leads to a nonconvex optimization problem, we develop a new numerical scheme for the SGP related nonconvex optimization problem and analyze the convergence property of the proposed algorithm under mild conditions. Extensive experiments demonstrate that the proposed algorithm performs favorably against the state-of-the-art unsupervised and semi-supervised methods.



Risheng Liu is supported by National Natural Science Foundation of China (NSFC) (Nos. 61300086, 61432003, 61672125), Fundamental Research Funds for the Central Universities (No. DUT15QY15), and the Hong Kong Scholar Program (No. XJ2015008). Zhixun Su is supported by NSFC (No. 61572099) and National Science and Technology Major Project (Nos. ZX20140419, 2014ZX04001011).

Supplementary material

416257_1_En_24_MOESM1_ESM.pdf (2.4 mb)
Supplementary material 1 (pdf 2488 KB)


  1. 1.
    Von Luxburg, U.: A tutorial on spectral clustering. Stat. Comput. 17, 395–416 (2007)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. TPAMI 22, 888–905 (2000)CrossRefGoogle Scholar
  3. 3.
    Hu, H., Lin, Z., Feng, J., Zhou, J.: Smooth representation clustering. In: CVPR, pp. 3834–3841 (2014)Google Scholar
  4. 4.
    Liu, R., Lin, Z., Su, Z.: Learning markov random walks for robust subspace clustering and estimation. Neural Netw. 59, 1–15 (2014)CrossRefzbMATHGoogle Scholar
  5. 5.
    He, X., Cai, D., Shao, Y., Bao, H., Han, J.: Laplacian regularized Gaussian mixture model for data clustering. TKDE 23, 1406–1418 (2011)Google Scholar
  6. 6.
    Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15, 1373–1396 (2003)CrossRefzbMATHGoogle Scholar
  7. 7.
    Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2323–2326 (2000)CrossRefGoogle Scholar
  8. 8.
    Yin, M., Gao, J., Lin, Z.: Laplacian regularized low-rank representation and its applications. TPAMI 38, 504–517 (2016)CrossRefGoogle Scholar
  9. 9.
    Belkin, M., Niyogi, P., Sindhwani, V.: Manifold regularization: a geometric framework for learning from labeled and unlabeled examples. JMLR 7, 2399–2434 (2006)MathSciNetzbMATHGoogle Scholar
  10. 10.
    Gao, S., Tsang, I.W.H., Chia, L.T.: Laplacian sparse coding, hypergraph laplacian sparse coding, and applications. TPAMI 35, 92–104 (2013)CrossRefGoogle Scholar
  11. 11.
    He, X., Yan, S., Hu, Y., Niyogi, P., Zhang, H.J.: Face recognition using Laplacianfaces. TPAMI 27, 328–340 (2005)CrossRefGoogle Scholar
  12. 12.
    Yang, Y., Wang, Z., Yang, J., Han, J., Huang, T.S.: Regularized l1-graph for data clustering. In: BMVC (2014)Google Scholar
  13. 13.
    Protter, M., Elad, M.: Image sequence denoising via sparse and redundant representations. TIP 18, 27–35 (2009)MathSciNetGoogle Scholar
  14. 14.
    Mairal, J., Bach, F., Ponce, J., Sapiro, G.: Online dictionary learning for sparse coding. In: ICML, pp. 689–696 (2009)Google Scholar
  15. 15.
    Chapelle, O., Schölkopf, B., Zien, A.: Semi-supervised Learning. MIT Press, Cambridge (2006)CrossRefGoogle Scholar
  16. 16.
    Zhu, X.: Semi-supervised learning. In: Encyclopedia of Machine Learning, pp. 892–897 (2011)Google Scholar
  17. 17.
    Zhu, X., Ghahramani, Z., Lafferty, J., et al.: Semi-supervised learning using Gaussian fields and harmonic functions. In: ICML, vol. 3, pp. 912–919 (2003)Google Scholar
  18. 18.
    Zhuang, L., Gao, H., Lin, Z., Ma, Y., Zhang, X., Yu, N.: Non-negative low rank and sparse graph for semi-supervised learning. In: CVPR, pp. 2328–2335 (2012)Google Scholar
  19. 19.
    Li, S., Fu, Y.: Low-rank coding with b-matching constraint for semi-supervised classification. In: IJCAI (2013)Google Scholar
  20. 20.
    Joachims, T., et al.: Transductive learning via spectral graph partitioning. In: ICML, vol. 3, pp. 290–297 (2003)Google Scholar
  21. 21.
    Belkin, M., Matveeva, I., Niyogi, P.: Regularization and semi-supervised learning on large graphs. In: Shawe-Taylor, J., Singer, Y. (eds.) COLT 2004. LNCS, vol. 3120, pp. 624–638. Springer, Heidelberg (2004). doi: 10.1007/978-3-540-27819-1_43 CrossRefGoogle Scholar
  22. 22.
    Yan, S., Wang, H.: Semi-supervised learning by sparse representation. In: SDM, pp. 792–801 (2009)Google Scholar
  23. 23.
    Li, C.G., Lin, Z., Zhang, H., Guo, J.: Learning semi-supervised representation towards a unified optimization framework for semi-supervised learning. In: ICCV, pp. 2767–2775 (2015)Google Scholar
  24. 24.
    Wright, J., Ganesh, A., Rao, S., Peng, Y., Ma, Y.: Robust principal component analysis: exact recovery of corrupted low-rank matrices via convex optimization. In: NIPS, pp. 2080–2088 (2009)Google Scholar
  25. 25.
    Liu, G., Lin, Z., Yan, S., Sun, J., Yu, Y., Ma, Y.: Robust recovery of subspace structures by low-rank representation. TPAMI 35, 171–184 (2013)CrossRefGoogle Scholar
  26. 26.
    Li, Z., Liu, J., Tang, J., Lu, H.: Robust structured subspace learning for data representation. TPAMI 37, 2085–2098 (2015)CrossRefGoogle Scholar
  27. 27.
    Zheng, M., Bu, J., Chen, C., Wang, C., Zhang, L., Qiu, G., Cai, D.: Graph regularized sparse coding for image representation. TIP 20, 1327–1336 (2011)MathSciNetGoogle Scholar
  28. 28.
    Liu, R., Lin, Z., Su, Z.: Linearized alternating direction method with parallel splitting and adaptive penalty for separable convex programs in machine learning. In: ACML, pp. 116–132 (2013)Google Scholar
  29. 29.
    Elhamifar, E., Vidal, R.: Sparse subspace clustering: algorithm, theory, and applications. TPAMI 35, 2765–2781 (2013)CrossRefGoogle Scholar
  30. 30.
    Nasihatkon, B., Hartley, R.: Graph connectivity in sparse subspace clustering. In: CVPR, pp. 2137–2144 (2011)Google Scholar
  31. 31.
    Tang, K., Dunson, D.B., Su, Z., Liu, R., Zhang, J., Dong, J.: Subspace segmentation by dense block and sparse representation. Neural Netw. 75, 66–76 (2016)CrossRefGoogle Scholar
  32. 32.
    Bresson, X., Laurent, T., Uminsky, D., von Brecht, J.: Multiclass total variation clustering. In: NIPS, pp. 1421–1429 (2013)Google Scholar
  33. 33.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3, 1–122 (2011)CrossRefzbMATHGoogle Scholar
  34. 34.
    Lin, Z., Liu, R., Su, Z.: Linearized alternating direction method with adaptive penalty for low-rank representation. In: NIPS, pp. 612–620 (2011)Google Scholar
  35. 35.
    Yuan, G., Ghanem, B.: l0tv: a new method for image restoration in the presence of impulse noise. In: CVPR, pp. 5369–5377 (2015)Google Scholar
  36. 36.
    Liu, G., Lin, Z., Yu, Y.: Robust subspace segmentation by low-rank representation. In: ICML, pp. 663–670 (2010)Google Scholar
  37. 37.
    Sim, T., Baker, S., Bsat, M.: The CMU pose, illumination, and expression (PIE) database. In: AFG, pp. 46–51 (2002)Google Scholar
  38. 38.
    Bartels, R.H., Stewart, G.W.: Solution of the matrix equation AX+XB=C. CACM 15, 820–826 (1972)CrossRefGoogle Scholar
  39. 39.
    Hull, J.J.: A database for handwritten text recognition research. TPAMI 16, 550–554 (1994)CrossRefGoogle Scholar
  40. 40.
    Tang, K., Liu, R., Su, Z., Zhang, J.: Structure-constrained low-rank representation. TNNLS 25, 2167–2179 (2014)Google Scholar
  41. 41.
    Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. 146, 459–494 (2014)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Jiangxin Dong
    • 1
  • Risheng Liu
    • 1
    Email author
  • Kewei Tang
    • 2
  • Yiyang Wang
    • 1
  • Xindong Zhang
    • 1
  • Zhixun Su
    • 1
  1. 1.Dalian University of TechnologyDalianChina
  2. 2.Liaoning Normal UniversityDalianChina

Personalised recommendations