Data-dependent kernel sparsity preserving projection and its application for semi-supervised classification

  • Ao Zhang
  • Xianwen Gao


Dimensionality reduction methods (DR) have been commonly used as a principled way to understand the high-dimensional data. In this paper, a novel semi-supervised nonlinear method called semi-supervised data-dependent kernel sparsity preserving projection (SDKSPP) is proposed for dimensionality reduction. To achieve performance improvements, SDKSPP adopts a data-dependent kernel (DK) instead of a standard kernel. The coefficients in DK are optimized with labeled samples by using the Fisher criterion. Then the labeled and unlabeled samples are mapped into a high dimensional space by DK. The sparse reconstructive relationship among the whole samples is calculated by minimizing a l1 regularization-related objective function. Finally, a transform matrix that can preserve this relationship is obtained to project the mapped data into a low-dimensional space. The effectiveness of the proposed method is tested and compared with seven methods on four popular datasets.


Sparsity preserving projection Data-dependent kernel Semi-supervised learning Dimensionality reduction 



This work is partially supported by the National Natural Science Foundation of China (No. 61573088, No. 61573087 and No. 61433004).


  1. 1.
    Amari S, Wu S (1999) Improving support vector machine classifiers by modifying kernel functions. Neural Netw 12(6):783–789CrossRefGoogle Scholar
  2. 2.
    Belhumeur PN, Hespanha JP, Kriengman DJ (1997) Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720CrossRefGoogle Scholar
  3. 3.
    Boyd S, Vandenberghe L (2004) Convex optimization. CambridgeGoogle Scholar
  4. 4.
    Cai D, He X, Han J (2007) Margin based semi-supervised elastic embedding for face image analysis. In: The IEEE international conference on computer vision, pp 1313–1320Google Scholar
  5. 5.
    Cai D, He X, Han J (2007) Semi-supervised discriminant analysis. In: International conference on computer vision.
  6. 6.
    Chen B, Liu H, Bao Z (2008) Optimizing the data-dependent kernel under a unified kernel optimization framework. Pattern Recogn 41(6):2107–2119CrossRefMATHGoogle Scholar
  7. 7.
    Cristianini N, Kandola J, Elisseeff A et al (2002) On kernel-target alignment. Adv Neural Inf Process Syst 179(5):367–373Google Scholar
  8. 8.
    Cristianini N, Ghaoui LE, Lanckriet GRG, Bartlett PL, Jordan MI (2004) Learning the kernel matrix with semi-definite programming. J Mach Learn Res 5(1):323–330MATHGoogle Scholar
  9. 9.
    Fan M, Gu N, Qiao H, Zhang B (2011) Sparse regularization for semi-supervised classification. Pattern Recogn 44(8):1777–1784CrossRefMATHGoogle Scholar
  10. 10.
    Friedman J, Hastie T, Tibshirani R (2010) Regularization paths for generalized linear models via coordinate descent. J Stat Softw 33(1):1CrossRefGoogle Scholar
  11. 11.
    Gao S, Tsang WH, Chia LT (2010) Kernel sparse representation for image classification and face recognition. In: European conference on computer vision, pp 1–14Google Scholar
  12. 12.
    Gao Q, Wang Q, Huang Y, Gao X, Hong X, Zhang H (2015) Dimensionality reduction by integrating sparse reduction and fisher criterion and its application. IEEE Trans Image Process 24(12):5684–5694MathSciNetCrossRefGoogle Scholar
  13. 13.
    Georghiades A (1997) Yale Face Database, Center for Computational Vision and Control at Yale University.
  14. 14.
    Gu N, Wang D, Fan M, Meng D (2014) A kernel-based sparsity preserving method for semi-supervised classification. Neurocomputing 139:345–356CrossRefGoogle Scholar
  15. 15.
    He Z, Li J (2015) Multiple data-dependent kernel for classification of hyperspectral images. Expert Syst Appl 42(3):1118–1135CrossRefGoogle Scholar
  16. 16.
    He X, Cai D, Han J (2008) Learning a maximum margin subspace for image retrieval. IEEE Trans Knowl Data Eng 20(2):189–201CrossRefGoogle Scholar
  17. 17.
    Hull JJ (1994) A database for handwritten text recognition research. IEEE Trans Pattern Anal Mach Intell 16(5):550–554CrossRefGoogle Scholar
  18. 18.
    Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. In: Proceedings of the IEEE, vol 86, no 11, pp 2278–2324Google Scholar
  19. 19.
    Lee MM, Keerthi SS, Ong CJ, Decoste D (2004) An efficient method for computing leave-one-out error in support vector machines with Gaussian kernels. IEEE Trans Neural Netw 15(3):750–757CrossRefGoogle Scholar
  20. 20.
    Lin C, Wang B, Zhao X, Pang M (2013) Optimizing kernel PCA using sparse representation-based classifier for MSTAR SAR image target recognition. Math Probl Eng 2013(6):707–724Google Scholar
  21. 21.
    Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2015) Action2Activity: recognizing complex activities from sensor data. In: Proceedings of the 24th international conference on artificial intelligence, pp 1617–1623Google Scholar
  22. 22.
    Liu Y, Liang Y, Liu S, Rosenblum D, Zheng Y (2016) Predicting urban water quality with ubiquitous data. arXiv:161009462
  23. 23.
    Liu Y, Zhang L, Nie L, Yan Y, Rosenblum DS (2016) Fortune teller: predicting your career path. In: Proceedings of the thirtieth AAAI conference on artificial intelligence, pp 201–207Google Scholar
  24. 24.
    Liu Y, Zheng Y, Liang Y, Liu S, Rosenblum DS (2016) Urban water quality prediction based on multi-task multi-view learning. In: Proceedings of the twenty-fifth international joint conference on artificial intelligence, pp 2576–2582Google Scholar
  25. 25.
    Lou S, Zhao X, Chuang Y, Zhang S (2016) Graph regularized sparsity discriminant analysis for face recognition. Neurocomputing 173(P2):290–297CrossRefGoogle Scholar
  26. 26.
    Luo L, Bao S, Mao J, Tang D (2016) Nonlinear process monitoring based on kernel global-local preserving projections. J Process Control 38:11–21CrossRefGoogle Scholar
  27. 27.
    Meng M, Wei J, Wang J, Ma Q, Wang X (2017) Adaptive semi-supervised dimensionality reduction based on pairwise constraints weighting and graph optimizing. Int J Mach Learn Cybern 8(3):793–805CrossRefGoogle Scholar
  28. 28.
    Motai Y, Yoshida H (2013) Principal composite kernel feature analysis: data-dependent kernel approach. IEEE Trans Knowl Data Eng 25(8):1863–1875CrossRefGoogle Scholar
  29. 29.
    Ong CS, Smola AJ, Williamson RC (2005) Learning the kernel with hyperkernels. J Mach Learn Res 6(1):1043–1071MathSciNetMATHGoogle Scholar
  30. 30.
    ORL face database. AT&T Laboratories, Cambridge.
  31. 31.
    Qiao L, Chen S, Tan X (2010) Sparsity preserving projections with applications to face recognition. Pattern Recogn 43(1):331–341CrossRefMATHGoogle Scholar
  32. 32.
    Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323CrossRefGoogle Scholar
  33. 33.
    Sugiyama M, Ide T, Nakajima S, Sese J (2006) Semi-supervised local Fisher discriminant analysis for dimensionality reduction. Mach Learn 78:35–61MathSciNetCrossRefGoogle Scholar
  34. 34.
    Tenenbaum JB, De SV, Langford JC (2000) A global geometric framework for nonlinear dimensionality reduction. Science 290(5500):2319CrossRefGoogle Scholar
  35. 35.
    Turk MA, Pentland AP (1991) Face recognition using eigenfaces. In: International conference on computer research and development, pp 302–306Google Scholar
  36. 36.
    Wright J, Yang A, Ganesh A, Sastry S, Ma Y (2009) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31 (2):210–227CrossRefGoogle Scholar
  37. 37.
    Xiong H, Swamy MN, Ahmad MO (2005) Optimizing the kernel in the empirical feature space. IEEE Trans Neural Netw 16(2):460–474CrossRefGoogle Scholar
  38. 38.
    Xiong H, Zhang Y, Chen XW (2007) Data-dependent kernel machines for microarray data classification. IEEE/ACM Trans Comput Biol Bioinform 4(4):583–595CrossRefGoogle Scholar
  39. 39.
    Yang Y, Wang Y, Xue X (2016) Discriminant sparse locality preserving projection for face recognition. Multimed Tools Appl 76(2):1–16Google Scholar
  40. 40.
    Yin J, Liu Z, Jin Z, Yang W (2012) Kernel sparse representation based classification. Neurocomputing 77(1):120–128CrossRefGoogle Scholar
  41. 41.
    Zhang L, Zhou WD (2016) Fisher-regularized support vector machine. Inf Sci 343–344:79–93MathSciNetCrossRefGoogle Scholar
  42. 42.
    Zhang D, Zhou ZH, Chen S (2007) Semi-supervised dimensionality reduction. In: SIAM international conference on data mining, pp 629–634Google Scholar
  43. 43.
    Zhang P, You X, Ou W, Chen CLP, Cheung YM (2016) Sparse discriminative multi-manifold embedding for one-sample face identification. Pattern Recogn 52(C):249–259CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.College of Information Science and EngineeringNortheastern UniversityShenyangChina

Personalised recommendations