Sparse Discriminant Analysis Based on the Bayesian Posterior Probability Obtained by L1 Regression

  • Akinori Hidaka
  • Takio Kurita
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7626)


Recently the kernel discriminant analysis (KDA) has been successfully applied in many applications. However, kernel functions are usually defined a priori and it is not known what the optimum kernel function for nonlinear discriminant analysis is. Otsu derived the optimum nonlinear discriminant analysis (ONDA) by assuming the underlying probabilities similar with the Bayesian decision theory. Kurita derived discriminant kernels function (DKF) as the optimum kernel functions in terms of the discriminant criterion by investigating the optimum discriminant mapping constructed by the ONDA. The derived kernel function is defined by using the Bayesian posterior probabilities. We can define a family of DKFs by changing the estimation method of the Bayesian posterior probabilities. In this paper, we propose a novel discriminant kernel function based on L1-regularized regression, called L1 DKF. L1 DKF is given by using the Bayesian posterior probabilities estimated by L1 regression. Since L1 regression yields a sparse representation for given samples, we can naturally introduce the sparseness into the discriminant kernel function. To introduce the sparseness into LDA, we use L1 DKF as the kernel function of LDA. In experiments, we show sparseness and classification performance of L1 DKF.


Kernel Function Linear Discriminant Analysis Sparse Representation Bayesian Posterior Probability Generalize Eigenvalue Problem 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Baudat, G., Anouar, F.: Generalized discriminant analysis using a kernel approach. Neural Computation 12(10), 2385–2404 (2000)CrossRefGoogle Scholar
  2. 2.
    Chow, C.K.: An optimum character recognition system using decision functions. IRE Trans. EC-6, 247–254 (1957)Google Scholar
  3. 3.
    Clemmensen, L., Hastie, T., Witten, D., Ersboll, B.: Sparse discriminant analysis (2011)Google Scholar
  4. 4.
    Fisher, R.A.: The Use of Multiple Measurements in Taxonomic Problems. Annals of Eugenics 7, 179–188 (1936)Google Scholar
  5. 5.
    Frank, A., Asuncion, A.: UCI Machine Learning Repository. University of California, School of Information and Computer Science,
  6. 6.
    Hidaka, A., Kurita, T.: Discriminant Kernels based Support Vector Machine. In: The First Asian Conference on Pattern Recognition (ACPR 2011), Beijing, China, November 28-30, pp. 159–163 (2011)Google Scholar
  7. 7.
    Kurita, T.: “Discriminant Kernels derived from the Optimum Nonlinear Discriminant Analysis. In: Proc. of 2011 International Joint Conference on Neural Networks, San Jose, California, USA, July 31-August 5 (2011)Google Scholar
  8. 8.
    Mika, S., Ratsch, G., Weston, J., Scholkopf, B., Smola, A., Muller, K.: Fisher discriminant analysis with kernels. In: Proc. IEEE Neural Networks for Signal Processing Workshop, pp. 41–48 (1999)Google Scholar
  9. 9.
    Otsu, N.: Nonlinear discriminant analysis as a natural extension of the linear case. Behavior Metrika 2, 45–59 (1975)Google Scholar
  10. 10.
    Otsu, N.: Mathemetical Studies on Feature Extraction In Pattern Recognition. Researches on the Electrotechnical Laboratory 818 (1981) (in Japanease)Google Scholar
  11. 11.
    Otsu, N.: Optimal linear and nonlinear solutions for least-square discriminant feature extraction. In: Proceedings of the 6th International Conference on Pattern Recognition, pp. 557–560 (1982)Google Scholar
  12. 12.
    Tibshirani, R.: Regression shrinkage and selection via the lasso. J. Royal. Statist. Soc B. 58(1), 267–288 (1996)MathSciNetzbMATHGoogle Scholar
  13. 13.
    Zou, H., Hastie, T., Tibshirani, R.: Sparse principal component analysis. Journal of Computational and Graphical Statistics 15(2), 262–286 (2006)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Akinori Hidaka
    • 1
    • 2
  • Takio Kurita
    • 1
    • 2
  1. 1.Tokyo Denki UniversityJapan
  2. 2.Hiroshima UniversityJapan

Personalised recommendations