A novel approach to semi-supervised learning for classical Fisher linear discriminant analysis is presented. It formulates the problem in terms of a constrained log-likelihood approach, where the semi-supervision comes in through the constraints. These constraints encode that the parameters in linear discriminant analysis fulfill particular relations involving label-dependent and label-independent quantities. In this way, the latter type of parameters, which can be estimated based on unlabeled data, impose constraints on the former. The former parameters are the class-conditional means and the average within-class covariance matrix, which are the parameters of interest in linear discriminant analysis. The constraints lead to a reduction in variability of the label-dependent estimates, resulting in a potential improvement of the semi-supervised linear discriminant over that of its regular supervised counterpart. We state upfront that some of the key insights in this contribution have been published previously in a workshop paper by the first author. The major contribution in this work is the basic observation that a semi-supervised linear discriminant analysis can be formulated in terms of a principled log-likelihood approach, where the previous solution employed an ad hoc procedure. With the current contribution, we move yet another step closer to a proper formulation of a semi-supervised version of this classical technique.


Linear Discriminant Analysis Label Data Unlabeled Data Unlabeled Sample Statistical Pattern Recognition 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Asuncion, A., Newman, D.: UCI machine learning repository (2007),
  2. 2.
    Ben-David, S., Lu, T., Pál, D.: Does unlabeled data provably help? Worst-case analysis of the sample complexity of semi-supervised learning. In: COLT 2008, pp. 33–44 (2008)Google Scholar
  3. 3.
    Chapelle, O., Schölkopf, B., Zien, A.: Semi-Supervised Learning. MIT Press, Cambridge (2006)Google Scholar
  4. 4.
    Cohen, I., Cozman, F., Sebe, N., Cirelo, M., Huang, T.: Semisupervised learning of classifiers: Theory, algorithms, and their application to human-computer interaction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1553–1567 (2004)Google Scholar
  5. 5.
    Cozman, F., Cohen, I.: Risks of semi-supervised learning. In: Semi-Supervised Learning, ch. 4. MIT Press (2006)Google Scholar
  6. 6.
    Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society. Series B (Methodological) 39(1), 1–38 (1977)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Fukunaga, K.: Introduction to Statistical Pattern Recognition. Academic Press (1990)Google Scholar
  8. 8.
    Hartley, H.O., Rao, J.N.K.: Classification and estimation in analysis of variance problems. Review of the International Statistical Institute 36(2), 141–147 (1968)MathSciNetzbMATHCrossRefGoogle Scholar
  9. 9.
    Liu, Q., Sung, A.H., Chen, Z., Liu, J., Huang, X., Deng, Y.: Feature selection and classification of MAQC-II breast cancer and multiple myeloma microarray gene expression data. PLoS ONE 4(12), e8250 (2009)CrossRefGoogle Scholar
  10. 10.
    Loog, M.: Constrained Parameter Estimation for Semi-supervised Learning: The Case of the Nearest Mean Classifier. In: Balcázar, J.L., Bonchi, F., Gionis, A., Sebag, M. (eds.) ECML PKDD 2010, Part II. LNCS, vol. 6322, pp. 291–304. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  11. 11.
    Loog, M.: Semi-supervised Linear Discriminant Analysis Using Moment Constraints. In: Schwenker, F., Trentin, E. (eds.) PSL 2011. LNCS (LNAI), vol. 7081, pp. 32–41. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  12. 12.
    Loog, M., Duin, R.P.W.: The Dipping Phenomenon. In: Gimel’ farb, G.L., Hancock, E., Imiya, A., Kudo, M., Kuijper, A., Omachi, S., Windeatt, T., Yamada, K. (eds.) SSPR & SPR 2012. LNCS, vol. 7626, pp. 310–317. Springer, Heidelberg (2012)Google Scholar
  13. 13.
    Mann, G.S., McCallum, A.: Generalized expectation criteria for semi-supervised learning with weakly labeled data. The Journal of Machine Learning Research 11, 955–984 (2010)MathSciNetzbMATHGoogle Scholar
  14. 14.
    McLachlan, G.J.: Iterative reclassification procedure for constructing an asymptotically optimal rule of allocation in discriminant analysis. Journal of the American Statistical Association 70(350), 365–369 (1975)MathSciNetzbMATHCrossRefGoogle Scholar
  15. 15.
    McLachlan, G.: Estimating the linear discriminant function from initial samples containing a small number of unclassified observations. Journal of the American Statistical Association 72(358), 403–406 (1977)MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    McLachlan, G.: Discriminant Analysis and Statistical Pattern Recognition. John Wiley & Sons (1992)Google Scholar
  17. 17.
    McLachlan, G., Ganesalingam, S.: Updating a discriminant function on the basis of unclassified data. Communications in Statistics - Simulation and Computation 11(6), 753–767 (1982)zbMATHCrossRefGoogle Scholar
  18. 18.
    Nigam, K., McCallum, A., Thrun, S., Mitchell, T.: Learning to classify text from labeled and unlabeled documents. In: Proceedings of the Fifteenth National Conference on Artificial Intelligence, pp. 792–799 (1998)Google Scholar
  19. 19.
    Salazar, R., Roepman, P., Capella, G., Moreno, V., Simon, I., Dreezen, C., Lopez-Doriga, A., Santos, C., Marijnen, C., Westerga, J., et al.: Gene expression signature to improve prognosis prediction of stage II and III colorectal cancer. Journal of Clinical Oncology 29(1), 17–24 (2011)CrossRefGoogle Scholar
  20. 20.
    Singh, A., Nowak, R., Zhu, X.: Unlabeled data: Now it helps, now it doesn’t. In: Advances in Neural Information Processing Systems, vol. 21 (2008)Google Scholar
  21. 21.
    Yarowsky, D.: Unsupervised word sense disambiguation rivaling supervised methods. In: Proceedings of the 33rd Annual Meeting on Association for Computational Linguistics, pp. 189–196 (1995)Google Scholar
  22. 22.
    Zhu, X., Goldberg, A.: Introduction to Semi-Supervised Learning. Morgan & Claypool Publishers (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Marco Loog
    • 1
  • Are C. Jensen
    • 2
  1. 1.Pattern Recognition LaboratoryDelft University of TechnologyDelftThe Netherlands
  2. 2.Department of InformaticsUniversity of OsloOsloNorway

Personalised recommendations