Advertisement

Discriminant Analysis

  • Ke-Lin DuEmail author
  • M. N. S. Swamy
Chapter

Abstract

Discriminant analysis plays an important role in statistical pattern recognition. LDA, originally derived by Fisher, is one of the most popular discriminant analysis techniques. Under the assumption that the class distributions are identically distributed Gaussians, LDA is Bayes optimal. Like PCA, LDA is widely applied to image retrieval, face recognition, information retrieval, and pattern recognition. This chapter is dedicated to discriminant analysis.

References

  1. 1.
    Baudat, G., & Anouar, F. (2000). Generalized discriminant analysis using a kernel approach. Neural Computation, 12, 2385–2404.CrossRefGoogle Scholar
  2. 2.
    Belhumeur, P. N., Hespanha, J. P., & Kriegman, D. J. (1997). Eigenfaces vs. fisherface: Recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7), 711–720.Google Scholar
  3. 3.
    Bian, W., & Tao, D. (2014). Asymptotic generalization bound of Fisher’s linear discriminant analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(12), 2325–2337.CrossRefGoogle Scholar
  4. 4.
    Cai, D., He, X., & Han, J. (2008). SRDA: An efficient algorithm for large-scale discriminant analysis. IEEE Transactions on Knowledge and Data Engineering, 20(1), 1–12.CrossRefGoogle Scholar
  5. 5.
    Cevikalp, H., Neamtu, M., Wilkes, M., & Barkana, A. (2005). Discriminative common vectors for face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(1), 4–13.CrossRefGoogle Scholar
  6. 6.
    Chatterjee, C., Roychowdhury, V. P., Ramos, J., & Zoltowski, M. D. (1997). Self-organizing algorithms for generalized eigen-decomposition. IEEE Transactions on Neural Networks, 8(6), 1518–1530.CrossRefGoogle Scholar
  7. 7.
    Chatterjee, C., & Roychowdhury, V. P. (1997). On self-organizing algorithms and networks for class-separability features. IEEE Transactions on Neural Networks, 8(3), 663–678.CrossRefGoogle Scholar
  8. 8.
    Chen, L.-F., Liao, H.-Y. M., Ko, M.-T., Lin, J.-C., & Yu, G.-J. (2000). A new LDA-based face recognition system which can solve the small sample size problem. Pattern Recognition, 33(10), 1713–1726.CrossRefGoogle Scholar
  9. 9.
    Chen, W. S., Yuen, P. C., & Huang, J. (2005). A new regularized linear discriminant analysis methods to solve small sample size problems. International Journal of Pattern Recognition and Artificial Intelligence, 19(7), 917–935.CrossRefGoogle Scholar
  10. 10.
    Ching, W.-K., Chu, D., Liao, L.-Z., & Wang, X. (2012). Regularized orthogonal linear discriminant analysis. Pattern Recognition, 45, 2719–2732.zbMATHCrossRefGoogle Scholar
  11. 11.
    Chu, D., Liao, L.-Z., Ng, M. K.-P., & Wang, X. (2015). Incremental linear discriminant analysis: A fast algorithm and comparisons. IEEE Transactions on Neural Networks and Learning Systems, 26(11), 2716–2735.MathSciNetCrossRefGoogle Scholar
  12. 12.
    Chu, D., & Thye, G. S. (2010). A new and fast implementation for null space based linear discriminant analysis. Pattern Recognition, 43(4), 1373–1379.zbMATHCrossRefGoogle Scholar
  13. 13.
    Clemmensen, L., Hastie, T., Witten, D., & Ersbøll, B. (2011). Sparse discriminant analysis. Technometrics, 53(4), 406–413.MathSciNetCrossRefGoogle Scholar
  14. 14.
    Cruz, C. S., & Dorronsoro, J. R. (1998). A nonlinear discriminant algorithm for feature extraction and data classification. IEEE Transactions on Neural Networks, 9(6), 1370–1376.CrossRefGoogle Scholar
  15. 15.
    Demir, G. K., & Ozmehmet, K. (2005). Online local learning algorithms for linear discriminant analysis. Pattern Recognition Letters, 26, 421–431.CrossRefGoogle Scholar
  16. 16.
    Ding, C., & Li, T. (2007). Adaptive dimension reduction using discriminant analysis and \(K\)-means clustering. In Proceedings of the 24th International Conference on Machine Learning (pp. 521–528). Corvallis, OR.Google Scholar
  17. 17.
    Duchene, L., & Leclerq, S. (1988). An optimal transformation for discriminant and principal component analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 10(6), 978–983.zbMATHCrossRefGoogle Scholar
  18. 18.
    Dufrenois, F., & Noyer, J. C. (2013). Formulating robust linear regression estimation as a one-class LDA criterion: Discriminative hat matrix. IEEE Transactions on Neural Networks and Learning Systems, 24(2), 262–273.CrossRefGoogle Scholar
  19. 19.
    Fisher, R. A. (1936). The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7, 179–188.CrossRefGoogle Scholar
  20. 20.
    Foley, D. H., & Sammon, J. W. (1975). An optimal set of discriminant vectors. IEEE Transactions on Computers, 24(3), 281–289.zbMATHCrossRefGoogle Scholar
  21. 21.
    Friedman, J. H. (1989). Regularized discriminant analysis. Journal of the American Statistical Association, 84(405), 165–175.MathSciNetCrossRefGoogle Scholar
  22. 22.
    Gkalelis, N., Mezaris, V., & Kompatsiaris, I. (2011). Mixture subclass discriminant analysis. IEEE Signal Processing Letters, 18(5), 319–322.CrossRefGoogle Scholar
  23. 23.
    Goldberger, J., Roweis, S., Hinton, G., & Salakhutdinov, R. (2005). Neighbourhood components analysis. In L. K. Saul, Y. Weiss, & L. Bottou (Eds.), Advances in neural information processing systems (Vol. 17, pp. 513–520). Cambridge: MIT Press.Google Scholar
  24. 24.
    Gonzalez, A., & Dorronsoro, J. R. (2007). Natural learning in NLDA networks. Neural Networks, 20, 610–620.zbMATHCrossRefGoogle Scholar
  25. 25.
    Gulmezoglu, M. B., Dzhafarov, V., Keskin, M., & Barkana, A. (1999). A novel approach to isolated word recognition. IEEE Transactions on Speech and Audio Processing, 7(6), 620–628.CrossRefGoogle Scholar
  26. 26.
    Hamsici, O. C., & Martinez, A. M. (2008). Bayes optimality in linear discriminant analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(4), 647–657.CrossRefGoogle Scholar
  27. 27.
    Hastie, T., Tibshirani, R., & Buja, A. (1994). Flexible discriminant analysis by optimal scoring. Journal of the American Statistical Association, 89(428), 1255–1270.MathSciNetzbMATHCrossRefGoogle Scholar
  28. 28.
    Hastie, T., & Tibshirani, R. (1995). Penalized discriminant analysis. Annals of Statistics, 23(1), 73–102.MathSciNetzbMATHCrossRefGoogle Scholar
  29. 29.
    Hastie, T., Tibshirani, R., & Friedman, J. H. (2001). The elements of statistical learning: Data mining, inference, and prediction. New York: Springer.zbMATHCrossRefGoogle Scholar
  30. 30.
    He, X., & Niyogi, P. (2004). Locality preserving projections. In S. Thrun, L. Saul, & B. Scholkopf (Eds.), Advances in neural information processing systems (Vol. 16, pp. 153–160). Cambridge: MIT Press.Google Scholar
  31. 31.
    He, X., Yan, S., Hu, Y., Niyogi, P., & Zhang, H.-J. (2005). Face recognition using Laplacianfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(3), 328–340.CrossRefGoogle Scholar
  32. 32.
    Howland, P., & Park, H. (2004). Generalizing discriminant analysis using the generalized singular value decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(8), 995–1006.CrossRefGoogle Scholar
  33. 33.
    Ji, S., & Ye, J. (2008). Generalized linear discriminant analysis: A unified framework and efficient model selection. IEEE Transactions on Neural Networks, 19(10), 1768–1782.CrossRefGoogle Scholar
  34. 34.
    Jin, Z., Yang, J., Hu, Z., & Lou, Z. (2001). Face recognition based on the uncorrelated discrimination transformation. Pattern Recognition, 34(7), 1405–1416.zbMATHCrossRefGoogle Scholar
  35. 35.
    Kan, M., Shan, S., Zhang, H., Lao, S., & Chen, X. (2016). Multi-view discriminant analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(1), 188–194.CrossRefGoogle Scholar
  36. 36.
    Kim, T.-K., & Kittler, J. (2005). Locally linear discriminant analysis for multimodally distributed classes for face recognition with a single model image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(3), 318–327.CrossRefGoogle Scholar
  37. 37.
    Kong, H., Wang, L., Teoh, E. K., Wang, J. G., & Venkateswarlu, R. (2004). A framework of 2D Fisher discriminant analysis: Application to face recognition with small number of training samples. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) (Vol. 2, pp. 1083–1088). San Diego, CA.Google Scholar
  38. 38.
    Kwak, N., & Lee, J.-W. (2010). Feature extraction based on subspace methods for regression problems. Neurocomputing, 73, 1740–1751.CrossRefGoogle Scholar
  39. 39.
    Kyperountas, M., Tefas, A., & Pitas, I. (2007). Weighted piecewise LDA for solving the small sample size problem in face verification. IEEE Transactions on Neural Networks, 18(2), 506–519.CrossRefGoogle Scholar
  40. 40.
    Li, H., Jiang, T., & Zhang, K. (2006). Efficient and robust feature extraction by maximum margin criterion. IEEE Transactions on Neural Networks, 17(1), 157–165.CrossRefGoogle Scholar
  41. 41.
    Li, M., & Yuan, B. (2005). 2D-LDA: A statistical linear discriminant analysis for image matrix. Pattern Recognition Letters, 26, 527–532.CrossRefGoogle Scholar
  42. 42.
    Li, Z., Lin, D., & Tang, X. (2009). Nonparametric discriminant analysis for face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(4), 755–761.CrossRefGoogle Scholar
  43. 43.
    Liu, C., & Wechsler, H. (2000). Robust coding scheme for indexing and retrieval from large face databases. IEEE Transactions on Image Processing, 9, 132–137.CrossRefGoogle Scholar
  44. 44.
    Liu, L.-P., Jiang, Y., & Zhou, Z.-H. (2009). Least square incremental linear discriminant analysis. In Proceedings of the 9th IEEE International Conference on Data Mining (pp. 298–306). Miami, FL.Google Scholar
  45. 45.
    Loog, M., Duin, R. P. W., & Haeb-Umbach, R. (2001). Multiclass linear dimension reduction by weighted pairwise Fisher criteria. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23, 762–766.CrossRefGoogle Scholar
  46. 46.
    Lu, J., Plataniotis, K. N., & Venetsanopoulos, A. N. (2003). Face recognition using LDA based algorithms. IEEE Transactions on Neural Networks, 14(1), 195–200.CrossRefGoogle Scholar
  47. 47.
    Martinez, A. M., & Kak, A. C. (2001). PCA versus LDA. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23, 228–233.CrossRefGoogle Scholar
  48. 48.
    Michie, D., Spiegelhalter, D. J., & Taylor, C. C. (Eds.). (1994). Machine learning, neural and statistical classification. New York: Ellis Horwood.zbMATHGoogle Scholar
  49. 49.
    Mao, J., & Jain, A. K. (1995). Artificial neural networks for feature extraction and multivariate data projection. IEEE Transactions on Neural Networks, 6(2), 296–317.CrossRefGoogle Scholar
  50. 50.
    Merchante, L., Grandvalet, Y., & Govaert, G. (2012). An efficient approach to sparse linear discriminant analysis. In Proceedings of the 29th International Conference on Machine Learning (pp. 1167-1174). Edinburgh, UK.Google Scholar
  51. 51.
    Moghaddam, B., Weiss, Y., & Avidan, S. (2006). Generalized spectral bounds for sparse LDA. In Proceedings of the 23th International Conference on Machine Learning (pp. 641–648). Pittsburgh, PA.Google Scholar
  52. 52.
    Muller, K. R., Mika, S., Ratsch, G., Tsuda, K., & Scholkopf, B. (2001). An introduction to kernel-based learning algorithms. IEEE Transactions on Neural Networks, 12(2), 181–201.CrossRefGoogle Scholar
  53. 53.
    Na, J. H., Park, M. S., & Choi, J. Y. (2010). Linear boundary discriminant analysis. Pattern Recognition, 43, 929–936.zbMATHCrossRefGoogle Scholar
  54. 54.
    Nagabhushan, P., Guru, D. S., & Shekar, B. H. (2006). (2D)\(^2\) FLD: An efficient approach for appearance based object recognition. Neurocomputing, 69, 934–940.CrossRefGoogle Scholar
  55. 55.
    Nenadic, Z. (2007). Information discriminant analysis: Feature extraction with an information-theoretic objective. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(8), 1394–1407.CrossRefGoogle Scholar
  56. 56.
    Okada, T., & Tomita, S. (1985). An optimal orthonormal system for discriminant analysis. Pattern Recognition, 18(2), 139–144.CrossRefGoogle Scholar
  57. 57.
    Pang, S., Ozawa, S., & Kasabov, N. (2004). One-pass incremental membership authentication by face classification. In D. Zhang & A. K. Jain (Eds.), Biometric authentication (pp. 155–161). New York: Springer.CrossRefGoogle Scholar
  58. 58.
    Peng, J., Zhang, P., & Riedel, N. (2008). Discriminant learning analysis. IEEE Transactions on Systems, Man, and Cybernetics Part B, 38(6), 1614–1625.CrossRefGoogle Scholar
  59. 59.
    Qiu, X., & Wu, L. (2006). Nearest neighbor discriminant analysis. International Journal of Pattern Recognition and Artificial Intelligence, 20(8), 1245–1259.CrossRefGoogle Scholar
  60. 60.
    Qiu, X., & Wu, L. (2007). Two-dimensional nearest neighbor discriminant analysis. Neurocomputing, 70, 2572–2575.CrossRefGoogle Scholar
  61. 61.
    Sharma, A., & Paliwal, K. K. (2008). A gradient linear discriminant analysis for small sample sized problem. Neural Processing Letters, 27, 17–24.CrossRefGoogle Scholar
  62. 62.
    Sharma, A., & Paliwal, K. K. (2008). Rotational linear discriminant analysis technique for dimensionality reduction. IEEE Transactions on Knowledge and Data Engineering, 20(10), 1336–1347.CrossRefGoogle Scholar
  63. 63.
    Skurichina, M., & Duin, R. P. W. (1996). Stabilizing classifiers for very small sample size. In Proceedings of International Conference on Pattern Recognition (pp. 891–896).Google Scholar
  64. 64.
    Srivastava, S., & Gupta, M. R. (2006). Distribution-based Bayesian minimum expected risk for discriminant analysis. In Proceedings of IEEE International Symposium on Information Theory (pp. 2294–2298).Google Scholar
  65. 65.
    Sugiyama, M. (2007). Dimensionality reduction of multimodal labeled data by local Fisher discriminant analysis. Journal of Machine Learning Research, 8, 1027–1061.zbMATHGoogle Scholar
  66. 66.
    Sugiyama, M., Ide, T., Nakajima, S., & Sese, J. (2010). Semi-supervised local Fisher discriminant analysis for dimensionality reduction. Machine Learning, 78, 35–61.MathSciNetCrossRefGoogle Scholar
  67. 67.
    Swets, D. L., & Weng, J. (1996). Using discriminant eigenfeatures for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(8), 831–836.CrossRefGoogle Scholar
  68. 68.
    Wang, F., Zhao, B., & Zhang, C. (2011). Unsupervised large margin discriminative projection. IEEE Transactions on Neural Networks, 22(9), 1446–1456.CrossRefGoogle Scholar
  69. 69.
    Witten, D. M., & Tibshirani, R. (2011). Penalized classification using Fisher’s linear discriminant. Journal of the Royal Statistical Society Series B, 73(5), 753–772.MathSciNetzbMATHCrossRefGoogle Scholar
  70. 70.
    Xiang, C., Fan, X. A., & Lee, T. H. (2006). Face recognition using recursive Fisher linear discriminant. IEEE Transactions on Image Processing, 15(8), 2097–2105.CrossRefGoogle Scholar
  71. 71.
    Xiong, H., Swamy, M. N. S., & Ahmad, M. O. (2005). Two-dimension FLD for face recognition. Pattern Recognition, 38(7), 1121–1124.CrossRefGoogle Scholar
  72. 72.
    Yan, J., Zhang, B., Yan, S., Yan, S., Yang, Q., Li, H., Chen, Z., Xi, W., Fan, W., Ma, W.-Y., & Cheng, Q. (2004). IMMC: Incremental maximum margin criterion. In Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 725–730). Seattle, WA.Google Scholar
  73. 73.
    Ye, J., Janardan, R., Park, C. H., & Park, H. (2004). An optimization criterion for generalized discriminant analysis on undersampled problems. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(8), 982–994.CrossRefGoogle Scholar
  74. 74.
    Ye, J., Janardan, R., & Li, Q. (2004). Two-dimensional linear discriminant analysis. Advances in neural information processing systems (Vol. 17, pp. 1569–1576). Vancouver, Canada.Google Scholar
  75. 75.
    Ye, J., Li, Q., Xiong, H., Park, H., Janardan, R., & Kumar, V. (2005). IDR/QR: An incremental dimension reduction algorithm via QR decomposition. IEEE Transactions on Knowledge and Data Engineering, 17(9), 1208–1222.CrossRefGoogle Scholar
  76. 76.
    Ye, J. (2005). Characterization of a family of algorithms for generalized discriminant analysis on undersampled problems. Journal of Machine Learning Research, 6, 483–502.MathSciNetzbMATHGoogle Scholar
  77. 77.
    Ye, J., & Li, Q. (2005). A two-stage linear discriminant analysis via QR-decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(6), 929–941.CrossRefGoogle Scholar
  78. 78.
    Ye, J. (2005). Generalized low rank approximations of matrices. Machine Learning, 61(1), 167–191.zbMATHCrossRefGoogle Scholar
  79. 79.
    Ye, J. (2007). Least squares linear discriminant analysis. In Proceedings of the 24th International Conference on Machine Learning (pp. 1087–1094). Corvallis, OR.Google Scholar
  80. 80.
    Ye, J., & Xiong, T. (2006). Computational and theoretical analysis of null space and orthogonal linear discriminant analysis. Journal of Machine Learning Research, 7, 1183–1204.MathSciNetzbMATHGoogle Scholar
  81. 81.
    Zaidi, N. A., Webb, G. I., Carman, M. J., Petitjean, F., & Cerquides, J. (2016). ALR\(^n\): Accelerated higher-order logistic regression. Machine Learning, 104(2), 151–194.MathSciNetzbMATHCrossRefGoogle Scholar
  82. 82.
    Zhang, T., Fang, B., Tang, Y. Y., Shang, Z., & Xu, B. (2010). Generalized discriminant analysis: A matrix exponential approach. IEEE Transactions on Systems, Man, and Cybernetics Part B, 40(1), 186–197.CrossRefGoogle Scholar
  83. 83.
    Zhang, H., Guan, C., & Li, Y. (2011). A linear discriminant analysis method based on mutual information maximization. Pattern Recognition, 44(4), 877–885.zbMATHCrossRefGoogle Scholar
  84. 84.
    Zhang, X., Chu, D., & Tan, R. C. E. (2016). Sparse uncorrelated linear discriminant analysis for undersampled problems. IEEE Transactions on Neural Networks and Learning Systems, 27(7), 1469–1485.MathSciNetCrossRefGoogle Scholar
  85. 85.
    Zhang, Y., & Yeung, D.-Y. (2011). Semisupervised generalized discriminant analysis. IEEE Transactions on Neural Networks, 22(8), 1207–1217.CrossRefGoogle Scholar
  86. 86.
    Zhao, H., & Yuen, P. C. (2008). Incremental linear discriminant analysis for face recognition. IEEE Transactions on Systems, Man, and Cybernetics Part B, 38(1), 210–221.CrossRefGoogle Scholar
  87. 87.
    Zheng, W. (2006). Class-incremental generalized discriminant analysis. Neural Computation, 18, 979–1006.MathSciNetzbMATHCrossRefGoogle Scholar
  88. 88.
    Zhu, M., & Martinez, A. M. (2006). Subclass discriminant analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(8), 1274–1286.CrossRefGoogle Scholar
  89. 89.
    Zhu, M., & Martinez, A. M. (2008). Pruning noisy bases in discriminant analysis. IEEE Transactions on Neural Networks, 19(1), 148–157.CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Electrical and Computer EngineeringConcordia UniversityMontrealCanada
  2. 2.Xonlink Inc.HangzhouChina

Personalised recommendations