Skip to main content
SpringerLink
Log in

Adaptive linear discriminant analysis for online feature extraction

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

In this paper, we present new adaptive algorithms for the computation of the square root of the inverse covariance matrix. In contrast to the current similar methods, these new algorithms are obtained from an explicit cost function that is introduced for the first time. The new adaptive algorithms are used in a cascade form with a well-known adaptive principal component analysis to construct linear discriminant features. The adaptive nature and fast convergence rate of the new adaptive linear discriminant analysis algorithms make them appropriate for online pattern recognition applications. All adaptive algorithms discussed in this paper are trained simultaneously using a sequence of random data. Experimental results using the synthetic and real multiclass, multidimensional input data demonstrate the effectiveness of the new adaptive algorithms to extract the optimal features for the purpose of classification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Fukunaga K.: Introduction to Statistical Pattern Recognition, 2nd edn. Academic Press, New York (1990)

    MATH  Google Scholar 

  2. Chen L., Liao H.M., Ko M., Lin J., Yu G.: New LDA based face recognition system which can solve the small sample size problem. Pattern Recognit. 33(10), 1713–1726 (2000)

    Article  Google Scholar 

  3. Lu J., Platantinos K.N., Venetsanopoulos A.N.: Face recognition using LDA-based algorithms. IEEE Trans. Neural Netw. 14(1), 195–200 (2003)

    Article  Google Scholar 

  4. Yu H., Yang J.: A direct LDA algorithm for high-dimensional data with application to face recognition. Pattern Recognit. 34(10), 2067–2070 (2001)

    Article  MATH  Google Scholar 

  5. Koren, Y., Carmel, L.: Visualization of labeled data using linear transformation. In: Proceedings of the Ninth IEEE Conference on Information Visualization pp. 121–128, Oct 2003

  6. Chang C., Ren H.: An Experimented-based quantitative and comparative analysis of target detection and image classification algorithms for hyper-spectral imagery. IEEE Trans. Geosci. Remote Sens. 38(2), 1044–1063 (2000)

    Article  Google Scholar 

  7. Aggarwal, C.C., Han, J., Wang, J., Yu, P.S.: On demand classification of data streams. In: Proceedings of ACM SIGKDD International Conference on Knowledge Discovery Data Mining pp. 503–508, Aug 2004

  8. Swets D.L., Weng J.J.: Using discriminant eigen-feature for image retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 18(8), 831–836 (1996)

    Article  Google Scholar 

  9. Zhao, W., Chellappa, R., Krishnaswamy, A.: Discriminant analysis of principal components for face recognition. In: Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition Nara, Japan, pp. 336–341 (1998)

  10. Belhumeur P.N., Hespanha J.P., Kriegman D.J.: Eigen-faces vs. Fisher-faces: recognition using class specific linear projection. IEEE Trans. Pattern Anal. Mach. Intell. 19, 711–720 (1997)

    Article  Google Scholar 

  11. Martinez A.M., Kak A.C.: PCA versus LDA. IEEE Trans. Pattern Anal. Mach. Intell. 23(2), 228–233 (2001)

    Article  Google Scholar 

  12. Hall, P., Marshall, A.D., Martin, R.: Incremental eigen analysis for classification. In: Proc. Brit. Machine Vision Conf. vol. 1, pp. 286–295 (1998)

  13. Hall P., Marshal A.D., Martin R.: Merging and splitting eigen space models. IEEE Trans. Pattern Anal. Mach. Intell. 22(9), 1042–1049 (2000)

    Article  Google Scholar 

  14. Miao Y., Hua Y.: Fast subspace tracking and neural network learning by a novel information criterion. IEEE Trans. Signal Process. 46, 1964–1979 (1998)

    Article  Google Scholar 

  15. Bannour S., Azimi-Sadjadi M.R.: Principal component extraction using recursive least squares learning. IEEE Trans. Neural Netw. 6, 457–469 (1995)

    Article  Google Scholar 

  16. Xu L.: Least mean square error reconstruction principle for self-organizing neural-net. Neural Netw. 6, 627–648 (1993)

    Article  Google Scholar 

  17. Weng J., Zhang Y., Hwang W.S.: Candid covariance free incremental principal component analysis. IEEE Trans. Pattern Anal. Mach. Intell. 25(8), 1034–1040 (2003)

    Article  Google Scholar 

  18. Oja E., Karhunen J.: On stochastic approximation of eigenvectors and eigen-values of the expectation of a random matrix. J. Math. Anal. Appl. 106, 69–84 (1990)

    Article  MathSciNet  Google Scholar 

  19. Oja E.: Principal components, minor components, and linear neural networks. Neural Netw. 5, 927–935 (1992)

    Article  Google Scholar 

  20. Sanger T.D.: Optimal unsupervised learning in a single-layer linear feed forward neural network. Neural Netw. 2, 459–473 (1989)

    Article  Google Scholar 

  21. Chatetrjee C., Kang A., Roychodhury V.: Algorithms for accelerated convergence of adaptive PCA. IEEE Trans. Neural Netw. 11(2), 338–355 (2000)

    Article  Google Scholar 

  22. Pang S., Ozawa S., Kasabov N.: Incremental linear discriminant analysis for classification of data streams. IEEE Trans. Syst. Man Cybernet. B 35(5), 905–914 (2005)

    Article  Google Scholar 

  23. Kim, T., Wong, S., Stenger, B., Kittler, J., Cipolla, R.: Incremental linear discriminant analysis using sufficient spanning set approximation. In: Proc. IEEE Int. Conf. Computer Vision and Pattern Recognition (CVPR) pp. 1–8, June 2007

  24. Ye, J., Li, Q., Xiong, H., Park, H., Janardan, R., Kumar, V.: IDR/QR: an incremental dimension reduction algorithm via QR decomposition. In: IEEE Trans. Knowl. Data Eng. 17(9), 1208–1222 (2005)

  25. Mao, J., Jain, A.K.: Discriminant analysis neural networks. In: Proc. IEEE Int. Conf. on Neural Networks CA, pp. 300–305 (1993)

  26. Chatterjee C., Roychowdhurry V.P.: On self-organizing algorithm and networks for class separability features. IEEE Trans. Neural Netw. 8(3), 663–678 (1997)

    Article  Google Scholar 

  27. Demir G.K., Ozmehmet K.: On-line local learning algorithm for linear discriminant analysis. Pattern Recognit. Lett. 26, 421–431 (2005)

    Article  Google Scholar 

  28. Abrishami Moghaddam H., Matinfar M., Sajad Sadough S.M., Amiri Zadeh K.: Algorithms and networks for accelerated convergence of adaptive LDA. Pattern Recognit. 38(4), 473–483 (2005)

    Article  MATH  Google Scholar 

  29. Abrishami Moghaddam H., Matinfar M.: Fast adaptive LDA using quasi-Newton algorithm. Pattern Recognit. Lett. 28(4), 613–621 (2007)

    Article  Google Scholar 

  30. Rao Y., Principe N., Wong J.C.: Fast RLS like algorithm for generalized eigen decomposition and its applications. J. VLSI Signal Process. Syst. 37(3), 333–344 (2004)

    Article  MATH  Google Scholar 

  31. Aliyari Ghassabeh Y., Abrishami Moghaddam H.: Adaptive algorithms and networks for optimal feature extraction from Gaussian data. Pattern Recognit. Lett. 31, 1331–1341 (2010)

    Article  Google Scholar 

  32. Hongo, H., Yasumoto, N., Niva, Y., Yamamoto, K.: Hierarchical face recognition using an adaptive discriminant space. In: Proc. IEEE Int. Conf. Comput. Commun. Control Power Eng. (TENCON) 1, 523–528 (2002)

  33. Ozawa S., Toh S.L., Abe S., Pang S., Kasabov N.: Incremental learning of feature space and classifier for face recognition. Neural Netw. 18, 575–584 (2005)

    Article  Google Scholar 

  34. Theodoridis S.: Pattern Recognition, 2nd edn. Academic Press, New York (2003)

    Google Scholar 

  35. Woelfel M., McDonough J.: Distant Speech Recognition. Wiley, New York (2009)

    Book  Google Scholar 

  36. Ljung L.: Analysis of recursive stochastic algorithms. IEEE Trans. Autom. Control 22, 551–575 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  37. Diamantaras K., Kung S.: Principal Component Neural Networks, Theory and Applications. Wiley, New York (1996)

    Google Scholar 

  38. Magnus J.R., Neudecker H.: Matrix Differential Calculus. Wiley, New York (1999)

    MATH  Google Scholar 

  39. Kushner H.J., Clarck D.S.: Stochastic Approximation Methods for Constrained and Unconstrained Systems. Springer, New York (1978)

    Book  Google Scholar 

  40. Benveniste A., Metivier M., Priouret P.: Adaptive Algorithms and Stochastic Approximations, 2nd edn. Academic Press, New York (1990)

    Book  MATH  Google Scholar 

  41. Okada T., Tomita S.: An optimal orthonormal system for discriminant analysis. Pattern Recognit. 18(2), 139–144 (1985)

    Article  Google Scholar 

  42. Hruschka E.R., Ebecken N.F.: Extracting rules from multilayer perceptrons in classification problems: a clustering-based approach. Neurocomputing 70, 384–397 (2006)

    Article  Google Scholar 

  43. Wang, C.H., Chi, Y.Y.: Dynamic optimal training of a three layer neural network with sigmoid function. In: Proc. IEEE Int. Conf. on Networking, Sensing and Control Fort Lauderdale, USA, pp. 392–397, Aug. 2006

  44. http://en.wikipedia.org/wiki/Iris_flower_data_set

  45. Georghiades A.S., Belhumeur P.N., Kriegman D.J.: From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 643–660 (2001)

    Article  Google Scholar 

  46. Rajkumar, N., Vijayakumar, S., Murukesh, C.: Intellectually combined face recognition using curvelet based principle component analysis for feature extraction and Bayesian classifier. In: Proc. IEEE Int. Conf. on Signal Processing, Communication, Computing and Networking Technologies (ICSCCN) Chennai, India, pp. 374–378, Sep. 2011

  47. Turk M., Pentland A.: Eigenfaces for face recognition. J. Cognit. Neurosci. 3(1), 71–86 (1991)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Queen’s University, Kingston, ON, Canada

    Youness Aliyari Ghassabeh & Hamid Abrishami Moghaddam

Authors
  1. Youness Aliyari Ghassabeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hamid Abrishami Moghaddam
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Youness Aliyari Ghassabeh.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Aliyari Ghassabeh, Y., Abrishami Moghaddam, H. Adaptive linear discriminant analysis for online feature extraction. Machine Vision and Applications 24, 777–794 (2013). https://doi.org/10.1007/s00138-012-0439-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-012-0439-z

Keywords

Search

Navigation