Advertisement

Principal Component Analysis

  • Ke-Lin DuEmail author
  • M. N. S. Swamy
Chapter

Abstract

Subspace learning techniques project high-dimensional data onto low-dimensional spaces. They are typically unsupervised. Well-known subspace learning algorithms are PCA, ICA, locality-preserving projection, and NMF. Discriminant analysis is a supervised subspace learning method and uses the data class label information. PCA is a classical statistical method for signal processing and data analysis. It is a feature extractor in the neural network processing setting, and is related to eigenvalue decomposition and singular value decomposition. This chapter introduces PCA, and the associated methods such as minor component analysis, generalized eigenvalue decomposition, singular value decomposition, factor analysis, and canonical correlation analysis.

References

  1. 1.
    Abed-Meraim, K., Attallah, S., Chkeif, A., & Hua, Y. (2000). Orthogonal Oja algorithm. IEEE Signal Processing Letters, 7(5), 116–119.CrossRefGoogle Scholar
  2. 2.
    Attallah, S., & Abed-Meraim, K. (2001). Fast algorithms for subspace tracking. IEEE Signal Processing Letters, 8(7), 203–206.CrossRefGoogle Scholar
  3. 3.
    Bannour, S., & Azimi-Sadjadi, M. R. (1995). Principal component extraction using recursive least squares learning. IEEE Transactions on Neural Networks, 6(2), 457–469.CrossRefGoogle Scholar
  4. 4.
    Bartlett, M. S. (1938). Further aspects of the theory of multiple regression. Proceedings of the Cambridge Philosophical Society, 34, 33–40.zbMATHCrossRefGoogle Scholar
  5. 5.
    Belohlavek, R., & Vychodil, V. (2010). Discovery of optimal factors in binary data via a novel method of matrix decomposition. Journal of Computer and System Sciences, 76(1), 3–20.MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Bourlard, H., & Kamp, Y. (1988). Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 59, 291–294.MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Cadima, J., & Jolliffe, I. (1995). Loadings and correlations in the interpretation of principal component analysis. Journal of Applied Statistics, 22(2), 203–214.MathSciNetCrossRefGoogle Scholar
  8. 8.
    Chatterjee, C., Roychowdhury, V. P., & Chong, E. K. P. (1998). On relative convergence properties of principal component analysis algorithms. IEEE Transactions on Neural Networks, 9(2), 319–329.CrossRefGoogle Scholar
  9. 9.
    Chauvin, Y. (1989). Principal component analysis by gradient descent on a constrained linear Hebbian cell. In Proceedings of the International Joint Conference on Neural Networks (pp. 373–380). Wanshington, DC.Google Scholar
  10. 10.
    Chatterjee, C., Roychowdhury, V. P., Ramos, J., & Zoltowski, M. D. (1997). Self-organizing algorithms for generalized eigen-decomposition. IEEE Transactions on Neural Networks, 8(6), 1518–1530.CrossRefGoogle Scholar
  11. 11.
    Chen, H., & Liu, R. W. (1994). An on-line unsupervised learning machine for adaptive feature extraction. IEEE Transactions on Circuits and Systems II, 41(2), 87–98.CrossRefGoogle Scholar
  12. 12.
    Chen, L. H., & Chang, S. (1995). An adaptive learning algorithm for principal component analysis. IEEE Transactions on Neural Networks, 6(5), 1255–1263.CrossRefGoogle Scholar
  13. 13.
    Chen, S., & Sun, T. (2005). Class-information-incorporated principal component analysis. Neurocomputing, 69, 216–223.CrossRefGoogle Scholar
  14. 14.
    Chen, T., Amari, S. I., & Lin, Q. (1998). A unified algorithm for principal and minor components extraction. Neural Networks, 11, 385–390.CrossRefGoogle Scholar
  15. 15.
    Chen, T., Amari, S. I., & Murata, N. (2001). Sequential extraction of minor components. Neural Processing Letters, 13, 195–201.zbMATHCrossRefGoogle Scholar
  16. 16.
    Chen, W., Er, M. J., & Wu, S. (2005). PCA and LDA in DCT domain. Pattern Recognition Letters, 26, 2474–2482.CrossRefGoogle Scholar
  17. 17.
    Chen, Y., & Hou, C. (1992). High resolution adaptive bearing estimation using a complex-weighted neural network. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) (Vol. 2, pp. 317–320). San Francisco, CA.Google Scholar
  18. 18.
    Cichocki, A., Swiniarski, R. W., & Bogner, R. E. (1996). Hierarchical neural network for robust PCA computation of complex valued signals. In Proceedings of the World Congress Neural Networks (pp. 818–821). San Diego, CA.Google Scholar
  19. 19.
    Collins, M., Dasgupta, S., & Schapire, R. E. (2002). A generalization of principal component analysis to the exponential family. In T. D. Dietterich, S. Becker, & Z. Ghahramani (Eds.), Advances in neural information processing systems (Vol. 14, pp. 617–624). Cambridge, MA: MIT Press.Google Scholar
  20. 20.
    d’Aspremont, A., Bach, F., & El Ghaoui, L. (2008). Optimal solutions for sparse principal component analysis. Journal of Machine Learning Research, 9, 1269–1294.MathSciNetzbMATHGoogle Scholar
  21. 21.
    Demir, G. K., & Ozmehmet, K. (2005). Online local learning algorithms for linear discriminant analysis. Pattern Recognition Letters, 26, 421–431.CrossRefGoogle Scholar
  22. 22.
    Diamantaras, K. I., & Kung, S. Y. (1994). Cross-correlation neural network models. IEEE Transactions on Signal Processing, 42(11), 3218–3323.CrossRefGoogle Scholar
  23. 23.
    Diamantaras, K. I., & Kung, S.-Y. (1994). Multilayer neural networks for reduced-rank approximation. IEEE Transactions on Neural Networks, 5(5), 684–697.CrossRefGoogle Scholar
  24. 24.
    Diamantaras, K. I., & Kung, S. Y. (1996). Principal component neural networks: Theory and applications. New York: Wiley.zbMATHGoogle Scholar
  25. 25.
    Douglas, S. C., Kung, S., & Amari, S. (1998). A self-stabilized minor subspace rule. IEEE Signal Processing Letters, 5(12), 328–330.CrossRefGoogle Scholar
  26. 26.
    Du, K.-L., & Swamy, M. N. S. (2004). Simple and practical cyclostationary beamforming algorithms. IEE Proceedings - Vision, Image and Signal Processing, 151(3), 175–179.CrossRefGoogle Scholar
  27. 27.
    Feng, D.-Z., Bao, Z., & Shi, W.-X. (1998). Cross-correlation neural network model for the smallest singular component of general matrix. Signal Processing, 64, 333–346.zbMATHCrossRefGoogle Scholar
  28. 28.
    Feng, D.-Z., Bao, Z., & Jiao, L.-C. (1998). Total least mean squares algorithm. IEEE Transactions on Signal Processing, 46(8), 2122–2130.CrossRefGoogle Scholar
  29. 29.
    Feng, D.-Z., Bao, Z., & Zhang, X.-D. (2001). A crossassociative neural network for SVD of nonsquared data matrix in signal processing. IEEE Transactions on Neural Networks, 12(5), 1215–1221.CrossRefGoogle Scholar
  30. 30.
    Feng, D.-Z., Zhang, X.-D., & Bao, Z. (2004). A neural network learning for adaptively extracting crosscorrelation features between two high-dimensional data streams. IEEE Transactions on Neural Networks, 15(6), 1541–1554.CrossRefGoogle Scholar
  31. 31.
    Feng, D.-Z., Zheng, W.-X., & Jia, Y. (2005). Neural network learning algorithms for tracking minor subspace in high-dimensional data stream. IEEE Transactions on Neural Networks, 16(3), 513–521.CrossRefGoogle Scholar
  32. 32.
    Feng, X., Kong, X., Duan, Z., & Ma, H. (2016). Adaptive generalized eigenpairs extraction algorithms and their convergence analysis. IEEE Transactions on Signal Processing, 64(11), 2976–2989.MathSciNetzbMATHCrossRefGoogle Scholar
  33. 33.
    Feng, X., Kong, X., Ma, H., & Si, X. (2017). A novel unified and self-stabilizing algorithm for generalized eigenpairs extraction. IEEE Transactions on Neural Networks and Learning Systems, 28(12), 3032–3044.MathSciNetCrossRefGoogle Scholar
  34. 34.
    Fiori, S., & Piazza, F. (1998). A general class of \(\psi \)-APEX PCA neural algorithms. IEEE Transactions on Circuits and Systems I, 47(9), 1394–1397.Google Scholar
  35. 35.
    Foldiak, P. (1989). Adaptive network for optimal linear feature extraction. In Proceedings of the International Joint Conference on Neural Networks (IJCNN) (Vol. 1, pp. 401–405). Washington, DC.Google Scholar
  36. 36.
    Fowler, J. E. (2009). Compressive-projection principal component analysis. IEEE Transactions on Image Processing, 18(10), 2230–2242.MathSciNetzbMATHCrossRefGoogle Scholar
  37. 37.
    Fu, Z., & Dowling, E. M. (1995). Conjugate gradient eigenstructure tracking for adaptive spectral estimation. IEEE Transactions on Signal Processing, 43(5), 1151–1160.CrossRefGoogle Scholar
  38. 38.
    Gao, K., Ahmad, M. O., & Swamy, M. N. S. (1992). A modified Hebbian rule for total least-squares estimation with complex valued arguments. In Proceedings of IEEE International Symposium on Circuits and Systems (pp. 1231–1234). San Diego, CA.Google Scholar
  39. 39.
    Gao, K., Ahmad, M. O., & Swamy, M. N. S. (1994). A constrained anti-Hebbian learning algorithm for total least-square estimation with applications to adaptive FIR and IIR filtering. IEEE Transactions on Circuits and Systems II, 41(11), 718–729.CrossRefGoogle Scholar
  40. 40.
    Golub, G. H., & van Loan, C. F. (1989). Matrix computation (2nd ed.). Baltimore, MD: John Hopkins University Press.zbMATHGoogle Scholar
  41. 41.
    Gottumukkal, R., & Asari, V. K. (2004). An improved face recognition technique based on modular PCA approach. Pattern Recognition Letters, 25(4), 429–436.CrossRefGoogle Scholar
  42. 42.
    Hall, P., & Martin, R. (1998). Incremental eigenanalysis for classification. In Proceedings of British Machine Vision Conference, (Vol. 1, pp. 286–295).Google Scholar
  43. 43.
    Hardoon, D. R., & Shawe-Taylor, J. (2011). Sparse canonical correlation analysis. Machine Learning, 83, 331–353.MathSciNetzbMATHCrossRefGoogle Scholar
  44. 44.
    Hassoun, M. H. (1995). Fundamentals of artificial neural networks. Cambridge, MA: MIT Press.zbMATHGoogle Scholar
  45. 45.
    Hastie, T., & Stuetzle, W. (1989). Principal curves. Journal of the American Statistical Association, 84, 502–516.MathSciNetzbMATHCrossRefGoogle Scholar
  46. 46.
    Hastie, T., Buja, A., & Tibshirani, R. (1995). Penalized discriminant analysis. Annals of Statistics, 23(1), 73–102.MathSciNetzbMATHCrossRefGoogle Scholar
  47. 47.
    Hebb, D. O. (1949). The organization of behavior. New York: Wiley.Google Scholar
  48. 48.
    Hegde, A., Principe, J. C., Erdogmus, D., & Ozertem, U. (2006). Perturbation-based eigenvector updates for on-line principal components analysis and canonical correlation analysis. Journal of VLSI Signal Processing, 45, 85–95.CrossRefGoogle Scholar
  49. 49.
    Horel, J. D. (1984). Complex principal component analysis: Theory and examples. Journal of Applied Meteorology and Climatology, 23, 1660–1673.CrossRefGoogle Scholar
  50. 50.
    Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24, 417–441.zbMATHCrossRefGoogle Scholar
  51. 51.
    Hotelling, H. (1936). Relations between two sets of variates. Biometrika, 28, 321–377.zbMATHCrossRefGoogle Scholar
  52. 52.
    Hoyle, D. C. (2008). Automatic PCA dimension selection for high dimensional data and small sample sizes. Journal of Machine Learning Research, 9, 2733–2759.zbMATHGoogle Scholar
  53. 53.
    Hsieh, W. W. (2007). Nonlinear principal component analysis of noisy data. Neural Networks, 20, 434–443.zbMATHCrossRefGoogle Scholar
  54. 54.
    Jankovic, M., & Ogawa, H. (2003). A new modulated hebb learning rule-Biologically plausible method for local computation of principal subspace. International Journal of Neural Systems, 13(4), 215–224.CrossRefGoogle Scholar
  55. 55.
    Jankovic, M., & Ogawa, H. (2004). Time-oriented hierarchical method for computation of principal components using subspace learning algorithm. International Journal of Neural Systems, 14(5), 313–323.CrossRefGoogle Scholar
  56. 56.
    Jankovic, M. V., & Ogawa, H. (2006). Modulated Hebb-Oja learning rule-A method for principal subspace analysis. IEEE Transactions on Neural Networks, 17(2), 345–356.CrossRefGoogle Scholar
  57. 57.
    Johnstone, I. M., & Lu, A. (2009). On consistency and sparsity for principal components analysis in high dimensions. Journal of the American Statistical Association, 104(486), 682–693.MathSciNetzbMATHCrossRefGoogle Scholar
  58. 58.
    Jolliffe, I., & Uddin, M. (2003). A modified principal component technique based on the lasso. Journal of Computational and Graphical Statistics, 12(3), 531–547.MathSciNetCrossRefGoogle Scholar
  59. 59.
    Journee, M., Nesterov, Y., Richtarik, P., & Sepulchre, R. (2010). Generalized power method for sparse principal component analysis. Journal of Machine Learning Research, 11, 517–553.MathSciNetzbMATHGoogle Scholar
  60. 60.
    Kaiser, A., Schenck, W., & Moller, R. (2010). Coupled singular value decomposition of a cross-covariance matrix. International Journal of Neural Systems, 20(4), 293–318.CrossRefGoogle Scholar
  61. 61.
    Kambhatla, N., & Leen, T. K. (1993). Fast non-linear dimension reduction. In Proceedings of IEEE International Conference on Neural Networks (Vol. 3, pp. 1213–1218). San Francisco, CA.Google Scholar
  62. 62.
    Kang, Z., Chatterjee, C., & Roychowdhury, V. P. (2000). An adaptive quasi-Newton algorithm for eigensubspace estimation. IEEE Transactions on Signal Processing, 48(12), 3328–3333.MathSciNetzbMATHCrossRefGoogle Scholar
  63. 63.
    Karasuyama, M., & Sugiyama, M. (2012). Canonical dependency analysis based on squared-loss mutual information. Neural Networks, 34, 46–55.zbMATHCrossRefGoogle Scholar
  64. 64.
    Karhunen, J., & Joutsensalo, J. (1995). Generalizations of principal component analysis, optimization problems, and neural networks. Neural Networks, 8(4), 549–562.CrossRefGoogle Scholar
  65. 65.
    Kettenring, J. R. (1971). Canonical analysis of several sets of variables. Biometrika, 58(3), 433–451.MathSciNetzbMATHCrossRefGoogle Scholar
  66. 66.
    Klami, A., Virtanen, S., Leppaaho, E., & Kaski, S. (2015). Group factor analysis. IEEE Transactions on Neural Networks and Learning Systems, 26(9), 2136–2147.MathSciNetCrossRefGoogle Scholar
  67. 67.
    Kong, X., Hu, C., & Han, C. (2010). On the discrete-time dynamics of a class of self-stabilizing MCA extraction algorithms. IEEE Transactions on Neural Networks, 21(1), 175–181.CrossRefGoogle Scholar
  68. 68.
    Kramer, M. A. (1991). Nonlinear principal component analysis using autoassociative neural networks. AIChE Journal, 37(2), 233–243.CrossRefGoogle Scholar
  69. 69.
    Kukharev, G., & Kamenskaya, E. (2010). Application of two-dimensional canonical correlation analysis for face image processing and recognition. Pattern Recognition and Image Analysis, 20(2), 210–219.CrossRefGoogle Scholar
  70. 70.
    Kung, S. Y. (1990). Constrained principal component analysis via an orthogonal learning network. In Proceedings of the IEEE International Symposium on Circuits and Systems (Vol. 1, pp. 719–722). New Orleans, LA.Google Scholar
  71. 71.
    Kung, S. Y., & Diamantaras, K. I. (1990). A neural network learning algorithm for adaptive principal components extraction (APEX). In Proceedings of International Conference on Acoustics, Speech, and Signal Processing (ICASSP) (pp. 861–864). Albuquerque, NM.Google Scholar
  72. 72.
    Kung, S. Y., Diamantaras, K. I., & Taur, J. S. (1994). Adaptive principal components extraction (APEX) and applications. IEEE Transactions on Signal Processing, 42(5), 1202–1217.CrossRefGoogle Scholar
  73. 73.
    Kursun, O., Alpaydin, E., & Favorov, O. V. (2011). Canonical correlation analysis using within-class coupling. Pattern Recognition Letters, 32, 134–144.CrossRefGoogle Scholar
  74. 74.
    Kwak, N. (2008). Principal component analysis based on L1-norm maximization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(9), 1672–1680.CrossRefGoogle Scholar
  75. 75.
    Kwok, J. T., & Zhao, H. (2003). Incremental eigendecomposition. In Proceedings of International Conference on Artificial Neural Networks (ICANN) (pp. 270–273). Istanbul, Turkey.Google Scholar
  76. 76.
    Leung, A. C. S., Wong, K. W., & Tsoi, A. C. (1997). Recursive algorithms for principal component extraction. Network, 8, 323–334.zbMATHCrossRefGoogle Scholar
  77. 77.
    Li, X., Pang, Y., & Yuan, Y. (2010). L1-norm-based 2DPCA. IEEE Transactions on Systems, Man, and Cybernetics Part B, 40(4), 1170–1175.CrossRefGoogle Scholar
  78. 78.
    Linsker, R. (1986). From basic network principles to neural architecture. Proceedings of the National Academy of Sciences of the USA, 83, 7508–7512, 8390–8394, 9779–8783CrossRefGoogle Scholar
  79. 79.
    Linsker, R. (1988). Self-organization in a perceptual network. IEEE Computer, 21(3), 105–117.CrossRefGoogle Scholar
  80. 80.
    Liu, Y., You, Z., & Cao, L. (2005). A simple functional neural network for computing the largest and smallest eigenvalues and corresponding eigenvectors of a real symmetric matrix. Neurocomputing, 67, 369–383.CrossRefGoogle Scholar
  81. 81.
    Ljung, L. (1977). Analysis of recursive stochastic algorithm. IEEE Transactions on Automatic Control, 22, 551–575.MathSciNetzbMATHCrossRefGoogle Scholar
  82. 82.
    Loeve, M. (1963). Probability theory (3rd ed.). New York: Van Nostrand.zbMATHGoogle Scholar
  83. 83.
    Lu, H., Plataniotis, K. N., & Venetsanopoulos, A. N. (2008). MPCA: Multilinear principal component analysis of tensor objects. IEEE Transactions on Neural Networks, 19(1), 18–39.CrossRefGoogle Scholar
  84. 84.
    Lu, H., Plataniotis, K. N. K., & Venetsanopoulos, A. N. (2009). Uncorrelated multilinear principal component analysis for unsupervised multilinear subspace learning. IEEE Transactions on Neural Networks, 20(11), 1820–1836.CrossRefGoogle Scholar
  85. 85.
    Lucchese, C., Orlando, S., & Perego, R. (2014). A unifying framework for mining approximate top-k binary patterns. IEEE Transactions on Knowledge and Data Engineering, 26(12), 2900–2913.CrossRefGoogle Scholar
  86. 86.
    Lucke, J., & Sahani, M. (2008). Maximal causes for non-linear component extraction. Journal of Machine Learning Research, 9, 1227–1267.MathSciNetzbMATHGoogle Scholar
  87. 87.
    Lv, J. C., Yi, Z., & Tan, K. K. (2007). Global convergence of GHA learning algorithm with nonzero-approaching adaptive learning rates. IEEE Transactions on Neural Networks, 18(6), 1557–1571.CrossRefGoogle Scholar
  88. 88.
    Mao, J., & Jain, A. K. (1995). Artificial neural networks for feature extraction and multivariate data projection. IEEE Transactions on Neural Networks, 6(2), 296–317.CrossRefGoogle Scholar
  89. 89.
    Mathew, G., & Reddy, V. U. (1996). A quasi-Newton adaptive algorithm for generalized symmetric eigenvalue problem. IEEE Transactions on Signal Processing, 44(10), 2413–2422.CrossRefGoogle Scholar
  90. 90.
    Mathew, G., Reddy, V. U., & Dasgupta, S. (1995). Adaptive estimation of eigensubspace. IEEE Transactions on Signal Processing, 43(2), 401–411.CrossRefGoogle Scholar
  91. 91.
    Miao, Y., & Hua, Y. (1998). Fast subspace tracking and neural network learning by a novel information criterion. IEEE Transactions on Signal Processing, 46(7), 1967–1979.CrossRefGoogle Scholar
  92. 92.
    Moller, R., & Hoffmann, H. (2004). An extension of neural gas to local PCA. Neurocomputing, 62, 305–326.CrossRefGoogle Scholar
  93. 93.
    Moller, R., & Konies, A. (2004). Coupled principal component analysis. IEEE Transactions on Neural Networks, 15(1), 214–222.CrossRefGoogle Scholar
  94. 94.
    Moller, R. (2006). First-order approximation of Gram-Schmidt orthonormalization beats deflation in coupled PCA learning rules. Neurocomputing, 69, 1582–1590.CrossRefGoogle Scholar
  95. 95.
    Morgan, D. R. (2004). Adaptive algorithms for solving generalized eigenvalue signal enhancement problems. Signal Processing, 84(6), 957–968.zbMATHCrossRefGoogle Scholar
  96. 96.
    Nguyen, T. D., & Yamada, I. (2013). Adaptive normalized quasi-Newton algorithms for extraction of generalized eigen-pairs and their convergence analysis. IEEE Transactions on Signal Processing, 61(6), 1404–1418.MathSciNetzbMATHCrossRefGoogle Scholar
  97. 97.
    Oja, E. (1982). A simplified neuron model as a principal component analyzer. Journal of Mathematical Biology, 15, 267–273.MathSciNetzbMATHCrossRefGoogle Scholar
  98. 98.
    Oja, E. (1992). Principal components, minor components, and linear neural networks. Neural Networks, 5, 929–935.CrossRefGoogle Scholar
  99. 99.
    Oja, E., & Karhunen, J. (1985). On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix. Journal of Mathematical Analysis and Applications, 104, 69–84.MathSciNetzbMATHCrossRefGoogle Scholar
  100. 100.
    Oja, E., Ogawa, H., & Wangviwattana, J. (1992). Principal component analysis by homogeneous neural networks. IEICE Transactions on Information and Systems, E75-D, 366–382.Google Scholar
  101. 101.
    Ouyang, S., Bao, Z., & Liao, G. (1999). Adaptive step-size minor component extraction algorithm. Electronics Letters, 35(6), 443–444.CrossRefGoogle Scholar
  102. 102.
    Ouyang, S., Bao, Z., & Liao, G. (2000). Robust recursive least squares learning algorithm for principal component analysis. IEEE Transactions on Neural Networks, 11(1), 215–221.CrossRefGoogle Scholar
  103. 103.
    Ouyang, S., Bao, Z., Liao, G. S., & Ching, P. C. (2001). Adaptive minor component extraction with modular structure. IEEE Transactions on Signal Processing, 49(9), 2127–2137.CrossRefGoogle Scholar
  104. 104.
    Ouyang, S., & Bao, Z. (2002). Fast principal component extraction by a weighted information criterion. IEEE Transactions on Signal Processing, 50(8), 1994–2002.MathSciNetzbMATHCrossRefGoogle Scholar
  105. 105.
    Ouyang, S., Ching, P. C., & Lee, T. (2003). Robust adaptive quasi-Newton algorithms for eigensubspace estimation. IEE Proceedings—Vision, Image and Signal Processing, 150(5), 321–330.CrossRefGoogle Scholar
  106. 106.
    Ozawa, S., Pang, S., & Kasabov, N. (2008). Incremental learning of chunk data for online pattern classification systems. IEEE Transactions on Neural Networks, 19(6), 1061–1074.CrossRefGoogle Scholar
  107. 107.
    Park, M. S., & Choi, J. Y. (2009). Theoretical analysis on feature extraction capability of class-augmented PCA. Pattern Recognition, 42, 2353–2362.zbMATHCrossRefGoogle Scholar
  108. 108.
    Pearlmutter, B. A., & Hinton, G. E. (1986). G-maximization: An unsupervised learning procedure for discovering regularities. In J. S. Denker (Ed.), AIP Conference Proceedings on Neural Networks for Computing (Vol. 151, pp. 333–338). Snowbird, UT: American Institute of Physics.CrossRefGoogle Scholar
  109. 109.
    Peng, D., Yi, Z., & Luo, W. (2007). Convergence analysis of a simple minor component analysis algorithm. Neural Networks, 20, 842–850.zbMATHCrossRefGoogle Scholar
  110. 110.
    Peng, D., Yi, Z., Lv, J. C., & Xiang, Y. (2008). A neural networks learning algorithm for minor component analysis and its convergence analysis. Neurocomputing, 71, 1748–1752.CrossRefGoogle Scholar
  111. 111.
    Rao, Y. N., Principe, J. C., & Wong, T. F. (2004). Fast RLS-like algorithm for generalized eigendecomposition and its applications. Journal of VLSI Signal Processing, 37, 333–344.zbMATHCrossRefGoogle Scholar
  112. 112.
    Rattan, S. S. P., & Hsieh, W. W. (2005). Complex-valued neural networks for nonlinear complex principal component analysis. Neural Networks, 18, 61–69.zbMATHCrossRefGoogle Scholar
  113. 113.
    Ren, C.-X., & Dai, D.-Q. (2010). Incremental learning of bidirectional principal components for face recognition. Pattern Recognition, 43, 318–330.zbMATHCrossRefGoogle Scholar
  114. 114.
    Ritter, H. (1995). Self-organizing feature maps: Kohonen maps. In M. A. Arbib (Ed.), The handbook of brain theory and neural networks (pp. 846–851). Cambridge, MA: MIT Press.Google Scholar
  115. 115.
    Robbins, H., & Monro, S. (1951). A stochastic approximation method. Annals of Mathematical Statistics, 22(3), 400–407.MathSciNetzbMATHCrossRefGoogle Scholar
  116. 116.
    Rubner, J., & Schulten, K. (1990). Development of feature detectors by self-organization. Biological Cybernetics, 62, 193–199.CrossRefGoogle Scholar
  117. 117.
    Rubner, J., & Tavan, P. (1989). A self-organizing network for principal-component analysis. Europhysics Letters, 10, 693–698.CrossRefGoogle Scholar
  118. 118.
    Rupnik, J., & Shawe-Taylor, J. (2010). Multi-view canonical correlation analysis. In Proceedings of Slovenian KDD Conference on Data Mining and Data Warehouses (SiKDD) (pp. 1–4).Google Scholar
  119. 119.
    Saegusa, R., Sakano, H., & Hashimoto, S. (2004). Nonlinear principal component analysis to preserve the order of principal components. Neurocomputing, 61, 57–70.CrossRefGoogle Scholar
  120. 120.
    Sanger, T. D. (1989). Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural Networks, 2, 459–473.CrossRefGoogle Scholar
  121. 121.
    Sanger, T. D. (1994). Two iterative algorithms for computing the singular value decomposition from input/output samples. In J. D. Cowan, G. Tesauro, & J. Alspector (Eds.), Advances in neural information processing systems (Vol. 6, pp. 144–151). San Francisco, CA: Morgan Kaufmann.Google Scholar
  122. 122.
    Spearman, C. (1904). General intelligence, objectively determined and measured. American Journal of Psychology, 15, 201–293.CrossRefGoogle Scholar
  123. 123.
    Sriperumbudur, B. K., Torres, D. A., & Lanckriet, G. R. G. (2011). A majorization-minimization approach to the sparse generalized eigenvalue problem. Machine Learning, 85, 3–39.MathSciNetzbMATHCrossRefGoogle Scholar
  124. 124.
    Sun, T., & Chen, S. (2007). Locality preserving CCA with applications to data visualization and pose estimation. Image and Vision Computing, 25, 531–543.CrossRefGoogle Scholar
  125. 125.
    Sun, L., Ji, S., & Ye, J. (2011). Canonical correlation analysis for multilabel classification: A least-squares formulation, extensions, and analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1), 194–200.CrossRefGoogle Scholar
  126. 126.
    Tang, Y., & Li, J. (2010). Notes on "Recurrent neural network model for computing largest and smallest generalized eigenvalue". Neurocomputing, 73, 1006–1012.CrossRefGoogle Scholar
  127. 127.
    Tipping, M. E., & Bishop, C. M. (1999). Mixtures of probabilistic principal component analyzers. Neural Computation, 11, 443–482.CrossRefGoogle Scholar
  128. 128.
    Tsagkarakis, N., Markopoulos, P. P., Sklivanitis, G., & Pados, D. A. (2018). \(L_1\)-norm principal-component analysis of complex data. IEEE Transactions on Signal Processing, 66(12), 3256–3267.MathSciNetzbMATHCrossRefGoogle Scholar
  129. 129.
    Ulfarsson, M. O., & Solo, V. (2008). Sparse variable PCA using geodesic steepest descent. IEEE Transactions on Signal Processing, 56(12), 5823–5832.MathSciNetzbMATHCrossRefGoogle Scholar
  130. 130.
    Ulfarsson, M. O., & Solo, V. (2011). Vector \(l_0\) sparse variable PCA. IEEE Transactions on Signal Processing, 59(5), 1949–1958.MathSciNetzbMATHCrossRefGoogle Scholar
  131. 131.
    Valizadeh, A., & Karimi, M. (2009). Fast subspace tracking algorithm based on the constrained projection approximation. EURASIP Journal on Advances in Signal Processing, 2009, Article ID 576972, 16 pages.Google Scholar
  132. 132.
    Warmuth, M. K., & Kuzmin, D. (2008). Randomized online PCA algorithms with regret bounds that are logarithmic in the dimension. Journal of Machine Learning Research, 9, 2287–2320.MathSciNetzbMATHGoogle Scholar
  133. 133.
    Wang, L., & Karhunen, J. (1996). A simplified neural bigradient algorithm for robust PCA and MCA. International Journal of Neural Systems, 7(1), 53–67.CrossRefGoogle Scholar
  134. 134.
    Wang, H. (2010). Local two-dimensional canonical correlation analysis. IEEE Signal Processing Letters, 17(11), 921–924.CrossRefGoogle Scholar
  135. 135.
    Weng, J., Zhang, Y., & Hwang, W.-S. (2003). Candid covariance-free incremental principal component analysis. IEEE Transactions on Pattern Analysis Machine Intelligence, 25(8), 1034–1040.CrossRefGoogle Scholar
  136. 136.
    Wu, J., & Zhou, Z. H. (2002). Face recognition with one training image per person. Pattern Recognition Letters, 23(14), 1711–1719.zbMATHCrossRefGoogle Scholar
  137. 137.
    Xu, D., Principe, J. C., & Wu, H. C. (1998). Generalized eigendecomposition with an on-line local algorithm. IEEE Signal Processing Letters, 5(11), 298–301.CrossRefGoogle Scholar
  138. 138.
    Xu, L. (1993). Least mean square error reconstruction principle for self-organizing neural-nets. Neural Networks, 6, 627–648.CrossRefGoogle Scholar
  139. 139.
    Xu, L., Krzyzak, A., & Oja, E. (1993). Rival penalized competitive learning for clustering analysis, RBF net, and curve detection. IEEE Transactions on Neural Networks, 4(4), 636–649.CrossRefGoogle Scholar
  140. 140.
    Xu, L., Oja, E., & Suen, C. Y. (1992). Modified Hebbian learning for curve and surface fitting. Neural Networks, 5, 441–457.CrossRefGoogle Scholar
  141. 141.
    Xu, L., & Yuille, A. L. (1995). Robust principal component analysis by self-organizing rules based on statistical physics approach. IEEE Transactions on Neural Networks, 6(1), 131–143.CrossRefGoogle Scholar
  142. 142.
    Yang, B. (1995). Projection approximation subspace tracking. IEEE Transactions on Signal Processing, 43(1), 95–107.CrossRefGoogle Scholar
  143. 143.
    Yang, B. (1995). An extension of the PASTd algorithm to both rank and subspace tracking. IEEE Signal Processing Letters, 2(9), 179–182.CrossRefGoogle Scholar
  144. 144.
    Yang, J., Zhang, D., Frangi, A. F., & Yang, J. Y. (2004). Two-dimensional PCA: A new approach to appearance-based face representation and recognition. IEEE Transactions on Pattern Analysis Machine Intelligence, 26(1), 131–137.CrossRefGoogle Scholar
  145. 145.
    Yang, J., Zhao, Y., & Xi, H. (2011). Weighted rule based adaptive algorithm for simultaneously extracting generalized eigenvectors. IEEE Transactions on Neural Networks, 22(5), 800–806.CrossRefGoogle Scholar
  146. 146.
    Yang, J., Chen, X., & Xi, H. (2013). Fast adaptive extraction algorithm for multiple principal generalized eigenvectors. International Journal of Intelligent Systems, 28, 289–306.CrossRefGoogle Scholar
  147. 147.
    Ye, J. (2005). Generalized low rank approximations of matrices. Machine Learning, 61, 167–191.zbMATHCrossRefGoogle Scholar
  148. 148.
    Ye, M. (2005). Global convergence analysis of a self-stabilizing MCA learning algorithm. Neurocomputing, 67, 321–327.CrossRefGoogle Scholar
  149. 149.
    Ye, M., Fan, X.-Q., & Li, X. (2006). A class of self-stabilizing MCA learning algorithms. IEEE Transactions on Neural Networks, 17(6), 1634–1638.CrossRefGoogle Scholar
  150. 150.
    Yi, Z., Fu, Y., & Tang, H. J. (2004). Neural networks based approach for computing eigenvectors and eigenvalues of symmetric matrix. Computers & Mathematics with Applications, 47, 1155–1164.MathSciNetzbMATHCrossRefGoogle Scholar
  151. 151.
    Yi, Z., Ye, M., Lv, J. C., & Tan, K. K. (2005). Convergence analysis of a deterministic discrete time system of Oja’s PCA learning algorithm. IEEE Transactions on Neural Networks, 16(6), 1318–1328.CrossRefGoogle Scholar
  152. 152.
    Yuille, A. L., Kammen, D. M., & Cohen, D. S. (1989). Quadrature and development of orientation selective cortical cells by Hebb rules. Biological Cybernetics, 61, 183–194.zbMATHCrossRefGoogle Scholar
  153. 153.
    Zhang, Y., & Ma, Y. (1997). CGHA for principal component extraction in the complex domain. IEEE Transactions on Neural Networks, 8(5), 1031–1036.CrossRefGoogle Scholar
  154. 154.
    Zhang, D., Zhou, Z. H., & Chen, S. (2006). Diagonal principal component analysis for face recognition. Pattern Recognition, 39, 140–142.CrossRefGoogle Scholar
  155. 155.
    Zhang, Q., & Leung, Y. W. (2000). A class of learning algorithms for principal component analysis and minor component analysis. IEEE Transactions on Neural Networks, 11(1), 200–204.CrossRefGoogle Scholar
  156. 156.
    Zhao, H., Yuen, P. C., & Kwok, J. T. (2006). A novel incremental principal component analysis and its application for face recognition. IEEE Transactions on Systems, Man, and Cybernetics, 36(4), 873–886.CrossRefGoogle Scholar
  157. 157.
    Zhao, S., Gao, C., Mukherjee, S., & Engelhardt, B. E. (2016). Bayesian group factor analysis with structured sparsity. Journal of Machine Learning Research, 17, 1–47.MathSciNetzbMATHGoogle Scholar
  158. 158.
    Zufiria, P. J. (2002). On the discrete-time dynamics of the basic Hebbian neural-network node. IEEE Transactions on Neural Networks, 13(6), 1342–1352.CrossRefGoogle Scholar
  159. 159.
    Zuo, W., Zhang, D., & Wang, K. (2006). Bidirectional PCA with assembled matrix distance metric for image recognition. IEEE Transactions on Systems, Man, and Cybernetics Part B, 36(4), 863–872.CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Electrical and Computer EngineeringConcordia UniversityMontrealCanada
  2. 2.Xonlink Inc.HangzhouChina

Personalised recommendations