, Volume 84, Issue 1, pp 164–185 | Cite as

Semi-sparse PCA

  • Lars Eldén
  • Nickolay TrendafilovEmail author


It is well known that the classical exploratory factor analysis (EFA) of data with more observations than variables has several types of indeterminacy. We study the factor indeterminacy and show some new aspects of this problem by considering EFA as a specific data matrix decomposition. We adopt a new approach to the EFA estimation and achieve a new characterization of the factor indeterminacy problem. A new alternative model is proposed, which gives determinate factors and can be seen as a semi-sparse principal component analysis (PCA). An alternating algorithm is developed, where in each step a Procrustes problem is solved. It is demonstrated that the new model/algorithm can act as a specific sparse PCA and as a low-rank-plus-sparse matrix decomposition. Numerical examples with several large data sets illustrate the versatility of the new model, and the performance and behaviour of its algorithmic implementation.


alternative factor analysis matrix decompositions least squares Stiefel manifold sparse PCA robust PCA 


  1. Absil, P.-A., Mahony, R., & Sepulchre, R. (2008). Optimization Algorithms on Matrix Manifolds. Princeton: Princeton University Press.CrossRefGoogle Scholar
  2. Adachi, K., & Trendafilov, N. (2017). Sparsest factor analysis for clustering variables: A matrix decomposition approach. Advances in Data Analysis and Classification, 12, 778–794.Google Scholar
  3. Aravkin, A., Becker, S., Cevher, V., & Olsen, P. (2014). A variational approach to stable principal component pursuit. In Conference on uncertainty in artificial intelligence (UAI).Google Scholar
  4. Armstrong, S. A., Staunton, J. E., Silverman, L. B., Pieters, R., den Boer, M. L., Minden, M. D., et al. (2002). MLL translocations specify a distinct gene expression profile that distinguishes a unique leukemia. Nature Genetics, 30, 41–47.CrossRefGoogle Scholar
  5. Cai, J.-F., Candès, E. J., & Shen, Z. (2008). A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization, 20, 1956–1982.CrossRefGoogle Scholar
  6. Candès, E. J., Li, X., Ma, Y., & Wright, J. (2009). Robust principal component analysis? Journal of ACM, 58, 1–37.CrossRefGoogle Scholar
  7. De Leeuw, J. (2004). Least squares optimal scaling of partially observed linear systems. In K. van Montfort, J. Oud, & A. Satorra (Eds.), Recent developments on structural equation models: Theory and applications (pp. 121–134). Dordrecht, NL: Kluwer Academic Publishers.CrossRefGoogle Scholar
  8. Edelman, A., Arias, T. A., & Smith, S. T. (1998). The geometry of algorithms with orthogonality constraints. SIAM Journal on Matrix Analysis and Applications, 20, 303–353.CrossRefGoogle Scholar
  9. Eldén, L. (2007). Matrix methods in data mining and pattern recognition. Philadelphia: SIAM.CrossRefGoogle Scholar
  10. Golub, G. H., & Van Loan, C. F. (2013). Matrix computations (4th ed.). Baltimore, MD: Johns Hopkins University Press.Google Scholar
  11. Harman, H. H. (1976). Modern factor analysis (3rd ed.). Chicago, IL: University of Chicago Press.Google Scholar
  12. Jolliffe, I. T., Trendafilov, N. T., & Uddin, M. (2003). A modified principal component technique based on the LASSO. Journal of Computational and Graphical Statistics, 12, 531–547.CrossRefGoogle Scholar
  13. Journée, M., Nesterov, Y., Richtárik, P., & Sepulchre, R. (2010). Generalized power method for sparse principal component analysis. Journal of Machine Learning Research, 11, 517–553.Google Scholar
  14. Lin, Z., Chen, M., Wu, L., & Ma, Y. (2009). The augmented Lagrange multiplier method for exact recovery of corrupted low-rank matrices. UIUC Technical Report, UILU-ENG-09-2215, November.Google Scholar
  15. Lin, Z., Ganesh, A., Wright, J., Wu, L., Chen, M., & Ma, Y. (2009). Fast convex optimization algorithms for exact recovery of a corrupted low-rank matrix. UIUC Technical Report, UILU-ENG-09-2214, August.Google Scholar
  16. Mulaik, S. A. (2005). Looking back on the factor indeterminacy controversies in factor analysis. In In A. Maydeu-Olivares & J. J. McArdle (Eds.), Contemporary Psychometrics (pp. 174–206). Mahwah, NJ: Lawrence Erlbaum Associates Inc.Google Scholar
  17. Mulaik, S. A. (2010). The foundations of factor analysis (2nd ed.). Boca Raton, FL: Chapman and Hall/CRC.Google Scholar
  18. Shen, H., & Huang, J. Z. (2008). Sparse principal component analysis via regularized low-rank matrix approximation. Journal of Multivariate Analysis, 99, 1015–1034.CrossRefGoogle Scholar
  19. Steiger, J. H. (1979). Factor indeterminacy in the 1930’s and the 1970’s: Some interesting parallels. Psychometrika, 44, 157–166.CrossRefGoogle Scholar
  20. Steiger, J. H., & Schonemann, P. H. (1978). A history of factor indeterminacy (pp. 136–178). Chicago, IL: University of Chicago Press.Google Scholar
  21. Trendafilov, N., Fontanella, S., & Adachi, K. (2017). Sparse exploratory factor analysis. Psychometrika, 82, 778–794.CrossRefGoogle Scholar
  22. Trendafilov, N. T., & Unkel, S. (2011). Exploratory factor analysis of data matrices with more variables than observations. Journal of Computational and Graphical Statistics, 20, 874–891.CrossRefGoogle Scholar
  23. Unkel, S., & Trendafilov, N. T. (2010). Simultaneous parameter estimation in exploratory factor analysis: An expository review. International Statistical Review, 78, 363–382.CrossRefGoogle Scholar
  24. Witten, D. M., Tibshirani, R., & Hastie, T. (2009). A penalized matrix decomposition, with applications to sparse principal components and canonical correlation. Biostatistics, 10, 515–534.CrossRefGoogle Scholar
  25. Yuan, X., & Yang, J. (2013). Sparse and low-rank matrix decomposition via alternating direction methods. Pacific Journal of Optimization, 9, 167–180.Google Scholar

Copyright information

© The Psychometric Society 2018

Authors and Affiliations

  1. 1.Department of MathematicsLinköping UniversityLinköpingSweden
  2. 2.School of Mathematics and StatisticsThe Open UniversityMilton KeynesUK

Personalised recommendations