Some inequalities contrasting principal component and factor analyses solutions

Abstract

Principal component analysis (PCA) and factor analysis (FA) are two time-honored dimension reduction methods. In this paper, some inequalities are presented to contrast the parameters’ estimates in PCA and FA. For this reason, we take advantage of the recently established matrix decomposition (MD) formulation of FA. In summary, the resulting inequalities show that (1) FA gives a better fit to a data set than PCA, (2) PCA extracts a larger amount of common “information” than FA, and (3) for each variable, its unique variance in FA is larger than its residual variance in PCA minus the one in FA. The resulting inequalities can be useful to suggest whether PCA or FA should be used for a particular data set. The answers can also be valid for the classic FA formulation not relying on the MD-FA definition, as both “types” FA provide almost equal solutions. Additionally, the inequalities give theoretical explanation of some empirically observed tendencies in PCA and FA solutions, e.g., that the absolute values of PCA loadings tend to be larger than those for FA loadings and that the unique variances in FA tend to be larger than the residual variances of PCA.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2

References

  1. Adachi, K. (2012). Some contributions to data-fitting factor analysis with empirical comparisons to covariance-fitting factor analysis. Journal of the Japanese Society of Computational Statistics,25, 25–38.

    MathSciNet  Article  Google Scholar 

  2. Adachi, K. (2015). A matrix-intensive approach to factor analysis. Journal of the Japan Statistical Society, Japanese Issue,44, 363–382. (in Japanese).

    MathSciNet  Google Scholar 

  3. Adachi, K. (2016). Matrix-based introduction to multivariate data analysis. Singapore: Springer.

    Google Scholar 

  4. Adachi, & Trendafilov. (2018). Some mathematical properties of the matrix decomposition solution in factor analysis. Psychometrika,83, 407–424.

    MathSciNet  Article  Google Scholar 

  5. Bentler, P. M., & Kano, Y. (1990). On the equivalence of factors and components. Multivariate Behavioral Research,25, 67–74.

    Article  Google Scholar 

  6. Eckart, C., & Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika,1, 211–218.

    Article  Google Scholar 

  7. Gower, J. C., & Dijksterhuis, G. B. (2004). Procrustes problems. Oxford: Oxford University Press.

    Google Scholar 

  8. Harman, H. H. (1976). Modern factor analysis (3rd ed.). Chicago: The University of Chicago Press.

    Google Scholar 

  9. Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. Journal of Educational Statistics,24, 417–441.

    MATH  Google Scholar 

  10. Jennrich, R. I. (2006). Rotation to simple loadings using component loss function: The oblique case. Psychometrika,71, 173–191.

    MathSciNet  Article  Google Scholar 

  11. Jolliffe, I. T. (2002). Principal component analysis (2nd ed.). New York: Springer.

    Google Scholar 

  12. Kaiser, H. F. (1958). The varimax criterion for analytic rotation in factor analysis. Psychometrika,23, 187–200.

    Article  Google Scholar 

  13. Mulaik, S. A. (2010). Foundations of factor analysis (2nd ed.). Boca Raton: CRC Press.

    Google Scholar 

  14. Mullen, F. (1939). Factors in the growth of girls seven to seventeen years of age. Ph.D Dissertation. University of Chicago, Department of Education.

  15. Ogasawara, H. (2000). Some relationships between factors and components. Psychometrika,65, 167–185.

    MathSciNet  Article  Google Scholar 

  16. Okamoto, M. (1969). Optimality of principal components. In P. R. Krishinaiah (Ed.), Multivariate analysis (Vol. II, pp. 673–687). New York: Academic Press.

    Google Scholar 

  17. Pearson, K. (1901). On lines and planes of closest fit to systems of points in space. Philosophical Magazines,2, 559–572.

    MATH  Google Scholar 

  18. Sato, M. (1990). Some remarks on principal component analysis as a substitute for factor analysis in monofactor cases. Journal of the Japan Statistical Society,20, 23–31.

    MathSciNet  MATH  Google Scholar 

  19. Sočan, G. (2003). The incremental value of minimum rank factor analysis. PhD Thesis, University of Groningen: Groningen.

  20. Spearman, C. (1904). “General Intelligence”, objectively determined and measured. American Journal of Psychology,15, 201–293.

    Article  Google Scholar 

  21. Stegeman, A. (2016). A new method for simultaneous estimation of the factor model parameters, factor scores, and unique parts. Computational Statistics and Data Analysis,99, 189–203.

    MathSciNet  Article  Google Scholar 

  22. Tanaka, Y., & Tarumi, T. (1995). Handbook for statistical analysis: Multivariate analysis (windows version). Tokyo: Kyoritsu-Shuppan. (in Japanese).

    Google Scholar 

  23. ten Berge, J. M. F., & Kiers, H. A. L. (1996). Optimality criteria for principal component analysis and generalizations. British Journal of Mathematical and Statistical Psychology,49, 335–345.

    MathSciNet  Article  Google Scholar 

  24. Thurstone, L. L. (1935). The vectors of mind. Chicago: University if Chicago Press.

    Google Scholar 

  25. Trendafilov, N. T., Unkel, S., & Krzanowski, W. (2013). Exploratory factor and principal component analyses: Some new aspects. Statistics and Computing,23, 209–220.

    MathSciNet  Article  Google Scholar 

  26. Unkel, S., & Trendafilov, N. T. (2010). Simultaneous parameter estimation in exploratory factor analysis: An expository review. International Statistical Review,78, 363–382.

    Article  Google Scholar 

Download references

Acknowledgements

Funding was provided by the Japan Society of the Promotion of Sciences [Grant (C)-18K11191]. The authors thank the anonymous reviewers for their useful comments.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Kohei Adachi.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Adachi, K., Trendafilov, N.T. Some inequalities contrasting principal component and factor analyses solutions. Jpn J Stat Data Sci 2, 31–47 (2019). https://doi.org/10.1007/s42081-018-0024-4

Download citation

Keywords

  • Matrix decomposition
  • Dimension reduction
  • Common parts
  • Unique parts
  • Loadings
  • Residuals