Abstract
Principal component analysis (PCA) and factor analysis (FA) are two time-honored dimension reduction methods. In this paper, some inequalities are presented to contrast the parameters’ estimates in PCA and FA. For this reason, we take advantage of the recently established matrix decomposition (MD) formulation of FA. In summary, the resulting inequalities show that (1) FA gives a better fit to a data set than PCA, (2) PCA extracts a larger amount of common “information” than FA, and (3) for each variable, its unique variance in FA is larger than its residual variance in PCA minus the one in FA. The resulting inequalities can be useful to suggest whether PCA or FA should be used for a particular data set. The answers can also be valid for the classic FA formulation not relying on the MD-FA definition, as both “types” FA provide almost equal solutions. Additionally, the inequalities give theoretical explanation of some empirically observed tendencies in PCA and FA solutions, e.g., that the absolute values of PCA loadings tend to be larger than those for FA loadings and that the unique variances in FA tend to be larger than the residual variances of PCA.
Similar content being viewed by others
References
Adachi, K. (2012). Some contributions to data-fitting factor analysis with empirical comparisons to covariance-fitting factor analysis. Journal of the Japanese Society of Computational Statistics, 25, 25–38.
Adachi, K. (2015). A matrix-intensive approach to factor analysis. Journal of the Japan Statistical Society, Japanese Issue, 44, 363–382. (in Japanese).
Adachi, K. (2016). Matrix-based introduction to multivariate data analysis. Singapore: Springer.
Adachi, & Trendafilov. (2018). Some mathematical properties of the matrix decomposition solution in factor analysis. Psychometrika, 83, 407–424.
Bentler, P. M., & Kano, Y. (1990). On the equivalence of factors and components. Multivariate Behavioral Research, 25, 67–74.
Eckart, C., & Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika, 1, 211–218.
Gower, J. C., & Dijksterhuis, G. B. (2004). Procrustes problems. Oxford: Oxford University Press.
Harman, H. H. (1976). Modern factor analysis (3rd ed.). Chicago: The University of Chicago Press.
Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. Journal of Educational Statistics, 24, 417–441.
Jennrich, R. I. (2006). Rotation to simple loadings using component loss function: The oblique case. Psychometrika, 71, 173–191.
Jolliffe, I. T. (2002). Principal component analysis (2nd ed.). New York: Springer.
Kaiser, H. F. (1958). The varimax criterion for analytic rotation in factor analysis. Psychometrika, 23, 187–200.
Mulaik, S. A. (2010). Foundations of factor analysis (2nd ed.). Boca Raton: CRC Press.
Mullen, F. (1939). Factors in the growth of girls seven to seventeen years of age. Ph.D Dissertation. University of Chicago, Department of Education.
Ogasawara, H. (2000). Some relationships between factors and components. Psychometrika, 65, 167–185.
Okamoto, M. (1969). Optimality of principal components. In P. R. Krishinaiah (Ed.), Multivariate analysis (Vol. II, pp. 673–687). New York: Academic Press.
Pearson, K. (1901). On lines and planes of closest fit to systems of points in space. Philosophical Magazines, 2, 559–572.
Sato, M. (1990). Some remarks on principal component analysis as a substitute for factor analysis in monofactor cases. Journal of the Japan Statistical Society, 20, 23–31.
Sočan, G. (2003). The incremental value of minimum rank factor analysis. PhD Thesis, University of Groningen: Groningen.
Spearman, C. (1904). “General Intelligence”, objectively determined and measured. American Journal of Psychology, 15, 201–293.
Stegeman, A. (2016). A new method for simultaneous estimation of the factor model parameters, factor scores, and unique parts. Computational Statistics and Data Analysis, 99, 189–203.
Tanaka, Y., & Tarumi, T. (1995). Handbook for statistical analysis: Multivariate analysis (windows version). Tokyo: Kyoritsu-Shuppan. (in Japanese).
ten Berge, J. M. F., & Kiers, H. A. L. (1996). Optimality criteria for principal component analysis and generalizations. British Journal of Mathematical and Statistical Psychology, 49, 335–345.
Thurstone, L. L. (1935). The vectors of mind. Chicago: University if Chicago Press.
Trendafilov, N. T., Unkel, S., & Krzanowski, W. (2013). Exploratory factor and principal component analyses: Some new aspects. Statistics and Computing, 23, 209–220.
Unkel, S., & Trendafilov, N. T. (2010). Simultaneous parameter estimation in exploratory factor analysis: An expository review. International Statistical Review, 78, 363–382.
Acknowledgements
Funding was provided by the Japan Society of the Promotion of Sciences [Grant (C)-18K11191]. The authors thank the anonymous reviewers for their useful comments.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Adachi, K., Trendafilov, N.T. Some inequalities contrasting principal component and factor analyses solutions. Jpn J Stat Data Sci 2, 31–47 (2019). https://doi.org/10.1007/s42081-018-0024-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42081-018-0024-4