# RMSEA, CFI, and TLI in structural equation modeling with ordered categorical data: The story they tell depends on the estimation methods

## Abstract

In structural equation modeling, application of the root mean square error of approximation (RMSEA), comparative fit index (CFI), and Tucker–Lewis index (TLI) highly relies on the conventional cutoff values developed under normal-theory maximum likelihood (ML) with continuous data. For ordered categorical data, unweighted least squares (ULS) and diagonally weighted least squares (DWLS) based on polychoric correlation matrices have been recommended in previous studies. Although no clear suggestions exist regarding the application of these fit indices when analyzing ordered categorical variables, practitioners are still tempted to adopt the conventional cutoff rules. The purpose of our research was to answer the question: Given a population polychoric correlation matrix and a hypothesized model, if ML results in a specific RMSEA value (e.g., .08), what is the RMSEA value when ULS or DWLS is applied? CFI and TLI were investigated in the same fashion. Both simulated and empirical polychoric correlation matrices with various degrees of model misspecification were employed to address the above question. The results showed that DWLS and ULS lead to smaller RMSEA and larger CFI and TLI values than does ML for all manipulated conditions, regardless of whether or not the indices are scaled. Applying the conventional cutoffs to DWLS and ULS, therefore, has a pronounced tendency not to discover model–data misfit. Discussions regarding the use of RMSEA, CFI, and TLI for ordered categorical data are given.

## Keywords

Structural equation modeling Ordered categorical data Diagonally weighted least squares Unweighted least squares Maximum likelihood Fit index## References

- Asparouhov, T., & Muthén, B. (2010).
*Simple second order chi-square correction*. Retrieved from www.statmodel.com/download/WLSMV_new_chi21.pdf - Barrett, P. (2007). Structural equation modelling: Adjudging model fit.
*Personality and Individual Differences*,*42*, 815–824.CrossRefGoogle Scholar - Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA.
*Structural Equation Modeling*,*13*, 186–203.CrossRefGoogle Scholar - De Beer, L. T., Pienaar, J., & Rothmann, S. (2014). Job burnout’s relationship with sleep difficulties in the presence of control variables: A self-report study.
*South African Journal of Psychology*,*44*, 454–466.CrossRefGoogle Scholar - Bentler, P. M. (1990). Comparative fit indexes in structural models.
*Psychological Bulletin*,*107*, 238–246. https://doi.org/10.1037/0033-2909.107.2.238 CrossRefGoogle Scholar - Bentler, P. M. (2008).
*EQS structural equation modeling software*. Encino, CA: Multivariate Software.Google Scholar - Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures.
*Psychological Bulletin*,*88*, 588–606. https://doi.org/10.1037/0033-2909.88.3.588 CrossRefGoogle Scholar - Brosseau-Liard, P. E., & Savalei, V. (2014). Adjusting incremental fit indexes for nonnormality.
*Multivariate Behavioral Research*,*49*, 460–470.CrossRefGoogle Scholar - Brosseau-Liard, P. E., Savalei, V., & Li, L. (2012). An investigation of the sample performance of two nonnormality corrections for RMSEA.
*Multivariate Behavioral Research*,*47*, 904–930.CrossRefGoogle Scholar - Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.),
*Testing structural equation models*(pp. 136–162). Newbury Park, CA: Sage.Google Scholar - Cudeck, R., & Browne, M. W. (1992). Constructing a covariance matrix that yields a specified minimizer and a specified minimum discrepancy function value.
*Psychometrika*,*57*, 357–369.CrossRefGoogle Scholar - Currier, J. M., & Holland, J. M. (2014). Involvement in abusive violence among Vietnam veterans: Direct and indirect associations with substance use problems and suicidality.
*Psychological Trauma: Theory, Research, Practice, and Policy*,*6*, 73–82.CrossRefGoogle Scholar - DiStefano, C., & Morgan, G. B. (2014). A comparison of diagonal weighted least squares robust estimation techniques for ordinal data.
*Structural Equation Modeling*,*21*, 425–438.CrossRefGoogle Scholar - Fernandez, S., & Moldogaziev, T. (2013). Employee empowerment, employee attitudes, and performance: Testing a causal model.
*Public Administration Review*,*73*, 490–506.CrossRefGoogle Scholar - Forero, C. G., Maydeu-Olivares, A., & Gallardo-Pujol, D. (2009). Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation.
*Structural Equation Modeling*,*16*, 625–641.CrossRefGoogle Scholar - Garrido, L. E., Abad, F. J., & Ponsoda, V. (2016). Are fit indexes really fit to estimate the number of factors with categorical variables? Some cautionary findings via Monte Carlo simulation.
*Psychological Methods*,*21*, 93–111.CrossRefGoogle Scholar - Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives.
*Structural Equation Modeling*,*6*, 1–55. https://doi.org/10.1080/10705519909540118 CrossRefGoogle Scholar - Iglesias, K., Burnand, B., & Peytremann-Bridevaux, I. (2014). PACIC Instrument: Disentangling dimensions using published validation models.
*International Journal for Quality in Health Care*,*26*, 250–260.CrossRefGoogle Scholar - Johnson, D. R., & Creech, J. C. (1983). Ordinal measures in multiple indicator models: A simulation study of categorization error.
*American Sociological Review*,*48*, 398–407.CrossRefGoogle Scholar - Jöreskog, K. G., & Sörbom, D. (1993).
*LISREL 8: Structural equation modeling with the SIMPLIS command language.*Chicago, IL: Scientific Software International.Google Scholar - Koziol, N. A. (2010).
*Evaluating measurement invariance with censored ordinal data: A Monte Carlo comparison of alternative model estimators and scales of measurement*. Unpublished master’s thesis, University of Nebraska, Lincoln, NE.Google Scholar - Lai, K., & Green, S. B. (2016). The problem with having two watches: Assessment of fit when RMSEA and CFI disagree.
*Multivariate Behavioral Research*,*51*220–239.CrossRefGoogle Scholar - Li, C. H. (2014).
*The performance of MLR, USLMV, and WLSMV estimation in structural regression models with ordinal variables*(Doctoral dissertation). Michigan State University, East Lansing, MI.Google Scholar - MacInnis, M. J., Lanting, S. C., Rupert, J. L., & Koehle, M. S. (2013). Is poor sleep quality at high altitude separate from acute mountain sickness? Factor structure and internal consistency of the Lake Louise Score Questionnaire.
*High Altitude Medicine and Biology*,*14*, 334–337.CrossRefGoogle Scholar - Marsh, H. W., Hau, K. T., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings.
*Structural Equation Modeling*,*11*, 320–341.CrossRefGoogle Scholar - Martínez-Rodríguez, S., Iraurgi, I., Gómez-Marroquin, I., Carrasco, M., Ortiz-Marqués, N., & Stevens, A. B. (2016). Psychometric properties of the Leisure Time Satisfaction Scale in family caregivers.
*Psicothema*,*28*, 207–213.Google Scholar - Maydeu-Olivares, A., Fairchild, A. J., & Hall, A. G. (2017). Goodness of fit in item factor analysis: Effect of the number of response alternatives.
*Structural Equation Modeling*,*24*, 495–505.CrossRefGoogle Scholar - McIntosh, C. N. (2007). Rethinking fit assessment in structural equation modelling: A commentary and elaboration on Barrett (2007).
*Personality and Individual Differences*,*42*, 859–867.CrossRefGoogle Scholar - Muthén, B. O., du Toit, S. H. C., & Spisic, D. (1997).
*Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes*(Unpublished technical report). Retrieved from www.statmodel.com/bmuthen/articles/Article_075.pdf - Muthén, L. K., & Muthén, B. O. (2015).
*Mplus user’s guide*(7th ed.). Los Angeles, CA: Muthén & Muthén.Google Scholar - Nguyen, T. Q., Poteat, T., Bandeen-Roche, K., German, D., Nguyen, Y. H., Vu, L. K., … Knowlton, A. R. (2016). The internalized homophobia scale for Vietnamese sexual minority women: Conceptualization, factor structure, reliability, and associations with hypothesized correlates.
*Archives of Sexual Behavior*,*45*, 1329–1346.CrossRefGoogle Scholar - Nye, C. D., & Drasgow, F. (2011). Assessing goodness of fit: Simple rules of thumb simply do not work.
*Organizational Research Methods*,*14*, 548–570.CrossRefGoogle Scholar - Olsson, U. (1979). Maximum likelihood estimation of the polychoric correlation coefficient.
*Psychometrika*,*44*, 443–460.CrossRefGoogle Scholar - Pearson, K. (1904).
*Mathematical contributions to the theory of evolution XIII: On the theory of contingency and its relation to association and normal correlation*(Drapers’ Co. Research Memoirs, Biometric Series, no. 1). Cambridge, UK: Cambridge University Press.Google Scholar - Pettersen, C., Nunes, K. L., & Cortoni, F. (2016). Does the factor structure of the aggression questionnaire hold for sexual offenders?
*Criminal Justice and Behavior*,*43*, 811–829.CrossRefGoogle Scholar - Rhemtulla, M., Brosseau-Liard, P. E., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions.
*Psychological Methods*,*17*, 354–373.CrossRefGoogle Scholar - SAS Institute. (2015).
*SAS/STAT 14.1 user’s guide.*Cary, NC: Author.Google Scholar - Sass, D. A., Schmitt, T. A., & Marsh, H. W. (2014). Evaluating model fit with ordered categorical data with a measurement invariance framework: A comparison of estimators.
*Structural Equation Modeling*,*21*, 167–180.CrossRefGoogle Scholar - Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In A. von Eye & C. C. Clogg (Eds.),
*Latent variables analysis: Applications for developmental research*(pp. 399–419). Thousand Oaks, CA: Sage.Google Scholar - Savalei, V. (2014). Understanding robust corrections in structural equation modeling.
*Structural Equation Modeling*,*21*, 149–160.CrossRefGoogle Scholar - Savalei, V., & Rhemtulla, M. (2013). The performance of robust test statistics with categorical data.
*British Journal of Mathematical and Statistical Psychology*,*66*, 201–223.CrossRefGoogle Scholar - Stander, F. W., Mostert, K., & de Beer, L. T. (2014). Organisational and individual strengths use as predictors of engagement and productivity.
*Journal of Psychology in Africa*,*24*, 403–409.CrossRefGoogle Scholar - Steiger, J. H. (1990). Structural model evaluation and modification: An interval estimation approach.
*Multivariate Behavioral Research*,*25*, 173–180.CrossRefGoogle Scholar - Steiger, J. H., & Lind, J. C. (1980).
*Statistically based tests for the number of common factors.*Paper presented at the Annual Meeting of the Psychometric Society, Iowa City, IA.Google Scholar - Tallis, G. M. (1962). The maximum likelihood estimation of correlation from contingency tables.
*Biometrics*,*18*, 342–353.CrossRefGoogle Scholar - Tucker, L. R., & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis.
*Psychometrika*,*38*, 1–10.CrossRefGoogle Scholar - Xia, Y., Yung, Y. F., & Zhang, W. (2016). Evaluating the selection of normal-theory weight matrices in the Satorra–Bentler correction of chi-square and standard errors.
*Structural Equation Modeling*,*23*, 585–594.CrossRefGoogle Scholar - Yang, Y., & Xia, Y. (2015). On the number of factors in exploratory factor analysis for ordered categorical data.
*Behavior Research Methods*,*47*, 756–772. https://doi.org/10.3758/s13428-014-0499-2 CrossRefGoogle Scholar - Yang-Wallentin, F., Jöreskog, K. G., & Luo, H. (2010). Confirmatory factor analysis of ordinal variables with misspecified models.
*Structural Equation Modeling*,*17*, 392–423.CrossRefGoogle Scholar - Yu, C.-Y. (2002).
*Evaluating cutoff criteria of model-fit indexes for latent variable models with binary and continuous outcomes*(Doctoral dissertation). University of California, Los Angeles, CA.Google Scholar - Yuan, K. H., & Marshall, L. L. (2004). A new measure of misfit for covariance structure models.
*Behaviormetrika*,*31*, 67–90.CrossRefGoogle Scholar - Zhang, W. (2008). A comparison of four estimators of a population measure of model fit in covariance structure analysis.
*Structural Equation Modeling*,*15*, 301–326.CrossRefGoogle Scholar