# Evaluating fit indices in a multilevel latent growth curve model: A Monte Carlo study

## Abstract

The multilevel latent growth curve model (MLGCM), which is subsumed by the multilevel structural equation modeling framework, has been advocated as a means of investigating individual and cluster trajectories. Still, how to evaluate the goodness of fit of MLGCMs has not been well addressed. The purpose of this study was to conduct a systematic Monte Carlo simulation to carefully investigate the effectiveness of (a) level-specific fit indices and (b) target-specific fit indices in an MLGCM, in terms of their independence from the sample size’s influence and their sensitivity to misspecification in the MLGCM that occurs in either the between-covariance, between-mean, or within-covariance structure. The design factors included the number of clusters, the cluster size, and the model specification. We used Mplus 7.4 to generate simulated replications and estimate each of the models. We appropriately controlled the severity of misspecification when we generated the simulated replications. The simulation results suggested that applying RMSEA_{T_S_COV}, TLI_{T _ S _ COV}, and SRMR_{B} maximizes the capacity to detect misspecifications in the between-covariance structure. Moreover, the use of RMSEA_{PS _ B}, CFI_{PS _ B}, and TLI_{PS _ B} is recommended for evaluating the fit of the between-mean structure. Finally, we found that evaluation of the within-covariance structure turned out to be unexpectedly challenging, because none of the within-level-specific fit indices (RMSEA_{PS _ W}, CFI_{PS _ W}, TLI_{PS _ W}, and SRMR_{W}) had a practically significant sensitivity.

## Keywords

Fit index Model evaluation Multilevel latent growth curve model Multilevel structural equation modeling## References

- Bentler, P. M. (1990). Comparative fit indexes in structural models.
*Psychological Bulletin*,*107*, 238–246. https://doi.org/10.1037/0033-2909.107.2.238 CrossRefPubMedGoogle Scholar - Bentler, P. M., & Dudgeon, P. (1996). Covariance structure analysis: Statistical practice, theory, and directions.
*Annual Review of Psychology*,*47*, 563–592.CrossRefPubMedGoogle Scholar - Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd). Hillsdale: Erlbaum.Google Scholar
- Cohen, J. (1992). A power primer.
*Psychological Bulletin*,*112*, 155–159. https://doi.org/10.1037/0033-2909.112.1.155 CrossRefPubMedGoogle Scholar - Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd). Mahwah: Erlbaum.Google Scholar
- Fan, X., & Sivo, S. A. (2005). Sensitivity of fit indexes to misspecified structural or measurement model components: Rationale of two-Index strategy revisited.
*Structural Equation Modeling*,*12*, 343–367.CrossRefGoogle Scholar - Gerbing, D. W., & Anderson, J. C. (1992). Monte Carlo evaluations of goodness of fit indices for structural equation models.
*Sociological Methods and Research*,*21*, 132–160.CrossRefGoogle Scholar - Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2nd). New York: Routledge.CrossRefGoogle Scholar
- Hox, J. J., & Maas, C. J. M. (2001). The accuracy of multilevel structural equation modeling with pseudobalanced groups and small samples.
*Structural Equation Modeling*,*8*, 157–174. https://doi.org/10.1207/S15328007SEM0802_1 CrossRefGoogle Scholar - Hox, J. J., Maas, C. J. M., & Brinkhuis, M. J. S. (2010). The effect of estimation method and sample size in multilevel structural equation modeling.
*Statistica Neerlandica*,*64*, 157–170. https://doi.org/10.1111/j.1467-9574.2009.00445.x CrossRefGoogle Scholar - Hsu, H.-Y., Kwok, O., Acosta, S., & Lin, J.-H. (2015). Detecting misspecified multilevel SEMs using common fit indices: A Monte Carlo study.
*Multivariate Behavioral Research*,*50*, 197–215. https://doi.org/10.1080/00273171.2014.977429 CrossRefPubMedGoogle Scholar - Hsu, H.-Y., Lin, J.-H., Kwok, O.-M., Acosta, S., & Willson, V. (2016). The impact of intraclass correlation on the effectiveness of level-specific fit indices in multilevel structural equation modeling: A Monte Carlo study.
*Educational and Psychological Measurement*,*77*, 5–31. https://doi.org/10.1177/0013164416642823 CrossRefPubMedPubMedCentralGoogle Scholar - Hu, L., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification.
*Psychological Methods*,*3*, 424–453. https://doi.org/10.1037/1082-989X.3.4.424 CrossRefGoogle Scholar - Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives.
*Structural Equation Modeling*,*6*, 1–55. https://doi.org/10.1080/10705519909540118 CrossRefGoogle Scholar - Ingels, S. J., Pratt, D. J., Herget, D. R., Dever, J. A., Fritch, L. B., Ottem, R., … Leinwand, S. (2013). Education Longitudinal Study of 2002: Base-year to first follow-up data file documentation (NCES 2014-361). Washington, DC: National Center for Education Statistics.Google Scholar
- Jackson, D. L. (2003). Revisiting sample size and number of parameter estimates: Some support for the N:q hypothesis.
*Structural Equation Modeling*,*10*, 128–141.CrossRefGoogle Scholar - Kaplan, D. (2009). Structural equation modeling: Foundations and extensions (2nd). Thousand Oaks: Sage.CrossRefGoogle Scholar
- Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd). New York: Guilford Press.Google Scholar
- Kwok, O.-M., West, S. G., & Green, S. B. (2007). The impact of misspecifying the within-subject covariance structure in multiwave longitudinal multilevel models: A Monte Carlo study.
*Multivariate Behavioral Research*,*42*, 557–592. https://doi.org/10.1080/00273170701540537 CrossRefGoogle Scholar - Leite, W. L., & Stapleton, L. M. (2011). Detecting growth shape misspecifications in latent growth models: An evaluation of fit indexes.
*Journal of Experimental Education*,*79*, 361–381.CrossRefGoogle Scholar - Lohr, S. L. (2009). Sampling design and analysis (2nd). Boston: Brooks/Cole.Google Scholar
- Lüdtke, O., Marsh, H. W., Robitzsch, A., & Trautwein, U. (2011). A 2 × 2 taxonomy of multilevel latent contextual models: Accuracy–bias trade-offs in full and partial error correction models.
*Psychological Methods*,*16*, 444–467. https://doi.org/10.1037/a0024376 CrossRefPubMedGoogle Scholar - Ma, X., & Ma, L. (2004). Modeling stability of growth between mathematics and science achievement during middle and high school.
*Evaluation Review*,*28*, 104–122. https://doi.org/10.1177/0193841X03261025 CrossRefPubMedGoogle Scholar - Ma, X., & Wilkins, J. M. (2007). Mathematics coursework regulates growth in mathematics achievement.
*Journal for Research in Mathematics Education*,*38*, 230–257.Google Scholar - Marsh, H. W., Balla, J. R., & McDonald, R. P. (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size.
*Psychological Bulletin*,*103*, 391–410.CrossRefGoogle Scholar - Marsh, H. W., Hau, K.-T., & Grayson, D. (2005). Goodness of fit in structural equation models. In A. Maydeu-Olivares & J. J. McArdle (Eds.), Contemporary psychometrics: A festschrift for Roderick P. McDonald. Multivariate applications book series (pp. 275–340). Mahwah: Erlbaum.Google Scholar
- McDonald, R. P., & Marsh, H. W. (1990). Choosing a multivariate model: Noncentrality and goodness of fit.
*Psychological Bulletin*,*107*, 247–255. https://doi.org/10.1037/0033-2909.107.2.247 CrossRefGoogle Scholar - Miller, J. D., Kimmel, L., Hoffer, T. B., & Nelson, C. (2000). Longitudinal study of American youth: User’s manual. Evanston: Northwestern University, International Center for the Advancement of Scientific Literacy.Google Scholar
- Muthén, B. O. (1997). Latent variable growth modeling with multilevel data. In M. Berkane (Ed.), Latent variable modeling and applications to causality (pp. 149–161). New York: Springer.CrossRefGoogle Scholar
- Muthén, B. O. (2004). Latent variable analysis: Growth mixture modeling and related techniques for longitudinal data. In D. Kaplan (Ed.), Handbook of quantitative methodology for the social sciences (pp. 345–368). Newbury Park: Sage.Google Scholar
- Muthén, B. O., & Asparouhov, T. (2011). Beyond multilevel regression modeling: Multilevel analysis in a general latent variable framework. In J. Hox & J. K. Roberts (Eds.), The handbook of advanced multilevel analysis. New York: Routledge.Google Scholar
- Muthén, B. O., & Satorra, A. (1995). Complex sample data in structural equation modeling.
*Sociological Methodology*,*25*, 267–316. https://doi.org/10.2307/271070 CrossRefGoogle Scholar - Muthén, L. K., & Muthén, B. O. (1998–2017).
*Mplus user’s guide*(8th). Los Angeles: Muthén & Muthén.Google Scholar - Neuman, W. L. (2009). Social research methods: Qualitative and quantitative approaches (7th). Boston: Allyn & Bacon.Google Scholar
- Ryu, E. (2014). Model fit evaluation in multilevel structural equation models.
*Frontiers in Psychology*,*5*, 81:1–9. https://doi.org/10.3389/fpsyg.2014.00081 Google Scholar - Ryu, E., & West, S. G. (2009). Level-specific evaluation of model fit in multilevel structural equation modeling.
*Structural Equation Modeling*,*16*, 583–601. https://doi.org/10.1080/10705510903203466 CrossRefGoogle Scholar - Scheaffer, R. L., Mendenhall, W., III, & Ott, R. L. (2005). Elementary survey sampling (6th). Belmont: Thomson Brooks/Cole.Google Scholar
- Schermelleh-Engel, K., Kerwer, M., & Klein, A. G. (2014). Evaluation of model fit in nonlinear multilevel structural equation modeling.
*Frontiers in Psychology*,*5*, 181:1–11. https://doi.org/10.3389/fpsyg.2014.00181 Google Scholar - Tourangeau, K., Nord, C., Lê, T., Sorongon, A. G., Hagedorn, M. C., Daly, P., & Najarian, M. (2015). Early Childhood Longitudinal Study, kindergarten class of 2010–11 (ECLS-K:2011), user’s manual for the ECLS-K:2011 kindergarten data file and electronic codebook, public version (NCES 2015-074). Washington, DC: National Center for Education Statistics.Google Scholar
- Tourangeau, K., Nord, C., Lê, T., Sorongon, A. G., & Najarian, M. (2009). Early Childhood Longitudinal Study, kindergarten class of 1998–99 (ECLS-K), combined user’s manual for the ECLS-K eighth-grade and K–8 full sample data files and electronic codebooks (NCES 2009-004). Washington, DC: National Center for Education Statistics. Retrieved from http://nces.ed.gov/ecls/data/ECLSK_K8_Manual_part1.pdf Google Scholar
- Tucker, J. S., & Lewis, C. (1973). The reliability coefficient for maximum likelihood factor analysis.
*Psychometrika*,*38*, 1–10.CrossRefGoogle Scholar - Widaman, K. F., & Thompson, J. S. (2003). On specifying the null model for incremental fit indices in structural equation modeling.
*Psychological Methods*,*8*, 16–37. https://doi.org/10.1037/1082-989X.8.1.16 CrossRefPubMedGoogle Scholar - Wu, J.-Y., Kwok, O., & Willson, V. L. (2015). Using design-based latent growth curve modeling with cluster-level predictor to address dependency.
*Journal of Experimental Education*,*82*, 431–454. https://doi.org/10.1080/00220973.2013.876226 CrossRefGoogle Scholar - Wu, W., & West, S. G. (2010). Sensitivity of fit indices to misspecification in growth curve models.
*Multivariate Behavioral Research*,*45*, 420–452.CrossRefPubMedGoogle Scholar - Wu, W., West, S. G., & Taylor, A. B. (2009). Evaluating model fit for growth curve models: Integration of fit indices from SEM and MLM frameworks.
*Psychological Methods*,*14*, 183–201. https://doi.org/10.1037/a0015858 CrossRefPubMedGoogle Scholar