# Synthesizing effects for multiple outcomes per study using robust variance estimation versus the three-level model

## Abstract

Primary studies increasingly report information that can be used to provide multiple effect sizes. Of interest in this study, primary studies might compare a treatment and a control group on multiple related outcomes that result in multiple dependent effect sizes to be synthesized. There are a number of ways to handle the resulting within-study “multiple-outcome” dependency. The present study focuses on use of the multilevel meta-analysis model (Van den Noortgate, López-López, Marín-Martínez, & Sánchez-Meca, 2013) and robust variance estimation (Hedges, Tipton, & Johnson, 2010) for handling this dependency, as well as for estimating outcome-specific mean effect sizes. We assessed these two approaches under various conditions that differed from each other in within-study sample size; the number of effect sizes per outcome; the number of outcomes per study; the number of studies per meta-analysis; the ratio of variances at Levels 1, 2, and 3; and the true correlation between pairs of effect sizes at the between-study level. Limitations and directions for future research are discussed.

## Keywords

Multilevel meta-analysis Robust variance estimation Multiple-outcome dependency Simulation study## References

- Abrami, P. C., Bernard, R. M., Borokhovski, E., Waddington, D. I., Wade, C. A., & Persson, T. (2015). Strategies for teaching students to think critically: A meta-analysis.
*Review of Educational Research*,*85*, 275–314.CrossRefGoogle Scholar - Assink, M., van der Put, C. E., Hoeve, M., de Vries, S. L. A., Stams, G. J. J. M., & Oort, F. J. (2015). Risk factors for persistent delinquent behavior among juveniles: A meta-analytic review.
*Clinical Psychology Review*,*42*, 47–61.CrossRefGoogle Scholar - Becker, B. J. (2000). Multivariate meta-analysis. In H. E. A. Tinsley & E. D. Brown (Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 499–525). Orlando, FL: Academic Press.CrossRefGoogle Scholar
- Becker, B. J., Hedges, L. V., & Pigott, T. D. (2004).
*Campbell Collaboration statistical analysis policy brief*(Campbell Collaboration resource document). Available from http://www.campbellcollaboration.org/ECG/policy_stat.asp - Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Hoboken, NJ: Wiley.CrossRefGoogle Scholar
- Cheung, M. W.-L. (2015). Meta-analysis: A structural equation modeling approach. Chichester, UK: Wiley.Google Scholar
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.Google Scholar
- Cooper, H., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis. New York, NY: Russell Sage Foundation.Google Scholar
- Fisher, Z., & Tipton, E. (2014). robumeta: An R-package for robust variance estimation in meta-analysis. Working paper, Department of Statistics, Northwestern University, Evanston, IL.Google Scholar
- Geeraert, L., Van den Noortgate, W., Grietens, H., & Onghena, P. (2004). The effects of early prevention programs for families with young children at risk for physical child abuse and neglect: A meta-analysis.
*Child Maltreatment*,*9*, 277–291.CrossRefGoogle Scholar - Glass, G. V. (1976). Primary, secondary, and meta-analysis of research.
*Educational Researcher*,*5*, 3–8.CrossRefGoogle Scholar - Gleser, L. J., & Olkin, I. (2009). Stochastically dependent effect sizes. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 357–376). New York, NY: Russell Sage Foundation.Google Scholar
- Hedges, L.. V. (1981). Distribution theory for Glass’s estimator of effect size and related estimators.
*Journal of Educational Statistics*,*6*, 107–128.Google Scholar - Hedges, L.. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.Google Scholar
- Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation of meta-regression with dependent effect size estimates.
*Research Synthesis Methods*,*1*, 39–65.CrossRefGoogle Scholar - Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis.
*Psychological Methods*,*3*, 486–504. doi: https://doi.org/10.1037/1082-989X.3.4.486 CrossRefGoogle Scholar - Hoogland, J. J., & Boomsma, A. (1998). Robustness studies in covariance structure modeling.
*Sociological Methods and Research*,*26*, 329–367.CrossRefGoogle Scholar - Hox, J. J., & Maas, C. J. (2001). The accuracy of multilevel structural equation modeling with pseudobalanced groups and small samples.
*Structural Equation Modeling*,*8*, 157–174.CrossRefGoogle Scholar - Hox, J. J., Moerbeek, M., & van de Schoot, R. (2018). Multilevel analysis: Techniques and applications (3rd ed.). New York, NY, Routledge.Google Scholar
- Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings
*.*Thousand Oaks, CA: Sage.CrossRefGoogle Scholar - Kenward, M. G., & Roger, J. H. (1997). Small sample inference for fixed effects from restricted maximum likelihood.
*Biometrics*,*53*, 983–997.CrossRefGoogle Scholar - Kisamore, J. L., & Brannick, M. T. (2008). An illustration of the consequences of meta-analysis model choice.
*Organizational Research Methods*,*11*, 35–53.CrossRefGoogle Scholar - Kreft, I. G. G., & De Leeuw, J. (1998). Introducing multilevel modeling. Newbury Park, CA: Sage.CrossRefGoogle Scholar
- Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling.
*Methodology*,*1*, 86–92. doi: https://doi.org/10.1027/1614-2241.1.3.86 CrossRefGoogle Scholar - Matt, G. E., & Cook, T. D. (2009). Threats to the validity of generalized inferences. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 538–557). New York, NY: Sage.Google Scholar
- McNeish, D. M., & Stapleton, L. M. (2016a). The effect of small sample size on two-level model estimates: A review and illustration.
*Educational Psychology Review*,*28*.Google Scholar - McNeish, D. M., & Stapleton, L. M. (2016b). Modeling clustered data with very few clusters.
*Multivariate Behavioral Research*,*51*, 495–518.CrossRefGoogle Scholar - McNeish, D., & Wentzel, K. R. (2017). Accommodating small sample sizes in three-level models when the third level is incidental.
*Multivariate Behavioral Research*,*52*, 200–215.CrossRefGoogle Scholar - Moeyaert, M., Ugille, M., Beretvas, S. N., Ferron, J., Bunuan, R., & Van den Noortgate, W. (2017) Methods for dealing with multiple outcomes in meta-analysis: a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis,
*International Journal of Social Research Methodology*,*20*, 559–572.CrossRefGoogle Scholar - National Research Council. (1992).
*Combining information: Statistical issues and opportunities for research*. Washington, DC: National Academy of Science Press.Google Scholar - Raudenbush, S. W. (1994). Random effects models. In H. Cooper & L. V. Hedges (Eds.), Handbook of research synthesis (pp. 301–322). New York, NY: Russell Sage Foundation.Google Scholar
- Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988). Modeling multivariate effect sizes.
*Psychological Bulletin*,*103*, 111–120. doi: https://doi.org/10.1037/0033-2909.103.1.111 CrossRefGoogle Scholar - Schmidt, F. L., Oh, I., & Hayes, T. L. (2009). Fixed- versus random-effects models in meta- analysis: Model properties and an empirical comparison of differences in results.
*British Journal of Mathematical and Statistical Psychology*,*62*, 97–128.CrossRefGoogle Scholar - Sheridan, S. M., Kim, E. M., Beretvas, S. N., Smith, T., & Park, S. Y. (2017).
*Family–school interventions and academic and social behavioral outcomes: What matters?*Paper presented at the annual meeting of American Psychological Association, Washington, DC.Google Scholar - Tanner-Smith, E. E., & Tipton, E. (2014). Robust variance estimation with dependent effect sizes: Practical considerations including a software tutorial in Stata and SPSS.
*Research Synthesis Methods*,*5*, 13–30. doi: https://doi.org/10.1002/jrsm.1091 CrossRefGoogle Scholar - Tipton, E. (2013). Robust variance estimation in meta-regression for binary dependent outcomes.
*Research Synthesis Methods*,*4*(2), 169–187. https://doi.org/10.1002/jrsm.1070 - Tipton, E. (2015). Small-sample adjustments for robust variance estimation with meta-regression.
*Psychological Methods*,*20*(3), 375–393. doi: https://doi.org/10.1037/met0000011 - Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013). Three-level meta-analyses of dependent effect sizes.
*Behavior Research Methods*,*45*, 576–594. doi: https://doi.org/10.3758/s13428-012-0261-6 CrossRefGoogle Scholar - Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2014). Meta-analysis of multiple outcomes: A multilevel approach.
*Behavior Research Methods*,*47*, 1274–1294.CrossRefGoogle Scholar - Van den Noortgate, W., & Onghena, P. (2003). Multilevel meta-analysis: A comparison with traditional meta-analytical procedures.
*Educational and Psychological Measurement*,*63*, 765–79.CrossRefGoogle Scholar - Van der Kleij, F. M., Feskens, R. C., & Eggen, T. J. (2015). Effects of feedback in a computer-based learning environment on students’ learning outcomes: A meta-analysis.
*Review of Educational Research*,*85*, 475–511.CrossRefGoogle Scholar - Viechtbauer, W. (2005). Bias and efficiency of meta-analytic variance estimators in the random-effects model.
*Journal of Educational and Behavioral Statistics*,*30*, 261–293.CrossRefGoogle Scholar