Behavior Research Methods

, Volume 51, Issue 1, pp 152–171 | Cite as

Synthesizing effects for multiple outcomes per study using robust variance estimation versus the three-level model

  • Sunyoung ParkEmail author
  • S. Natasha Beretvas


Primary studies increasingly report information that can be used to provide multiple effect sizes. Of interest in this study, primary studies might compare a treatment and a control group on multiple related outcomes that result in multiple dependent effect sizes to be synthesized. There are a number of ways to handle the resulting within-study “multiple-outcome” dependency. The present study focuses on use of the multilevel meta-analysis model (Van den Noortgate, López-López, Marín-Martínez, & Sánchez-Meca, 2013) and robust variance estimation (Hedges, Tipton, & Johnson, 2010) for handling this dependency, as well as for estimating outcome-specific mean effect sizes. We assessed these two approaches under various conditions that differed from each other in within-study sample size; the number of effect sizes per outcome; the number of outcomes per study; the number of studies per meta-analysis; the ratio of variances at Levels 1, 2, and 3; and the true correlation between pairs of effect sizes at the between-study level. Limitations and directions for future research are discussed.


Multilevel meta-analysis Robust variance estimation Multiple-outcome dependency Simulation study 


  1. Abrami, P. C., Bernard, R. M., Borokhovski, E., Waddington, D. I., Wade, C. A., & Persson, T. (2015). Strategies for teaching students to think critically: A meta-analysis. Review of Educational Research, 85, 275–314.CrossRefGoogle Scholar
  2. Assink, M., van der Put, C. E., Hoeve, M., de Vries, S. L. A., Stams, G. J. J. M., & Oort, F. J. (2015). Risk factors for persistent delinquent behavior among juveniles: A meta-analytic review. Clinical Psychology Review, 42, 47–61.CrossRefGoogle Scholar
  3. Becker, B. J. (2000). Multivariate meta-analysis. In H. E. A. Tinsley & E. D. Brown (Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 499–525). Orlando, FL: Academic Press.CrossRefGoogle Scholar
  4. Becker, B. J., Hedges, L. V., & Pigott, T. D. (2004). Campbell Collaboration statistical analysis policy brief (Campbell Collaboration resource document). Available from
  5. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Hoboken, NJ: Wiley.CrossRefGoogle Scholar
  6. Cheung, M. W.-L. (2015). Meta-analysis: A structural equation modeling approach. Chichester, UK: Wiley.Google Scholar
  7. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.Google Scholar
  8. Cooper, H., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis. New York, NY: Russell Sage Foundation.Google Scholar
  9. Fisher, Z., & Tipton, E. (2014). robumeta: An R-package for robust variance estimation in meta-analysis. Working paper, Department of Statistics, Northwestern University, Evanston, IL.Google Scholar
  10. Geeraert, L., Van den Noortgate, W., Grietens, H., & Onghena, P. (2004). The effects of early prevention programs for families with young children at risk for physical child abuse and neglect: A meta-analysis. Child Maltreatment, 9, 277–291.CrossRefGoogle Scholar
  11. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.CrossRefGoogle Scholar
  12. Gleser, L. J., & Olkin, I. (2009). Stochastically dependent effect sizes. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 357–376). New York, NY: Russell Sage Foundation.Google Scholar
  13. Hedges, L.. V. (1981). Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107–128.Google Scholar
  14. Hedges, L.. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.Google Scholar
  15. Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation of meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65.CrossRefGoogle Scholar
  16. Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486–504. doi: CrossRefGoogle Scholar
  17. Hoogland, J. J., & Boomsma, A. (1998). Robustness studies in covariance structure modeling. Sociological Methods and Research, 26, 329–367.CrossRefGoogle Scholar
  18. Hox, J. J., & Maas, C. J. (2001). The accuracy of multilevel structural equation modeling with pseudobalanced groups and small samples. Structural Equation Modeling, 8, 157–174.CrossRefGoogle Scholar
  19. Hox, J. J., Moerbeek, M., & van de Schoot, R. (2018). Multilevel analysis: Techniques and applications (3rd ed.). New York, NY, Routledge.Google Scholar
  20. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.CrossRefGoogle Scholar
  21. Kenward, M. G., & Roger, J. H. (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53, 983–997.CrossRefGoogle Scholar
  22. Kisamore, J. L., & Brannick, M. T. (2008). An illustration of the consequences of meta-analysis model choice. Organizational Research Methods, 11, 35–53.CrossRefGoogle Scholar
  23. Kreft, I. G. G., & De Leeuw, J. (1998). Introducing multilevel modeling. Newbury Park, CA: Sage.CrossRefGoogle Scholar
  24. Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1, 86–92. doi: CrossRefGoogle Scholar
  25. Matt, G. E., & Cook, T. D. (2009). Threats to the validity of generalized inferences. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 538–557). New York, NY: Sage.Google Scholar
  26. McNeish, D. M., & Stapleton, L. M. (2016a). The effect of small sample size on two-level model estimates: A review and illustration. Educational Psychology Review, 28.Google Scholar
  27. McNeish, D. M., & Stapleton, L. M. (2016b). Modeling clustered data with very few clusters. Multivariate Behavioral Research, 51, 495–518.CrossRefGoogle Scholar
  28. McNeish, D., & Wentzel, K. R. (2017). Accommodating small sample sizes in three-level models when the third level is incidental. Multivariate Behavioral Research, 52, 200–215.CrossRefGoogle Scholar
  29. Moeyaert, M., Ugille, M., Beretvas, S. N., Ferron, J., Bunuan, R., & Van den Noortgate, W. (2017) Methods for dealing with multiple outcomes in meta-analysis: a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis, International Journal of Social Research Methodology, 20, 559–572.CrossRefGoogle Scholar
  30. National Research Council. (1992). Combining information: Statistical issues and opportunities for research. Washington, DC: National Academy of Science Press.Google Scholar
  31. Raudenbush, S. W. (1994). Random effects models. In H. Cooper & L. V. Hedges (Eds.), Handbook of research synthesis (pp. 301–322). New York, NY: Russell Sage Foundation.Google Scholar
  32. Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988). Modeling multivariate effect sizes. Psychological Bulletin, 103, 111–120. doi: CrossRefGoogle Scholar
  33. Schmidt, F. L., Oh, I., & Hayes, T. L. (2009). Fixed- versus random-effects models in meta- analysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology, 62, 97–128.CrossRefGoogle Scholar
  34. Sheridan, S. M., Kim, E. M., Beretvas, S. N., Smith, T., & Park, S. Y. (2017). Family–school interventions and academic and social behavioral outcomes: What matters? Paper presented at the annual meeting of American Psychological Association, Washington, DC.Google Scholar
  35. Tanner-Smith, E. E., & Tipton, E. (2014). Robust variance estimation with dependent effect sizes: Practical considerations including a software tutorial in Stata and SPSS. Research Synthesis Methods, 5, 13–30. doi: CrossRefGoogle Scholar
  36. Tipton, E. (2013). Robust variance estimation in meta-regression for binary dependent outcomes. Research Synthesis Methods, 4(2), 169–187.
  37. Tipton, E. (2015). Small-sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20(3), 375–393. doi:
  38. Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013). Three-level meta-analyses of dependent effect sizes. Behavior Research Methods, 45, 576–594. doi: CrossRefGoogle Scholar
  39. Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2014). Meta-analysis of multiple outcomes: A multilevel approach. Behavior Research Methods, 47, 1274–1294.CrossRefGoogle Scholar
  40. Van den Noortgate, W., & Onghena, P. (2003). Multilevel meta-analysis: A comparison with traditional meta-analytical procedures. Educational and Psychological Measurement, 63, 765–79.CrossRefGoogle Scholar
  41. Van der Kleij, F. M., Feskens, R. C., & Eggen, T. J. (2015). Effects of feedback in a computer-based learning environment on students’ learning outcomes: A meta-analysis. Review of Educational Research, 85, 475–511.CrossRefGoogle Scholar
  42. Viechtbauer, W. (2005). Bias and efficiency of meta-analytic variance estimators in the random-effects model. Journal of Educational and Behavioral Statistics, 30, 261–293.CrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2018

Authors and Affiliations

  1. 1.Graduate School of PsychologyCalifornia Lutheran UniversityThousand OaksUSA
  2. 2.College of EducationUniversity of Texas at AustinAustinUSA

Personalised recommendations