Abstract
In this article we use Monte Carlo analysis to assess the small sample behaviour of the OLS, the weighted least squares (WLS) and the mixed effects meta-estimators under several types of effect size heterogeneity, using the bias, the mean squared error and the size and power of the statistical tests as performance indicators. Specifically, we analyse the consequences of heterogeneity in effect size precision (heteroskedasticity) and of two types of random effect size variation, one where the variation holds for the entire sample, and one where only a subset of the sample of studies is affected. Our results show that the mixed effects estimator is to be preferred to the other two estimators in the first two situations, but that WLS outperforms OLS and mixed effects in the third situation. Our findings therefore show that, under circumstances that are quite common in practice, using the mixed effects estimator may be suboptimal and that the use of WLS is preferable.
Article PDF
Similar content being viewed by others
References
Abreu M, De Groot HLF, Florax RJGM (2005) A meta-analysis of beta-convergence: the legendary 2%. J Econ Surv 19:389–420
Bijmolt THA, Pieters RGM (2001) Meta-analysis in marketing when studies contain multiple measurements. Mark Lett 12: 157–169
Brockwell SE, Gordon IR (2001) A comparison of statistical methods for meta-analysis. Stat Med 20: 825–840
Brons MRA, Nijkamp P, Pels E, Rietveld P (2008) A meta-analysis of the price elasticity of gasoline demand: a SUR approach. Energy Econ 30: 2105–2122
De Dominicis L, Florax RJGM, De Groot HLF (2008) Meta-analysis of the relationship between income inequality and economic growth. Scott J Political Econ 55: 654–682
Field AP (2001) Meta-analysis of correlation coefficients: a Monte Carlo comparison of fixed- and random-effects methods. Psychol Methods 6: 161–180
Greene WH (2000) Econometric analysis, 4th edn. Prentice-Hall, Upper Saddle Rivern, New Jersey
Hedges LV (1994) Fixed effects models. In: Cooper H, Hedges LV (eds) The handbook of research synthesis. Russell Sage Foundation, New York
Hedges LV, Olkin I (1985) Statistical methods for meta-analysis. Academic Press, Orlando, Florida
Higgins JPT, Thompson SG (2004) Controlling the risk of spurious findings from meta-regression. Stat Med 23: 1663–1682
Hunt M (1997) How science takes stock: the story of meta-analysis. Russell Sage Foundation, New York
Koetse MJ, De Groot HLF, Florax RJGM (2008) Capital-energy substitution and shifts in factor demand: a meta-analysis. Energy Econ 30: 2236–2251
Koetse MJ, De Groot HLF, Florax RJGM (2009) A meta-analysis of the investment-uncertainty relationship. South Econ J 76: 283–306
Koetse MJ, Florax RJGM, De Groot HLF (2005) Correcting for primary study misspecifications in meta-analysis. Tinbergen Institute Discussion Paper 05-029/3, Tinbergen Institute, Amsterdam
Kuhnert R, Böhning D (2007) A comparison of three different models for estimating relative risk in meta-analysis of 3 clinical trials under unobserved heterogeneity. Stat Med 26: 2277–2296
Nijkamp P, Poot J (2004) Meta-analysis of the effect of fiscal policies on long-run growth. Eur J Political Econ 20: 91–124
Nijkamp P, Poot J (2005) The last word on the wage curve? A meta-analytic assessment. J Econ Surv 19: 421–450
Oswald FL, Johnson JW (1998) On the robustness, bias, and stability of statistics from meta-analysis of correlation coefficients: some initial Monte Carlo findings. J Appl Psychol 83: 164–178
Roberts CJ, Stanley TD (2005) Meta-regression analysis: issues of publication bias in economics. Blackwell, Oxford
Sanchez-Meca J, Marin-Martinez F (1997) Homogeneity tests in meta-analysis: a Monte Carlo comparison of statistical power and Type I error. Qual Quant 31: 385–399
Sanchez-Meca J, Marin-Martinez F (1998) Weighting by inverse variance or by sample size in meta-analysis: a simulation study. Educ Psychol Meas 58: 211–220
Stanley TD (2001) Wheat from chaff: meta-analysis as quantitative literature review. J Econ Perspect 15: 131–150
Stanley TD (2008) Meta-regression methods for detecting and estimating empirical effects in the presence of publication selection. Oxf Bull Econ Stat 70: 103–127
Stanley TD, Jarrell SB (1989) Meta-regression analysis: a quantitative method of literature surveys. J Econ Surv 3: 54–67
Sutton AJ, Abrams KR, Sheldon TA, Song F (2000) Methods for meta-analysis in medical research. Wiley, New York
Weichselbaumer D, Winter-Ebmer R (2005) A meta-analysis of the international gender wage gap. J Econ Surv 19: 479–511
Acknowledgments
This research is supported through the program ‘Stimulating the Adoption of Energy-Efficient Technologies’, funded by the Netherlands Organization for Scientific Research (NWO) and the Dutch Ministry of Economic Affairs (SenterNovem). We are grateful to two anonymous referees for useful comments on an earlier version of this article. The usual disclaimer applies.
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Koetse, M.J., Florax, R.J.G.M. & de Groot, H.L.F. Consequences of effect size heterogeneity for meta-analysis: a Monte Carlo study. Stat Methods Appl 19, 217–236 (2010). https://doi.org/10.1007/s10260-009-0125-0
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10260-009-0125-0
Keywords
- Effect size heterogeneity
- Meta-analysis
- Monte Carlo analysis
- OLS meta-estimator
- WLS meta-estimator
- Mixed effects meta-estimator
- Small sample performance