Missing Data in Meta-analysis: Strategies and Approaches

Abstract

This chapter provides an overview of missing data issues that can occur in a meta-analysis. Common approaches to missing data in meta-analysis are discussed. The chapter focuses on the problem of missing data in moderators of effect size. The examples demonstrate the use of maximum likelihood methods and multiple imputation, the only two methods that produce unbiased estimates under the assumption that data are missing at random. The methods discussed in this chapter are most useful in testing the sensitivity of results to missing data.

Appendix

7.1.1 Computing Packages for Computation of the Multiple Imputation Results

There are a number of options for obtaining multiple imputation results in a meta-analysis model. Two freeware programs are available. The first is the program Norm by Schafer and available at http://www.stat.psu.edu/~jls/misoftwa.html. The Norm program runs as a stand alone program on Windows 95/98/NT. The second is a program available in R by Honaker et al. called Amelia II and available at http://gking.harvard.edu/amelia/. Schafer’s norm program was used for the example given earlier.

The program SAS includes two procedures, one for generating the multiple imputations, PROC MI, and a second for analyzing the completed data sets, PROC MIANALYZE. For obtaining the weighted regression results for meta-analysis, the SAS procedure PROC MIANALYZE will have limited utility since the standard errors of the weighted regression coefficients will need to be adjusted as detailed by Lipsey and Wilson (2001). Below is an illustration of the use of PROC MI for the leadership data.

7.1.1.1 R Programs

One program available in R for generating multiple imputations is Amelia II (Honaker et al. 2011). Directions for using the program are available at http://gking.harvard.edu/amelia/. Once the program is loaded into R, the following command was used to generate m = 5 imputed data sets.

> a.out < −amelia(leadimp, m = 5, idvars = "ID")

The imputed data sets can be saved for export into another program to complete the analyses using the command,

>write.amelia(obj = a.out, file.stem = "outdata").

where “obj” refers to the name given to the object with the imputed data sets (the result of using the command Amelia), and “file.stem” provides the name of the data sets that will be written from the program.

Table 7.12 are the weighted regression estimates for the effect size model from each imputation obtained in Amelia. The two variables missing observations are average age of subjects and percent of male leaders. There is variation among the five data sets in their estimates of the regression coefficients. This variation signals that there is some uncertainty in the data set due to missing observations.

Table 7.12 Regression estimates from each imputation generated using Amelia

Table 7.13 provides the multiply-imputed estimates for the linear model of effect size. These estimates were combined in Excel, and are fairly consistent with the earlier multiple imputation analysis using Schafer’s Norm program. None of the coefficients are significantly different from zero.

Table 7.13 Multiply-imputed estimates from Amelia

7.1.1.2 SAS Proc MI

The SAS procedure PROC MI provides a number of options for analyzing data with missing data. For the example illustrated in this chapter, we use the Monte Carlo Markov Chain with a single chain for the multiple imputations. We also use the EM estimates as the initial starting values for the MCMC analysis. The commands below were used with the leadership data to produce the five imputed data sets:

proc mi data = work.leader out = work.leaderimp seed = 101897;

var year ageave perlead gen2 sizeorg2 rndm2 effsize;

mcmc;

The first line of the command gives the name of the data set to use, the name of the created SAS data set with the imputations, and the seed number for the pseudo-random number generator. The second command line provides the variables to use in the imputations. Note that the effect size is included in this analysis. The third line specifies the use of Markov Chain Monte Carlo to obtain the estimates of the joint posterior distribution as described by Rubin (1987). Note that the number of imputations are not specified; the default number of imputed data sets generated is five, the number recommended by Schafer (1997).

SAS Proc MI provides a number of useful tables, including one outlining the missing data patterns and the group means for each variable within each missing data pattern. Once the imputations are generated, the procedure gives the estimates for the mean and standard error of the variables with missing data as illustrated below.

Multiple Imputation Parameter Estimates

Variable	Mean	SE	95% confidence limits		DF
Average age of sample	44.109	1.619	40.341	47.877	7.596
Percent of male leaders	65.691	2.898	59.743	71.640	26.869

Variable	Minimum	Maximum	Mu0	t for Mean = Mu0	Pr > |t|
Average age of sample	42.659	45.481	0	27.25	<.0001
Percent of male leaders	64.586	67.390	0	22.67	<.0001

To obtain the weighted regression results for each imputation, we use Proc Reg with weights. The command lines are shown below.

proc reg data = work.leaderimp outest = work.regout covout;

model effsize = year ageave perlead gen2 sizeorg2 rndm2;

weight wt;

by _Imputation_;

run;

The lines given above use the SAS data set generated by Proc MI, and estimate the coefficients for the effect size model using weighted regression. The results are computed for each imputation as indicated in the by statement. Table 7.14 provides the weighted regression results for each imputation.

Table 7.14 Multiple imputations generated using SAS Proc MI

Table 7.15 gives the multiply-imputed estimates for the weighted regression results. As in the prior analyses, none of the regression coefficients were significantly different from zero.

Table 7.15 Multiply-imputed estimates generated by SAS