Including Individual Participant Data in Meta-analysis

Abstract

This chapter introduces methods for including individual participant data in a traditional meta-analysis. Since meta-analyses use data aggregated to the study, there is potential for aggregation bias, finding relationships between the effect size and study characteristics that may hold only at the level of the study. The potential of aggregation bias may limit the application of meta-analysis results to practice and policy. This chapter provides an example of including publicly available data in a traditional meta-analysis.

Appendix: SAS Code for Meta-analyses Using a Mix of IPD and AD

8.1.1 SAS Code for Simple Random Effects Model Using the Two-Step Method

Below is the code needed to run a random effects meta-analysis with the raw correlation. The model for this analysis is given in (8.5). The data for this analysis should include a variable “outcome” that holds the effect size for each study (in this case, the correlation coefficient), a variable that provides a unique identifier for each study, labeled “study” in this example. We also need the variances for each effect size estimate, in this example, the estimated variance of the correlation coefficient as given in (8.6). The first line of code calls the SAS procedure Proc Mixed. The options in this line, noclprint and covtest are options for output. The noclprint suppresses the printing of class level information, or the list of all studies in this example. The covtest prints out the standard errors and test statistics for the variance and covariance parameters. The class statement indicates that “study” is a class variable, which will be designated as the random effect later in the code. The model statement asks for a simple random effects model with “outcome” as the dependent variable. The option solution prints out the fixed-effects parameter estimates, which in this example is the estimate of the mean effect size (“outcome”). The random statement designates the random effect, and the option solution prints the estimate of the random effect, or in this case, the random effects variance. The repeated statement specifies the covariance matrix for the error terms, and in this case, allows between group heterogeneity. The parms statement provides the starting values for the covariance parameters. The first value is the starting value for the overall variance component for this model. The next 42 elements are the within-study estimates of the variance of the effect size, here the correlation. The option eqcons fixes the variances for the 42 studies in the analysis since these are considered known in a meta-analysis model.

proc mixed noclprint covtest;

class study;

model outcome =/solution;

random study/solution;

repeated/group = study;

parms

(0.05)

(.000071) (.000311) (.001584) (.008978) (.000222) (.000041) (.000072)

(.004656) (.002213) (.007193) (.002426) (.008054) (.002803) (.003252)

(.000616) (.005791) (.000557) (.001512) (.000676) (.001345) (.002302)

(.002746) (.003254) (.007465) (.010415) (.004995) (.008849) (.002823)

(.002765) (.000083) (.000221) (.002669) (.000665) (.002237) (.000598)

(.000924) (.000813) (.001663) (.003136) (.001659) (.027218) (.005051)

/eqcons=2 to 43;

run;

8.1.1.1 Output from Two-Stage Simple Random Effects Model

Table 8.2 gives the estimate of the variance component in the first line along with its standard error and test of significance. Table 8.3 provides the random effects mean for the overall effect size.

Table 8.2 Estimate of variance component from SAS Proc Mixed

Table 8.3 Estimate of random effects mean from SAS Proc Mixed

8.1.2 SAS Code for Meta-regression Using the Two-Stage Method

The code here is the same as for the aggregated data simple random effects model except for the addition of a moderator in the model statement. The model for this analysis is given in (8.8). The moderator in this example is permin, the percent of minority students in the study sample. As in the example above, restricted maximum likelihood is the default estimation method.

proc mixed noclprint covtest;

class study;

model outcome = permin/solution;

random study/solution;

repeated/group = study;

parms

(0.05)

(.000071) (.000311) (.001584) (.008978) (.000222) (.000041) (.000072)

(.004656) (.002213) (.007193) (.002426) (.008054) (.002803) (.003252)

(.000616) (.005791) (.000557) (.001512) (.000676) (.001345) (.002302)

(.002746) (.003254) (.007465) (.010415) (.004995) (.008849) (.002823)

(.002765) (.000083) (.000221) (.002669) (.000665) (.002237) (.000598)

(.000924) (.000813) (.001663) (.003136) (.001659) (.027218) (.005051)

/eqcons=2 to 43;

run;

8.1.2.1 Output from Meta-regression Using the Two-Stage Method

Table 8.4 gives the estimate of the variance component in the first line along with its standard error and test of significance conditional on the moderator permin. Table 8.5 provides the random effects mean for the overall effect size given the percent of minority students in the sample. Note that the slope for permin is not statistically significant, and thus in this analysis, the percent of the minority students in the sample is not associated with the correlation between SES and achievement in these studies.

Table 8.4 Estimate of variance component from SAS Proc Mixed for meta-regression

Table 8.5 Estimates for meta-regression using SAS Proc Mixed

8.1.3 SAS Code for Simple Random Effects Model Using the One-Stage Model

The code below estimates a simple random effects model with both IPD and AD data. The data is set up as in Table 8.6. Two of the AD studies are given in the first two lines of the table. The AD studies contribute one observation (Person =1), does not provide individual level data (IPD = 0), has a value of the standardized SES of 1, and has an outcome equal to the correlation estimated in that study. The last lines provide examples of data from individuals in the NELS and NLSY data set. Each individual has a person identification number, has a value of IPD=1 since individual level data is provided, and also provides a standardized value for SES and for the outcome which is achievement in our example.

Table 8.6 Example of data for a mixed IPD and AD analysis

Given the values for the variables in Table 8.6, the SAS code follows the general structure of the examples for the AD random effects models. The first line calls the SAS Proc Mixed, with option cl and noclprint defined as in earlier sections. The option method = reml specifies the estimation method as restricted maximum likelihood (the default). Since we have two levels in our model – the within-study level for the studies that provide IPD, and the between-study level, we have two class variables, person and study. The model statement reflects the model discussed in (8.13). The outcome for the IPD studies, achievement in its standardized form, will be modeled with study as a factor, and zses as a predictor. Since ipd =0 for the AD studies and zses = 1, the model for the AD studies will be equivalent to the model given for simple random effects as given above. The options for the model command are noint which fits a no-intercept model, s which is short for solution and provides the solution for the fixed effects parameters. The option cl produces the confidence limits for the covariance parameter estimates, and covb gives the estimated variance-covariance matrix of the fixed-effects parameters. The random command line indicates that study is a random effect. The command line for repeated specifies the variance-covariance matrix for the mixed model. The option type = un specifies that the variance-covariance matrix is unstructured. The option subject = study(person) indicates that the variable persons is nested within study. The option group=study indicates that observations having the same value of study are at the same level and should have the same covariance parameters. The parms command provides a number of values for the variance parameters. The first three values after parms are the starting values for the random effects variances, the first for the overall random effects variance between studies, the second two for the NELS and the NLSY studies. The next set of parameters is the within-study variances for the outcome or effect size.

proc mixed noclprint cl method=reml;

class person study;

model outcome = study*ipd zses/noint s cl covb;

random study;

repeated/type = un subject = study(person) group = study;

parms

(0.05) (0.05) (0.05)

(0.001584129) (0.008978216) (0.000222147) (4.14666E-05)

(7.23005E-05) (0.004655734) (0.002213119) (0.007192762)

(0.000221388) (0.002668852) (0.000664976) (0.002236663)

(0.002425768) (0.008054047) (0.002802574) (0.00325176)

(0.000616328) (0.005790909) (0.000557348) (0.000597936)

(0.000923595) (0.001511538) (0.000676163) (0.001345186)

(0.002302425) (0.002746439) (0.003253767) (0.000812903)

(0.007464759) (0.010414583) (0.004995399) (0.008849115)

(0.002822828) (0.002765381) (8.29676E-05) (0.001659)

(0.001662966) (0.003136358) (0.02721837) (0.005050641)

/eqcons= 4 to 43;

run;

8.1.3.1 Output from One-Stage Simple Random Effects Model

The important parts of this output are under Covariance Parameter Estimates, which gives the estimate of the variance component for the simple random effects model with both individual-level and study-level data. The table below looks different from the one in the simple random effects model with only study-level data since one option used here was cl, the confidence limits for the covariance parameters. The second table gives the estimate of the random effects mean effect size, as the solution for the fixed effects for the variable Zses (Tables 8.7 and 8.8).

Table 8.7 Estimate of variance component from SAS Proc Mixed for simple random effects model using the one-step method

Table 8.8 Estimates for simple random effects model using SAS Proc Mixed

8.1.4 SAS Code for a Meta-regression Model Using the One-Step Method

The SAS code for a meta-regression moderator model in the one-step method includes the same code as for the simple random effects models as given above. The main difference occurs in the “model” statement, where we add a number of variables and interactions to the model. The “ipd*min” variable is the within-study effect of being a minority student on the association between achievement and SES within the IPD studies. The dummy code for minority student has also been centered at the study mean. The factor “idp*zses*min” is the interaction effect between SES and being a minority student on student achievement. The “zses*permin” effect is the between-study association of the percent of minority students with the study-level correlation between achievement and SES. The rest of the command lines are the same as in the one-step method for the simple random effects model.

proc mixed cl noclprint method=reml;

class person study;

model outcome = newid*ipd zses ipd*min ipd*zses*min zses*permin/noint s cl covb;

random study;

repeated/type = un subject = study(person) group = study;

parms

(0.05) (0.05) (0.05)

(0.001584129) (0.008978216) (0.000222147) (4.14666E-05)

(7.23005E-05) (0.004655734) (0.002213119) (0.007192762)

(0.000221388) (0.002668852) (0.000664976) (0.002236663)

(0.002425768) (0.008054047) (0.002802574) (0.00325176)

(0.000616328) (0.005790909) (0.000557348) (0.000597936)

(0.000923595) (0.001511538) (0.000676163) (0.001345186)

(0.002302425) (0.002746439) (0.003253767) (0.000812903)

(0.007464759) (0.010414583) (0.004995399) (0.008849115)

(0.002822828) (0.002765381) (8.29676E-05) (0.001659)

(0.001662966) (0.003136358) (0.02721837) (0.005050641)

/eqcons= 4 to 43;

run;

8.1.4.1 Output for Meta-regression Using the One-Step Method

Table 8.9 below gives the conditional random variance component for the model, 0.0319, which is similar to the meta-regression estimate using the two-step method. In Table 8.10, we see the estimates for the multilevel model for the one-step method. The overall estimate of the random effects conditional mean is 0.362, similar to the two-step result, with the overall estimate of the between-study effect of percent of minority students as −0.167, which in this analysis is statistically significant. The two within-study factors, being a minority study, and the interaction between minority status and SES, are also both significant as seen in the lines just before the bottom of the second table.

Table 8.9 Estimate of the variance component from SAS Proc Mixed for meta-regression model using one-step method

Table 8.10 Estimates for meta-regression in one-step method