Abstract
Individual participant data (IPD) meta-analysis is a meta-analysis in which the individual-level data for each study are obtained and used for synthesis. A common challenge in IPD meta-analysis is when variables of interest are measured differently in different studies. The term harmonization has been coined to describe the procedure of placing variables on the same scale in order to permit pooling of data from a large number of studies. Using data from an IPD meta-analysis of 19 adolescent depression trials, we describe a multiple imputation approach for harmonizing 10 depression measures across the 19 trials by treating those depression measures that were not used in a study as missing data. We then apply diagnostics to address the fit of our imputation model. Even after reducing the scale of our application, we were still unable to produce accurate imputations of the missing values. We describe those features of the data that made it difficult to harmonize the depression measures and provide some guidelines for using multiple imputation for harmonization in IPD meta-analysis.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Achenbach, T.M. (1991). Manual for the Child Behavior Checklist/4-18 and 1991 profile [Computer software manual]. Burlington VT: University of Vermont, Department of Psychiatry.
Bauer, D.J., & Hussong, A.M. (2009). Psychometric approaches for developing commensurate measures across independent studies: Traditional and new models. Psychological Methods, 14, 101–125.
Brown, C.H., Brincks, A., Huang, S., Perrino, T., Cruden, G., Pantin, H., & Sandler, I. (2016). Two-year in impact of prevention programs on adolescent depression: An integrative data analysis approach. Prevention Science. doi:10.1007/s11121-016-0737-1.
Clarke, G.N., Lewinsohn, P.M., Hops, H., & Seeley, J.R. (1992). A self-and parent-report measure of adolescent depression: The Child Behavior Checklist Depression scale (CBCL-D). Behavioral Assessment, 14.
Cowles, M.K., & Carlin, B.P. (1996). Markov chain monte carlo convergence diagnostics: A comparative review. Journal of the American Statistical Association, 91, 883–904.
Curran, P.J. (2009). The seemingly quixotic pursuit of a cumulative psychological science: Introduction to the special issue. Psychological Methods, 14, 77–80.
Curran, P.J., & Hussong, A.M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–100.
Curran, P.J., Hussong, A.M., Cai, L., Huang, W., Chassin, L., Sher, K. J., & Zucker, R.A. (2008). Pooling data from multiple longitudinal studies: The role of item response theory in integrative data analysis. Developmental Psychology, 44, 365–380.
Dagne, G.A., Brown, C.H., Howe, G., Kellam, S.G., & Liu, L. (2016). Testing moderation in network meta-analysis with individual participant data. Statistics in Medicine, 34, 2485–2502.
Eaton, W.W., Smith, C., Ybarra, M., Muntaner, C., & Tien, A. (2004). Center for epidemiologic studies depression scale: review and revision (CESD and CESD-r). In Maruish, M. E. (Ed.) The Use of Psychological Testing for Treatment Planning and Outcomes Assessment: Volume 3: Instruments for Adults. 3rd edn (pp. 363–377). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.
Ebesutani, C., Bernstein, A., Martinez, J.I., Chorpita, B.F., & Weisz, J.R. (2011). The youth self report: Applicability and validity across younger and older youths. Journal of Clinical Child and Adolescent Psychology, 40, 338–346.
Gelman, A., Carlin, J.B., Stern, H.S., & Rubin, D.B. (2004). Bayesian data analysis, 2nd edn. Boca Raton, FL: Chapman and Hall/CRC press.
Gelman, A., King, G., & Liu, C. (1998). Not asked and not answered: Multiple imputation for multiple surveys. Journal of the American Statistical Association, 93, 846–857.
Gelman, A., Meng, X.-L., & Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6, 733–760.
Gelman, A., & Rubin, D.B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 457–472.
Geweke, J. (1992). Evaluating the accuracy of sampling-based approaches to calculating posterior moments. In Bernado, J. M., Berger, J. O. , Dawid, A. P., & Smith, A. F. (Eds.) Bayesian statistics, (Vol. 4 pp. 169–193). Oxford: Clarendon Press.
Griffith, L.E., Van Den Heuvel, E., Fortier, I., Sohel, N., Hofer, S.M., Payette, H., & et al. (2015). Statistical approaches to harmonize data on cognitive measures in systematic reviews are rarely reported. Journal of Clinical Epidemiology, 68, 154–162.
Hamilton, M. (1960). A rating scale for depression. Journal of Neurology, Neurosurgery, and Psychiatry, 23, 56–62.
Harel, O., & Zhou, X.-H. (2007). Multiple imputation: Review of theory, implementation and software. Statistics in Medicine, 26, 3057–3077.
He, Y., & Zaslavsky, A.M. (2012). Diagnosing imputation models by applying target analyses to posterior replicates of completed data. Statistics in Medicine, 31, 1–18.
Helsel, W.J., & Matson, J.L. (1984). The assessment of depression in children: The internal structure of the Child Depression Inventory CDI. Behaviour Research and Therapy, 22, 289–298.
Howe, G.W., Dagne, G., Brown, C.H., Brincks, A., & Beardslee, W. (2017). Evaluating construct equivalence and harmonizing measurement of adolescent depression when synthesizing results across multiple studies. In preparation.
Hussong, A. M., Curran, P. J., & Bauer, D. J. (2013). Integrative data analysis in clinical psychology research. Annual Review of Clinical Psychology, 9, 61–89.
Kline, D., Andridge, R., & Kaizar, E. (2015). Comparing multiple imputation methods for systematically missing subject-level data. Research Synthesis Methods, 1–13.
Kovacs, M. (1984). The Children’s Depression Inventory CDI. Psychopharmacology Bulletin, 21, 995–998.
Mayes, T.L., Bernstein, I.H., Haley, C.L., Kennard, B.D., & Emslie, G.J. (2010). Psychometric properties of the Children’s Depression Rating Scale-revised in adolescents. Journal of Child and Adolescent Psychopharmacology, 20, 513–516.
National Institutes of Health (2003). Final NIH statement on sharing research data. (http://grants.nih.gov/grants/guide/notice-files/NOT-OD-03-032.html [Accessed 3-March-2014]).
National Science Foundation (2011). Dissemination and sharing of research results. (http://www.nsf.gov/pubs/policydocs/pappguide/nsf11001/aag_6.jsp#VID4 [Accessed 3-March-2014]).
Perrino, T., Howe, G., Sperling, A., Beardslee, W., Sandler, I., Shern, D., & Brown, C.H. (2013). Advancing science through collaborative data sharing and synthesis. Perspectives on Psychological Science, 8, 433–444.
Poznanski, E.O., Freeman, L.N., & Mokros, H.B. (1985). Children’s depression rating scale-revised (September 1984). Psychopharmacology Bulletin, 21, 979–989.
Quay, H.C., & Peterson, D.R. (1996). Revised Behavior Problem Checklist. Odessa, FL: Psychological Assessment Resources.
R Core Team (2012). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Retrieved from http://www.R-project.org/ (ISBN 3-90005 1-07-0).
Radloff, L.S. (1977). The CES-D scale a self-report depression scale for research in the general population. Applied Psychological Measurement, 1, 385–401.
Radloff, L.S. (1991). The use of the Center for Epidemiologic Studies Depression Scale in adolescents and young adults. Journal of Youth and Adolescence, 20, 149–166.
Rässler, S (2003). A non-iterative Bayesian approach to statistical matching. Statistica Neerlandica, 57, 58–74.
Resche-Rigon, M., White, I.R., Bartlett, J.W., Peters, S.A., & Thompson, S.G. (2013). Multiple imputation for handling systematically missing confounders in meta-analysis of individual participant data. Statistics in Medicine, 32, 4890–4905.
Riley, R.D., Lambert, P.C., & Abo-Zaid, G. (2010). Meta-analysis of individual participant data: Rationale, conduct, and reporting. BMJ: British Medical Journal, 521–525.
Rodwell, L., Lee, K.J., Romaniuk, H., & Carlin, J.B. (2014). Comparison of methods for imputing limited-range variables: A simulation study. BMC Medical Research Methodology, 14, 57.
Rubin, D.B. (1987). Multiple imputation for nonresponse in surveys. New York: John Wiley and Sons.
Schafer, J.L., & Yucel, R.M. (2002). Computational strategies for multivariate linear mixed-effects models with missing values. Journal of Computational and Graphical Statistics, 11, 437– 457.
Schifeling, T.A., & Reiter, J.P (2015). Incorporating marginal prior information in latent class models.
Siddique, J., Reiter, J.P., Brincks, A., Gibbons, R.D., Crespi, C.M., & Brown, C.H. (2015). Multiple imputation for harmonizing longitudinal non-commensurate measures in individual participant data meta-analysis. Statistics in Medicine, 34, 3399– 3414.
Zhao, J.H., & Schafer, J.L. (2013). Pan: Multiple imputation for multivariate panel or clustered data [Computer software manual]. (R package version 0.9).
Acknowledgments
We gratefully acknowledge the National Institute of Mental Health Collaborative Synthesis for Adolescent Depression Trials Study Team, comprised of our many colleagues who generously provided their data to be used in this study, obtained access to key datasets, reviewed coding decisions, or provided substantive or methodologic recommendations. We also thank NIH for their support through Grant Number R01MH040859 (Collaborative Synthesis for Adolescent Depression Trials, Brown PI), and the following grants: Siddique-NCI CA154862-01, Garber, Brent, Beardslee, Clarke et al. NIMH MH64735, MH6454, MH64717, Gillham et al.– NIMH MH52270, Garber et al.– William T. Grant Foundation 961730, Dishion et al.– NIDA DA07031 and DA13773, Gillham et al.– NIMH MH52270, Szapocznik et al.– NIMH MH61143, Pantin et al.– NIDA DA017462, Prado et al.– NIDA DA025894, Prado et al.– CDC U01PS000671, Stormshak et al.– NIDA DA018374, Sandler et al.– NIMH MH49155, Wolchik et al.– NIMH MH068685, Young et al.– NARSAD, Spoth et al. – NIDA DA 007029, Clarke et al.– NIMH MH 48118, Young et al.– NIMH MH071320, Beardslee et al.– NIMH MH48696, VanVoorhees et al.– NIMH MH072918, and Gonzales et al. NIMH MH64707. The content of this paper is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies nor that of our collaborators who provided access to their data.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interests
The authors declare that they have no conflict of interest.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent
Informed consent was obtained from all individual participants included in the study.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Siddique, J., de Chavez, P.J., Howe, G. et al. Limitations in Using Multiple Imputation to Harmonize Individual Participant Data for Meta-Analysis. Prev Sci 19 (Suppl 1), 95–108 (2018). https://doi.org/10.1007/s11121-017-0760-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-017-0760-x