Advertisement

Prevention Science

, Volume 14, Issue 2, pp 144–156 | Cite as

Methods for Synthesizing Findings on Moderation Effects Across Multiple Randomized Trials

  • C. Hendricks Brown
  • Zili Sloboda
  • Fabrizio Faggiano
  • Brent Teasdale
  • Ferdinand Keller
  • Gregor Burkhart
  • Federica Vigna-Taglianti
  • George Howe
  • Katherine Masyn
  • Wei Wang
  • Bengt Muthén
  • Peggy Stephens
  • Scott Grey
  • Tatiana Perrino
  • Prevention Science and Methodology Group
Article

Abstract

This paper presents new methods for synthesizing results from subgroup and moderation analyses across different randomized trials. We demonstrate that such a synthesis generally results in additional power to detect significant moderation findings above what one would find in a single trial. Three general methods for conducting synthesis analyses are discussed, with two methods, integrative data analysis and parallel analyses, sharing a large advantage over traditional methods available in meta-analysis. We present a broad class of analytic models to examine moderation effects across trials that can be used to assess their overall effect and explain sources of heterogeneity, and present ways to disentangle differences across trials due to individual differences, contextual level differences, intervention, and trial design.

Keywords

Meta-analysis Parallel data analysis Integrative data analysis Variation in impact Subgroup analyses 

Notes

Acknowledgements

We would like to thank our colleagues in the Prevention Science and Methodology Group (PSMG) for their suggestions in the development of this paper. This project was funded by a National Institute on Drug Abuse supplement to the Prevention Science and Methodology Group for the US-EU Drug Abuse Prevention Project (R01MH040859; Brown, Sloboda, Muthén, Masyn, Wang), Robert Wood Johnson Foundation (No. 039223, 040371) for Sloboda, Stephens, Grey, Teasdale. The EU-Drug Abuse Prevention Project is funded by the European Commission (European Public Health programme 2002 grant # SPC 2002376, Faggiano, Vigna-Taglianti), and the parallel data analyses supported by the European Monitoring Centre for Drugs and Drug Addiction (Burkhart, CT.09.RES.005.1.0: Keller). We would also like to thank J G Perpich LLC for their support in the use of the NIDA International Virtual Collaboratory funded through N44DA000000-00409, for their logistical support in developing our international collaboration. A version of this paper was presented by the first author at the “Foundational Issues in Examining Subgroup Effects in Experiments” Interagency Federal Methodological Meeting: Subgroup Analysis in Prevention and Intervention Research, Washington, DC.

References

  1. Asparouhov, T., & Muthén, B. O. (2008). Multilevel mixture models. In G. R. Hancock & K. M. Samuelsen (Eds.), Advances in latent variable mixture models (pp. 27–51). Charlotte, NC: Information Age.Google Scholar
  2. Barker, L. E., Luman, E. T., McCauley, M. M., & Chu, S. Y. (2002). Assessing equivalence: An alternative to the use of difference tests for measuring disparities in vaccination coverage. American Journal of Epidemiology, 156, 1056–1061.PubMedCrossRefGoogle Scholar
  3. Bauer, D. J., & Hussong, A. M. (2009). Psychometric approaches for developing commensurate measures across independent studies: Traditional and new models. Psychological Methods, 14, 101–125.PubMedCrossRefGoogle Scholar
  4. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate—a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B-(Statistical Methodology), 57, 289–300.Google Scholar
  5. Berlin, J. A., Santanna, J., Schmid, C. H., Szczech, L. A., & Feldman, H. I. (2002). Individual patient-versus group-level data meta-regressions for the investigation of treatment effect modifiers: Ecological bias rears its ugly head. Statistics in Medicine, 21, 371–387.PubMedCrossRefGoogle Scholar
  6. Bridge, J. A., Iyengar, S., Salary, C. B., Barbe, R. P., Birmaher, B., Pincus, H. A., et al. (2007). Clinical response and risk for reported suicidal ideation and suicide attempts in pediatric antidepressant treatment: A meta-analysis of randomized controlled trials. Journal of the American Medical Association, 297, 1683–1696.PubMedCrossRefGoogle Scholar
  7. Brown, C. H. (1993). Analyzing preventive trials with generalized additive models. American Journal of Community Psychology, 21, 635–664.PubMedCrossRefGoogle Scholar
  8. Brown, C. H., & Liao, J. (1999). Principles for designing randomized preventive trials in mental health: An emerging developmental epidemiology paradigm. American Journal of Community Psychology, 27, 673–710.PubMedCrossRefGoogle Scholar
  9. Brown, E. C., Hawkins, J. D., Arthur, M. W., Briney, J. S., & Abbott, R. D. (2007). Effects of Communities That Care on prevention services systems: Findings from the community youth development study at 1.5 years. Prevention Science, 8, 180–191.PubMedCrossRefGoogle Scholar
  10. Brown, C. H., Wang, W., Kellam, S. G., Muthén, B. O., Petras, H., Toyinbo, P., et al. (2008a). Methods for testing theory and evaluating impact in randomized field trials: Intent-to-treat analyses for integrating the perspectives of person, place, and time. Drug and Alcohol Dependence, 95, S74–S104.CrossRefGoogle Scholar
  11. Brown, C. H., Wang, W., & Sandler, I. (2008b). Examining how context changes intervention impact: The use of effect sizes in multilevel meta-analysis. Child Development Perspectives, 2, 198–205.CrossRefGoogle Scholar
  12. Coie, J. D., Watt, N. F., West, S. G., Hawkins, J. D., Asarnow, J. R., Markman, H. J., et al. (1993). The science of prevention: A conceptual framework and some directions for a national research program. American Psychologist, 48, 1013–1022.PubMedCrossRefGoogle Scholar
  13. Cook, T. D. (1992). Meta-analysis for explanation: A casebook. New York: Russell Sage.Google Scholar
  14. Cooper, H. (1999). Synthesizing research: A guide for literature reviews (3rd ed.). Thousand Oaks, CA: Sage.Google Scholar
  15. Cooper, H. M. (2010). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Los Angeles: Sage.Google Scholar
  16. Cooper, H., & Patall, E. A. (2009). The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. Psychological Methods, 14, 165–176.PubMedCrossRefGoogle Scholar
  17. Cooper, H. M., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage.Google Scholar
  18. Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–100.PubMedCrossRefGoogle Scholar
  19. Curran, P. J., Hussong, A. M., Cai, L., Huang, W., Chassin, L., Sher, K. J., et al. (2008). Pooling data from multiple longitudinal studies: The role of item response theory in integrative data analysis. Developmental Psychology, 44, 365–380.PubMedCrossRefGoogle Scholar
  20. DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177–188.PubMedCrossRefGoogle Scholar
  21. Dillman Carpentier, F. R., Mauricio, A. M., Gonzales, N. A., Millsap, R. E., Meza, C. M., Dumka, L. E., et al. (2007). Engaging Mexican origin families in a school-based preventive intervention. Journal of Primary Prevention, 28, 521–546.PubMedCrossRefGoogle Scholar
  22. Dishion, T. J., Spracklen, K. M., Andrews, D. W., & Patterson, G. R. (1996). Deviancy training in male adolescents friendships. Behavior Therapy, 27, 373–390.CrossRefGoogle Scholar
  23. Dishion, T. J., McCord, J., & Poulin, F. (1999). When interventions harm. Peer groups and problem behavior. American Psychologist, 54, 755–764.PubMedCrossRefGoogle Scholar
  24. Dishion, T. J., Burraston, B., & Poulin, F. (2001). Peer group dynamics associated with iatrogenic effects in group interventions with high-risk young adolescents. In C. Erdley & D. W. Nangle (Eds.), Damon’s new directions in child development: The role of friendship in psychological adjustment (pp. 79–92). San Francisco: Jossey-Bass.Google Scholar
  25. Dolan, L. J., Kellam, S. G., Brown, C. H., Werthamer-Larsson, L., Rebok, G. W., Mayer, L. S., et al. (1993). The short-term impact of two classroom-based preventive interventions on aggressive and shy behaviors and poor achievement. Journal of Applied Developmental Psychology, 14, 317–345.CrossRefGoogle Scholar
  26. Durlak, J. A., & Wells, A. M. (1997). Primary prevention mental health programs for children and adolescents: A meta-analytic review. American Journal of Community Psychology, 25, 115–152.PubMedCrossRefGoogle Scholar
  27. Elliott, D. S., & Mihalic, S. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science, 5, 47–53.PubMedCrossRefGoogle Scholar
  28. Ennett, S. T., Tobler, N. S., Ringwalt, C. L., & Flewelling, R. L. (1994). How effective is drug abuse resistance education? A meta-analysis of Project DARE outcome evaluations. American Journal of Public Health, 84, 1394–1401.PubMedCrossRefGoogle Scholar
  29. Faggiano, F., Vigna-Taglianti, F., Versino, E., Zambon, A., Borraccino, A., & Lemma, P. (2005). School-based prevention for illicit drugs’ use. Cochrane Database of Systematic Reviews, 2, Art. No.: CD003020.Google Scholar
  30. Faggiano, F., Vigna-Taglianti, F. D., Versino, E., Zambon, A., Borraccino, A., & Lemma, P. (2008). School-based prevention for illicit drugs use: A systematic review. Preventive Medicine, 46, 385–396.PubMedCrossRefGoogle Scholar
  31. Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., et al. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6, 151–175.PubMedCrossRefGoogle Scholar
  32. Fournier, J. C., DeRubeis, R. J., Hollen, S. D., Dimdjian, S., Amsterdam, J. D., Shelton, R. C., et al. (2010). Antidepressant drug effects and depression severity: Patient-level meta-analysis. Journal of the American Medical Association, 303, 47–53.PubMedCrossRefGoogle Scholar
  33. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8.Google Scholar
  34. Glass, G. V. (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351–379.Google Scholar
  35. Glass, T. A., & McAtee, M. J. (2006). Behavioral science at the crossroads in public health: Extending horizons, envisioning the future. Social Science & Medicine, 62, 1650–1671.CrossRefGoogle Scholar
  36. Glass, G. V., & Smith, M. L. (1978). Meta-analysis of research on the relationship of class-size and achievement: The Class Size and Instruction Project. National Institute of Education (DHEW), Washington, DC.Google Scholar
  37. Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.Google Scholar
  38. Hastie, T., & Tibshirani, R. (1990). Generalized additive models (1st ed.). London: Chapman and Hall.Google Scholar
  39. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic.Google Scholar
  40. Higgins, J. P. T., & Green, S. (Eds.). (2008). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2 [updated September 2009]: The Cochrane Collaboration. Retrieved from http://www.cochrane-handbook.org
  41. Higgins, J. P., Whitehead, A., Turner, R. M., Omar, R. Z., & Thompson, S. G. (2001). Meta-analysis of continuous outcome data from individual patients. Statistics in Medicine, 20, 2219–2241.PubMedCrossRefGoogle Scholar
  42. Hofer, S. M., & Piccinin, A. M. (2009). Integrative data analysis through coordination of measurement and analysis protocol across independent longitudinal studies. Psychological Methods, 14, 150–164.PubMedCrossRefGoogle Scholar
  43. Horowitz, J. L., & Garber, J. (2006). The prevention of depressive symptoms in children and adolescents: A meta-analytic review. Journal of Consulting and Clinical Psychology, 74, 401–415.PubMedCrossRefGoogle Scholar
  44. Howe, G. W., Reiss, D., & Yuh, J. (2002). Can prevention trials test theories of etiology? Development and Psychopathology, 14, 673–693.PubMedCrossRefGoogle Scholar
  45. Hussong, A., Cai, L., Curran, P., Flora, D., Chassin, L., & Zucker, R. (2008). Disaggregating the distal, proximal, and time-varying effects of parent alcoholism on children’s internalizing symptoms. Journal of Abnormal Child Psychology, 36, 335–346.PubMedCrossRefGoogle Scholar
  46. Ialongo, N. S., Werthamer, L., Kellam, S. G., Brown, C. H., Wang, S., & Lin, Y. (1999). Proximal impact of two first-grade preventive interventions on the early risk behaviors for later substance abuse, depression, and antisocial behavior. American Journal of Community Psychology, 27, 599–641.PubMedCrossRefGoogle Scholar
  47. Ialongo, N., Poduska, J., Werthamer, L., & Kellam, S. (2001). The distal impact of two first-grade preventive interventions on conduct problems and disorder in early adolescence. Journal of Emotional & Behavioral Disorders, 9, 146–160.CrossRefGoogle Scholar
  48. Jané-Llopis, E., Hosman, C., Jenkins, R., & Anderson, P. (2003). Predictors of efficacy in depression prevention programmes. Meta-analysis. British Journal of Psychiatry, 183, 384–397.PubMedCrossRefGoogle Scholar
  49. Kellam, S. G., & Langevin, D. J. (2003). A framework for understanding “evidence” in prevention research and programs. Prevention Science, 4, 137–153.PubMedCrossRefGoogle Scholar
  50. Kellam, S. G., Koretz, D., & Moscicki, E. K. (1999). Core elements of developmental epidemiologically based prevention research. American Journal of Community Psychology, 27, 463–482.PubMedCrossRefGoogle Scholar
  51. Kellam, S. G., Brown, C. H., Poduska, J. M., Ialongo, N. S., Wang, W., Toyinbo, P., et al. (2008). Effects of a universal classroom behavior management program in first and second grades on young adult behavioral, psychiatric, and social outcomes. Drug and Alcohol Dependence, 95, S5–S28.PubMedCrossRefGoogle Scholar
  52. Klein, A., & Moosbrugger, H. (2000). Maximum likelihood estimation of latent interaction effects with the LMS method. Psychometrika, 65, 457–474.CrossRefGoogle Scholar
  53. Klein, A. G., & Muthén, B. (2006). Modeling heterogeneity of growth depending on initial status. Journal of Educational and Behavioral Statistics, 31, 357–375.CrossRefGoogle Scholar
  54. Kraemer, H. C., Wilson, G. T., Fairburn, C. G., Agras, W. S., Kraemer, H. C., Wilson, G. T., et al. (2002). Mediators and moderators of treatment effects in randomized clinical trials. Archives of General Psychiatry, 59, 877–883.PubMedCrossRefGoogle Scholar
  55. Lipsey, M. W. (2003). Those confounded moderators in meta-analysis: Good, bad, and ugly. Annals of the American Academy of Political and Social Science, 587, 69–81.CrossRefGoogle Scholar
  56. Mosteller, F., & Colditz, G. A. (1996). Understanding research synthesis (meta-analysis). Annual Review of Public Health, 17, 1–23.PubMedCrossRefGoogle Scholar
  57. Muthén, B. (2001). Latent variable mixture modeling. In G. A. Marcoulides & R. E. Schumacker (Eds.), New developments and techniques in structural equation modeling (pp. 1–33). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  58. Muthén, B., & Asparouhov, T. (2003a). Advances in latent variable modeling, Part I: Integrating multilevel and structural equation modeling using Mplus. Manucript in Preparation.Google Scholar
  59. Muthén, B., & Asparouhov, T. (2003b). Modeling interactions between latent and observed continuous variables using maximum-likelihood estimation in Mplus. Mplus Web Notes, #6. Retrieved from http://www.statmodel.com/download/webnotes/webnote6.pdf
  60. Muthén, B. O., & Asparouhov, T. (2008). Growth mixture modeling: Analysis with non-Gaussian random effects. In G. Fitzmaurice, M. Davidian, G. Verbeke, & G. Molenberghs (Eds.), Longitudinal data analysis: A handbook of modern statistical methods (pp. 143–165). Boca Raton, FL: Chapman & Hall/CRC.Google Scholar
  61. Muthén, B. O., & Asparouhov, T. (2009). Multilevel regression mixture analysis. Journal of the Royal Statistical Society, 172, 639–657.CrossRefGoogle Scholar
  62. Muthén, L. K., & Muthén, B. O. (1999). Mplus user’s guide version 2 (2nd ed.). Los Angeles: Authors.Google Scholar
  63. Muthén, B., & Muthén, L. K. (2000). Integrating person-centered and variable-centered analyses: Growth mixture modeling with latent trajectory classes. Alcoholism: Clinical & Experimental Research, 24, 882–891.CrossRefGoogle Scholar
  64. Muthén, L. K., & Muthén, B. O. (2007). Mplus: Statistical analysis with latent variables: User’s guide (4th ed.). Los Angeles: Authors.Google Scholar
  65. Muthén, B. O., & Shedden, K. (1999). Finite mixture modeling with mixture outcomes using the EM algorithm. Biometrics, 55, 463–469.PubMedCrossRefGoogle Scholar
  66. O’Connell, M. E., Boat, T., & Warner, K. E. (Eds.). (2009). Mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: National Academy.Google Scholar
  67. Shadish, W. R., & Sweeney, R. B. (1991). Mediators and moderators in meta-analysis: There’s a reason we don’t let dodo birds tell us which psychotherapies should have prizes. Journal of Consulting and Clinical Psychology, 59, 883–893.PubMedCrossRefGoogle Scholar
  68. Shrout, P. E. (2009). Short and long views of integrative data analysis: Comments on contributions to the special issue. Psychological Methods, 14, 177–181.PubMedCrossRefGoogle Scholar
  69. Sloboda, Z., Stephens, R. C., Stephens, P. C., Grey, S. F., Teasdale, B., Hawthorne, R. D., et al. (2009). The Adolescent Substance Abuse Prevention Study: A randomized field trial of a universal substance abuse prevention program. Drug and Alcohol Dependence, 102, 1–10.PubMedCrossRefGoogle Scholar
  70. Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752–760.PubMedCrossRefGoogle Scholar
  71. Tein, J.-Y., Sandler, I. N., MacKinnon, D. P., & Wolchik, S. A. (2004). How did it work? Who did it work for? Mediation in the context of a moderated prevention effect for children of divorce. Journal of Consulting & Clinical Psychology, 72, 617–624.CrossRefGoogle Scholar
  72. Tobler, N. S. (1986). Meta-analysis of 143 adolescent drug prevention programs: Quantitative outcome results of program participants compared to a control or comparison group. Journal of Drug Issues, 16, 537–567.Google Scholar
  73. Van Horn, M. L., Jaki, T., Masyn, K., Ramey, S. L., Smith, J. A., & Antaramian, S. (2009). Detecting differential effects: Applying regression mixture models to identify variations in the influence of family resources on academic achievement. Developmental Psychology, 45, 1298–313.PubMedCrossRefGoogle Scholar
  74. Wilson, D. B., & Lipsey, M. W. (2003). The role of method in treatment effectiveness research: Evidence from meta-analysis. In A. E. Kazdin (Ed.), Methodological issues & strategies in clinical research (3rd ed., pp. 589–615). Washington, DC: American Psychological Association.Google Scholar
  75. Wolchik, S., Sandler, I., Weiss, L., & Winslow, E. (2007). New Beginnings: An empirically-based intervention program for divorced mothers to help children adjust to divorce. In J. M. Briesmeister & C. E. Schaefer (Eds.), Handbook of parent training: Helping parents prevent and solve problem behaviors (3rd ed., pp. 25–66). Hoboken, NJ: John Wiley & Sons.Google Scholar

Copyright information

© Society for Prevention Research 2011

Authors and Affiliations

  • C. Hendricks Brown
    • 1
  • Zili Sloboda
    • 2
  • Fabrizio Faggiano
    • 3
  • Brent Teasdale
    • 4
  • Ferdinand Keller
    • 5
  • Gregor Burkhart
    • 6
  • Federica Vigna-Taglianti
    • 7
  • George Howe
    • 8
  • Katherine Masyn
    • 9
  • Wei Wang
    • 10
  • Bengt Muthén
    • 11
  • Peggy Stephens
    • 12
  • Scott Grey
    • 13
  • Tatiana Perrino
    • 1
  • Prevention Science and Methodology Group
  1. 1.University of MiamiMiller School of MedicineMiamiUSA
  2. 2.JBS InternationalRockvilleUSA
  3. 3.Avogadro UniversityNovaraItaly
  4. 4.Georgia State UniversityAtlantaUSA
  5. 5.University of UlmUlmGermany
  6. 6.European Monitoring Centre for Drugs and Drug AddictionLisbonPortugal
  7. 7.Piedmont Centre for Drug Addiction EpidemiologyGrugliascoItaly
  8. 8.George Washington UniversityWashingtonUSA
  9. 9.Harvard UniversityCambridgeUSA
  10. 10.University of South FloridaTampaUSA
  11. 11.UCLALos AngelesUSA
  12. 12.Akron UniversityAkronUSA
  13. 13.Kent State UniversityKentUSA

Personalised recommendations