Journal of Behavioral Education

, Volume 24, Issue 1, pp 133–151 | Cite as

Using a Meta-analytic Technique to Assess the Relationship between Treatment Intensity and Program Effects in a Cluster-Randomized Trial

  • Joshua R. Polanin
  • Dorothy L. Espelage
Original Paper


School bullying and delinquent behaviors are persistent and pervasive problems for schools, and have lasting effects for all individuals involved (Copeland et al., JAMA Psychiatry 70:419–426, 2013; Espelage et al., J Res Adolesc 24(2):337–349, 2013a). As a result, policymakers and practitioners have attempted to thwart these ill-effects using school-based interventions. Recent meta-analyses have found, however, that these programs produce only moderate effects (Ttofi and Farrington, J Exp Criminol 7:27–56, 2011). Consequently, it is important to investigate further the reasons for such findings. One promising analysis is to assess the relation between treatment intensity variables and program outcomes. Unfortunately, few treatment intensity variables have been utilized in the school-based prevention literature, and it is often cumbersome to model the relation between treatment intensity and outcomes. The purpose of this project, therefore, is to explicate novel measures of treatment intensity and delineate a relatively new meta-analytic technique to model the relation between the variables and program effects. The context for this project is a large-scale, multi-site, cluster-randomized trial; 36 schools and 3,616 students participated in three waves of data collection. The results indicated that, for the second wave of data collection, stronger treatment effects were found when teachers and program implementers spent a greater amount of time prepping lessons, provided additional financial resources, and received outside consultation and support.


Bullying Victimization Treatment intensity Meta-analysis Robust variance estimation 



Research for the current study was supported by the Centers for Disease Control & Prevention (#1U01/CE001677) to Dorothy Espelage (PI) at the University of Illinois at Urbana-Champaign. Opinions expressed herein do not necessarily reflect those of the Centers for Disease Control & Prevention, or related offices within.


  1. American Educational Research Association. (2013). Prevention of bullying in schools, colleges, and universities: Research report and recommendations. Washington, DC: American Educational Research Association.Google Scholar
  2. Borenstein, M., Hedges, L. V., Higgins, J., & Rothstein, H. (2009). Introduction to meta-analysis. West Sussex: Wiley.CrossRefGoogle Scholar
  3. Brown, E. C., Low, S., Smith, B. H., & Haggerty, K. P. (2011). Outcomes from a school-randomized controlled trial of STEPS to RESPECT: A Bullying Prevention Program. School Psychology Review, 40, 423–443.Google Scholar
  4. Committee for Children. (2008). Second step: Student Success through Prevention Program. Seattle, WA: Author.Google Scholar
  5. Copeland, W. E., Wolke, D., Angold, A., & Costello, E. J. (2013). Adult psychiatric outcomes of bullying and being bullies by peers in childhood and adolescence. JAMA Psychiatry, 70, 419–426.CrossRefPubMedCentralPubMedGoogle Scholar
  6. Cordray, D. S. (2014). Fidelity of implementation.IES summer training institute for cluster-randomized control trials. Lecture conducted at Northwestern University, Evanston, IL.Google Scholar
  7. Cordray, D. S., & Pion, G. M. (2006). Treatment strength and integrity: Models and methods. In R. R. Bootzin & P. E. McKnight (Eds.), Strengthening research methodology: Psychological measurement and evaluation (pp. 103–124). Washington, DC: American Psychological Association.CrossRefGoogle Scholar
  8. Crowley, D. M., Coffman, D. L., Feinberg, M. E., Greenberg, M. T., & Spoth, R. L. (2014). Evaluating the impact of implementation factors on family-based prevention programming: Methods for strengthening causal inference. Prevention Science, 15(2), 246–255.CrossRefPubMedGoogle Scholar
  9. Daly, E. J., Martens, B. K., Barnett, D., Witt, J. C., & Olson, S. C. (2007). Varying intervention delivery in response to intervention: Confronting and resolving challenges with measurement, instruction, and intensity. School Psychology Review, 36(4), 562–581.Google Scholar
  10. Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350.CrossRefPubMedGoogle Scholar
  11. Espelage, D. L. (2013). Why are bully prevention programs failing in U.S. schools? Journal of Curriculum and Pedagogy, 10, 121–123.CrossRefGoogle Scholar
  12. Espelage, D. L., Basile, K. C., & Hamburger, M. E. (2012). Bullying experiences and co-occurring sexual violence perpetration among middle school students: Shared and unique risk factors. Journal of Adolescent Health, 50, 60–65.CrossRefPubMedGoogle Scholar
  13. Espelage, D. L., & Holt, M. L. (2001). Bullying and victimization during early adolescence: Peer influences and psychosocial correlates. Journal of Emotional Abuse, 2, 123–142.CrossRefGoogle Scholar
  14. Espelage, D. L., Holt, M. K., & Henkel, R. R. (2003). Examination of peer-group contextual effects on aggression during early adolescence. Child Development, 74, 205–220.CrossRefPubMedGoogle Scholar
  15. Espelage, D. L., Low, S., Polanin, J. R., & Brown, E. C. (2013a). The impact of a middle school program to reduce aggression, victimization, and sexual violence. Journal of Adolescent Health, 53(2), 180–186.CrossRefPubMedGoogle Scholar
  16. Espelage, D. L., Low, S., Rao, M. A., Hong, J. S., & Little, T. (2013b). Family violence, bullying, fighting, and substance use among adolescents: A longitudinal transactional model. Journal of Research on Adolescence, 24(2), 337–349.CrossRefGoogle Scholar
  17. Fisher, Z., & Tipton, E. (2014). robumeta: Robust variance meta-regression (Version 1.0). Retrieved from:
  18. Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65.CrossRefGoogle Scholar
  19. Higgins, J., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558.CrossRefPubMedGoogle Scholar
  20. Hulleman, C. S., & Cordray, D. S. (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2, 88–110.CrossRefGoogle Scholar
  21. Inc, Survey Monkey. (2014). SuveryMonkey [computer software]. Palo Alto, CA: SurveyMonkey Inc.Google Scholar
  22. Kazdin, A. E. (2009). Understanding how and why psychotherapy leads to change. Psychotherapy Research, 19(4), 418–428.Google Scholar
  23. Kochenderfer, B. J., & Ladd, G. W. (1996). Peer victimization: Cause or consequence of school maladjustment? Child Development, 67, 1305–1317.CrossRefPubMedGoogle Scholar
  24. Konstantopoulos, S. (2011). How consistent are class size effects? Evaluation Review, 35, 71–92.CrossRefPubMedGoogle Scholar
  25. Merrell, K. W., Gueldner, B. A., Ross, S. W., & Isava, D. M. (2008). How effective are school bullying intervention programs? A meta-analysis of intervention research. School Psychology Quarterly, 23, 26–42.CrossRefGoogle Scholar
  26. Musher-Eizenman, D. R., Boxer, P., Danner, S., Dubow, E. F., Goldstein, S. E., & Heretick, D. M. (2004). Social-cognitive mediators of the relation of environmental and emotion regulation factors to children’s aggression. Aggressive Behavior, 30, 389–408.CrossRefGoogle Scholar
  27. Olweus, D. (2005). A useful evaluation design, and effects of the Olweus bullying prevention program. Psychology, Crime & Law, 11, 389–402.CrossRefGoogle Scholar
  28. Paunonen, S. V. (1984). Optimizing the validity of personality assessments: The importance of aggregation and item content. Journal of Research in Personality, 18, 411–431.CrossRefGoogle Scholar
  29. Pigott, T. D. (2012). Advances in meta-analysis. New York: Springer.CrossRefGoogle Scholar
  30. Pituch, K. A., Whittaker, T. A., & Stapleton, L. M. (2005). A comparison of methods to test for mediation in multisite experiments. Multivariate Behavioral Research, 40(1), 1–23.CrossRefGoogle Scholar
  31. Polanin, J. R., Espelage, D. L., & Pigott, T. D. (2012). A meta-analysis of school-based bullying prevention programs’ effects on bystander intervention behavior. School Psychology Review, 41, 47–65.Google Scholar
  32. Poteat, V. P., & Espelage, D. L. (2005). Exploring the relation between bullying and homophobic verbal content: The homophobic content agent target (HCAT) scale. Violence and Victims, 20, 513–528.CrossRefPubMedGoogle Scholar
  33. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage Publications.Google Scholar
  34. Rigby, K. (2001). Health consequences of bullying and its prevention in schools. In J. Juvonen & S. Graham (Eds.), Peer harassment in school: The plight of the vulnerable and victimized (pp. 310–331). New York: Guilford Press.Google Scholar
  35. Rigby, K., & Slee, P. T. (1993). Dimensions of interpersonal relation among Australian children and implications for psychological well-being. The Journal of Social Psychology, 133, 33–42.CrossRefPubMedGoogle Scholar
  36. Rivers, I., Poteat, V. P., Noret, N., & Ashurst, N. (2009). Observing bullying at school: The mental health implications of witness status. School Psychology Quarterly, 24, 211–223.CrossRefGoogle Scholar
  37. Ross, S. W., Horner, R. H., & Higbee, T. (2009). Bully prevention in positive behavior support. Journal of Applied Behavior Analysis, 42, 747–759.CrossRefPubMedCentralPubMedGoogle Scholar
  38. Smith, P. K. (1997). Bullying in schools: The UK experience and the Sheffield anti-bullying project. The Irish Journal of Psychology, 18, 191–201.CrossRefGoogle Scholar
  39. Srabstein, J. C., & Leventhal, B. L. (2010). Prevention of bullying-related morbidity and mortality: A call for public health policies. Bulletin of the World Health Organization, 77, 403–404.CrossRefGoogle Scholar
  40. Sweeting, H., Young, R., West, P., & Der, G. (2006). Peer victimization and depression in early–mid adolescence: A longitudinal study. British Journal of Educational Psychology, 76, 577–594.CrossRefPubMedGoogle Scholar
  41. Ttofi, M. M., & Farrington, D. P. (2011). Effectiveness of school-based programs to reduce bullying: A systematic and meta-analytic review. Journal of Experimental Criminology, 7, 27–56.CrossRefGoogle Scholar
  42. Ttofi, M. M., Farrington, D. P., & Lösel, F. (2012). School bullying as a predictor of violence later in life: A systematic review and meta-analysis of prospective longitudinal studies. Aggression and Violent Behavior, 17, 405–418.CrossRefGoogle Scholar
  43. Unlu, F., Bozzi, L., Layzer, C., Smith, A., Price, C., & Hurtig, R. (2013, March). Linking implementation fidelity to impacts in an RCT. Paper presented at the annual meeting of Society for Research on Educational Effectiveness, Washington, DC. Abstract retrieved from
  44. U.S. Department of Education. (2011). Analysis of state bullying laws and policies. Washington, DC: U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service.Google Scholar
  45. vanGeel, M., Vedder, P., & Tanilon, J. (2014). Relationship between peer victimization, cyberbullying, and suicide in children and adolescents: A meta-analysis. JAMA pediatrics, Advanced Online Publication. doi: 10.1001/jamapediatrics.2013.4143.
  46. Warren, S. F., Fey, M. E., & Yoder, P. J. (2007). Differential treatment intensity research: A missing link to creating optimally effective communication interventions. Mental Retardation and Developmental Disabilities Research Reviews, 13(1), 70–77.CrossRefPubMedGoogle Scholar
  47. Whitaker, D. J., Rosenbluth, B., Valle, L. A., & Sanchez, E. (2004). Expect Respect: A school-based intervention to promote awareness and effective responses to bullying and sexual harassment. In D. L. Espelage & S. M. Swearer (Eds.), Bullying in American schools: A social-ecological perspective on prevention and intervention (pp. 327–350). Mahwah, NJ: Taylor & Francis.Google Scholar
  48. Wickham, H. (2009). ggplot2: Elegant graphics for data analysis. New York: Springer.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Peabody Research InstituteVanderbilt UniversityNashvilleUSA
  2. 2.Department of Educational PsychologyUniversity of Illinois at Urbana-ChampaignUrbana-ChampaignUSA

Personalised recommendations