The Journal of Primary Prevention

, Volume 29, Issue 6, pp 489–501 | Cite as

Measuring Quality of Delivery in a Substance Use Prevention Program

  • Steven Giles
  • Julia Jackson-Newsom
  • Melinda M. Pankratz
  • William B. Hansen
  • Christopher L. Ringwalt
  • Linda Dusenbury
Original Paper


The purpose of this study was to develop and validate an observation measure designed to capture teachers’ use of interactive teaching skills within the delivery of the All Stars substance use prevention program. Coders counted the number of times teachers praised and encouraged students, accepted and used students’ ideas, asked questions, self-disclosed personal anecdotes, and corrected student misbehavior. These teacher behaviors loaded on three factors: classroom management, acknowledgment, and student-centered methods. Classroom management was negatively related to student engagement. Acknowledgment was negatively related to students’ normative beliefs. Student-centered methods were positively related to student idealism and normative beliefs, and marginally predicted decreases in student marijuana use. Editors’ Strategic Implications: The authors provide a promising approach to studying pedagogical prevention approaches, and they also link teaching processes to student outcomes. This study of program delivery should be of general interest (i.e., not limited to substance use prevention) to practitioners and researchers.


Interactivity Program delivery Prevention Substance use 



This study was supported by a grant from the National Institute on Drug Abuse, Grant No. 5 R01 DA016098, “Promoting Fidelity Using Remote and Onsite Support.” We would like to acknowledge the contribution of our coders—Terry Dumansky, Sara Patterson, and Grant Bollmer. The work was tedious but their insights and acumen were invaluable. We are also indebted to Dr. Inez Drummond of Chicago Public Schools for her support and to all the teachers who collaborated with us on this study.


  1. Abbott, R. D., O’Donnell, I., Hawkins, D., Hill, K. G., Kosterman, R., & Catalano, R. F. (1998). Changing teaching practices to promote achievement and bonding to school. American Journal of Orthopsychiatry, 68, 542–552. doi: 10.1037/h0080363.PubMedCrossRefGoogle Scholar
  2. Bosworth, K., & Sailes, J. (1993). Content and teaching strategies in 10 selected drug abuse prevention curricula. Journal of School Health, 63, 247–253.PubMedGoogle Scholar
  3. Botvin, G. J., Dusenbury, L., Baker, E., James-Ortiz, S., & Kerner, J. (1989). A skills training approach to smoking prevention among Hispanic youth. Journal of Behavioral Medicine, 12, 279–296. doi: 10.1007/BF00844872.PubMedCrossRefGoogle Scholar
  4. Dusenbury, L., Brannigan, R., Hansen, W. B., Walsh, J., & Falco, M. (2005). Quality of implementation: Developing measures crucial to understanding the diffusion of preventive interventions. Health Education Research, 20, 308–313. doi: 10.1093/her/cyg134.PubMedCrossRefGoogle Scholar
  5. Dusenbury, L., & Falco, M. (1995). Eleven components of effective drug abuse prevention curricula. Journal of School Health, 65, 420–425.PubMedCrossRefGoogle Scholar
  6. Flanders, N. (1970). Analyzing teacher behavior. Reading, MA: Addison-Wesley.Google Scholar
  7. Hansen, W. B. (1996). Pilot test results comparing the All Stars program with seventh grade D.A.R.E.: Program integrity and mediating variable analysis. Substance Use and Misuse, 31, 1359–1377. doi: 10.3109/10826089609063981.PubMedCrossRefGoogle Scholar
  8. Hansen, W. B., Graham, J. W., Wolkenstein, B. H., & Rohrbach, L. A. (1991). Program integrity as a moderator of prevention program effectiveness: Results for fifth-grade students in the Adolescent Alcohol Prevention Trial. Journal of Studies on Alcohol, 52, 568–579.PubMedGoogle Scholar
  9. Hansen, W. B., & McNeal, R. B. (1999). Drug education practice: Results of an observational study. Health Education Research, 14, 85–97. doi: 10.1093/her/14.1.85.PubMedCrossRefGoogle Scholar
  10. Harachi, T. W., Abbott, R. D., Catalano, R. F., Haggerty, K. P., & Fleming, C. B. (1999). Opening the black box: Using process evaluation measures to assess implementation and theory building. American Journal of Community Psychology, 27, 711–731. doi: 10.1023/A:1022194005511.PubMedCrossRefGoogle Scholar
  11. Harrington, N. G., Giles, S. M., Hoyle, R. H., Feeney, G. J., & Yungbluth, S. C. (2001). Evaluation of the All Stars character education and problem behavior prevention program: Pretest-posttest effects on mediator and outcome variables for middle school students. Health Education & Behavior, 28, 533–546. doi: 10.1177/109019810102800502.CrossRefGoogle Scholar
  12. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46. doi: 10.1037/1082-989X.1.1.30.CrossRefGoogle Scholar
  13. McNeal, R. B., Hansen, W. B., Harrington, N. G., & Giles, S. M. (2004). How All Stars works: An examination of program effects on mediating variables. Health Education & Behavior, 31, 165–178. doi: 10.1177/1090198103259852.CrossRefGoogle Scholar
  14. Muthén, L. K., & Muthén, B. O. (2007). Mplus statistical software (Version 5.0). Los Angeles, CA: Muthén & Muthén.Google Scholar
  15. Pentz, M. A., Trebow, E. A., Hansen, W. B., MacKinnon, D. P., Dwyer, J. H., Johnson, C. A., et al. (1990). Effects of program implementation on adolescent drug use behavior. Evaluation Review, 14, 264–289. doi: 10.1177/0193841X9001400303.CrossRefGoogle Scholar
  16. Ringwalt, C. L., Pankratz, M. M., Hansen, W. B., Dusenbury, L., Jackson-Newsom, J., Giles, S. M., Brodish, P. (2007). The potential of coaching as a strategy to improve the effectiveness of school-based substance use prevention curricula. Health Education & Behavior, July 25, 2007. doi: 10.1177/1090198107303311.
  17. Shrout, P. E., & Fleiss, J. J. (1979). Intraclass correlations: Uses in assessing reliability. Psychological Bulletin, 86, 420–428. doi: 10.1037/0033-2909.86.2.420.PubMedCrossRefGoogle Scholar
  18. Sobol, D. F., Dent, C. W., Gleason, L., Brannon, B. R., Johnson, C. A., & Flay, B. R. (1989). The integrity of smoking prevention curriculum delivery. Health Education Research, 4, 59–67. doi: 10.1093/her/4.1.59.CrossRefGoogle Scholar
  19. Substance Abuse and Mental Health Services Administration. (2008). SAMHSA model programs: Effective substance abuse and mental health programs for every community. Retrieved 22 July 2008.
  20. Tobler, N. (1986). Meta-analysis of 143 adolescent drug prevention programs: Quantitative outcome results of program participants compared to a control or comparison group. Journal of Drug Issues, 16, 537–567.Google Scholar
  21. Tobler, N. S. (1992). Drug prevention programs can work: Research findings. Journal of Addictive Diseases, 11(3), 1–28. doi: 10.1300/J069v11n03_01.PubMedCrossRefGoogle Scholar
  22. Tobler, N. S. (2000). Lessons learned. Journal of Primary Prevention, 20, 261–274. doi: 10.1023/A:1021362620740.CrossRefGoogle Scholar
  23. Tobler, N. S., Lessard, T., Marshall, D., Ochshorn, P., & Roona, M. (1999). Effectiveness of school-based drug prevention programs for marijuana use. School Psychology International, 20, 105–137. doi: 10.1177/0143034399201008.CrossRefGoogle Scholar
  24. Tobler, N. S., & Stratton, H. (1997). Effectiveness of school-based drug prevention programs: A meta-analysis of the research. Journal of Primary Prevention, 18, 71–128. doi: 10.1023/A:1024630205999.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Steven Giles
    • 1
  • Julia Jackson-Newsom
    • 2
  • Melinda M. Pankratz
    • 3
  • William B. Hansen
    • 2
  • Christopher L. Ringwalt
    • 3
  • Linda Dusenbury
    • 2
  1. 1.Wake Forest UniversityWinston-SalemUSA
  2. 2.Tanglewood ResearchGreensboroUSA
  3. 3.Pacific Institute for Research and EvaluationChapel HillUSA

Personalised recommendations