Abstract
This study assesses the implementation quality of Think Smart, a school-based drug prevention curriculum that was designed to reduce use of harmful legal products (HLPs; e.g., inhalants and over-the-counter drugs), alcohol, tobacco, and other drugs among 5th- and 6th-grade students in frontier Alaska. Participating in the study were eight communities that took part in a larger randomized control trial to assess the short-term effects of the Think Smart curriculum. Video-recorded observations of the 12 core and 3 booster lessons were conducted in 20 classrooms. Ninety-five sessions were randomly selected from 228 usable videodiscs, and two pairs of researchers observed each video recording to code level of dosage, adherence to curriculum design, and teachers’ delivery skills. Inter-rater reliability for all implementation quality measures was very high. An expert panel consisting of 16 scientists reviewed the results of the implementation study and concluded that the level of dosage and adherence to the curriculum design was at least as high as those yielded by similar studies. However, the panel assessed the delivery quality to be only marginal in comparison to results of other studies. The experts concluded that the implementation quality of the Think Smart curriculum was adequate even though the teachers’ delivery skills were only marginal. A bootstrapping analysis, in which 1,000 samples were drawn for each implementation quality result, found the expert judgments to be reliable. The authors conclude that despite some limitations, video-recorded observations, as well as expert judgment, provide strong methodologies that should be considered for future implementation quality studies.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Backer, T. E. (2001). Finding the balance: Program fidelity and adaptation in substance abuse prevention. Rockville, MD: Center for Substance Abuse Prevention.
Berman, P., Greenwood, P. W., McLaughlin, M. W., & Pincus, J. (1975). Federal programs supporting educational change. Santa Monica, CA: RAND.
Berman, P., & McLaughlin, M. W. (1976). Implementation of educational innovation. The Educational Forum, 40, 345–370.
Bierman, K. L., Cole, J. D., Dodge, K. A., Greenberg, M. T., Lochman, J. E., & McMahon, R. J. (1997). Implementing a comprehensive program for the prevention of conduct problems in rural communities: The Fast Track experience. American Journal of Community Psychology, 25, 493–514.
Bosworth, K. (2000). Protective schools: Linking drug abuse prevention with student success. A guide for educators, policy makers, and families. Tucson: University of Arizona.
Botvin, G. J., Griffin, K. W., Diaz, T., Scheier, L. M., Williams, C., & Epstein, J. A. (2000). Preventing illicit drug use in adolescents: Long-term follow-up data from a randomized control trial of a school population. Addictive Behaviors, 25, 769–774.
Brandon, P. R., Taum, A. K. H., Young, D. B., Pottenger, F. M., III, & Speitel, T. W. (2008). The complexity of measuring the quality of program implementation with observations. American Journal of Evaluation, 29, 235–250.
Centers for Disease Control and Prevention. (1994). Guidelines for school health programs to prevent tobacco use and addiction. MMWR, 1994, 43.
Cohen, J. (1960). A coefficient for agreement for nominal scales. Education and Psychological Measurement, 20, 37–46.
CSAP (2001). Model programs: Bringing effective prevention to every community. Retrieved May 30, 2007, from http://modelprograms.samhsa.gov/model.htm.
Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45.
Domitrovich, C. E., & Greenberg, M. T. (2000). The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational and Psychological Consultation, 11, 193–221.
Drug Strategies (1999). Making the grade: A guide to drug prevention programs (Revised Edition). Washington, DC: Drug Strategies.
Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18, 237–256.
Dusenbury, L., Brannigan, R., Hansen, W. B., Walsh, J., & Falco, M. (2005). Quality of implementation: Developing measures crucial to understanding the diffusion of preventive interventions. Health Education Research, 20, 308–313.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378–382.
Fullan, M., & Pomfret, A. (1977). Research on curriculum and instruction implementation. Review of Educational Research, 52, 7–30.
Giles, S. M., Jackson-Newsom, J., Pankratz, M. M., Hansen, W. B., Ringwalt, C. L., & Dusenbury, L. (2008). Measuring quality of delivery in a substance use prevention program. Journal of Primary Prevention, 29, 489–501.
Gingiss, P. M., Robert-Graw, C., & Boer, M. (2006). Bridge-it: A system for predicting implementation fidelity for school-based tobacco prevention programs. Prevention Science, 7, 197–207.
Hansen, W. B., & McNeal, R. B. (1999). Drug education practice: Results of an observational study. Health Education Research, 14, 85–97.
Johnson, K. W. (1990). Impact of a university-government partnership in criminal justice: A linkage model for the 1990s. The Journal of Criminal Justice Education, 1, 167–194.
Johnson, K. W., Holder, H., Ogilvie, K. A., Collins, D. A., Courser, M., Miller, B., et al. (2007). A community prevention intervention to combat inhalants and other harmful legal products among pre-adolescents. Journal of Drug Education, 37, 227–247.
Johnson, K. W., Shamblen, S. R., Ogilvie, K. A., Collins, D. A., & Saylor, B. L. (2009). Preventing youth’s use of inhalants and other harmful legal products in frontier Alaskan communities: A randomized trial. Prevention Science, 10, 298–312.
Lillehoj, C. J., Griffin, K. W., & Spoth, R. (2004). Program provider and observer ratings of school-based preventive intervention implementation: Agreement and relation to youth outcomes. Health Education and Behavior, 31, 242–257.
McCormick, L., Steckler, A., & McLeroy, K. (1995). Diffusion of innovations in schools: A study of adoption and implementation of school-based tobacco prevention curricula. American Journal of Health Promotion, 9, 210–219.
McGrew, J. H., Bond, G. R., Dietzen, L., & Salyers, M. (1994). Measuring the fidelity of implementation of a mental health program model. Journal of Consulting and Clinical Psychology, 62, 670–678.
Melde, C., Esbensen, F. A., & Tusinski, K. (2006). Addressing program fidelity using onsite observations and program provider descriptions of program delivery. Evaluation Review, 30, 714–740.
Meyer, M. A., & Booker, J. M. (2001). Eliciting and analyzing expert judgment: A practical guide. Philadelphia: Society for Industrial and Applied Mathematics.
National Center for Frontier Communities (2007). List of frontier counties from 2000 U.S. Census. Retrieved May 25, 2007, from http://frontierus.org/2000census.htm.
NCLB (2004). Safe and drug-free schools and communities act, state grants, guidance for state and local implementation of programs. Retrieved May 1, 2009, from www.ed.gov/programs/dvpformula/guidance.doc.
Nevo, D. (1989). Expert opinion in program evaluation. New Directions for Program Evaluation, 42, 85–93.
NREPP (2007). SAMHSA’S National registry of evidence based programs and practices: Finding interventions. Retrieved May 30, 2007, from www.nrepp.samhsa.gov/find.asp.
Ogilvie, D. C., Coulehan, H. M., Ogilvie, K. A., & Johnson, K. W. (2006). Think Smart: Problem solving for Alaskan youth, A curriculum to prevent substance abuse including inhalants and other harmful everyday legal products. Anchorage, AK: Akeela.
OJJDP. (2002). Blueprints for violence prevention. Retrieved May 30, 2007, from www.colorado.edu/cspv/blueprints/model/overview.html.
Pankratz, M. M., Jackson-Newsom, J., Giles, S. M., Ringwalt, C. L., Bliss, K., & Bell, M. L. (2006). Implementation fidelity in a teacher-led alcohol use prevention curriculum. Journal of Drug Education, 36, 317–333.
Ridgely, M. S., & Jerrell, J. M. (1996). Analysis of three interventions for substance abuse treatment of severely mentally ill people. Community Mental Health Journal, 32, 561–572.
Ringwalt, C. L., & Bliss, K. (2006). The cultural tailoring of a substance use prevention curriculum for American Indian youth. Journal of Drug Education, 36, 159–177.
Ringwalt, C. L., Ennett, S., Rohrbach, L. A., Simons-Rudolph, A., Vincus, A., & Thorne, J. (2003). Factors associated with fidelity to substance use prevention curriculum guides on the nation's middle schools. Health Education & Behavior, 30, 375–391.
Ringwalt, C. L., Pankratz, M. M., Jackson-Newsom, J., Gottfredson, N. C., Hansen, W. B., Giles, S. M., et al. (2009). Three year trajectory of teachers' fidelity to a drug prevention curriculum. Prevention Science (in press).
Robertson, E. B., David, S. L., & Rao, S. A. (2003). Preventing drug use among children and adolescents: A research-based guide for parents, educators, and community leaders (2nd ed.). Bethesda, MD: National Institute on Drug Abuse.
Rohrbach, L. A., Dent, C. W., Skara, S., Sun, P., & Sussman, S. (2007). Fidelity of implementation in Project Towards No Drug Abuse (TND): A comparison of classroom teachers and program specialists. Prevention Science, 8, 125–132.
Schinke, S. P., Orlandi, M. A., Botvin, G. J., Gilchrist, L. D., Trimble, J. E., & Locklear, V. S. (1988). Preventing substance abuse among American-Indian adolescents: A bicultural competence skills approach. Journal of Counseling Psychology, 35, 87–90.
Schinke, S. P., Tepavac, L., & Cole, K. C. (2000). Preventing substance use among Native American youth: Three-year results. Addictive Behaviors, 25, 387–397.
Shrout, P., & Fleiss, J. L. (1979). Intraclass correlation: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.
Sobol, D. F., Rohrbach, L. A., Dent, C. W., Gleason, L., Brannon, B. R., Johnson, C. A., et al. (1989). The integrity of smoking prevention curriculum delivery. Health Education Research, 4, 59–67.
Spoth, R., Redmond, C., & Shin, C. (1988). Direct and indirect latent-variable parenting outcomes of two universal family-focused preventative interventions: Extending a public health oriented research base. Journal of Consulting and Clinical Psychology, 66, 385–399.
Spoth, R., Guyll, M., Lillehoj, C. J., Redmond, C., & Greenberg, M. (2007). Prosper study of evidence-based intervention implementation quality by community-university partnerships. Journal of Community Psychology, 35, 981–999.
Tricker, R., & Davis, L. G. (1988). Implementing drug education in schools: An analysis of the costs and teacher perceptions. Journal of School Health, 58, 181–185.
U.S. DOE (2001). Safe, disciplined, and drug-free schools expert panel. Retrieved May 30, 2007, from www.ed.gov/admins/lead/safety/exemplary01/index.html.
Yeaton, W. H., & Sechrest, L. (1981). Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity and effectiveness. Journal of Consulting and Clinical Psychology, 49, 156–167.
Acknowledgements
Research for and preparation of this manuscript were supported by the National Institute on Drug Abuse Grant #R01 DA019640, K. Johnson, PI. The authors wish to express their gratitude to Diane Ogilvie of Akeela, Inc. who collaborated on adapting the curriculum and managed its implementation during the study as well as the Alaska schools and their teachers, staff, students, and administrators that made this study possible. A debt of gratitude is also owed to the coders of curriculum. The authors finally wish to acknowledge the 16 expert panel members, some of whom also made valuable suggestions to improving an earlier version of the manuscript. These members are Drs. Paul Brandon, Linda Dusenbury, Abby Fagan, Steve Giles, Phyllis Gingiss, Mark Greenberg, Bill Hansen, Laura Hill, Anja Kurki, Torill Larsen, Catherine Lillehoj, Melinda Pankratz, Luanne Rohrbach, Richard Spoth, Steve Sussman, and Jennifer Unger.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Johnson, K.W., Ogilvie, K.A., Collins, D.A. et al. Studying Implementation Quality of a School-Based Prevention Curriculum in Frontier Alaska: Application of Video-Recorded Observations and Expert Panel Judgment. Prev Sci 11, 275–286 (2010). https://doi.org/10.1007/s11121-010-0174-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-010-0174-5