Abstract
The present study examined the feasibility of an innovative technology designed to assess implementation fidelity of the Early Risers conduct problems prevention program across 27 geographically dispersed school sites. A multidimensional construct of fidelity was used to assess the quantity of services provided (exposure), the degree to which program strategies conformed to the manual (adherence), and how well implementers delivered the program (quality of delivery). The measurement technology featured a fidelity monitoring system that required (a) weekly reporting on a web-based documentation system to assess program exposure and adherence, and (b) five annually administered telephone interviews with a technical assistant to assess quality of program implementation. The results showed that the fidelity monitoring system was feasible, with all sites achieving 100% compliance in completion of their required on-line reporting and on average over 80% of the required teleconference interviews. User feedback indicated satisfaction with the web-based program. The system was successful in measuring multiple indices of fidelity. The strengths and limitations of measuring fidelity at a distance with web-based and teleconferencing technologies are discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The Early Risers program has evolved over a 10-year period and has been continually refined through feedback from efficacy and effectiveness research trials. The goals of the program, conceptual theory of change, and delivery system have remained constant over that time. However, the labels used to identify the components have changed to more accurately reflect the specific target of the intervention (e.g., child skills, child school support, parent skills, and family support). In addition, within some components, curriculum changes have been made based on scientific evidence, feasibility, and acceptability.
There has been confusion between the terms, exposure and dose (or dosage) in the literature of fidelity of program implementation. Whereas dose may indicate the amount of program content received by participants, we define exposure as the amount of program content delivered by program implementers to participants, regardless of whether a given participant actually attended the program component.
References
August, G. J., Bloomquist, M. L., Realmuto, G. M., & Hektner, J. M. (2007). The Early Risers “Skills for Success” Program: A targeted intervention for preventing conduct problems and substance abuse in aggressive elementary school children. In P. Tolan, J. Szapocznik, & S. Sambrano (Eds.), Preventing youth substance abuse (pp. 137–158). Washington, DC: American Psychological Association.
Blakeley, C. H., Mayer, J. P., Gottschalk, R. G., Schmitt, N., Davidson, W. S., Roitman, D. B., et al. (1987). The fidelity-adaptation debate: Implications for the implementation for public sector social programs. American Journal of Community Psychology, 15, 253–268, doi:10.1007/BF00922697.
Botvin, G. J., Baker, E., Dusenberry, L., Tortu, S., & Botvin, E. M. (1990). Preventing adolescent drug abuse through a multimodal cognitive-behavioral approach: Results of a 3-year study. Journal of Consulting and Clinical Psychology, 58, 437–446, doi:10.1037/0022-006X.58.4.437.
Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation efforts out of control? Clinical Psychology Review, 18, 23–45, doi:10.1016/S0272-7358(97)00043-3.
Dumas, J. E., Lynch, A. M., Laughlin, J. E., Smith, E. P., & Prinz, R. J. (2001). Promoting intervention fidelity: Conceptual issues, methods, and preliminary results from the Early Alliance Prevention Trial. American Journal of Preventive Medicine, 20, 38–47, doi:10.1016/S0749-3797(00)00272-5.
Durlak, J. A. (1997). Successful prevention programs for children and adolescents. New York: Plenum.
Dusenbury, L., Branningan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18, 237–256, doi:10.1093/her/18.2.237.
Greenberg, M. T., & Kusche, C. A. (1998). Promoting Alternative Thinking Strategies, Book 10: Blueprint for violence prevention. University of Colorado: Institute of Behavioral Sciences.
Gresham, F. M., MacMillian, D. L., Beebe-Frankenberger, M. E., & Bocian, K. M. (2000). Treatment integrity in learning disabilities intervention research: Do we really know how treatments are implemented? Learning Disabilities Research and Practice, 15, 198–205, doi:10.1207/SLDRP1504_4.
Hansen, W. B., Graham, J. W., Wolkenstein, B. H., & Rohrbach, L. A. (1991). Journal of Studies on Alcohol, 52, 568–579.
Hill, L. G., Maucione, K., & Hood, B. K. (2007). A focused approach to assessing program fidelity. Prevention Science, 8, 25–34, doi:10.1007/s11121-006-0051-4.
Kam, C. M., Greenberg, M. T., & Walls, C. T. (2003). Examining the role of implementation quality in school-based prevention using the PATHS curriculum. Prevention Science, 4, 55–63, doi:10.1023/A:1021786811186.
Kendall, P. C., & Beidas, R. S. (2007). Smoothing the trail for dissemination of evidence-based practices for youth: Flexibility within fidelity. Professional Psychology, Research and Practice, 38, 13–20, doi:10.1037/0735-7028.38.1.13.
Lillehoj, C. J., Griffin, K. W., & Spoth, R. (2004). Program provider and observer ratings of school-based preventive intervention implementation: Agreement and relation to youth outcomes. Health Education and Behavior, 31, 242–257, doi:10.1177/1090198103260514.
Moncher, F., & Prinz, R. (1991). Treatment fidelity in outcome studies. Clinical Psychology Review, 11, 247–266, doi:10.1016/0272-7358(91)90103-2.
Ogden, T., Forgatch, M. S., Askeland, E., Patterson, G. R., & Bullock, B. M. (2005). Implementation of parent management training at the national level: The Case of Norway. Journal of Social Work Practice, 19, 317–329, doi:10.1080/02650530500291518.
Pentz, M. A., Trebow, E. A., Hansen, W. B., MacKinnon, D. P., Dqyer, J. H., Johnson, C. A., et al. (1990). Effects of program implementation on adolescent drug use behavior: The Midwestern Prevention Project (MPP). Evaluation Review, 14, 264–289, doi:10.1177/0193841X9001400303.
Rohrbach, L. A., Graham, J. W., & Hansen, W. B. (1993). Diffusion of a school-based substance abuse prevention program: Predictors of program implementation. Preventive Medicine, 22, 237–260, doi:10.1006/pmed.1993.1020.
Small, M., Domitrovich, C., Camplese, K., & Prabhu, V. (2006). Interactive system for program implementation and real-time evaluation: Using technology to improve implementation quality of EBIs. Paper presented at the annual convention of the Society for Prevention Research, San Antonio, TX, May.
Sobol, D., Rorbach, L., Dent, C., Gleason, L., Brannon, B., Johnson, C. A., et al. (1989). The integrity of smoking prevention curriculum delivery. Health Education Research, 4, 59–67, doi:10.1093/her/4.1.59.
Yeaton, W. H., & Sechrest, L. (1981). Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49, 156–167, doi:10.1037/0022-006X.49.2.156.
Author information
Authors and Affiliations
Corresponding author
Additional information
This study was supported by a grant to Gerald J. August from the National Institute on Drug Abuse (DA016067). The authors would like to thank Ms. Sarah Coleman, the project manager, for her major contribution to this effort.
Rights and permissions
About this article
Cite this article
Lee, CY.S., August, G.J., Realmuto, G.M. et al. Fidelity at a Distance: Assessing Implementation Fidelity of the Early Risers Prevention Program in a Going-to-Scale Intervention Trial. Prev Sci 9, 215–229 (2008). https://doi.org/10.1007/s11121-008-0097-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-008-0097-6