Skip to main content

Advertisement

SpringerLink
Log in

Exploring Provider Use of a Digital Implementation Support System for School Mental Health: A Pilot Study

  • Original Article
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

This pilot study explored provider use of an online system, Centervention, to support the delivery of empirically supported school-based mental health interventions (ESIs); and associations between components of this system [resources, training, technical assistance (TA), feedback loops], implementation indicators, and student outcomes. Multilevel modeling data were collected from 39 providers implementing ESIs with 758 students. Training, TA, and progress monitoring predicted ESI adherence, and perceived value of resources and TA influenced student responsiveness. Greater adherence was predictive of better socio-emotional outcomes. Interviews with 15 providers illuminated how they used these four Centervention support strategies. Implications for digital implementation support research are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. Initially, the project team was planning on capturing provider use of this variable. Unfortunately, usage data on resources were unavailable due to unanticipated issues with the data usage tracker for the Resource Center pages. The Resource Center pages were not all coded to drive a change in the URL when clicked on by the user; in other words, the number of clicks on Resource Center pages could not be accurately captured. Provider perceptions of the value of ESI resources were included instead, based on previous research supporting that quality of resources are important for ESI implementation.

  2. The kappa value was based on the two coders coding the presence or absence of 67 codes in each of the transcripts included in the 20% check (i.e., 3 interviews). The unit of analysis was the provider/transcript; consequently, the kappa value was computed based on 201 as the total number of opportunities to agree on presence or absence of a particular code. Kappa values ranged from 0.81 to 1 per transcript/provider.

  3. The inclusion of the implementation support strategies reduced the conditional variance at the group level from 0.024 to 0.022, while variance at the provider level was reduced from 0.043 to 0.012.

  4. The inclusion of the implementation support strategies reduced the conditional variance at both the group level (from 0.13 to 0.083) and the provider level (from 0.14 to 0.089).

  5. As could be expected, the addition of adherence, engagement, and attendance to the model did not alter the conditional variance at the student level (0.35 to 0.35) (attendance, the only student-level variable, had minimal variability). However, the variance at the group level decreased from 0.51 to 0.48.

References

  • 3C Institute. Centervention. Retrieved October 31, 2014, from https://centervention.org.

  • Anderson, N. B. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285.

    Article  Google Scholar 

  • Baron, M., & Graham, C. (2007). Identifying key factors in the adoption of innovative practices. Technology and Teacher Education Annual, 18(2), 772.

    Google Scholar 

  • Berkel, C., Mauricio, A. M., Schoenfelder, E., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12, 23–33. doi:10.1007/s11121-010-0186-1.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bickman, L., Douglas, S. R., De Andrade, A. R. V., Tomlinson, M., Gleacher, A., Olin, S., & Hoagwood, K. (2016). Implementing a measurement feedback system: A tale of two sites. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 410–425. doi:10.1007/s10488-015-0647-8.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bickman, L., Kelley, S., Breda, C., De Andrade, A., & Riemer, M. (2011). Effects of routine feedback to clinicians on youth mental health outcomes: A randomized trial. Psychiatric Services, 62(12), 1423–1429. doi:10.1176/appi.ps.002052011.

    Article  PubMed  Google Scholar 

  • Bickman, L., Lyon, A. R., & Wolpert, M. (2016). Achieving precision mental health through effective assessment, monitoring, and feedback processes: Introduction to the special issue. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 271–276. doi:10.1007/s10488-016-0718-5.

    Article  PubMed  PubMed Central  Google Scholar 

  • Brightwood, L. H., & DeRosier, M. E. (2007). Lifestories for kids: Enhancing character development and social skills through storytelling (Grades 3–5). Cary, NC: 3C Institute for Social Development.

    Google Scholar 

  • Brightwood, L. H., DeRosier, M. E., & Maschauer, E. L. (2012). Adventures in emotional literacy (AEL): Building social relationships through enhanced communication, cooperation, and confidence (Grade 3). Cary, NC: 3C Institute for Social Development.

    Google Scholar 

  • Brightwood, L. H., DeRosier, M. E., Maschauer, E. L., & Wilson, M. E. (2009). Adventures in emotional literacy (AEL): Building social relationships through enhanced communication, cooperation, and confidence (Grades K-2). Cary, NC: 3C Institute for Social Development.

    Google Scholar 

  • Bumbarger, B., & Perkins, D. F. (2008). After randomized trials: Issues related to dissemination of evidence-based interventions. Journal of Children’s Services, 3, 53–61. doi:10.1108/17466660200800012.

    Google Scholar 

  • Craig, A. B., Leary, K. A., Parker, J., McMillen, J. S., & DeRosier, M. E. (2015). Effectiveness of an emotional literacy intervention for the elementary classroom. (Unpublished manuscript).

  • Crosse, S., Williams, B., Hagen, C. A., Harmon, M., Ristow, L., DiGaetano, R., … & Derzon, J. H. (2011). Prevalence and implementation fidelity of research-based prevention programs in public schools (Final Report). Washington, D.C.: U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service. Retrieved from http://files.eric.ed.gov/fulltext/ED529062.pdf.

  • DeRosier, M. E. (2004). Building relationships and combating bullying: Effectiveness of a school-based social skills group intervention. Journal of Clinical Child and Adolescent Psychology, 33, 125–130. doi:10.1207/S15374424JCCP3301_18.

    Article  Google Scholar 

  • DeRosier, M. E. (2007). Social Skills Group Intervention (S.S.GRIN): Group interventions and exercises for enhancing children’s communication, cooperation, and confidence (Grades K-5). Cary, NC: 3C Institute for Social Development.

    Google Scholar 

  • DeRosier, M. E., & Brightwood, L. H. (2007). Lifestories for kids: Enhancing character development and social skills through storytelling (Grades K-2). Cary, NC: 3C Institute for Social Development.

    Google Scholar 

  • DeRosier, M. E., & Marcus, S. R. (2005). Building friendships and combating bullying: Effectiveness of S.S.GRIN at one-year follow-up. Journal of Clinical Child and Adolescent Psychology, 34, 140–150. doi:10.1207/s15374424jccp3401_13.

    Article  PubMed  Google Scholar 

  • DeRosier, M. E., & Mercer, S. (2007). Improving student social behavior: The effectiveness of a school-based character education program. Journal of Research in Character Education, 5, 131–148.

    Google Scholar 

  • Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M., Hoagwood, K., Buckley, J. A., Olin, S., … & Ialongo, N. S. (2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion, 1, 6–28. doi:10.1080/1754730X.2008.9715730.

    Article  PubMed  PubMed Central  Google Scholar 

  • Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. doi:10.1007/s10464-008-9165-0.

    Article  PubMed  Google Scholar 

  • Farmer, E. M., Burns, B. J., Phillips, S. D., Angold, A., & Costello, E. J. (2003). Pathways into and through mental health services for children and adolescents. Psychiatric Services, 54(1), 60–66. doi:10.1176/appi.ps.54.1.60.

    Article  PubMed  Google Scholar 

  • Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.

    Google Scholar 

  • Forman, S. G., Olin, S. S., Hoagwood, K. E., Crowe, M., & Saka, N. (2009). Evidence-based interventions in schools: Developers’ views of implementation barriers and facilitators. School Mental Health, 1, 26–36. doi:10.1007/s12310-008-9002-5.

    Article  Google Scholar 

  • Forman, S. G., Shapiro, E. S., Codding, R. S., Gonzales, J. E., Reddy, L. A., Rosenfield, S. A., … & Stoiber, K. C. (2013). Implementation science and school psychology. School Psychology Quarterly, 28, 77–100. doi:10.1037/spq0000019.

    Article  PubMed  Google Scholar 

  • Frambach, R. T., & Schillewaert, N. (2002). Organizational innovation adoption: A multi-level framework of determinants and opportunities for future research. Journal of Business Research, 55(2), 163–176. doi:10.1016/S0148-2963(00)00152-1.

    Article  Google Scholar 

  • Gill, P., Stewart, K., Treasure, E., & Chadwick, B. (2008). Methods of data collection in qualitative research: Interviews and focus groups. British Dental Journal, 204, 291–295. doi:10.1038/bdj.2008.192.

    Article  CAS  PubMed  Google Scholar 

  • Gingiss, P. M., Roberts-Gray, C., & Boerm, M. (2006). Bridge-It: A system for predicting implementation fidelity for school-based tobacco prevention programs. Prevention Science, 7, 197–207. doi:10.1007/s11121-006-0038-1.

    Article  PubMed  Google Scholar 

  • Gresham, F. M., Cook, C. R., Collins, T., Dart, E., Rasetshwane, K., Truelson, E., & Grant, S. (2010). Developing a change-sensitive brief behavior rating scale as a progress monitoring tool for social behavior: An example using the Social Skills Rating System—Teacher Form. School Psychology Review, 39, 364–379.

    Google Scholar 

  • Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33, 665–679. doi:10.1007/s10802-005-7646-2.

    Article  PubMed  Google Scholar 

  • Henderson, J. L., MacKay, S., & Peterson-Badali, M. (2006). Closing the research-practice gap: Factors affecting adoption and implementation of a children’s mental health program. Journal of Clinical Child and Adolescent Psychology, 35, 2–12. doi:10.1207/s15374424jccp3501_1.

    Article  PubMed  Google Scholar 

  • Herschell, A. D., Kolko, D. J., Baumann, B. L., & Davis, A. C. (2010). The role of therapist training in the implementation of psychosocial treatments: A review and critique with recommendations. Clinical Psychology Review, 30, 448–466. doi:10.1016/j.cpr.2010.02.005.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hoagwood, K., Atkins, M., & Ialongo, N. (2013). Unpacking the black box of implementation: The next generation for policy, research, and practice. Administration and Policy in Mental Health, 40, 451–455. doi:10.1007/s10488-013-0512-6.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15, 1277–1288. doi:10.1177/1049732305276687.

    Article  PubMed  Google Scholar 

  • Hubley, S., Woodcock, E. A., Dimeff, L. A., & Dimidjian, S. (2015). Disseminating behavioural activation for depression via online training: Preliminary steps. Behavioural and Cognitive Psychotherapy, 43(02), 224–238. doi:10.1017/S1352465813000842.

    Article  PubMed  Google Scholar 

  • Jensen, P. S., & Foster, M. (2010). Closing the research to practice gap in children’s mental health: Structures, solutions, and strategies. Administration and Policy in Mental Health and Mental Health Services Research, 37, 111–119. doi:10.1007/s10488-010-0286-z.

    Article  PubMed  Google Scholar 

  • Jensen-Doss, A., & Hawley, K. M. (2010). Understanding barriers to evidence-based assessment: Clinician attitudes toward standardized assessment tools. Journal of Clinical Child and Adolescent Psychology, 39(6), 885–896. doi:10.1080/15374416.2010.517169.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kreft, I. G., De Leeuw, J., & Aiken, L. S. (1995). The effect of different forms of centering in hierarchical linear models. Multivariate Behavioral Research, 30, 1–21.

    Article  CAS  PubMed  Google Scholar 

  • Langberg, J. M., & Smith, B. H. (2006). Developing evidence-based interventions for deployment into school settings: A case example highlighting key issues of efficacy and effectiveness. Evaluation and Program Planning, 29, 323–334. doi:10.1016/j.evalprogplan.2006.02.002.

    Article  PubMed  Google Scholar 

  • Langley, A. K., Nadeem, E., Kataoka, S. H., Stein, B. D., & Jaycox, L. H. (2010). Evidence-based mental health programs in schools: Barriers and facilitators of successful implementation. School Mental Health, 2, 105–113. doi:10.1007/s12310-010-9038-1.

    Article  PubMed  PubMed Central  Google Scholar 

  • Leeman, J., Calancie, L., Hartman, M. A., Escoffery, C. T., Herrmann, A. K., Tague, L. E., … & Samuel-Hodge, C. (2015). What strategies are used to build practitioners’ capacity to implement community-based interventions and are they effective? A systematic review. Implementation Science, 10, 80. doi:10.1186/s13012-015-0272-7.

    Article  PubMed  PubMed Central  Google Scholar 

  • Levin, B. L., Hennessy, K. D., & Petrila, J. (2010). Mental health services: A public health perspective (3rd edn.). New York: Oxford University Press.

    Book  Google Scholar 

  • Lochman, J. E., Boxmeyer, C., Powell, N., Qu, L., Wells, K., & Windle, M. (2009). Dissemination of the Coping Power program: Importance of intensity of counselor training. Journal of Consulting and Clinical Psychology, 77, 397. doi:10.1037/a0014514.

    Article  PubMed  Google Scholar 

  • Lyon, A. R., & Lewis, C. C. (2016). Designing health information technologies for uptake: Development and implementation of measurement feedback systems in mental health service delivery. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 344–349. doi:10.1007/s10488-015-0704-3.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lyon, A. R., Lewis, C. C., Boyd, M. R., Hendrix, E., & Liu, F. (2016a). Capabilities and characteristics of digital measurement feedback systems: Results from a comprehensive review. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 441–466. doi:10.1007/s10488-016-0719-4.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lyon, A. R., Ludwig, K., Wasse, J. K., Bergstrom, A., Hendrix, E., & McCauley, E. (2016b). Determinants and functions of standardized assessment use among school mental health clinicians: A mixed methods evaluation. Administration and Policy in Mental Health and Mental Health Services Research, 43(1), 122–134. doi:10.1007/s10488-015-0626-0.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lyon, A. R., Pullmann, M. D., Whitaker, K., Ludwig, K., Wasse, J. K., & McCauley, E. (2017). A digital feedback system to support implementation of measurement-based care by school-based mental health clinicians. Journal of Clinical Child and Adolescent Psychology, 00, 1–12. doi:10.1080/15374416.2017.1280808.

    Article  Google Scholar 

  • Lyon, A. R., Wasse, J. K., Ludwig, K., Zachry, M., Bruns, E. J., Unützer, J., & McCauley, E. (2016c). The contextualized technology adaptation process (CTAP): Optimizing health information technology to improve mental health systems. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 394–409. doi:10.1007/s10488-015-0637-x.

    Article  PubMed  PubMed Central  Google Scholar 

  • McHugh, K. R., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65, 73–84. doi:10.1037/a0018121.

    Article  PubMed  Google Scholar 

  • McIntyre, L. L., Gresham, F. M., Digennaro, F. D., & Reed, D. D. (2007). Treatment integrity of school-based interventions with children in Journal of Applied Behavioral Analysis studies from 1991 to 2005. Journal of Applied Behavior Analysis, 40, 659–672. doi:10.1901/jaba.2007.659-672.

    Article  PubMed  Google Scholar 

  • Meyers, D. C., Durlak, J. A., & Wandersman, A. (2012). The Quality Implementation Framework: A synthesis of critical steps in the implementation process. American Journal of Community Psychology, 50(3–4), 462–480. doi:10.1007/s10464-012-9522-x.

    Article  PubMed  Google Scholar 

  • Mihalic, S. F., Fagan, A. A., & Argamaso, S. (2008). Implementing the LifeSkills Training drug prevention program: Factors related to implementation fidelity. Implementation Science, 3, 1–16. doi:10.1186/1748-5908-3-5.

    Article  Google Scholar 

  • Nadeem, E., Cappella, E., Holland, S., Coccaro, C., & Crisonino, G. (2016). Development and piloting of a classroom-focused measurement feedback system. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 379–393. doi:10.1007/s10488-015-0651-z.

    Article  PubMed  PubMed Central  Google Scholar 

  • O’Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K–12 curriculum intervention research. Review of Educational Research, 78, 33–84. doi:10.3102/0034654307313793.

    Article  Google Scholar 

  • Palinkas, L. A., Aarons, G. A., Horwitz, S., Chamberlain, P., Hurlburt, M., & Landsverk, J. (2011). Mixed method designs in implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 44–53. doi:10.1007/s10488-010-0314-z.

    Article  PubMed  Google Scholar 

  • Powell, B. J., McMillen, J. C., Hawley, K. M., & Proctor, E. K. (2013). Mental health clinicians’ motivation to invest in training: Results from a practice-based research network survey. Psychiatric Services, 64, 816–818. doi:10.1176/appi.ps.003602012.

    Article  PubMed  Google Scholar 

  • Powell, B. J., Proctor, E. K., & Glass, J. E. (2014). A systematic review of strategies for implementing empirically supported mental health interventions. Research on Social Work Practice, 24, 192–212. doi:10.1177/1049731513505778.

    Article  PubMed  Google Scholar 

  • Proctor, E. K., & Brownson, R. C. (2012). Measurement issues in dissemination and implementation research. In R. C. Brownson, G. A. Colditz & E. K. Proctor (Eds.), Dissemination and implementation research in health: Translating science to practice (pp. 261–280). New York, NY: Oxford University Press, Inc. doi:10.1093/acprof:oso/9780199751877.003.0013.

    Chapter  Google Scholar 

  • Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013). Implementation strategies: Recommendations for specifying and reporting. Implementation Science, 8(1), 1. doi:10.1186/1748-5908-8-139.

    Article  Google Scholar 

  • Rohrbach, L. A., Gunning, M., Sun, P., & Sussman, S. (2010). The project towards no drug abuse (TND) dissemination trial: Implementation fidelity and immediate outcomes. Prevention Science, 11, 77–88. doi:10.1007/s11121-009-0151-z.

    Article  PubMed  PubMed Central  Google Scholar 

  • Shea, C. M., Jacobs, S. R., Esserman, D. A., Bruce, K., & Weiner, B. J. (2014). Organizational readiness for implementing change: A psychometric assessment of a new measure. Implementation Science, 9, 15. doi:10.1186/1748-5908-9-7.

    Article  Google Scholar 

  • Sopko, K. M. (2006). School mental health services in the United States. InForum, 1–9. Available from National Association of State Directors of Special Education website: http://nasdse.org/DesktopModules/DNNspot-Store/ProductFiles/180_a75a2595-fb39-4f38-ac15-46b4d9bc45b6.pdf.

  • Tibbits, M. K., Bumbarger, B. K., Kyler, S. J., & Perkins, D. F. (2010). Sustaining evidence-based interventions under real-world conditions: Results from a large-scale diffusion project. Prevention Science, 11, 252–262. doi:10.1007/s11121-010-0170-9.

    Article  PubMed  PubMed Central  Google Scholar 

  • Todd, N. R., Allen, N. E., & Javdani, S. (2012). Multilevel modeling: Method and application for community-based research. In L. A. Jason & D. S. Glenwick (Eds.), Methodological approaches to community-based research (pp. 167–186). Washington D.C.: American Psychological Association. 10.1037/13492-010.

    Chapter  Google Scholar 

  • Wandersman, A., Chien, V. H., & Katz, J. (2012). Toward an evidence-based system for innovation support for implementing innovations with quality: Tools, training, technical assistance, and quality assurance/quality improvement. American Journal of Community Psychology, 50, 445–459. doi:10.1007/s10464-012-9509-7.

    Article  PubMed  Google Scholar 

  • Weiner, B. J., Amick, H., & Lee, S.-Y. D. (2008). Conceptualization and measurement of organizational readiness for change: A review of the literature in health services research and other fields. Medical Care Research and Review, 65(4), 379–436. doi:10.1177/1077558708317802.

    Article  PubMed  Google Scholar 

Download references

Funding

This study received support from a grant from the National Institute of Mental Health (2R44MH084375-03).

Author information

Authors and Affiliations

  1. 3C Institute, 4364 S Alston Ave, Durham, NC, 27713, USA

    Melanie Livet, Mary Yannayon, Kelly Kocher, James Upright & Janey McMillen

  2. University of North Carolina-Chapel Hill, Chapel Hill, NC, 27514, USA

    Kelly Sheppard

Authors
  1. Melanie Livet
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mary Yannayon
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kelly Sheppard
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kelly Kocher
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. James Upright
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Janey McMillen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Melanie Livet.

Ethics declarations

Conflict of interest

Five of the six authors who conducted this study were employees of the 3C Institute at the time of the study. Both the software and the ESIs tested in this study are commercially available. In order to minimize conflict of interest concerns, analyses were performed by author K.S., who had no competing financial interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 882 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Livet, M., Yannayon, M., Sheppard, K. et al. Exploring Provider Use of a Digital Implementation Support System for School Mental Health: A Pilot Study. Adm Policy Ment Health 45, 362–380 (2018). https://doi.org/10.1007/s10488-017-0829-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-017-0829-7

Keywords

Search

Navigation