Exploring Provider Use of a Digital Implementation Support System for School Mental Health: A Pilot Study

  • Melanie Livet
  • Mary Yannayon
  • Kelly Sheppard
  • Kelly Kocher
  • James Upright
  • Janey McMillen
Original Article


This pilot study explored provider use of an online system, Centervention, to support the delivery of empirically supported school-based mental health interventions (ESIs); and associations between components of this system [resources, training, technical assistance (TA), feedback loops], implementation indicators, and student outcomes. Multilevel modeling data were collected from 39 providers implementing ESIs with 758 students. Training, TA, and progress monitoring predicted ESI adherence, and perceived value of resources and TA influenced student responsiveness. Greater adherence was predictive of better socio-emotional outcomes. Interviews with 15 providers illuminated how they used these four Centervention support strategies. Implications for digital implementation support research are discussed.


Implementation Outcomes EBSIS Technology Implementation strategies 



This study received support from a grant from the National Institute of Mental Health (2R44MH084375-03).

Compliance with Ethical Standards

Conflict of interest

Five of the six authors who conducted this study were employees of the 3C Institute at the time of the study. Both the software and the ESIs tested in this study are commercially available. In order to minimize conflict of interest concerns, analyses were performed by author K.S., who had no competing financial interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Supplementary material

10488_2017_829_MOESM1_ESM.docx (883 kb)
Supplementary material 1 (DOCX 882 KB)


  1. 3C Institute. Centervention. Retrieved October 31, 2014, from
  2. Anderson, N. B. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285.CrossRefGoogle Scholar
  3. Baron, M., & Graham, C. (2007). Identifying key factors in the adoption of innovative practices. Technology and Teacher Education Annual, 18(2), 772.Google Scholar
  4. Berkel, C., Mauricio, A. M., Schoenfelder, E., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12, 23–33. doi: 10.1007/s11121-010-0186-1.CrossRefPubMedPubMedCentralGoogle Scholar
  5. Bickman, L., Douglas, S. R., De Andrade, A. R. V., Tomlinson, M., Gleacher, A., Olin, S., & Hoagwood, K. (2016). Implementing a measurement feedback system: A tale of two sites. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 410–425. doi: 10.1007/s10488-015-0647-8.CrossRefPubMedPubMedCentralGoogle Scholar
  6. Bickman, L., Kelley, S., Breda, C., De Andrade, A., & Riemer, M. (2011). Effects of routine feedback to clinicians on youth mental health outcomes: A randomized trial. Psychiatric Services, 62(12), 1423–1429. doi: 10.1176/ Scholar
  7. Bickman, L., Lyon, A. R., & Wolpert, M. (2016). Achieving precision mental health through effective assessment, monitoring, and feedback processes: Introduction to the special issue. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 271–276. doi: 10.1007/s10488-016-0718-5.CrossRefPubMedPubMedCentralGoogle Scholar
  8. Brightwood, L. H., & DeRosier, M. E. (2007). Lifestories for kids: Enhancing character development and social skills through storytelling (Grades 3–5). Cary, NC: 3C Institute for Social Development.Google Scholar
  9. Brightwood, L. H., DeRosier, M. E., & Maschauer, E. L. (2012). Adventures in emotional literacy (AEL): Building social relationships through enhanced communication, cooperation, and confidence (Grade 3). Cary, NC: 3C Institute for Social Development.Google Scholar
  10. Brightwood, L. H., DeRosier, M. E., Maschauer, E. L., & Wilson, M. E. (2009). Adventures in emotional literacy (AEL): Building social relationships through enhanced communication, cooperation, and confidence (Grades K-2). Cary, NC: 3C Institute for Social Development.Google Scholar
  11. Bumbarger, B., & Perkins, D. F. (2008). After randomized trials: Issues related to dissemination of evidence-based interventions. Journal of Children’s Services, 3, 53–61. doi: 10.1108/17466660200800012.Google Scholar
  12. Craig, A. B., Leary, K. A., Parker, J., McMillen, J. S., & DeRosier, M. E. (2015). Effectiveness of an emotional literacy intervention for the elementary classroom. (Unpublished manuscript).Google Scholar
  13. Crosse, S., Williams, B., Hagen, C. A., Harmon, M., Ristow, L., DiGaetano, R., … & Derzon, J. H. (2011). Prevalence and implementation fidelity of research-based prevention programs in public schools (Final Report). Washington, D.C.: U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service. Retrieved from
  14. DeRosier, M. E. (2004). Building relationships and combating bullying: Effectiveness of a school-based social skills group intervention. Journal of Clinical Child and Adolescent Psychology, 33, 125–130. doi: 10.1207/S15374424JCCP3301_18.CrossRefGoogle Scholar
  15. DeRosier, M. E. (2007). Social Skills Group Intervention (S.S.GRIN): Group interventions and exercises for enhancing children’s communication, cooperation, and confidence (Grades K-5). Cary, NC: 3C Institute for Social Development.Google Scholar
  16. DeRosier, M. E., & Brightwood, L. H. (2007). Lifestories for kids: Enhancing character development and social skills through storytelling (Grades K-2). Cary, NC: 3C Institute for Social Development.Google Scholar
  17. DeRosier, M. E., & Marcus, S. R. (2005). Building friendships and combating bullying: Effectiveness of S.S.GRIN at one-year follow-up. Journal of Clinical Child and Adolescent Psychology, 34, 140–150. doi: 10.1207/s15374424jccp3401_13.CrossRefPubMedGoogle Scholar
  18. DeRosier, M. E., & Mercer, S. (2007). Improving student social behavior: The effectiveness of a school-based character education program. Journal of Research in Character Education, 5, 131–148.Google Scholar
  19. Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M., Hoagwood, K., Buckley, J. A., Olin, S., … & Ialongo, N. S. (2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion, 1, 6–28. doi: 10.1080/1754730X.2008.9715730.CrossRefPubMedPubMedCentralGoogle Scholar
  20. Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. doi: 10.1007/s10464-008-9165-0.CrossRefPubMedGoogle Scholar
  21. Farmer, E. M., Burns, B. J., Phillips, S. D., Angold, A., & Costello, E. J. (2003). Pathways into and through mental health services for children and adolescents. Psychiatric Services, 54(1), 60–66. doi: 10.1176/ Scholar
  22. Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.Google Scholar
  23. Forman, S. G., Olin, S. S., Hoagwood, K. E., Crowe, M., & Saka, N. (2009). Evidence-based interventions in schools: Developers’ views of implementation barriers and facilitators. School Mental Health, 1, 26–36. doi: 10.1007/s12310-008-9002-5.CrossRefGoogle Scholar
  24. Forman, S. G., Shapiro, E. S., Codding, R. S., Gonzales, J. E., Reddy, L. A., Rosenfield, S. A., … & Stoiber, K. C. (2013). Implementation science and school psychology. School Psychology Quarterly, 28, 77–100. doi: 10.1037/spq0000019.CrossRefPubMedGoogle Scholar
  25. Frambach, R. T., & Schillewaert, N. (2002). Organizational innovation adoption: A multi-level framework of determinants and opportunities for future research. Journal of Business Research, 55(2), 163–176. doi: 10.1016/S0148-2963(00)00152-1.CrossRefGoogle Scholar
  26. Gill, P., Stewart, K., Treasure, E., & Chadwick, B. (2008). Methods of data collection in qualitative research: Interviews and focus groups. British Dental Journal, 204, 291–295. doi: 10.1038/bdj.2008.192.CrossRefPubMedGoogle Scholar
  27. Gingiss, P. M., Roberts-Gray, C., & Boerm, M. (2006). Bridge-It: A system for predicting implementation fidelity for school-based tobacco prevention programs. Prevention Science, 7, 197–207. doi: 10.1007/s11121-006-0038-1.CrossRefPubMedGoogle Scholar
  28. Gresham, F. M., Cook, C. R., Collins, T., Dart, E., Rasetshwane, K., Truelson, E., & Grant, S. (2010). Developing a change-sensitive brief behavior rating scale as a progress monitoring tool for social behavior: An example using the Social Skills Rating System—Teacher Form. School Psychology Review, 39, 364–379.Google Scholar
  29. Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33, 665–679. doi: 10.1007/s10802-005-7646-2.CrossRefPubMedGoogle Scholar
  30. Henderson, J. L., MacKay, S., & Peterson-Badali, M. (2006). Closing the research-practice gap: Factors affecting adoption and implementation of a children’s mental health program. Journal of Clinical Child and Adolescent Psychology, 35, 2–12. doi: 10.1207/s15374424jccp3501_1.CrossRefPubMedGoogle Scholar
  31. Herschell, A. D., Kolko, D. J., Baumann, B. L., & Davis, A. C. (2010). The role of therapist training in the implementation of psychosocial treatments: A review and critique with recommendations. Clinical Psychology Review, 30, 448–466. doi: 10.1016/j.cpr.2010.02.005.CrossRefPubMedPubMedCentralGoogle Scholar
  32. Hoagwood, K., Atkins, M., & Ialongo, N. (2013). Unpacking the black box of implementation: The next generation for policy, research, and practice. Administration and Policy in Mental Health, 40, 451–455. doi: 10.1007/s10488-013-0512-6.CrossRefPubMedPubMedCentralGoogle Scholar
  33. Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15, 1277–1288. doi: 10.1177/1049732305276687.CrossRefPubMedGoogle Scholar
  34. Hubley, S., Woodcock, E. A., Dimeff, L. A., & Dimidjian, S. (2015). Disseminating behavioural activation for depression via online training: Preliminary steps. Behavioural and Cognitive Psychotherapy, 43(02), 224–238. doi: 10.1017/S1352465813000842.CrossRefPubMedGoogle Scholar
  35. Jensen, P. S., & Foster, M. (2010). Closing the research to practice gap in children’s mental health: Structures, solutions, and strategies. Administration and Policy in Mental Health and Mental Health Services Research, 37, 111–119. doi: 10.1007/s10488-010-0286-z.CrossRefPubMedGoogle Scholar
  36. Jensen-Doss, A., & Hawley, K. M. (2010). Understanding barriers to evidence-based assessment: Clinician attitudes toward standardized assessment tools. Journal of Clinical Child and Adolescent Psychology, 39(6), 885–896. doi: 10.1080/15374416.2010.517169.CrossRefPubMedPubMedCentralGoogle Scholar
  37. Kreft, I. G., De Leeuw, J., & Aiken, L. S. (1995). The effect of different forms of centering in hierarchical linear models. Multivariate Behavioral Research, 30, 1–21.CrossRefPubMedGoogle Scholar
  38. Langberg, J. M., & Smith, B. H. (2006). Developing evidence-based interventions for deployment into school settings: A case example highlighting key issues of efficacy and effectiveness. Evaluation and Program Planning, 29, 323–334. doi: 10.1016/j.evalprogplan.2006.02.002.CrossRefPubMedGoogle Scholar
  39. Langley, A. K., Nadeem, E., Kataoka, S. H., Stein, B. D., & Jaycox, L. H. (2010). Evidence-based mental health programs in schools: Barriers and facilitators of successful implementation. School Mental Health, 2, 105–113. doi: 10.1007/s12310-010-9038-1.CrossRefPubMedPubMedCentralGoogle Scholar
  40. Leeman, J., Calancie, L., Hartman, M. A., Escoffery, C. T., Herrmann, A. K., Tague, L. E., … & Samuel-Hodge, C. (2015). What strategies are used to build practitioners’ capacity to implement community-based interventions and are they effective? A systematic review. Implementation Science, 10, 80. doi: 10.1186/s13012-015-0272-7.CrossRefPubMedPubMedCentralGoogle Scholar
  41. Levin, B. L., Hennessy, K. D., & Petrila, J. (2010). Mental health services: A public health perspective (3rd edn.). New York: Oxford University Press.CrossRefGoogle Scholar
  42. Lochman, J. E., Boxmeyer, C., Powell, N., Qu, L., Wells, K., & Windle, M. (2009). Dissemination of the Coping Power program: Importance of intensity of counselor training. Journal of Consulting and Clinical Psychology, 77, 397. doi: 10.1037/a0014514.CrossRefPubMedGoogle Scholar
  43. Lyon, A. R., & Lewis, C. C. (2016). Designing health information technologies for uptake: Development and implementation of measurement feedback systems in mental health service delivery. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 344–349. doi: 10.1007/s10488-015-0704-3.CrossRefPubMedPubMedCentralGoogle Scholar
  44. Lyon, A. R., Lewis, C. C., Boyd, M. R., Hendrix, E., & Liu, F. (2016a). Capabilities and characteristics of digital measurement feedback systems: Results from a comprehensive review. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 441–466. doi: 10.1007/s10488-016-0719-4.CrossRefPubMedPubMedCentralGoogle Scholar
  45. Lyon, A. R., Ludwig, K., Wasse, J. K., Bergstrom, A., Hendrix, E., & McCauley, E. (2016b). Determinants and functions of standardized assessment use among school mental health clinicians: A mixed methods evaluation. Administration and Policy in Mental Health and Mental Health Services Research, 43(1), 122–134. doi: 10.1007/s10488-015-0626-0.CrossRefPubMedPubMedCentralGoogle Scholar
  46. Lyon, A. R., Pullmann, M. D., Whitaker, K., Ludwig, K., Wasse, J. K., & McCauley, E. (2017). A digital feedback system to support implementation of measurement-based care by school-based mental health clinicians. Journal of Clinical Child and Adolescent Psychology, 00, 1–12. doi: 10.1080/15374416.2017.1280808.CrossRefGoogle Scholar
  47. Lyon, A. R., Wasse, J. K., Ludwig, K., Zachry, M., Bruns, E. J., Unützer, J., & McCauley, E. (2016c). The contextualized technology adaptation process (CTAP): Optimizing health information technology to improve mental health systems. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 394–409. doi: 10.1007/s10488-015-0637-x.CrossRefPubMedPubMedCentralGoogle Scholar
  48. McHugh, K. R., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65, 73–84. doi: 10.1037/a0018121.CrossRefPubMedGoogle Scholar
  49. McIntyre, L. L., Gresham, F. M., Digennaro, F. D., & Reed, D. D. (2007). Treatment integrity of school-based interventions with children in Journal of Applied Behavioral Analysis studies from 1991 to 2005. Journal of Applied Behavior Analysis, 40, 659–672. doi: 10.1901/jaba.2007.659-672.CrossRefPubMedGoogle Scholar
  50. Meyers, D. C., Durlak, J. A., & Wandersman, A. (2012). The Quality Implementation Framework: A synthesis of critical steps in the implementation process. American Journal of Community Psychology, 50(3–4), 462–480. doi: 10.1007/s10464-012-9522-x.CrossRefPubMedGoogle Scholar
  51. Mihalic, S. F., Fagan, A. A., & Argamaso, S. (2008). Implementing the LifeSkills Training drug prevention program: Factors related to implementation fidelity. Implementation Science, 3, 1–16. doi: 10.1186/1748-5908-3-5.CrossRefGoogle Scholar
  52. Nadeem, E., Cappella, E., Holland, S., Coccaro, C., & Crisonino, G. (2016). Development and piloting of a classroom-focused measurement feedback system. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 379–393. doi: 10.1007/s10488-015-0651-z.CrossRefPubMedPubMedCentralGoogle Scholar
  53. O’Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K–12 curriculum intervention research. Review of Educational Research, 78, 33–84. doi: 10.3102/0034654307313793.CrossRefGoogle Scholar
  54. Palinkas, L. A., Aarons, G. A., Horwitz, S., Chamberlain, P., Hurlburt, M., & Landsverk, J. (2011). Mixed method designs in implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 44–53. doi: 10.1007/s10488-010-0314-z.CrossRefPubMedGoogle Scholar
  55. Powell, B. J., McMillen, J. C., Hawley, K. M., & Proctor, E. K. (2013). Mental health clinicians’ motivation to invest in training: Results from a practice-based research network survey. Psychiatric Services, 64, 816–818. doi: 10.1176/ Scholar
  56. Powell, B. J., Proctor, E. K., & Glass, J. E. (2014). A systematic review of strategies for implementing empirically supported mental health interventions. Research on Social Work Practice, 24, 192–212. doi: 10.1177/1049731513505778.CrossRefPubMedGoogle Scholar
  57. Proctor, E. K., & Brownson, R. C. (2012). Measurement issues in dissemination and implementation research. In R. C. Brownson, G. A. Colditz & E. K. Proctor (Eds.), Dissemination and implementation research in health: Translating science to practice (pp. 261–280). New York, NY: Oxford University Press, Inc. doi: 10.1093/acprof:oso/9780199751877.003.0013.CrossRefGoogle Scholar
  58. Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013). Implementation strategies: Recommendations for specifying and reporting. Implementation Science, 8(1), 1. doi: 10.1186/1748-5908-8-139.CrossRefGoogle Scholar
  59. Rohrbach, L. A., Gunning, M., Sun, P., & Sussman, S. (2010). The project towards no drug abuse (TND) dissemination trial: Implementation fidelity and immediate outcomes. Prevention Science, 11, 77–88. doi: 10.1007/s11121-009-0151-z.CrossRefPubMedPubMedCentralGoogle Scholar
  60. Shea, C. M., Jacobs, S. R., Esserman, D. A., Bruce, K., & Weiner, B. J. (2014). Organizational readiness for implementing change: A psychometric assessment of a new measure. Implementation Science, 9, 15. doi: 10.1186/1748-5908-9-7.CrossRefGoogle Scholar
  61. Sopko, K. M. (2006). School mental health services in the United States. InForum, 1–9. Available from National Association of State Directors of Special Education website:
  62. Tibbits, M. K., Bumbarger, B. K., Kyler, S. J., & Perkins, D. F. (2010). Sustaining evidence-based interventions under real-world conditions: Results from a large-scale diffusion project. Prevention Science, 11, 252–262. doi: 10.1007/s11121-010-0170-9.CrossRefPubMedPubMedCentralGoogle Scholar
  63. Todd, N. R., Allen, N. E., & Javdani, S. (2012). Multilevel modeling: Method and application for community-based research. In L. A. Jason & D. S. Glenwick (Eds.), Methodological approaches to community-based research (pp. 167–186). Washington D.C.: American Psychological Association.  10.1037/13492-010.CrossRefGoogle Scholar
  64. Wandersman, A., Chien, V. H., & Katz, J. (2012). Toward an evidence-based system for innovation support for implementing innovations with quality: Tools, training, technical assistance, and quality assurance/quality improvement. American Journal of Community Psychology, 50, 445–459. doi: 10.1007/s10464-012-9509-7.CrossRefPubMedGoogle Scholar
  65. Weiner, B. J., Amick, H., & Lee, S.-Y. D. (2008). Conceptualization and measurement of organizational readiness for change: A review of the literature in health services research and other fields. Medical Care Research and Review, 65(4), 379–436. doi: 10.1177/1077558708317802.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.3C InstituteDurhamUSA
  2. 2.University of North Carolina-Chapel HillChapel HillUSA

Personalised recommendations