Abstract
Health information technologies have become a central fixture in the mental healthcare landscape, but few frameworks exist to guide their adaptation to novel settings. This paper introduces the contextualized technology adaptation process (CTAP) and presents data collected during Phase 1 of its application to measurement feedback system development in school mental health. The CTAP is built on models of human-centered design and implementation science and incorporates repeated mixed methods assessments to guide the design of technologies to ensure high compatibility with a destination setting. CTAP phases include: (1) Contextual evaluation, (2) Evaluation of the unadapted technology, (3) Trialing and evaluation of the adapted technology, (4) Refinement and larger-scale implementation, and (5) Sustainment through ongoing evaluation and system revision. Qualitative findings from school-based practitioner focus groups are presented, which provided information for CTAP Phase 1, contextual evaluation, surrounding education sector clinicians’ workflows, types of technologies currently available, and influences on technology use. Discussion focuses on how findings will inform subsequent CTAP phases, as well as their implications for future technology adaptation across content domains and service sectors.
Similar content being viewed by others
References
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23.
Aarons, G. A., Green, A. E., Palinkas, L. A., Self-Brown, S., Whitaker, D. J., Lutzker, J. R., & Chaffin, M. J. (2012). Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implementation Science, 7(32), 1–9.
Atkinson, N. L. (2007). Developing a questionnaire to measure perceived attributes of eHealth innovations. American Journal of Health Behavior, 31(6), 612–621.
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human-Computer Interaction, 24(6), 574–594. doi:10.1080/10447310802205776.
Becker, E. M., & Jensen-Doss, A. (2013). Computer-assisted therapies: Examination of therapist-level barriers to their use. Behavior Therapy, 44(4), 614–624.
Bickman, L. (2008). A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry, 47(10), 1114.
Bickman, L., Kelley, S. D., Breda, C., de Andrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatric Services, 62, 1423–1429.
Bickman, L., Kelley, S. D., & Athay, M. (2012). The technology of measurement feedback systems. Couple and Family Psychology: Review and Practice, 1, 274–284.
Borntrager, C., & Lyon, A. R. (2015). Client progress monitoring and feedback in school-based mental health. Cognitive & Behavioral Practice, 22, 74–86.
Brooke, J. (1996). SUS-A quick and dirty usability scale. In P. W. Jordan, B. Thomas, I. L. McClelland, & B. Weerdmeester (Eds.), Usability evaluation in industry (pp. 189–194). Bristol, PA: Taylor & Francis Inc.
Butler, K. A., Haselkorn, M., Bahrami, A., & Schroder, K. (2011). Introducing the MATH method and toolsuite for evidence‐based HIT. In Paper presented at the 2nd Annual AMA/IEEE EMBS Medical Technology Conference, Boston, MA. Published online at https://www.uthouston.edu/dotAsset/fcc91d1b-3a16-495b-9809-37e2108ed5e2.pdf.
Chambers, D., Glasgow, R., & Stange, K. (2013). The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implement Science, 8(1), 117.
Connors, E. H., Arora, P., Curtis, L., & Stephan, S. H. (2015). Evidence-based assessment in school mental health. Cognitive and Behavioral Practice, 22, 60–73.
Cook, J. M., Biyanova, T., & Coyne, J. C. (2009). Barriers to adoption of new treatments: An internet study of practicing community psychotherapists. Administration and Policy in Mental Health and Mental Health Services Research, 36(2), 83–90.
Courage, C., & Baxter, K. (2005). Understanding your users: A practical guide to user requirements methods, tools, and techniques. San Francisco, CA: Morgan Kaufmann.
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implement Science, 4(1), 50.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340.
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Management Science, 35(8), 982–1003.
De Jong, M., & Van Der Geest, T. (2000). Characterizing web heuristics. Technical Communication, 47(3), 311–326.
DeSantis, L., & Ugarriza, D. N. (2000). The concept of theme as used in qualitative nursing research. Western Journal of Nursing Research, 22(3), 351–372.
Diaper, D., & Stanton, N. (Eds.). (2003). The handbook of task analysis for human-computer interaction. Boca Raton: CRC Press.
Farmer, E. M., Burns, B. J., Phillips, S. D., Angold, A., & Costello, E. J. (2003). Pathways into and through mental health services for children and adolescents. Psychiatric Services, 54(1), 60–66.
Few, S. (2006). Information dashboard design: The effective visual communication of data. O’Reilly.
Flanagan, M. E., Saleem, J. J., Millitello, L. G., Russ, A. L., & Doebbeling, B. N. (2013). Paper-and computer-based workarounds to electronic health record use at three benchmark institutions. Journal of the American Medical Informatics Association, 20(e1), e59–e66.
Foster, S., Rollefson, M., Doksum, T., Noonan, D., Robinson, G., & Teich, J. (2005). School Mental Health Services in the United States, 2002–2003 (DHHS Pub. No. (SMA) 05–4068). Rockville, MD: Center for Mental Health Services, Substance Abuse and Mental Health Services Administration.
Friese, S. (2012). ATLAS.ti 7 user manual. Berlin: ATLAS. ti Scientific Software Development GmbH.
Furukawa, M. F., King, J., Patel, V., Hsiao, C.-J., Adler-Milstein, J., & Jha, A. K. (2014). Despite substantial progress in EHR adoption, health information exchange and patient engagement remain low in office settings. Health Affairs, 33(9), 1672–1679.
Gance-Cleveland, B., & Yousey, Y. (2005). Benefits of a school-based health center in a preschool. Clinical Nursing Research, 14(4), 327–342.
Glasgow, R. E., Phillips, S. M., & Sanchez, M. A. (2013). Implementation science approaches for integrating eHealth research into practice and policy. International Journal of Medical Informatics, 83, e1–e11.
Glasgow, R. E., Kessler, R. S., Ory, M. G., Roby, D., Gorin, S. S., & Krist, A. (2014). Conducting rapid, relevant research: Lessons learned from the my own health report project. American Journal of Preventive Medicine, 47(2), 212–219.
González, M. P., Lorés, J., & Granollers, A. (2008). Enhancing usability testing through datamining techniques: A novel approach to detecting usability problem patterns for a context of use. Information and Software Technology, 50(6), 547–568.
Grossman, T., Fitzmaurice, G., & Attar, R. (2009). A survey of software learnability: Metrics, methodologies and guidelines. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 649–658.
Hackos, J. T., & Redish, J. (1998). User and task analysis for interface design. New York, NY: Wiley.
Health Information Technology for Economic and Clinical Health Act of 2009, Title XIII of Division A and Title IV of Division B of the American Recovery and Reinvestment Act of 2009 (ARRA), Pub. L. No. 111-5, 123 Stat. 226 (Feb 17, 2009), codified at 42 U.S.C. §§300jj et seq.; §§17901 et seq.
Heeks, R. (2006). Health information systems: Failure, success and improvisation. International Journal of Medical Informatics, 75(2), 125–137.
Hill, C. E., Thompson, B. J., & Williams, E. N. (1997). A guide to conducting consensual qualitative research. The counseling Psychologist, 25(4), 517–572.
Hill, C. E., Knox, S., Thompson, B. J., Nutt Williams, E., & Hess, S. A. (2005). Consensual qualitative research: An update. Journal of Counseling Psychology, 52, 196–205.
Holden, R. J., & Karsh, B. T. (2010). The technology acceptance model: its past and its future in health care. Journal of Biomedical Informatics, 43(1), 159–172.
Holtzblatt, K., Wendell, J. B., & Wood, S. (2004). Rapid contextual design: A how-to guide to key techniques for user-centered design. San Francisco: Elsevier.
Hornbæk, K. (2006). Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies, 64(2), 79–102. doi:10.1016/j.ijhcs.2005.06.002.
Hornbæk, K., & Law, E. L. C. (2007). Meta-analysis of correlations among usability measures. In Proceedings of the SIGCHI conference on Human factors in computing systems.
Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288.
International Standards Organization. (1998). Ergonomic requirements for office work with visual display terminals (VDTs)—Part 11: Guidance on usability. International Organization for Standardization, 9241–11.
International Standards Organization. (2010). Ergonomics of human-system interaction—Part 2010: Human centered design for interactive systems. International Organization for Standardization.
Kokkonen, E. W., Davis, S. A., Lin, H. C., Dabade, T. S., Feldman, S. R., & Fleischer, A. B., Jr. (2013). Use of electronic medical records differs by specialty and office settings. Journal of the American Medical Informatics Association, 20(1), 33–38.
Krug, S. (2014). Usability testing on 10 cents a day (pp. 110–141). Don’t make me think: A common sense approach to web usability, revisited.
Lambert, M. J., Whipple, J. L., Hawkins, E. J., Vermeersch, D. A., Nielsen, S. L., & Smart, D. W. (2003). Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clinical Psychology: Science and Practice, 10, 288–301.
Lewis, J. R. (1994). Sample size for usability studies: Additional considerations. Human Factors, 36, 368–378.
Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1), 57–78.
Lewis, J. R. (2002). Psychometric evaluation of the PSSUQ using data from five years of usability studies. International Journal of Human-Computer Interaction, 14(3–4), 463–488.
Lyon, A. R., & Lewis, C. C. Designing health information technologies for uptake: Development and implementation of measurement feedback systems in mental health service delivery. Introduction to the special section. Administration and Policy in Mental Health and Mental Health Services Research (this issue).
Lyon, A. R., Borntrager, C., Nakamura, B., & Higa-McMillan, C. (2013). From distal to proximal: Routine educational data monitoring in school-based mental health. Advances in School Mental Health Promotion, 6(4), 263–279.
Lyon, A. R., Dorsey, S., Pullmann, M., Silbaugh-Cowdin, J., & Berliner, L. (2015). Clinician use of standardized assessments following a common elements psychotherapy training and consultation program. Administration and Policy in Mental Health and Mental Health Services Research, 42, 47–60.
Lyon, A. R., Ludwig, K., Knaster Wasse, J., Bergstrom, A., Hendrix, E., & McCauley, E. Determinants and functions of standardized assessment use among school mental health clinicians: A mixed methods evaluation. Administration and Policy in Mental Health and Mental Health Services Research (in press).
McLellan, S., Muddimer, A., & Peres, S. C. (2012). The effect of experience on system usability scale ratings. Journal of Usability Studies, 7(2), 56–67.
Michel-Verkerke, M. B., & Spil, T. A. M. (2008). The USE IT-adoption-model to predict and evaluate adoption of information and communication technology in healthcare. Methods of Information in Medicine, 47(3), 260–269.
Mohr, D. C., Burns, M. N., Schueller, S. M., Clarke, G., & Klinkman, M. (2013). Behavioral intervention technologies: Evidence review and recommendations for future research in mental health. General Hospital Psychiatry, 35(4), 332–338.
Norman, D. A., & Draper, S. W. (Eds.). (1986). User centered system design: New perspectives on human-computer interaction. Hillsdale, NJ: Lawrence Erlbaum Associates.
Owens, J., Lyon, A. R., Brandt, N. E., Maisa Warner, M., Nadeem, E., Spiel, C., & Wagner, M. (2014). Implementation science in school mental health: Key constructs and a proposed research agenda. School Mental Health, 6, 99–111.
Palinkas, L. A., Aarons, G. A., Horwitz, S., Chamberlain, P., Hurlburt, M., & Landsverk, J. (2011). Mixed method designs in implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 44–53.
Patient Protection and Affordable Care Act of 2010, Pub. L. No. 111-148, § 6301, 124 Stat. 727 (2010).
Pringle, B., Chambers, D., & Wang, P. S. (2010). Toward enough of the best for all: Research to transform the efficacy, quality, and reach of mental health care for youth. Administration and Policy in Mental Health and Mental Health Services Research, 37(1), 191–196.
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38, 65–76.
Rausch, T., & Leigh Jackson, J. (2007). Using clinical workflows to improve medical device/system development. In High Confidence Medical Devices, Software, and Systems and Medical Device Plug-and-Play Interoperability, 2007. HCMDSS-MDPnP. Joint Workshop on IEEE (pp. 133–134).
Rogers, E. M. (2003). Diffusions of innovations (5th ed.). New York, NY: Free Press.
Rosenbaum, S. (1989). Usability evaluations versus usability testing: When and why? IEEE Transactions on Professional Communication, 32(4), 210–216.
Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: how to plan, design, and conduct effective tests. Indianapolis, IN: Wiley.
Sauro, J. (2011). A practical guide to the sytem usability scalel background, benchmarks & best practices. New York: CreateSpace Independent Publishing Platform.
Shehabuddeen, N. T. M. H., & Probert, D. R. (2004). Excavating the technology landscape: Deploying technology intelligence to detect early warning signals. Proceedings of the IEEE Engineering Management Society, 1, 332–336.
Shekelle, P., Morton, S. C., & Keeler, E. B. (2006). Costs and benefits of health information technology. Rockville, MD: Agency for Healthcare Research and Quality.
Tabak, R. G., Khoong, E. C., Chambers, D., & Brownson, R. C. (2013). Models in dissemination and implementation research: Useful tools in public health services and systems research. Frontiers in Public Health Services and Systems Research, 2(1), 8.
Tullis, T., & Albert, B. (2013). Measuring the user experience: Collecting, analyzing, and presenting usability metrics (2nd ed.). Burlington, MA: Morgan Kaufmann.
Tullis, T. S., & Stetson, J. N. (2004). A comparison of questionnaires for assessing website usability. In Usability Professional Association Conference, Minneapolis, MN.
Turner, C. W., Lewis, J. R., & Nielsen, J. (2006). Determining usability test sample size. International Encyclopedia of Ergonomics and Human Factors, 3, 3084–3088.
Unützer, J., Katon, W., Williams, J. W., Jr, Callahan, C. M., Harpole, L., Hunkeler, E. M., & Langston, C. A. (2001). Improving primary care for depression in late life: the design of a multicenter randomized trial. Medical Care, 39(8), 785–799.
Unützer, J., Katon, W., Callahan, C. M., Williams, J. W., Jr, Hunkeler, E., Harpole, L., & Impact Investigators. (2002). Collaborative care management of late-life depression in the primary care setting: a randomized controlled trial. JAMA, 288(22), 2836–2845.
Unützer, J., Chan, Y. F., Hafer, E., Knaster, J., Shields, A., Powers, D., & Veith, R. C. (2012). Quality improvement with pay-for-performance incentives in integrated behavioral health care. American Journal of Public Health, 102(6), e41–e45.
U.S. Department of Health and Human Services, Office of the National Coordinator. (2014a). Health IT Glossary. Retrieved Oct 5, 2014, from http://www.healthit.gov/unintendedconsequences/content/glossary.html.
U.S. Department of Health and Human Services, Office of the National Coordinator. (2014b). About the Blue Button Initiative. Retrieved Oct 5, 2014, from http://www.healthit.gov/patients-families/blue-button/about-blue-button.
Vredenburg, K., Isensee, S., Righi, C., & Design, U. C. (2001). User centered design: An integrated approach. Englewood Cliffs: Prentice Hall.
Walker, J., Pan, E., Johnston, D., Alder-Milstein, J., Bates, D. W., & Middleton, B. (2005). The value of health care information exchange and interoperability. Health Affairs, W5, 10–18.
Walker, S. C., Kerns, S. E., Lyon, A. R., Bruns, E. J., & Cosgrove, T. J. (2010). Impact of school-based health center use on academic outcomes. Journal of Adolescent Health, 46(3), 251–257.
Williams, J. W., Katon, W., Lin, E. H., Nöel, P. H., Worchel, J., Cornell, J., & Unützer, J. (2004). The effectiveness of depression care management on diabetes-related outcomes in older patients. Annals of Internal Medicine, 140(12), 1015–1024.
Wolpert, M., Curtis-Tyler, K., & Edbrooke-Childs, J. A qualitative exploration of patient and clinician views on patient reported outcome measures in child mental health and diabetes services. Administration and Policy in Mental Health and Mental Health Services Research (in press).
Zhou, R. (2007). How to quantify user experience: fuzzy comprehensive evaluation model based on summative usability testing. In N. Aykin (Ed.), Usability and Internationalization. Global and local user interfaces (pp. 564–573). Heidelberg: Springer.
Acknowledgments
This publication was made possible in part by funding from Grant Number K08 MH095939, awarded to the first author from the National Institute of Mental Health (NIMH). The authors would also like to thank the school-based mental health provider participants, Seattle Children’s Hospital, and Public Health – Seattle and King County for their support of this project. Dr. Lyon is an investigator with the Implementation Research Institute (IRI), at the George Warren Brown School of Social Work, Washington University in St. Louis; through an award from the National Institute of Mental Health (R25 MH080916) and the Department of Veterans Affairs, Health Services Research & Development Service, Quality Enhancement Research Initiative (QUERI).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lyon, A.R., Wasse, J.K., Ludwig, K. et al. The Contextualized Technology Adaptation Process (CTAP): Optimizing Health Information Technology to Improve Mental Health Systems. Adm Policy Ment Health 43, 394–409 (2016). https://doi.org/10.1007/s10488-015-0637-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-015-0637-x