Implementing Implementation: Integrating the Measurement of Implementation and Effectiveness in Complex Service Systems

  • Wei Wu TanEmail author
  • Colleen Jeffreys
  • Arno Parolini


Implementation science is concerned with the effective deployment and sustainment of evidence-based practices in service delivery systems, with the ultimate goal of providing services to bring about optimal outcomes for clients. As the field of implementation science matures and research moves towards sustainment of implementation strategies within complex dynamic systems of care, the integration of implementation research into real-life practice settings will become increasingly important. Such an integrated approach relies on the ability to measure implementation success over time and to learn about causal mechanisms of implementation in service delivery systems. However, despite a continuing emphasis on the importance of high-quality data to support successful implementation efforts, data collection for implementation practice remains an under-researched frontier of implementation science. To this end, this chapter describes a causal approach of implementation research predicated on recognising implementation as a system component, with interventions and their implementation forming integral parts of dynamic systems of care with multiple stakeholders. This approach is facilitated by a 5-step implementation research framework, with a high-quality data system that integrates research with operational components to enable a holistic view of stakeholder incentives in what we denote as the Implementation Space. By considering the full implementation space from the beginning, data can be purposefully collected, stored and used. Such data systems will enable a process of learning by supporting a cascading and dynamic model of continuous quality improvement and practice optimisation through the Plan-Do-Study-Act (PDSA) cycle across all domains of the implementation space.


Implementation Evidence-based practice Causal mechanisms Evaluation Measurements Health care systems Systems of care Data system Service delivery data model Continuous quality improvement 


  1. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23. Scholar
  2. Bollen, K. A., & Pearl, J. (2013). Eight myths about causality and structural equation models. In S. L. Morgan (Ed.), Handbook of causal analysis for social research (pp. 301–328). Dordrecht: Springer Netherlands. Scholar
  3. Chambers, D. (2012). Foreword. In R. C. Brownson, G. A. Colditz, & E. K. Proctor (Eds.), Dissemination and implementation research in health: Translating science to practice (pp. vii–vix). Oxford: Oxford University Press.Google Scholar
  4. Chambers, D., Wilson, P., Thompson, C., Harden, M., Coiera, E. (2012) Social Network Analysis in Healthcare Settings: A Systematic Scoping Review. PLoS ONE 7(8):e41911Google Scholar
  5. Chaudoir, S. R., Dugan, A. G., & Barr, C. H. (2013). Measuring factors affecting implementation of health innovations: A systematic review of structural, organizational, provider, patient, and innovation level measures. Implementation Science, 8(1), 22. Scholar
  6. Cook, T. J., & Dobson, L. D. (1982). Reaction to reexamination: More on type III error in program evaluation. Evaluation and Program Planning, 5(2), 119–121. Scholar
  7. Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012). Effectiveness-implementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care, 50(3), 217–226. Scholar
  8. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50. Scholar
  9. Deming, W. E. (1950). Elementary principles of the statistical control of quality: A series of lectures. Nippon Kagaku Gijutsu Remmei. Retrieved from
  10. Deming, W. E. (1986). Out of the crisis. Cambridge, MA: Cambridge University Press. Retrieved from Scholar
  11. Dobson, D., & Cook, T. J. (1980). Avoiding type III error in program evaluation: Results from a field experiment. Evaluation and Program Planning, 3(4), 269–276. Scholar
  12. Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2014). Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS). Implementation Science, 9(1), 157. Scholar
  13. Fernandez, M. E., Walker, T. J., Weiner, B. J., Calo, W. A., Liang, S., Risendal, B., … Kegler, M. C. (2018). Developing measures to assess constructs from the inner setting domain of the consolidated framework for implementation research. Implementation Science, 13(1), 52. Scholar
  14. Heckman, J. J., & Pinto, R. (2015). Causal analysis after Haavelmo. Econometric Theory, 31(1), 115–151. Scholar
  15. Heckman, J. J., & Vytlacil, E. (2005). Structural equations, treatment effects, and econometric policy evaluation. Econometrica, 73(3), 669–738. Scholar
  16. Heckman, J. J., & Vytlacil, E. J. (2007). Chapter 70 Econometric evaluation of social programs, part I: Causal models, structural models and econometric policy evaluation. In J. J. Heckman & E. E. Leamer (Eds.), Handbook of econometrics, Vol. 6 (pp. 4779–4874). North Holland: Elsevier. Scholar
  17. Holmes, B.J., Finegood, D.T., Riley, B.L., Best, A. (2014). Systems thinking in dissemination and implementation research. In R. C. Brownson, G. A. Colditz, & E. K. Proctor (Eds.), Dissemination and Implementation Research in Health: Translating Science to Practice (pp. 175–91). New York; Oxford: Oxford University Press.Google Scholar
  18. Kimball, R., Ross, M., Thornthwaite, W., Mundy, J., & Becker, B. (2008). In Tom (Ed.), The data warehouse lifecycle toolkit: Practical techniques for building data warehouse and business intelligence systems (2nd ed.). Indianapolis: Wiley.Google Scholar
  19. Krcmar, H. (2005). Informations management (4th ed.). Berlin, Germany: Springer.Google Scholar
  20. Langley, G. J., Nolan, K. M., & Nolan, T. W. (1994). The Foundation of Improvement. Quality Progress, 27(6), 81–86.Google Scholar
  21. Lewis, C. C., Stanick, C. F., Martinez, R. G., Weiner, B. J., Kim, M., Barwick, M., & Comtois, K. A. (2015). The society for implementation research collaboration instrument review project: A methodology to promote rigorous evaluation. Implementation Science, 10(1), 2. Scholar
  22. Maglio, P.P., & Mabry, P.L. (2011). Agent-based models and systems science approaches to public health. Am J Prev Med, 40(3):392–394.Google Scholar
  23. Maier, R. (2007). Knowledge management systems: Information and communication technologies for knowledge management (3rd ed.). Berlin, Germany: Springer.Google Scholar
  24. Moen, R. D., & Norman, C. L. (2010). Circling back. Quality Progress, 11, 20.Google Scholar
  25. Mas-Colell, A., Whinston, M. D., & Green, J. R. (1995). Microeconomic theory. New York: Oxford University Press.Google Scholar
  26. Parolini A., Tan W.W., Shlonsky A. (2019). Decision-based models of the implementation of interventions in systems of healthcare: Implementation outcomes and intervention effectiveness in complex service environments. PLOS ONE 14(10): e0223129. Scholar
  27. Pearl, J. (2009). Causality: Models, reasoning, and inference (2nd ed.). New York: Cambridge University Press.CrossRefGoogle Scholar
  28. Pipino, L. L., Lee, Y. W., & Wang, R. Y. (2002). Data quality assessment. Communications of the ACM, 45(4).
  29. Proctor, E. (2014). Dissemination and Implementation Research. In Encyclopedia of social work. Oxford University Press. Retrieved from
  30. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., … Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. Scholar
  31. Rezmovic, E. L. (1982). Program implementation and evaluation results: A reexamination of type III error in a field experiment. Evaluation and Program Planning, 5(2), 111–118. Scholar
  32. Roberts, M. S. (2015) Dynamic Simulation in Health Care Comes of Age. Value in Health 18 (2):143–144Google Scholar
  33. Skrondal, A., & Rabe-Hesketh, S. (2004). Generalized latent variable modeling: Multilevel, longitudinal, and structural equation models. Boca Raton, FL: Chapman & Hall/CRC.CrossRefGoogle Scholar
  34. Sterman, J. D. (1994). Learning in and about complex systems. System Dynamics Review, 10(2–3), 291–330. Scholar
  35. Sterman, J. D. (2006). Learning from evidence in a complex world. American Journal of Public Health, 96(3), 505–514. Scholar
  36. Tabak, R. G., Khoong, E. C., Chambers, D. A., & Brownson, R. C. (2012). Bridging research and practice. Models for dissemination and implementation research. American Journal of Preventive Medicine, 43, 337–350.CrossRefGoogle Scholar
  37. Wand, Y., & Wang, R. Y. (1996). Anchoring data quality dimensions in ontological foundations. Communications of the ACM, 39(11), 86–95. Scholar
  38. Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems, 12(4), 5–33. Scholar
  39. White, H., & Chalak, K. (2013). Identification and identification failure for treatment effects using structural systems. Econometric Reviews, 32(3), 273–317. Scholar
  40. White, H., & Lu, X. (2011). Causal diagrams for treatment effect estimation with application to efficient covariate selection. The Review of Economics and Statistics, 93(4), 1453–1459. Scholar
  41. Wulczyn, F., Clinch, R., Coulton, C., Keller, S., Moore, J., Muschkin, C., … Barghaus, K. (2017). Establishing a standard data model for large-scale IDS use (actionable intelligence for social policy, expert panel report). Actionable intelligence for social policy, University of Pennsylvania. Retrieved from

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Wei Wu Tan
    • 1
    • 2
    Email author
  • Colleen Jeffreys
    • 1
  • Arno Parolini
    • 1
    • 2
  1. 1.Department of Social WorkThe University of MelbourneMelbourneAustralia
  2. 2.California Child Welfare Indicators Project, School of Social WelfareUniversity of California at BerkeleyCaliforniaUSA

Personalised recommendations