Advancing Implementation Science Measurement

Abstract

The quality of implementation science and the efficiency of implementation practice are undermined by several measurement issues. First, few measures have evidence of fundamental psychometric properties, making it difficult to have confidence in study findings. Second, to accumulate knowledge, we need measures that can be used across studies; yet, the majority of measures are setting-, population-, or intervention-specific and are used only once. Third, there are no minimal reporting standards for measures, which contribute to the lack of psychometric evidence. Finally, only recently have measure developers begun to address pragmatic qualities of measures. This chapter will address each issue and make recommendations for how to address them in the service of advancing implementation science.

Glossary

Absorptive Capacity (pg. 232)

An organization’s ability to access and effectively use information (Emmons et al., 2012)

Adoption (pg. 233)

The complete or partial decision to proceed with the implementation of an innovation as a distinct process preceding but separate from actual implementation (Wisdom et al., 2014)

Barrier (pg. 237)

Factor that obstructs changes in targeted professional behaviors or healthcare delivery processes, such as the cost and complexity of the intervention (Krause et al., 2014)

Collaboration (pg. 230)

One of five core tenets of dissemination and implementation (D&I) research proposed by Glasgow et al. (2012), defined as the use of interdisciplinary research teams and research-practice collaborations (Glasgow et al., 2012)

Concurrent Validity (pg. 231)

The degree to which an instrument distinguishes groups it should theoretically distinguish. Concurrent validity is not demonstrated if there is no reasonable hypothesized difference among groups on the instrument (Weiner et al., 2008)

Construct Validity (pg. 232)

The degree to which inferences can legitimately be made from an instrument to the theoretical construct that it purportedly measures (Weiner et al., 2008)

Content Validity (pg. 231)

The ability of the selected items to reflect the variables of the construct in the measure (Zamanzadeh et al., 2015)

Convergent Validity (pg. 231)

The degree to which an instrument performs in a similar manner to other instruments that purportedly measure the same construct (e.g., two measures show a strong positive correlation). Convergent validity is most often assessed through confirmatory factor analysis (Weiner et al., 2008)

Criterion Validity (pg. 232)

An empirical check on the performance of an instrument against some criteria (Weiner et al., 2008)

Cumulative Knowledge (pg. 230)

One of five core tenets of dissemination & implementation (D&I) research proposed by Glasgow et al. (2012), defined as the need to create resources and funds of accessible knowledge about dissemination & implementation (D&I) research and its findings (Glasgow et al., 2012)

Discriminant Validity (pg. 231)

The degree to which an instrument performs in a different manner to other instruments that purportedly measure different constructs. Discriminant validity is most often assessed through confirmatory factor analysis (Weiner et al., 2008)

Efficiency (pg. 230)

One of five core tenets of dissemination & implementation (D&I) research proposed by Glasgow et al. (2012), defined as the use of efficient methods to study dissemination & implementation (D&I) (Glasgow et al., 2012)

Facilitator (pg. 237)

Factor that enables changes in targeted professional behaviors or healthcare delivery processes, such as the cost and complexity of the intervention (Krause et al., 2014)

Fidelity (pg. 228)

The competent and reliable delivery of an intervention as intended in the original design (Ibrahim & Sidani, 2015)

Homonymy (pg. 242)

Models that define the same construct in different ways (i.e., two discrepant definitions for the same term) (Martinez et al., 2014)

Implementation Outcomes (pg. 232)

Effects of deliberate and purposive actions to implement new treatments, practices, and services (Proctor et al., 2009)

Improved Capacity (pg. 230)

One of five core tenets of dissemination & implementation (D&I) research proposed by Glasgow et al. (2012), defined as a necessary increase in the capacity to train future dissemination & implementation (D&I) researchers and share advances made through implementation science with all stakeholders (Glasgow et al., 2012)

Intermediary (pg. 229)

Known as trainers, internal and external facilitators, implementation practitioners and purveyors. Intermediaries provide training and consultation and otherwise assist community settings to implement evidence-based practices (Lewis et al., 2016)

Internal Consistency (pg. 233)

Refers to whether several items that propose to measure the same general construct produce similar scores

Inter-rater Reliability (pg. 231)

A measure of reliability used to assess the degree to which different judges or raters agree in their assessment decisions (Reliability and Validity n.d.)

Leadership (pg. 229)

The ability of an individual to influence, motivate, and enable others to contribute toward the effectiveness and success of organizations of which they are members (House et al., 1999)

Managerial Relations (pg. 232)

Alliances between groups within an organization to promote change (Zuckerman et al., 1990)

Mixed Methods (pg. 228)

Focus on collecting, analyzing and merging both quantitative and qualitative data into one or more studies. The central premise of these designs is that the use of quantitative and qualitative approaches in combination provides a better understanding of research issues than either approach alone (Robins et al., 2008)

Norms (pg. 228)

The sample size, and the mean (M) and standard deviation (SD) for the instrument results (C. C. Lewis, Fischer, et al., 2015)

Organizational Climate (pg. 232)

Organizational members’ perceptions of their work environment (Emmons et al., 2012)

Organizational Readiness for Change (pg. 231)

The extent to which organizational members are psychologically and behaviorally prepared to implement organizational change (Weiner et al., 2008)

Penetration (pg. 228)

The integration of a practice within a service setting and its subsystems (Proctor et al., 2009)

Pragmatic Measure (pg. 228)

One that has relevance to stakeholders and is feasible to use in most real-world settings to assess progress (Glasgow & Riley, 2013)

Predictive Validity (pg. 232)

The degree to which the instrument can predict or correlate with an outcome of interest measured at some time in the future (Lewis, Fischer, et al., 2015)

Predictor (pg. 233)

Variable that may anticipate implementation effectiveness (Jacobs et al., 2015)

Proxy Measure (pg. 231)

Indirect measures of clinical practice, such as a review of medical records or interviewing the clinician (Hrisos et al., 2009)

Qualitative Methods (pg. 228)

Used to explore and obtain depth of understanding as to the reasons for success or failure to implement evidence-based practice or to identify strategies for facilitating implementation (Teddlie & Tashakkori, 2003)

Quantitative Methods (pg. 228)

Uused to test and confirm hypotheses based on an existing conceptual model and obtain breadth of understanding of predictors of successful implementation (Teddlie & Tashakkori, 2003)

Reliability (pg. 231)

The extent to which a measure (or its set of items) produces the same results (on repeated measures)

Responsiveness (pg. 233)

The ability of an instrument to detect clinically important changes in the construct it measures over time (Lewis, Fischer, et al., 2015)

Rigor and Relevance (pg. 230)

One of five core tenets of dissemination & implementation (D&I) research proposed by Glasgow et al. (2012), defined as the use of rigorous research methods that address critical questions in relevant contexts (Glasgow et al., 2012)

Social Network (pg. 233)

The pattern of relations and interactions that exist among people, organizations, communities, or other social systems (Valente, 1996)

Stakeholder (pg. 229)

An individual, group, or organization who may affect, be affected by, or perceive itself to be affected by a decision, activity, or outcome of a project, program, or portfolio

Structural Validity (pg. 233)

The degree to which all the test items rise and fall together; or to which, by contrast, perhaps, one set of test items rise and fall together in one pattern, and another group of test items rises and falls in a different pattern (Lewis, Fischer, et al., 2015)

Synonymy (pg. 242)

Models that define different constructs the same way (i.e., two unique terms assigned the same definition) (Martinez et al., 2014)

Test-retest Reliability (pg. 233)

A measure of reliability obtained by administering the same test twice over a period of time to a group of individuals. The scores from Time 1 and Time 2 can then be correlated in order to evaluate the test for stability over time (“Reliability and Validity,” n.d.)

Theory of Planned Behavior (pg. 237)

States that people’s behavior is determined by their intention to perform a given behavior (Casper, 2007)

Translation Validity (pg. 232)

The degree to which an instrument accurately translates (or carries) the meaning of the construct (Weiner et al., 2008)

Usability (pg. 233)

The ease of administration, which is calculated by the total number of items on the measure being rated (Lewis, Fischer, et al., 2015)

Validity (pg. 231)

The quality of the inferences, claims or decisions drawn from the scores of an instrument

Vision (pg. 232)

An idea of a valued outcome which represents a higher order goal and a motivating force at work (Farr & West, 1990)