Evaluation of VHP Program
A thorough and methodical evaluation of any program is imperative in determining its effectiveness. Prior to this chapter, a multitude of evidence-based practices and the importance of their application in promoting and ensuring vessel health and preservation have been described. The question is, how do we promote and ensure adherence to these sound practices and guidelines that when implemented will improve patient clinical outcomes and patient experience? How do we best implement these practice changes? And, what is the “right” way to evaluate components of a vessel health and preservation program? The objective of this chapter is to describe implementation strategies and to explore strategies for evaluation of a VHP program.
KeywordsEvaluation Audit and feedback Implementing change Audit and feedback strategies
The last quadrant of the VHP model includes processes to evaluate the success of your implementation, identify gaps and weaknesses, consider evaluation of new products to meet a need, and focus on patient and clinician satisfaction measurements. Improvement of care is impossible without an established process of evaluation. Right evaluation for VHP program application includes outcome measurement of complications, observation of policy performance, and plan to provide education for staff departments and units with negative outcomes or practice deficiencies. A multimodal quality program applies education of guidelines and recommendations ensuring that practices are consistent, and that staff is well informed. Adding new products should involve an established process for evaluation and trialing to ensure performance at the company stated levels. Integrated within a VHP program is the evaluation of products, supplies, and technology, both existing and considering new product trial testing. Each product should undergo periodic assessment to determine performance according to the facility needs and expected application. Each of these areas of evaluation requires measurement criteria, periodic checks for application, and compliance, all to ensure a successful and high-quality program.
According to the VHP model, the right evaluation must include audits and outcome monitoring which lead to identification of educational needs and use of products that can contribute to improved outcomes (Moureau and Carr 2018). Outcomes include complication rates, observation of compliance to organizational policies and procedures, and evaluation of clinician competencies. Examples of specific outcome measurements include catheter-related bloodstream infection rates, venous thrombosis rates in patients with PICCs, peripheral IV-related complications, and catheter occlusion rates. Process outcomes that might be evaluated in a VHP program include appropriate use of PICCs versus midline catheters, site care, and dressings performed in accordance with policy, intactness of VAD dressings, prompt removal of VADs when no longer necessary, and compliance with adhering to the components of the central line insertion checklist.
Outcomes are not just measured but are used to validate performance or to identify needed areas for quality improvement interventions. Audit and feedback are a widely used and recommended strategy in implementation and evaluation of practice changes. It is defined as a summary of clinical performance that may include recommendations for action; information is gathered over a specified period of time and used to increase group awareness of theirs and/or others’ practice (RNAO 2012; Ivers et al. 2012). Audit and feedback are usually used in conjunction with other interventions, namely, educational strategies. As stated by Ray-Barruel (2017), ongoing audits are used to ensure compliance with care and will provide the evidence required to ensure that an organization is on track to excellence in VAD care. Clearly, without attention to program evaluation, deficiencies and patient outcomes would not be identified, and opportunities for performance improvement including necessary clinician education would not be identified.
20.2 Implementation and Quality Improvement Strategies
Organizational: May include changes in professional roles or skill mix changes, for example, champion leaders or implementation of a specialty team.
- Regulatory: For example, government required healthcare initiatives.
Example: Requirements for infection or other outcome reporting.
- Financial: Constraints of reimbursement based on clinical outcomes or patient satisfaction. This is a strong motivator for implementing practice changes!
Example: In the United States, central line-associated bloodstream infection (CLABSI) rates in hospitals are publicly reported on the Medicare.gov website (“hospital compare”). Hospitals face penalties for high CLABSI rates, and any vascular access device-related BSI is considered a preventable event with treatment not reimbursable under medicare. The patient experience which includes satisfaction measures also factors into reimbursement.
Professional interventions: The focus of this chapter may include educational strategies, the use of local opinion leaders, and the use of audit and feedback and reminders.
Engage: Includes multidisciplinary teams that include frontline and senior leadership in the process and outcome improvement plan, identification of local champions (e.g., vascular access/infusion clinicians), and focus on a culture of safety.
Educate: The use of a variety of educational methods and strategies.
Execute: Use quality improvement methodologies (e.g., plan-do-study-act) to structure prevention efforts, to standardize care processes (e.g. guidelines, bundles), and to create “redundancy” such as with visual reminders, checklists, posters, screen saver messages, etc.
Evaluate: Include audits of both process (e.g. compliance with insertion bundle) and outcome (CLABSI rate) measures and link to initial and ongoing competency assessments and the critical step of providing feedback to all healthcare staff. Examples include compliance with insertion checklists with graphs showing cumulative compliance with process measures.
In the Infusion Therapy Standards of Practice, the Infusion Nurses Society addresses the importance of quality improvement programs including monitoring of adverse outcomes including infection rates, reporting such data regularly to both clinicians and leadership, and minimizing and eliminating barriers to change and improvement (Gorski et al. 2016). The Standards for Infusion Therapy from the Royal College of Nursing (2016) include similar recommendations and specifically address the need for auditing as an ongoing process in order to monitor, maintain, and improve clinical practice in infusion therapy and the need for timely dissemination of audit results to develop a culture of learning and quality improvement. The epic3 guidelines (Loveday et al. 2014) recommend the use of quality improvement interventions to support and manage VADs including the use of audit and feedback of compliance with practice guidelines.
Potentially negative attitudes by clinical staff, often related to organizational problems or issues such as inadequate staffing, high patient acuity levels, and organizational changes.
Lack of interest or lack of knowledge which may also be impacted by many competing clinical priorities (e.g., reduce incidence of falls, pressure ulcers, nonvascular-associated infections).
Guidelines or practice changes are not integrated or “hard-wired” into organization practices or policies/procedures (RNAO 2012).
It is important to acknowledge that the presence of policies does not result in adherence to best practices. In a survey of 975 US hospitals enrolled in the National Health and Safety Network, data were provided on the presence of policies in 1534 ICU units. There was a widespread presence of policies relative to CLABSI prevention (87–97%); however adherence to such policies ranged from 37 to 71% (Stone et al. 2014).
Another potential barrier to implementing evidence-based practices is how and in what way do key persons assess and champion the evidence. If the evidence is believed to be weak, implementation of a practice is not likely. The perceived strength of evidence relative to evidence-based infection prevention practices was studied in a survey of infection prevention personnel (Saint et al. 2013). Relative to CLABSI prevention, 90% or more survey respondents perceived the following evidence as strong: skin antisepsis with chlorhexidine, maximal sterile barrier precautions, and avoiding the femoral site. The researchers recommend that when translating evidence into practice, consideration should be given as to how key individuals assess the strength of evidence.
Conduction of an organizational survey may be a helpful strategy for identifying facilitators and barriers. Once understood, it is important to build upon identified facilitators and examine ways to tackle the barriers. Examples include addressing the educational needs and the level/type of education needed, providing adequate time for implementation, and exploring motivational strategies.
20.3 Exploration and Effective Use of Audit and Feedback Strategies
As stated earlier, audit and feedback are a widely used and recommended strategy in implementation of practice changes. A systematic review of literature was performed to explore features of educational interventions that led to competence in aseptic central line insertion and care in acute care settings (Cherry et al. 2010). Based upon a review of 47 studies that met the researchers’ inclusion criteria, educational interventions were most effective and more likely to result in outcome improvement when used in conjunction with audit and feedback as well as availability of the clinical supplies consistent with the education. Education alone is not enough to improve outcomes and clinical practice!
Baseline performance is low to begin with.
It is provided more than once.
It is provided by a supervisor or colleague.
It is provided verbally as well as in writing.
There are clear goals/targets and an action plan.
In a follow-up analysis to the Cochrane review, Ivers et al. (2014) state that audit and feedback are one of the most studied quality improvement interventions. While again asserting that audit and feedback lead to small but potentially important practice changes, they state that “the appropriate question is not: ‘can audit and feedback improve professional practice?’ but: ‘how the effect of audit and feedback interventions can be optimized?’” (Ivers et al. 2014). Citing evidence that repeated feedback is more effective, the researchers opine that studies continue to evaluate interventions after only one cycle of feedback. There remains the need to identify the key, active ingredients of audit and feedback that will lead to a greater impact of interventions, and produce more generalizable outcomes.
Using a modified grounded theory approach involving 72 clinicians, including nurses, in ICU settings, the use of audit and feedback was perceived as fragmented and variable, not transparent, and provided information that was neither timely nor “actionable” (Sinuff et al. 2015). To improve the process, the researchers suggest the following: providing rationale for practice changes and for the audit process, incorporating feedback into daily activities such as rounds, developing audit and feedback criteria that are specific and transparent, and providing information that can be translated into specific actions. Furthermore, the feedback must be timely. For example, finding out that there were two-line infections in the previous month gives little information if the patient is not identified and what were potential breaches in expected practice. Participants in the study feedback should provide specific actions “which can be acted upon more immediately” (Sinuff et al. 2015: 396).
Relevance: The need to understand the purpose of audit and feedback and the link between the audit criteria and patient outcomes.
Timing and feedback: Some nurses felt that the audit process occurred too early during the shift, and they were not able to complete the expected work in such limited time.
Individual factors: A perception of criticism resulted in greater tendency to respond in a more negative manner.
Summarizing implications for nurse managers, the researchers suggest making the audit and feedback process relevant and transparent in its purpose, involving nurses in planning and developing the process and providing feedback with clearly evident goals and action plans to make feedback less personal (Christina et al. 2016).
Strategies for improving the audit and feedback process
The audit process is transparent and relevant and includes:
• Rationale for practice changes
• Rationale for the audit process
• There is a link between audit criteria and patient outcomes
• Provide both written and verbal feedback
• Information obtained can be translated into clear goals and action plans
• Feedback is provided more than once
20.4 Examples: Successful Integration of Audit and Feedback into Practice
A quality improvement project implemented in a two-site hospital system involved the use of nurse-specific report cards providing feedback on central line management (Morrison et al. 2017). Using both visual and documentation audits, the authors sought to understand if known risk factors (e.g., maintaining dry intact dressing, maintaining IV set integrity) associated with central line-associated bloodstream infection (CLABSI) could be decreased. Using feedback intervention theory, information that is timely, detail focused, based on goals, and aimed at changing behavior and improving performance was provided. Nurses were provided a four-part computer-based educational module, and a trial of an alcohol disinfection cap was initiated before the audits. During the auditing period, additional interventions included house-wide implementation of the alcohol disinfection cap, updating of policies and procedures, and redesigned central line dressing/implanted port needle insertion trays. The weekly visual audit consisted of observation of dressings (e.g., dry and intact, chlorhexidine-impregnated dressings, dressings dated) and administration sets (e.g., labeling, use of disinfection caps, no looping) and also a documentation review (e.g., site assessment and care, catheter patency, dressing change date corresponds to date on dressing).
Over the 16-week period, 487 nurses received a “report card,” which included 620 central lines; 113 failed the visual audit with 54% having at least one CLABSI contributing factor. Over time, there was a decrease in CLABSI contributing factors which the authors believed resulted from the personalized report cards. The nurses’ responses to the report cards were briefly discussed and included justification of failures (not my fault), excuses (float nurse), and dismay. Outcome data included a decrease in CLABSIs; statistical significance of this outcome was not reported. Based upon the evaluation of this quality improvement study data, additional organizational changes were made including a face-to-face line care module for newly hired nurses, mandatory annual CLABSI prevention education, immediate intervention to correct failures at the time of the audit, and redesign of the electronic medical record IV documentation (Morrison et al. 2017).
In another example, a reduction in CLABSIs was reported over a 5-year period. Daily rounds of all central lines in an ICU setting revealed a substantial number of occluded lines as well as inadequate clinical practices including unlabeled dressings, improperly placed chlorhexidine foam dressings, and breaches in aseptic technique (Matocha 2013). Outcome data was used to support practice changes, product changes, and education and competency training.
In an emergency department, placement and care of peripheral IV catheters (PIV) was addressed using audit and feedback (Fakih et al. 2012). A quasi-experimental pre- and post-study methodology was used. Pre-study data included 10–20 observations of PIV placement per week, evaluation of documentation, and about 10 intravenous medication administrations observed per week. Steps observed include hand hygiene, skin antisepsis including air drying, maintaining aseptic technique during placement, and application of a sterile dressing. Implementation included education with a pre- and posttest as well as the same observations done pre-study. Two hundred twenty completed PIV placements were observed during the study period. Very poor compliance and lack of knowledge was evident relative to proper insertion technique pre-implementation (4.8%) and improved over time (30.9% during implementation, 31.7% post-implementation).
While the researchers found that audit and feedback were associated with a statistically significant improvement in documentation and compliance with PIV procedural steps, it is notable that overall compliance with the PIV insertion steps only improved to a little over 30%. When breaking down compliance into the five steps, hand hygiene prior to patient contact was about 20% pre-implementation and remained under 40% during implementation and post-implementation. Compliance was much greater, approaching or greater than 80% with the other PIV steps.
In a study of compliance with a transfusion bundle, monthly team audit and feedback were compared to monthly team audit and feedback plus individual feedback (Borgert et al. 2016). Compliance was significantly improved when the individual nurse was provided timely feedback in addition to the monthly team audit and feedback.
Implementing and sustaining evidence-based practice changes is challenging but is imperative in providing the best care and outcomes for the patients we serve. Increasingly, organizations are facing regulatory and reimbursement requirements and constraints that compel changes in clinical practice. Key interventions steps in implementation include involvement of a team that includes leadership, local champions, and clinical experts, an understanding of both positive characteristics of an organization that support practice change and barriers to change, and employment of sound quality improvement strategies. It is important to realize that education and well-written policies and procedures, while important and certainly needed as part of any implementation plan, are not enough.
What next steps could be taken in promoting the updated peripheral IV catheter policy?
How might adherence to the new policy changes be measured?
How should the measurement process be presented to the staff nurses?
How might the changes be addressed in a daily practice?
Summary of Key Points
When planning to implement changes in practice, put together a team that includes leadership, local champions, and clinical experts.
Explore and understand organizational characteristics that will support practice changes as well as any barriers to change.
Staff education and policies/procedures are necessary in implementation of practice changes, but they are not enough.
The right evaluation of a vessel health and preservation program must include audit and feedback that includes both process and outcome measures.
Audit and feedback must be provided in a timely manner, provided more than once, provided by a supervisor or colleague both verbally and in writing.
Clear goals/targets and an action plan should be part of the audit and feedback process.
The audit and feedback process should be transparent to and understood by clinicians.
- Flodgren G, Conterno LO, Mayhew A, Omar O, Pereira CR, Shepperd S. Interventions to improve professional adherence to guidelines for prevention of device-related infections. Cochrane Database Syst Rev. 2013;(3):CD006559. https://doi.org/10.1002/14651858.CD006559.pub2.
- Gorski LA, Hadaway L, Hagle M, McGoldrick M, Orr M, Doellman D. 2016 infusion therapy standards of practice. J Infus Nurs. 2016;39(1S):S1–S159.Google Scholar
- Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O’Brien MA, Johansen M, Grimshaw J, Oxman AD. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;(6):CD000259. https://doi.org/10.1002/14651858.CD000259.pub3.
- Ivers NM, Grimshaw JM, Jamtvedt G, Flottorp S, O’Brien MA, French SD, Young J, Odgaard-Jensen J. Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care. J Gen Intern Med. 2014;29(11):1534–41.CrossRefGoogle Scholar
- Registered Nurses of Ontario. Toolkit: implementation of best practice guidelines. 2nd ed. Toronto, ON: Registered Nurses of Ontario; 2012.Google Scholar
- Royal College of Nursing. Standards for infusion therapy. 4th ed. London: Royal College of Nursing; 2016.Google Scholar
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.