Background

The identification of adverse events (AE), defined by the Institute of Medicine as “an injury caused by medical management rather than the underlying condition of the patient” [1] has become an essential component of patient safety and clinical governance within healthcare [1, 2]. Emergency care has previously been identified as an area of significant concern regarding the prevalence of AEs [1]. However, the majority of this focus has been on the in-hospital setting, with little understanding of the identification and incidence of AEs in the prehospital environment. The potential for AEs to occur within emergency medical services (EMS) is significant, given that prehospital clinicians are required to make prompt decisions, with little to no supporting patient information, and implement care with limited resources, in a multitude of challenging environments.

A number of methods that have been employed to monitor for AEs in healthcare rely on either voluntary reporting systems, direct observation, complaints, mortality and morbidity review, or patient care documentation review [3,4,5,6]. While these methods may be effective for specific patients or defined high risk procedures, there is little evidence to suggest they provide a comprehensive or robust system of AE detection [3,4,5,6]. A novel approach known as the Trigger Tool (TT) methodology has been shown to be a more time-effective, cost-effective and sensitive means of identifying AEs when compared with more traditional methods [7,8,9].

The early development and testing of EMS specific triggers for the identification of AEs and Harm has been previously described [10]. The primary focus for these triggers was on the the low risk/high frequency case cohort of an EMS organisation. High risk/low frequency cases such as those that make up a key clinical care pathway (e.g. ST-elevation myocardial infarction); high-risk procedures (e.g., surgical airway); the administration of high-risk medication (e.g., paralytic agents); infrequent procedures (e.g., intraosseous cannulation) and high acuity inter-facility transport cases were excluded given that they are generally included as part of a 100% audit framework, as a result of their high potential for AEs and harm. The remaining low risk/high frequency cases, which generally make up the significant majority of EMS calls, are often neglected from audit, and as such, were the target population. In addition, the focus of audit in high risk cases is largely aimed at identifying individual occurrences of AEs and harm for case/risk management purposes, whereas the purpose of the EMS trigger tool is to act as a global measure of system performance and to facilitate general system improvement.

The aim of this study was to test the use of EMS triggers, using a sampling and review methodology, and to report on the incidence rates and types of AEs and harm they identify.

Methodology

The EMS triggers were previously developed over several derivation rounds and a single large sample analysis for performance assessment/test characteristics [10]. As a retrospective sampling strategy for the targeted identification of cases at risk for potential AEs and harm, the EMS triggers demonstrated both a superior sensitivity and specificity compared to a random sampling strategy for audit case identification (See Table 1 for EMS Trigger Items).

Table 1 Emergency medical services trigger tool items
Full size table

In order to operationalise the EMS triggers, we adapted the processes developed by the Institute for Healthcare Improvement (IHI) for use with the Global Trigger Tool (GTT) [11], to an EMS setting. The adapted EMS trigger tool (EMSTT) process was applied using a stepwise approach, outlined by Adler et al. in their implementation of the GTT [12], to the analysis of 36 consecutive samples of patient care records over an 18-month period (n = 710).

Setting

The study was conducted within the Hamad Medical Corporation Ambulance Service (HMCAS), the government-funded national ambulance service of Qatar. HMCAS is a two-tiered provider (Ambulance Paramedic and Critical Care Paramedic) that serves a population of approximately 2.6 million people with an average daily call rate of between 600 and 700 emergency and non-emergency calls. At the time of writing, HMCAS was transitioning from a paper-based care record system to an electronic patient care record.

Institutional review board

Ethical approval to conduct the study was granted by the Medical Research Centre of the Hamad Medical Corporation, Qatar.

Trigger tool methodology

Sampling

Consistent with the sampling strategy of the GTT, the EMSTT process utilises a sampling methodology of small consecutive samples over time. Cases considered for analysis included those where there was patient contact with EMS and for whom a Patient Care Record (PCR) was generated. Excluded PCRs included: those reviewed as part of a key clinical care pathway (e.g., ST-Elevation Myocardial Infarction); high-risk procedures (e.g., surgical airway); the administration of high-risk medication (e.g., paralytic agents); infrequent procedures (e.g., intraosseous cannulation); and inter-facility transport records. Of the remaining cases, two samples per month were selected by random number table for the 18-month test period (one sample for each half of the month). Each sample consisted of 30 PCRs, with the intent to review the first 20 cases, and the additional 10 PCRs included in case any of the first 20 met exclusion criteria (See Additional files 1 and 2). The sampling process was provided to a documentation clerk, who identified the PCRs meeting inclusion criteria, and supplied paper copies to the primary reviewers for review.

Review process

Each review round was conducted independently by two primary reviewers (IH, BP) during scheduled and protected review time. The primary reviewers were each operational Critical Care Paramedics employed by HMCAS, with > 15 years EMS experience each, who were additionally involved in clinical governance and education within the service. Individual record reviews were limited to 10 min each. Following each review round, the two primary reviewers met to compare findings, reach consensus, and summarize the results. In cases where consensus could not be reached, a third reviewer, the HMCAS Medical Director or Head of Professions (LAS/NC) were consulted to determine an outcome. Each review round was part of an iterative process and allowed for continued refinement of the EMSTT methodology and data collection processes (See Additional files 1 and 2).

Data collection process

Each record was manually reviewed for the presence of triggers only. If a trigger was found, the record was further reviewed for the occurrence of AEs and/or harm. Records that did not contain a trigger were not reviewed further. The National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) classification system was used to categorise AEs and harm for EMSTT positive cases. Given the in-hospital focus of the MERP system, Category C was modified to include cases where it was unclear if harm had occurred given the short duration of care provided by EMS (Table 2) [13]. Beginning with Sample 10 (Month 5) an additional, EMS specific classification system developed by Patterson et al. [14], the Adverse Event Severity Rating Index, was introduced to run concurrently with the NCC MERP system (Table 2). This allowed a fit for purpose EMS-specific system to be used for AE classification, as opposed to relying solely on the in-hospital focused NCC MERP.

Table 2 Harm classification systems
Full size table

Consecutive sample analysis

The EMSTT was applied over an 18-month period as part of a consecutive sample analysis using the operational processes reported above. Records for review were identified via the HMCAS centralised electronic database (Microsoft Access 2010, Redwood, WA) and cases meeting exclusion criteria were removed prior to sampling. All data was captured on a standardised data capture template and summary report template (Microsoft Excel 2010, Redwood, WA) for analysis and reporting.

Data analysis and reporting process

Inter-rater reliability was measured for each trigger item and level of classification overall. In addition, univariate descriptive analysis was conducted for all continuous and categorical variables (i.e.: trigger items and levels of harm classification). Lastly, a reporting standard was developed that included outcome measures for future potential benchmarking purposes and data visualisation methods for analysis and performance monitoring. The three outcome measures identified included: Triggers per 10,000 Patient Encounters, AEs per 10,000 Patient Encounters and Harm per 10,000 Patient Encounters. For the purpose of the reporting standard, a patient encounter was defined as: “An interaction between a patient and EMS healthcare provider(s) for the purpose of providing healthcare service(s) or assessing the health status of a patient.” All outcome measures were calculated and plotted on Statistical Process Control (SPC) Charts using Minitab Version 17 (2010, State College PA) and employed the Nelson Rules for detecting special cause variation [15] SPC is a branch of statistics that combines time series analysis methods with graphical presentation of data to understand variation and yield insight into whether the variation is due to chance or assignable causes [16]. The primary tool of SPC, the control chart, provides researchers and practitioners with a method of better understanding and communicating data from healthcare improvement efforts by enabling the study of variation and differentiation of special from common cause variation [17]. The data presented in this study meets all requirements associated with the use of U-charts [15].

Results

The EMSTT identified 194 individual triggers among the 720 sampled PCRs. There was a high level of inter-rater agreement across all items ranging from 85.6–100% depending on the trigger item (Table 3). Out of a total of 40 items analysed, 4 were observed to have poor reliability (κ < 0.41), 8 were observed to have moderate reliability (κ = 0.41–0.6), 3 substantial reliability (κ = 0.61–0.8), and 14 near perfect reliability (κ = 0.81–1) [18]. Six cases in total required referral to a third reviewer, all of which were from the Clinical group. All disagreements were primarily attributed to PCR illegibility.

Table 3 Inter-rater reliability
Full size table

Consistent with the results reported in the development of the EMSTT, the Clinical group made up the majority of triggers found in our analysis [n = 190 (97.9%)] (Table 4).Three triggers: Change in Systolic Blood Pressure Greater Than 20%; Temp > 38 °C without subsequent reduction; and SpO2 < 94% without supplemental Oxygen or SpO2 < 85% without assisted ventilation accounted for 92.8% (n = 180) of the triggers found throughout the analysis. Change in Systolic Blood Pressure Greater Than 20% was the most common trigger amongst the Clinical group, as well as overall [n = 108 (55.7%)].

Table 4 Results summary 1 – trigger items
Full size table

The three primary outcomes measures were calculated and plotted on U charts for the study period. An average rate of 8.20 Triggers per 10,000 Patient Encounters (Fig. 1), 2.48 AEs per 10,000 Patient Encounters (Fig. 2) and 0.34 Harm events per 10,000 Patient Encounters (Fig. 3) were found during the analysis. While one sample generated data outside the control limits in the Harm events per 10,000 Patient Encounters chart, no obvious attributable cause for this variation could be immediately identified.

Fig. 1
figure 1

Triggers per 10,000 Patient Encounters

Full size image
Fig. 2
figure 2

Adverse Events per 10,000 Patient Encounters

Full size image
Fig. 3
figure 3

Harm per 10,000 Patient Encounters

Full size image

When classifying harm using the NCC MERP Category A (Circumstances or Events that have the capacity to cause error) was the most common assigned class [n = 147 (20.7%)] (Table 5). In total, only three NCC MERP categories were assigned to all cases in which triggers were found: Category A, Category B (An error occurred but the error did not reach the patient) [n = 36 (5.1%)]; and Category C (An error occurred that reached the patient, but did not cause patient harm) [n = 11 (1.5%)], modified in this study to include cases in which it was unclear if harm had occurred due to the short duration of care provided by EMS. As with the development of the EMSTT, for reporting and analysis purposes, the latter classification was counted as positive for the occurrence of Harm.

Table 5 Results summary 2 – harm classification system 1
Full size table

When classifying harm using the AE Severity Rating Index, implemented with the fifth sample, Rating Code 7 (No AE identified) was the most common reported class in trigger positive cases [n = 67 (12%)] (Table 6). Similar to the NCC MERP system, three categories made up the majority of assigned categories in trigger positive cases, including: Code 7; Code 6 (AE with potential to cause harm with no fault) [n = 11 (2%)]; and Code 4 (AE with potential to cause harm as a result of commission) [n = 11 (2%)].

Table 6 Results summary 2 – harm classification system 2
Full size table

Limitations

Our study separated the definitions of AEs and harm into separate concepts, aligning the definition of AE more synonymously with error. However, there is a recent trend towards the definition of AE to be used interchangeably and associated more with the definition of harm. The reasons for this are two-fold. Firstly, error reporting has become less prominent, given its perceived “dilution” of the reporting on harm events. In addition, early results using the trigger tool methodology reported a relatively low conversion of error into harm events, reducing the emphasis on reporting the incidence of error, and focusing more on identifying actual cases involving patient harm [19]. Finally, harm can occur in the absence of an error in the delivery of a patient’s care. These events began to be referred to as AEs and were seen as synonymous with harm given their untoward outcome, despite the potential absence of a trigger or error in patient care [19].

In order to account for this recent change in the understanding of AEs, an additional measure was calculated to align with the evolving change in thinking towards these definitions - Modified AEs per 10,000 Patient Encounters values were calculated by adding AEs and harm, and the results plotted on a U chart for comparison. While this has the potential to skew the results observed for this study, the value of each of these modified measures remained relatively low, with 2.82 Modified Adverse Events per 10,000 Patient Encounters (Fig. 4) observed over the study period.

Fig. 4
figure 4

Modified Adverse Events per 10,000 Patient Encounters

Full size image

Consistent with the development of the EMS triggers, our study excluded several records during the sampling process, including evidence-based care pathways and high-risk/infrequent medications and procedures. Although these cases represent a small proportion of all EMS calls, the potential exists that their exclusion skewed the results of the analysis. The removal of these records was deliberate, given the aim of the tool for use in the high frequency/low risk cases often excluded from regular EMS audit programs. This limitation represents an avenue for further research and evaluation of the EMSTT in a sample where such exclusion criteria have not been applied.

Lastly, the aim of this research was limited to the development of implementation, data collection and reporting processes, and to test the EMSTT using these processes over time. As such, the study did not include tests for association or correlation between trigger items and rates of AEs and/or Harm. Similarly, this study did not include efforts aimed at improving or reducing the impact of the most common triggers. These remain as areas for future research.

Discussion

The TT methodology has seen considerable success as a quick, reliable and simple, yet comprehensive method of detecting AEs within healthcare. These desirable characteristics have led to the successful development of trigger tools within a variety of disciplines, including general healthcare [7], surgical care [20], primary care [21], intensive care [22], paediatric care [23] as well as pharmacy [24] and laboratory services [25]. However, integrating such a system into regular operation can be a complex task and should be viewed as equally important as the content of the tool itself [11, 12]. Despite the robust planning and processes put in place, several issues became apparent that hindered data collection for several months, requiring constant intervention and refinement to correct, and included:

  • Communication of the process and operational definitions emerged as a central component both internally within the project team, and externally with all departments the project could potentially impact.

  • Dedicated, protected time for each step within the process was essential to complete the required reviews. When the time required to complete each review round was specifically scheduled, review rounds were quicker and easier to complete.

  • The review team should be kept to a small, dedicated number of staff, to allow for consistency and familiarity of the tool and processes to develop over time. This proved to be beneficial from a resources and time investment vs. output point of view.

  • The importance of a contextually relevant and appropriate AE/Harm classification system became increasingly apparent throughout the analysis. As comprehensive as the NCC MERP system is, the nuances of the EMS environment made application of this system difficult. The fit for purpose AE Severity Index, while more subjective, provided a simple and easy to implement alternative that warrants further examination.

  • Documentation adequacy and legibility created difficulties in separating deficiencies in care from deficiencies in documentation. For such a system to be successful, it is essential that a review of documentation quality be carried out in parallel, or that a service have sufficient recourse to minimise the effect of poor documentation. The increased uptake of electronic medical records may limit the effects of some of these issues.

  • Given the above-mentioned period of transition towards an electronic PCR, the topic of automation was discussed amongst the project team on several occasions. As the project progressed, it became increasingly apparent that the iterative manual review of cases over time added to the learning experience in trying to understand the interaction of triggers to adverse events and harm. It additionally offered insight into future potential trigger items, knowledge that would have otherwise been lost had the process become automated.

This study was amongst the first to test and analyse a trigger tool and its operational processes specific to the EMS setting. The incidence of trigger items observed in this analysis echoed the observations reported in the development of the EMS triggers, with the significant majority of trigger positive cases comprising the same three triggers. The incidence of AEs and Harm identified was relatively low, including the modified measures, when compared to the development of the EMS triggers. However, these differences could potentially be attributed to differences in sampling, with the focus on small consecutive samples in this analysis, compared to the single large sample focus with the initial development. The focus for this study was modelled on the sampling approach used by the GTT, which has shown reliable predictability compared with larger samples [11]. Nonetheless, the greater ease and shorter duration of time required to assess EMS PCRs, using the EMSTT, remains an avenue for further study employing larger samples.

No other peer-reviewed research exists examining the application of a TT over time, within the EMS setting. As such, there is no benchmark with which to compare the results observed in this study. Despite this, it stands to reason that variation in service models found in EMS around the world could have the potential to impact the results of the application of the EMSTT in other settings. This represents a significant opportunity for future research, to better understand the prevalence and incidence of AEs within EMS. While many in-hospital TTs exist, variation in patient cohorts, area of service, length of exposure to the healthcare service studied and varying outcome measures make comparison with the results of this study impossible.

Conclusion

TTs represent a potential strategy towards the successful identification of the incidence of AE and harm within healthcare. With a better understanding of the case types and causes of AEs and Harm, targeted improvement projects can be designed and implemented to advance quality and patient safety. This study is amongst the first to test and analyse an EMS specific TT over time, in order to measure the rate of AE/Harm in this setting. With sufficient focus on implementation and data collection, and the inclusion of a contextually relevant classification system, the EMSTT represents a potentially successful strategy towards identifying the rate of AE/Harm within EMS.