Impact statements

  • Non-clinical ground rules disagree with risk prioritisation algorithms that are based on clinically important risk factors.

  • This comparison can be used to provide pharmacists with a better foundation on which to decide between risk prioritisation algorithms in a clinical setting.

  • Medicine Risk Score and a modified version of Assessment of Risk Tool had a moderate agreement in their risk categorisation.

Introduction

Medication reviews (MRs) aim to improve the quality, safety, and appropriate use of medications by finding and solving drug-related problems (DRP) [1,2,3,4]. Important risk factors for DRPs include polypharmacy, female gender, high risk medications, renally eliminated drugs and being elderly (most often 65 years of age and older) [2, 3]. Some of these risk factors are also used as criteria for patient prioritisation for MRs [5, 6].

At the joint emergency department (JED) at Nordsjællands Hospital in Denmark, pharmacists provide medication reviews (MRs) to hospitalised patients. Risk prioritisation algorithms could help find patients for whom the pharmacist's resources are best utilised [7, 8]. Several risk prioritisation algorithms have been developed, such as the Brighton Adverse Drug Reactions Risk model and the Medicines Optimisation Assessment Tool. These risk prioritisation algorithms include different items used to calculate the risk of DRP or measure other similar outcomes [9,10,11,12,13,14,15,16,17,18,19,20].

The JED at Nordsjællands Hospital has a high patient flow and only one pharmacist providing MRs on any given day, so it was essential to select the patients that would benefit the most from MRs [7, 21]. Currently, pharmacists at JED do not use a prioritisation algorithm but use non-clinical ground rules developed by JED. The non-clinical ground rules prioritise patients with the longest stay and who have not had their medications transferred to the electronic health record (EHR) [22]. Therefore, it was important to examine the current non-clinical ground rules and compare current practice with the risk categorisation of the prioritisation algorithms.

This study used the algorithm Medicine Risk Score (MERIS) developed in Denmark using literature search and the Delphi method and a modified version of Assessment of Risk Tool (ART) developed in New Zealand using a group consensus method. MERIS consists of three items: reduced renal function, number of drugs and number of drugs with different categories of risk of harm and risk of interaction. ART consists of 38 items divided into five categories: patient profile, patient encounter, clinical profile—patients with chronic diseases, high-risk medications and laboratory values. Both algorithms provide risk categories by adding points from the fulfilled items [23,24,25,26].

Aim

To examine how the selection of patients by the clinical pharmacists in an emergency department for MRs compared with the categorisation provided by MERIS and a modified version of ART (mART). Furthermore, examine the agreement between MERIS and mART.

Ethics approval

No ethical approval was required according to Danish law [27], however the pharmacists were anonymous and all material was stored confidentially. Data collection and processing were in compliance with the European General Data Protection.

Method

Study design

A cross-sectional study was conducted at the JED at Nordsjællands Hospital in Denmark. Patient data relevant for the algorithms were collected while observing the pharmacists working at the JED, who provided MRs daily from 8 a.m. to 5 p.m. All admitted patients 18 years of age and older were included the study, as well as patients who stayed overnight and patients admitted until approximately 4:45 p.m. Afterwards, patient data were entered into the risk prioritisation algorithms MERIS and the modified ART (mART). When ART was developed, it had 38 items [25]. However, some of these items were specific to the New Zealand health care setting or not suitable for a fully automatic algorithm, or relevant in the JED. Therefore, 12 items were removed to make the mART used in this study. Examples of items removed were: being from specific New Zealand native populations, poor English proficiency, history of psychiatric disease, and having specific laboratory values taken over several days. See electronic supplementary material 1 for the mART items.

Data collection

The first author (SGK), a student in her last semester prior to graduating as a pharmacist, followed six pharmacists working at JED. Each pharmacist was observed for three days from March 9th to May 6th, 2022, on weekdays and weekends. The pharmacists ranked their MRs as ‘not important’, ‘important’ or ‘very important’, based on their clinical insight. The pharmacists did not change their typical workday during the data collection, meaning that they continued to attend meetings and precept students. These days, they did not manage to prioritize and complete as many MRs as they might have otherwise. The overall study included both the patients who were actively deselected for MRs and those admitted patients whom the pharmacists did not have time to consider.

The algorithms Medicine Risk Score (MERIS) and a modified version of Assessment of Risk Tool (mART) were used. Besides the relevant information for the different items in MERIS and mART, information about the patients’ age and gender were collected. The patients’ medication lists were extracted manually from the EHR and entered into an Excel spreadsheet. All medication formulations were counted (i.e. tablets, cremes, eyedrops). Duplicate medications and combination medications were counted once per active ingredient. The medication lists were taken from the EHR and not from patient interviews. Unless otherwise stated it was assumed that every drug on the list was used by the patient.

Data analysis

A frequency analysis of the total mART risk scores made the cut-off values for the risk categories of the mART. As observed in the original ART development the 10%, 15%, and 75% cut-offs were used [25]. The mART cut-off values were based on a frequency analysis of the total ART risk scores. Patients with a risk score above 36 were defined as high-risk, 25–36 as medium-risk, and 24 or under were low-risk. In the original version of ART the cut-off values were: above 22 were defined as high-risk, 11–22 as medium-risk, and 10 or under were low-risk [25].

The mean ages of patients in the high versus those in the low-risk categories of the algorithms were compared as were mean risk scores of patients who had either been selected or deselected by the pharmacists. Proportions of females were compared in the algorithms' high versus low-risk categories. Finally, the pharmacists’ ratings of the importance of the MRs were compared to the risk categorisations provided by the algorithms.

The sensitivity and specificity of the MERIS and mART were calculated using the pharmacists’ selection and deselection as the gold standard. For mART the medium and high-risk categories were merged. A true positive and a true negative were when the algorithms and the pharmacists agreed and a false positive and a false negative when they disagreed. The sensitivity, specificity and Kappa were calculated twice, respectively including and excluding patients with missing laboratory values.

The pharmacists’ numbers of patients selected and deselected for MRs were compared among them and their risk categories against MERIS and mART. In addition, the ratings of importance were compared among the individual pharmacists.

T-tests were used for comparing means, Z-tests for proportions. The Kappa Statistics test assessed the agreement between the algorithms’ risk categorisation. Finally, Kruskal–Wallis H, Chi square and Monte Carlo significance tests were used for nominal variables. All statistical analyses were done in SPSS version 28.0.0.0.

Results

In total, 1133 patients were included in the analysis of MERIS and mART, where pharmacists actively selected or deselected from 450 patients. Approximately 70% of the patients were categorised as low-risk on both the MERIS and mART tools (Table 1).

Table 1 Distributions of patients included in the study in each risk category with the original ART, the modified ART, and MERIS (n = 1133)
Full size table

Statistically significant differences were observed in the mean age between the risk categories for MERIS and mART, where patients classified as high-risk had a higher mean age. Similarly, patients selected by pharmacists had a higher mean age than those deselected for MRs. No statistically significant gender differences were observed with this categorisation (Table 2).

Table 2 Characteristics of patients in the study according to risk categories in MERIS and the modified ART and according to whether they were selected or deselected for medication review by pharmacists
Full size table

No statistically significant differences were found between the risk scores of selected and deselected patients for MRs in the high-risk category of MERIS, indicating that the risk scores were similar. This also applied to the high-risk and medium-risk categories of mART. However, for the low-risk categories of MERIS and mART, statistically significant differences were seen between patients who were selected and deselected by pharmacists, where the highest risk score was observed in the selected patients (Table 3).

Table 3 Risk scores of the patients who were selected or deselected by pharmacists for medication review in each of the risk categories of MERIS and the modified ART (n = 450)
Full size table

A statistically significant difference was observed between the pharmacists’ rating of importance of providing MR to the patient and the categorization of patients by both algorithms. However, for both algorithms there was a larger number of ‘not important’ ratings by the pharmacists in the low-risk categories, indicating that there might be some agreement between the pharmacists and low-risk categorisation by the algorithms (Table 4).

Table 4 The pharmacists’ rating of the importance of patients receiving a medication review compared to the different risk categories of MERIS and the modified ART (n = 230)
Full size table

The sensitivity and specificity for MERIS were 37.8% and 73.6%, respectively when the patients with missing laboratory values were included. When these patients were removed the values were 37.9% and 72.0%. The sensitivity and specificity for mART were 33.0% and 75.9% respectively, and 33.3% and 74.0% when removing patients with missing laboratory values. The agreement of risk categorisation between MERIS and mART (grouping the high-risk and medium-risk groups of mART) was statistically significant (Kappa = 0.58), indicating a moderate agreement [28]. The same applied to patients when excluding patients with missing laboratory values (Table 5).

Table 5 Distribution of patients across the different risk categories comparing MERIS and the modified ART, both when patients with missing laboratory values were included and excluded
Full size table

When looking at the items of mART, three were not fulfilled by any of the patients. These were the following: a positive Clostridium difficile toxin culture, a therapeutic drug monitoring test, and an activated partial thromboplastin time of over 100 s. Generally, the items relating to laboratory were fulfilled by a low proportion of patients. The estimated glomerular filtration rate (eGFR) < 30 ml/min and P-glucose > 11 mmol/L were the items most frequently used (5.7% and 3.6% of all patients). The most frequently fulfilled mART items were: having more than 8 drugs and being over 75 years old (approximately 40% of the patients). In MERIS every item was fulfilled by at least one patient. The most frequently fulfilled items were drugs with low and medium risk of interaction and drugs with medium risk of harm (96.9% and 82.1%).

No statistically significant difference was observed between the individual pharmacists’ number of selected and deselected patients. However, Pharmacist 3 stood out as selecting the most patients for MRs and deselecting the fewest patients. A statistically significant difference between the individual pharmacists’ rating of importance of the provided MRs was observed. Pharmacist 1 had the largest proportion of ‘very important’ MRs, whereas Pharmacist 3 had the largest proportion of ‘not important’ MRs. No statistically significant difference between the individual pharmacist’s number of MRs in each risk category of the two algorithms. For most of the pharmacists there was a larger proportion of MRs in the low-risk category of both algorithms (Table 6).

Table 6 Characteristics of the individual pharmacists in the study according to selection or deselection of patients for medication reviews, rating of importance, and the risk categories of MERIS and ART (n = 230)
Full size table

Discussion

Statement of key findings and interpretation

The cut-off values of mART corresponded to the distribution of patients in MERIS. Since mART had fewer items than the original ART and the maximum score was lower, naturally the risk category cut-off values changed. The mART cut-off values were higher than those of the original algorithm, which could be due to this patient population having higher risk scores than the population used in the development of original ART.

A higher mean age in the high-risk categories is not surprising as high age is an item in mART. Higher age is associated with using more medications, which could explain why those in the high-risk category of MERIS also had a higher mean age, as MERIS includes number of drugs as an item, but not age [29]. The same difference was observed in the pharmacists’ selection, indicating that they may believe that older patients are more likely to have a DRP. No gender differences were observed in the risk categories, despite other studies showing that more women experienced polypharmacy [29,30,31].

Patients selected or deselected by pharmacists were at a similar risk of a DRP according to MERIS and mART. This indicates that the pharmacists disagreed with the risk categorisation provided by the algorithms. The non-clinical ground rules could also have influenced the selection and deselection, as the rules did not consider clinical risk factors [5]. It could also be a coincidence that the deselected patient had high risk scores since they were deselected for various reasons. However, in the low-risk categories of MERIS and ART significant differences were observed between the selected and deselected patients, where the selected had a higher risk score. This indicates that the pharmacists selected patients at higher risk within the low-risk category. The study also showed differences between the pharmacists’ rating of importance and the risk categorisation provided by the algorithms, indicating that they disagreed with the risk categorisation.

The sensitivity and specificity of MERIS and mART were similar, indicating an agreement in the risk categorisation by the algorithms. Previous studies found that the sensitivity and specificity of MERIS were 64 to 81% and 52 to 75% [23, 32]. The specificity of MERIS for this study was similar. However, the sensitivity was lower, which could be due to using the pharmacists as the gold standard as their non-clinical ground rules did not necessarily take the risk of DRP into account. No sensitivity and specificity values have been reported earlier for mART. However, the same issue of the pharmacists’ non-clinical ground rules could also influence the relatively low sensitivity found in this study. A moderate agreement of risk categorisation by MERIS and mART was found. With the distribution of patients across the risk categories, it seems probable that the algorithms would agree since approximately 30% were at high-risk, and 70% were at low-risk.

The laboratory items of the mART were the items most often missing from the EHR. It could be due to the study taking place at the JED, where the patients did not stay for long and few had blood work done. Including many laboratory items is probably not applicable in this setting since mART was not explicitly developed for emergency departments [25]. The most fulfilled items of mART match those from its validation study [26]. All the items of MERIS were fulfilled and easy to collect from the EHR compared to mART.

Differences were found between the individual pharmacists in the number of patients they each selected or deselected for MRs. Research indicates an association between clinical experience and how pharmacists prioritise patients, where the more experienced often take the reason for admission into account [22]. Clinical experience did not seem to be a factor in how pharmacists prioritized patients in this study, although pharmacists with the least clinical experience rated the largest proportion of patients as ‘very important’, indicating that they may generally feel more responsibility for providing MRs. All pharmacists rated a relatively large proportion of patients as ‘not important’, which again could be due to their non-clinical ground rules. Differences between pharmacists could be because they made decisions at different speeds or some did not necessarily follow the non-clinical ground rules, or due to alternating workdays with administrative or education tasks.

No difference was observed between the individual pharmacists’ number of MRs in the different risk categories of the algorithms. However, all the pharmacists had a larger number of MRs for patients in the low-risk categories, which could be due to their adhering to the non-clinical ground rules. This could also be due to the algorithms' categorisation of approximately 70% of patients in the low-risk category, which could explain why a higher proportion of low-risk patients were selected by pharmacists.

Strengths and limitations

One of the study’s strengths was that the data were collected on three alternate days for each pharmacist to get a larger data set with fewer day-to-day variations. To our knowledge a comparison of MERIS and ART has not been done before on the same patient population.

A limitation of the study was that the medication lists were based on the EHR, which was not always correct or updated. A recent Danish study has shown that 81% of the patients included had at least one discrepancy between EHR and what the patients reported taking at home [33]. If this applied to the patients in this study, it could bias the risk categorisation. This could have been avoided by basing the medication lists on patient interviews. Also, the pharmacists did not rate the same patients, so it was not possible to conclude strongly about their similarities or differences. An independent expert could have rated the patients for a more objective comparison. However, since the pharmacists were supposed to work according to the same non-clinical ground rules and provide a similar performance, it seemed reasonable to put them in the same group to get a more realistic picture. Conducting this study only in one ED is a limitation and needs to be considered if these results are transferred.

A further limitation was a potential observer effect, as the pharmacists knew that the observer was noting the number of patients selected and deselected for MRs. This could have influenced how the pharmacists worked by making them work faster and selecting a higher number of patients than they normally would. The observation could also have influenced which patients the pharmacists selected, although it was not possible to know in which way this would affect the results.

Further research

Future research should have the pharmacists use one of these risk algorithms to examine how it affects their prioritisation and the importance of the MRs they then provide. It could be interesting to further characterize the patients where the algorithms disagreed in the risk categorisation and delve into the disagreements between the pharmacists’ categorizations and risk algorithms. Further work should also examine if there could be an association between the risk categorisation by the algorithms and the reason for admission.

Conclusion

The study found that the pharmacists disagreed with the risk categorisation provided by MERIS and mART. This could be due to the pharmacists' non-clinical ground rules, which did not consider clinical risk factors when selecting patients for MRs. However, MERIS and mART were found to have similar sensitivity, specificity, and moderate agreement in their risk categorisation. Further research should focus on how clinical algorithms affect the selection of patients and the importance of the MRs carried out by pharmacists.