Detecting Signals of Disproportionate Reporting from Singapore’s Spontaneous Adverse Event Reporting System: An Application of the Sequential Probability Ratio Test

Abstract

Introduction

The ability to detect safety concerns from spontaneous adverse drug reaction reports in a timely and efficient manner remains important in public health.

Objective

This paper explores the behaviour of the Sequential Probability Ratio Test (SPRT) and ability to detect signals of disproportionate reporting (SDRs) in the Singapore context.

Methods

We used SPRT with a combination of two hypothesised relative risks (hRRs) of 2 and 4.1 to detect signals of both common and rare adverse events in our small database. We compared SPRT with other methods in terms of number of signals detected and whether labelled adverse drug reactions were detected or the reaction terms were considered serious. The other methods used were reporting odds ratio (ROR), Bayesian Confidence Propagation Neural Network (BCPNN) and Gamma Poisson Shrinker (GPS).

Results

The SPRT produced 2187 signals in common with all methods, 268 unique signals, and 70 signals in common with at least one other method, and did not produce signals in 178 cases where two other methods detected them, and there were 403 signals unique to one of the other methods. In terms of sensitivity, ROR performed better than other methods, but the SPRT method found more new signals. The performances of the methods were similar for negative predictive value and specificity.

Conclusions

Using a combination of hRRs for SPRT could be a useful screening tool for regulatory agencies, and more detailed investigation of the medical utility of the system is merited.

Appendix: Statistical Details of the Sequential Probability Ratio Test (SPRT)

1.1 Introduction

The SPRT was developed by A. Wald in the 1940s [14, 15] and was mainly used in process monitoring. A review of the pharmacovigilance literature showed that the SPRT may have useful properties not shared by other methods [16,17,18]. It has been used in the context of scanning electronic health records, particularly for vaccine studies, but has not had extensive evaluation for spontaneous reports. Specifically, SPRT compares two hypotheses based on the likelihood of observing the data given those hypotheses [19]. Unlike the other methods, SPRT is based on the difference between (rather than the ratio of) the observed and expected values. However, there is limited research conducted on SPRT to evaluate its performance in a Spontaneous Reports System (SRS) database similar to Singapore’s. A previous evaluation of SPRT used a single alternative hypothesis on simulated data [20]. On theoretical grounds, at least, the methods that do not allow for this form of multiple testing will have a higher rate of false positive findings—false signals. The method is essentially applied to a 2 × 2 table of data in the same way as other methods and where the emphasis is on obtaining an expected count assuming no association between the drug and the adverse event (AE). The method is applied to a single drug and adverse event and a determination of whether there is evidence of an association is made. The process is repeated, doing the same calculations for the next possible pairing and so on, for every drug–AE combination that occurs in the database.

1.2 The Methods of Use for SPRT in Signal Detection

For any drug–AE pair, the analysis of quantitative data mining methods can be based on 2 × 2 contingency tables (Table 4). We focus on one particular drug–AE pair and include the totals of reports of other drugs and AEs.

Table 4 2 x 2 contingency tables of AE and drugs

The number of reports listing drug and the AE are the observed values, and the expected value for each drug–AE pair is calculated (just as for a chi-square test), assuming independence, by the formula:

$$ E = \frac{{((a + c){ \times }(a + b))}}{(a + b + c + d)} $$

(1)

We assume the continuously accumulating data in SRS is described by the Poisson distribution [17] which is widely used in pharmacoepidemiology and pharmacovigilance.

The SPRT uses the log likelihood ratio (LLR) for the Poisson distribution calculated [19] by the formula:

$$ {\text{LLR}} = O \times \log_{e} ({\text{hRR}}) - \, E \times ({\text{hRR}} - 1), $$

(2)

where for any drug–AE pair,

• O is the observed number of reports, which is equal to cell a in Table 4.

• hRR is the hypothesised relative risk of interest to constitute a signal.

• E is the expected count as calculated at (1).

This is a very simple calculation which uses the observed and expected counts multiplied by factors related to the hypothesised relative risk.

1.3 Varying Thresholds in Signal Detection

A and B are two thresholds in the SPRT method that determine, in this context, whether a signal has been detected.

The signalling rules for SPRT are:

• If LLR > B, then reject H ₀ and accept H ₁.

• If LLR < A, then reject H ₁ and accept H ₀.

In the conventional use of SPRT in process control one can stop monitoring if there is convincing evidence that there is no effect. However, in the case of post marketing surveillance, monitoring continues indefinitely, as signals can still occur subsequently due to various factors such as change in pattern of drug use and prescribing, so the threshold A is irrelevant in this context.

The threshold is calculated using α and β; α (type I error), the probability of incorrect rejection of a true null hypothesis, also called a “false positive”. β (type II error) is the probability of accepting a false null hypothesis, also called a “false negative”. 1 − β gives the power of the test.

The values A and B are obtained using the formulae:

$$ A = \log_{e} \{ \beta /(1{-}\alpha )\} $$

(3)

$$ B = \log_{e} \{ (1{-}\beta )/\alpha \} . $$

(4)

To minimize the false positives, the value of α should be made small. To lessen the false negatives, the value of β should also be made small. We have used conventional α and β values at 0.05 and 0.20, respectively. This gives the approximate threshold value to be 2.77 for B. Different threshold values based on different choices of α and β can also be derived (see Table 5).

Table 5 Different threshold values based on different choices of α and β

1.4 Observed and Expected Counts and their Effect on LLR

In order to understand how SPRT works, it is helpful to see what happens to the LLR when its parameters change. A graph of LLR is shown in Fig. 2 with varying values of hRR (in the range of 1–6) with a fixed observed count of 3 and an expected count of 1. This graph shows that the most likely value of hRR, (its maximum LLR) is at hRR of 3, and the likelihood of other values of hRR fall off the further hRR is from 3.

Fig. 2

Graph of LLR with varying values of hRR for fixed O = 3 and E = 1. LLR log likelihood ratio, O observed number of reports, E expected values for each drug–AE pair

1.5 Observed and Expected Counts and the Relation to hRR

In order to detect a signal, with a threshold value of 2.77 (i.e. α at 0.05 and β values at 0.20) and expected count of 1, using an alternative hypothesis hRR of 2 the minimum observed count is 6 (see Fig. 3). If the hRR is increased to 3, the LLR will cross the threshold when the observed count is greater than or equal to 5.

Fig. 3

LLR vs observed values for fixed hRR = 2 and E = 1. LLR log likelihood ratio, O observed number of reports, hRR hypothesised relative risk (risk of interest/relative risk in the alternative), E expected values for each drug–AE pair. Threshold = 2.77

If we reduce the expected value to zero (though we cannot ever have exactly zero in practice, but it can be very close to zero), then we see that the LLR formula in (2) would approximately be equal to LLR = O × log _e (hRR). Hence, we can make LLR arbitrarily large with any observed count (and an extremely small expected count), including a low one, cross the threshold by having a sufficiently high value of the alternative hypothesis for hRR. Although this at first seems counter-intuitive, a very high hRR becomes more likely than the null even with a low observed count with an extremely small expected count. This is because the SPRT is a test comparing two simple hypotheses. Table 6 gives the hRR for the alternative hypothesis with a corresponding minimum observed count needed to cross a threshold of 2.77. It was observed that when SPRT was applied with a low hRR of 2 as the alternative hypothesis to our SRS database, it resulted in low numbers of SDRs. This could be explained as more than 90% of the drug–AE pairs in our SRS had less than 10 reports (see Fig. 4).

Table 6 Varying values of hRR with corresponding qualifying SDRs

Fig. 4

Distribution of number of reports received for drug–AE pairs (1993–2013)

In all databases of spontaneous reports, low cell counts occur very frequently (and zero counts are the most frequent of all as noted above), and this is a particular issue with a relatively small database in Singapore compared to the US FDA Adverse Event Reporting System or the European EudraVigilance database. On the other hand, if the alternative hypothesis for hRR is sufficiently high then signals can be obtained even with a single observed count. However, this means that with higher observed and expected values, but with lower values of hRR, then signals become more difficult to detect.

1.6 Use of Two Hypothesised Values

Initially SPRT with hRR = 2 was applied as has been used by others in references cited above. However, rare events with relatively small values of the observed and small values of the expected count, did not produce signals. Figure 5 shows that the numbers of reports rise when the value of hRR reaches 4.1. This is because the minimum value for the observed count to be a signal with an expected close to zero, becomes 2 and so potentially many more cells have a count of 2 than a count of 3. Rarer AEs, such that although they only occur twice, but have a very small expected value and hence were not detected as a signal using SPRT with hRR = 2 could be signalled more easily. Figure 6 shows the overlap of significant pairs detected by SPRT with hRR = 2 and hRR = 4.1.

Fig. 5

Total number of SDRs with increasing hRR for data from 1993 to 2013. hRR hypothesised relative risk (risk of interest/relative risk in the alternative), SDR signals of disproportionate reporting, SPRT Sequential Probability Ratio Test

Fig. 6

Venn diagrams showing overlap of SDRs detected by SPRT with hRR = 2 and hRR = 4.1 (1993–2013). hRR hypothesised relative risk (risk of interest/relative risk in the alternative), SDR signals of disproportionate reporting, SPRT Sequential Probability Ratio Test

Signals with a notable excess based on a low number of reports with very low expected values are of medical interest as well as those with larger numbers of reports, so we decided to include all the SDRs generated by SPRT with hRR = 2 or with hRR = 4.1, if either crossed the threshold of 2.77.

All these calculations are repeated using each drug–AE combination that exists. The calculations are quite simple; the complexity is in doing them for every combination and classifying each pair as giving rise to a signal or not.

1.7 Comparison of SPRT with Other Methods

In our study, three methods in common use i.e. reporting odds ratio (ROR), Bayesian Confidence Propagation Neural Network (BCPNN) and Gamma Poisson Shrinker (GPS) were compared with SPRT. Virtually all those using these methods use a criterion based on a minimum count of drug–AE pairs (N ≥ 3) so this was also imposed here. The reason is that the analyses of ROR and GPS methods with small cell counts with one or two reports can result in statistical instability associated with disproportionalities. If the criterion of N ≥ 3 was not applied, ROR and GPS methods would generate many SDRs which are false positives. When we did not specify N ≥ 3 for GPS, the method signalled several drug–AE pairs with two counts. As explained in the study by Ang et al. [21], using a stricter threshold criterion for these methods will not give a better performance. Table 7 gives the formulae and thresholds for all the four methods compared in this study. Table 8 shows the formulae for calculating the sensitivity, specificity, positive predictive value and negative predictive value.

Table 7 Formulas and thresholds of methods used

Table 8 Concepts of sensitivity, specificity, positive predictive value, and negative predictive value