FormalPara Key Points

We present a framework for the selection of drugs and outcomes based on the study design of interest; to optimise the performance of real-world data signal detection studies.

This framework aims to address bias relating to drugs and outcomes not being suited to the study design of choice.

We applied this framework to build a reference set for a study aiming to assess the performance of the self-controlled case series with active comparators.

1 Introduction

Epidemiological methods are differentially valid depending on the nature of the drug and outcome [1]. Although this is well understood in traditional epidemiology, it has not fully been considered in real-world data signal detection studies, especially at the design stage. While it is clearly impractical to conduct bespoke analyses for every individual drug-outcome pair in a transparent auditable manner [2], many signal detection studies have been conducted with minimal consideration of the drug-event pairs to be analysed. Authors implemented a generic approach in the same way for all the drug-event or vaccine-event pairs, leading to suboptimal performances of signal detection methods, with high numbers of false-positive and false-negative findings [3]. In recent years there have been multiple efforts to standardise and harmonise elements of study design and reporting through the introduction of frameworks and other guidance.

The US Food and Drug Administration’s Mini-Sentinel Taxonomy Work Group initiated the approach of considering a systematic framework for tailoring the study design choice to the specific characteristics of drug-outcome pairs in hypothesis-testing studies using real-world data (RWD). Their work holds relevance for signal detection, although adaptations would be needed to include specificities of signal detection. The aim of the Mini-Sentinel Taxonomy Work Group was “To characterize analytic methods suitable for signal refinement and to provide clarity and practical advice for choosing the appropriate signal refinement methodology for the Mini-Sentinel System” [4]. They proposed a framework for how, based on attributes of the drug of interest (DOI) and health outcomes of interest (HOI), one could and should decide between a between- and a within-person comparison method. Another independent sentinel component developed a structured decision table for method selection in vaccine signal detection [5]. Similar criteria to the Mini-Sentinel study were used, but some were removed because only vaccines were considered, and they provided more granularity on methods with specific choices. Although this was developed for vaccine signal detection, this is also relevant to drug safety signal detection with some adjustments, to account for, for example, the common difference between healthy and diseased indicated populations and duration of use.

The Mini-Sentinel study findings translate to the impact of the choice of epidemiological methods on signal detection performance, and specifically the ability to anticipate where there will be suboptimal signal detection performance for a specific method given the nature of the exposure and outcome. This could be considered in two ways, either (1) picking the DOIs and HOIs first (drug/outcome-based approach), similarly to the Mini-Sentinel work, or (2) choosing a method and selecting the DOIs and HOIs based on the characteristics of that method (method-based approach).

Unless a prospective open-ended evaluation approach is taken and signals are reviewed as they are identified [6], the most common approach to the evaluation of signal detection is retrospectively examining method performance to some external benchmark reference set [7, 8]. One of the problems with assessing signal detection methods in RWD-based analyses is that most studies to date have used broad reference sets containing a wide range of exposures and outcomes for evaluation; irrespective of the characteristics of the method they were trying to evaluate. Ensuring a priori-defined appropriate references sets (i.e. a list of drug-outcome pairs classified as positive or negative controls) of DOIs and HOIs will enable more effective and accurate method assessments. This should therefore account for the nature of the DOIs and HOIs and other aspects that affect the performance of the method. More broadly, ensuring the choice of DOIs and HOIs is suitable for the chosen study design can increase performances of signal detection methods in large datasets.

There is therefore a clear need to develop a framework to enable consistent and transparent approaches in the choice of DOIs and HOIs for signal detection studies, which is not only for signal detection studies to investigate drug safety but can also be applied to the development of reference sets for signal detection method evaluation. The framework presented here will be applicable to signal detection studies where a list of predefined outcomes of potential interest has been decided. This could be done for example, by combining lists of general important adverse reactions in pharmacovigilance [9], known adverse reactions for other drugs within the same class and pharmacologically plausible adverse reactions. For more open-ended signal detection without a priori events of interest, other solutions would be needed to obtain a list of outcomes, but that is not the focus of this paper.

2 Purpose

In this study, we aim to build a systematic framework for the selection of appropriate DOIs and HOIs for signal detection studies based on a study design of interest. As a motivating example, we illustrate how the framework is applied to build a reference set for a study aiming to assess the performance of self-controlled designs with active comparators.

3 Methods

3.1 Presentation of the General Framework

Our framework is based on the characteristics presented in the published studies presented above [5, 10]. Key characteristics for design choice were: (1) the strength of confounding (both between person and within person); (2) the circumstances that could lead to misclassification of: the exposure or the timing of the HOI; and (3) the sustained or acute nature of the exposure. The final report presented a scoring system to evaluate whether a within-person comparison or a between-person comparison would be more appropriate, depending on the DOI, the HOI and other characteristics of the study [10]. The full list is presented in the Electronic Supplementary Material (ESM). We reviewed all the criteria and assessed their relevance for signal detection. For example, in a signal detection study, the number of comparator groups or the relationship between frequency of DOI use and incidence of the HOI are unknown. Building on their list of criteria, we added other characteristics, such as the public health importance, that were specific to signal detection but were not considered in existing studies.

We propose a method-based approach to choose suitable exposures and outcomes for a specific method of interest, recognising that a single method is not optimal for all exposures and outcomes. This approach enables a single study design to be applied for several types of drugs and outcomes if they share similar characteristics, as well as allows an assessment of performance of the chosen design. It also explicitly involves stating which exposures and outcomes are not suitable for the chosen design.

Our general framework proposition is presented in Fig. 1. Starting from a method of interest, it enables the choice of appropriate drugs and outcomes to conduct a signal detection study with this method. It is based on:

  • The application of study design requirements, which includes:

    1. o

      transient or sustained nature of the exposure;

    2. o

      long- or short-term nature of the outcome: length of the risk window;

    3. o

      immediate- or delayed-risk window;

    4. o

      susceptibility to within- and between-person confounding.

  • The database(s) of interest.

  • The clinical importance of the outcome(s).

Fig. 1
figure 1

Summary of the general framework. 1Depending on the coding system (ICD-10, Read Codes), similar clinical terms for outcomes could be grouped

Full size image

3.2 Application of the Framework: A Self-Controlled Study with Active Comparators

We anticipate that the framework should be both method and applied example agnostic. We set out to examine the application of the framework for an exemplar, to develop a reference set for a study aiming to assess the performance of the self-controlled case series (SCCS) for signal detection. However, the general principles we propose are equally applicable to both fully exploratory signal detection and method performance assessment.

We chose for this initial assessment self-controlled designs, as they were found to be the highest-performing methods for signal detection among other designs (e.g. cohort studies, disproportionality analyses) in a recent literature review [3]. Specifically, we selected for testing the SCCS, which is a case-only design comparing the event rate during exposed and unexposed time within the same individual [11]. One previous limitation with this method was the lack of tools to deal with time-varying confounding, such as the use of active comparators, but this has been recently implemented [12] for epidemiological studies and this could potentially improve their signal detection performance further. We therefore decided to use SCCS with active comparators in this example.

The main database of interest is the Clinical Research Practice Datalink (CPRD) Aurum, which contains records from general practitioners in the UK. The study will be replicated in the Systeme National des Donnees de Sante, a nationwide public claims database from France, and in IBM Marketscan, a private claims database from the USA.

Using the framework, we aimed to select drugs and outcomes for which the SCCS is expected to work well. This is to be able to assess the performance of this method using the reference set developed here.

3.3 Selection Process for the Case Study

We adapted our proposed general framework presented above as follows for the case study, taking into account the assumptions of the SCCS and the presence of active comparators (Fig. 2).

Fig. 2
figure 2

Overview of the selection process

Full size image

4 Results

Initially, 273 outcomes were found on the drugs’ labels and considered for inclusion. After applying our proposed framework to the case study, 28 outcomes were selected in the reference set. A flowchart of inclusion is presented in Fig. 3. The list of outcomes excluded at each step can be found in the Appendix in the ESM. In total, 104 positive controls (individual drug-outcome pairs) and 58 negative controls were included in the reference set.

Fig. 3
figure 3

Application of the framework for the selection of the reference set in the study. CPRD Clinical Research Practice Datalink, SCCS self-controlled case series

Full size image

The positive control outcomes included in the reference set are listed below by organ class. The list of the negative control outcomes is the same with the addition of hip fracture and gastrointestinal bleeding, as explained in the methods section. Table 1 illustrates the full list of the positive and negative control outcomes for ciprofloxacin and clarithromycin as examples for fluoroquinolones and macrolides, respectively.

  • Infections and infestations: pneumonia and cellulitis

  • Blood and lymphatic system disorders: thrombocythemia and pancytopenia

  • Immune system disorder: anaphylactic shock

  • Psychiatric disorders: delirium

  • Nervous system disorders: intracranial hypertension, amnesia, vertigo, gait disturbance and peripheral neuropathy

  • Eye disorders: transient vision loss and uveitis

  • Ear and labyrinth disorders: tinnitus and hearing loss

  • Cardiac disorders: syncope, atrial fibrillation and arrhythmias

  • Vascular disorders: vasculitis and phlebitis

  • Gastrointestinal disorders: pancreatitis and dysphagia

  • Skin and subcutaneous tissue disorders: dermatitis and petechiae

  • Musculoskeletal connective tissue and bone disorders: tendon rupture and tendinitis

  • Renal and urinary disorders: renal failure

  • General disorders and administration-site conditions: oedema

Table 1 List of positive and negative control outcomes for ciprofloxacin and clarithromycin
Full size table

5 Discussion

There is a need to take into account characteristics of the study design to choose DOIs and HOIs to optimise the performance of drug safety signal detection studies. This can be done through the application of a taxonomy framework. The aim of this work was to describe such a framework, to outline the main general principles considered and to apply these for the selection of a reference set in a self-controlled signal detection study. This work was based on previous studies but tailored to our specific method-based approach and to the specificities of signal detection. The general principles can be applied to any method but specific implementation needs to be adapted depending on the chosen design.

Previous work from the Sentinel project, despite not being directly related to drug safety signal detection, provides a good basis. We used Sentinel principles as the foundation for our framework, by reviewing their criteria and including only those relevant to signal detection. We added an assessment of the suitability of the DOIs and HOIs in the database of interest, as it is necessary to make sure the study is feasible or to choose a database accordingly.

While there is a clear need to change the way reference sets are developed for a method evaluation based on the exposure and outcomes of interest as well as the data source, these must be clearly defined a priori. Explanation of the choice of reference set should be consistent and transparent, in line with the renewed focus on enhanced replicability and transparency in real-world evidence generation from healthcare databases [22].

5.1 Application of the Framework for the Selection of a Reference Set

Several criteria from the Sentinel project were reused either directly or indirectly regarding the study requirements: the exposure use pattern, the onset and duration of exposure risk window, and the degree of misclassification through the coverage in the database of interest. We did not consider the strength of within- and between-person confounding because in our study time-invariant confounding is handled by the self-controlled design, and part of the time-variant confounding is handled by the short observation period, and the use of active comparators. If between-person comparative methods are chosen, these would be important to be considered when selecting the DOIs and HOIs. By applying this taxonomy framework, we are able to utilise a single study design to investigate a very broad list of preselected HOIs, covering a range of organ classes and different levels of seriousness.

The Mini-Sentinel work recommended that when assumptions are met, self-controlled designs should be used in priority because within-person confounding is handled [23]. If assumptions are not met, an alternative design should be chosen. Our work provided an example of checking SCCS assumptions and an extension to accommodate active comparators.

5.2 Challenges in Implementing the Framework

Although we attempted to present the choice of criteria in a systematic approach, some criteria still relied on clinical opinion and human judgement to be implemented, introducing some degree of subjectivity in the process, which was highlighted when applying the framework to the case study. Importantly, our proposed framework encourages the systematic documentation of the decision-making process, which could lead to high transparency of signal detection-related research studies.

In the application of the framework for the reference set, the original number of adverse events in the label investigated was 273, but the final list contains only 28 outcomes after application of the framework. However, it must be recognised that not all adverse events listed in a product label are of equal public health importance, and that many outcomes were excluded from our reference set for reasons such as their mild self-limiting nature. Here, we have selected those that the assessors considered to be of highest public health importance, as well as those that are most suited to the constraints of the study. The reference set we obtained contains HOIs and DOIs appropriately covered in the CPRD. We have also checked that the final list of drugs and outcomes was also well covered in the other databases (Systeme National des Donnees de Sante and MarketScan) used in the study. This will enable the use of the same reference set in several nationwide databases, leading to multi-database comparisons of the results of this study.

There is a potential misclassification and imbalance between the number of positive and negative controls [24] in the reference set obtained. Indeed, labels do not necessarily represent true causal associations, and an outcome absent from the label could still be associated with a DOI if the association is unknown. We identified a much larger number of positive controls than negative controls, which was because of the strict criteria we chose to ensure the quality of the negative controls. More broadly, we recognise that routine signal detection within a RWD database should be considered one of many tools in the broader signal detection armoury, with unique strengths and limitations.

5.3 Adaptation of This Framework

This framework can be implemented at a broader scale in signal detection studies. Our work was method based (SCCS with active comparators as an example) but it is also possible to adapt a similar framework to drug- or outcome-based approaches, as well as to study designs other than SCCS. Depending on the drug(s) and/or outcome(s) of interest, their characteristics and coverage in the available database, one can choose the appropriate design and type of analysis. To apply our framework, a list of pre-specified outcomes of interest is needed. We selected the outcomes of interest based on drug labels, which is not necessarily useful when designing a non-performance assessment signal detection study. Further, there is no reason why this framework could not be applied to vaccine signal detection as well. Where a range of different drugs and outcomes are of interest that do not share characteristics amenable to a single study design, several study designs may need to be considered to optimise the potential for signal detection using routinely collected health data.

6 Conclusions

Here, we propose a framework for the optimal selection of HOIs and DOIs in signal detection studies given a chosen study design and have applied it to an example using the SCCS design with the novel active comparator method to assess its performance. This framework could also be used for investigating drug safety only, including other drug or vaccine outcome pairs, as well as for other study designs.

This framework will enable the evaluation of optimal method performance by removing outcomes from the investigation pool that are not suited to the method or database of choice, which we believe will promote better decision making about the choice of potential signal detection methods. This framework will be increasingly useful as signal detection in RWD becomes more prevalent and clarity on performance, in addition to issues like transparency, will be critical for trusted routine use. In future work, wide reference sets could be developed that one could easily use to pull the HOIs and DOIs of relevance for a given RWD signal detection study.