Evaluating performance of electronic healthcare records and spontaneous reporting data in drug safety signal detection
- 692 Downloads
Background Electronic reporting and processing of suspected adverse drug reactions (ADRs) is increasing and has facilitated automated screening procedures. It is crucial for healthcare professionals to understand the nature and proper use of data available in pharmacovigilance practice. Objectives To (a) compare performance of EU-ADR [electronic healthcare record (EHR) exemplar] and FAERS [spontaneous reporting system (SRS) exemplar] databases in detecting signals using “positive” and “negative” drug-event reference sets; and (b) evaluate the impact of timing bias on sensitivity thresholds by comparing all data to data restricted to the time before a warning/regulatory action. Methods Ten events with known positive and negative reference sets were selected. Signals were identified when respective statistics exceeded defined thresholds. Main outcome measure Performance metrics, including sensitivity, specificity, positive predictive value and accuracy were calculated. In addition, the effect of regulatory action on the performance of signal detection in each data source was evaluated. Results The sensitivity for detecting signals in EHR data varied depending on the nature of the adverse events and increased substantially if the analyses were restricted to the period preceding the first regulatory action. Across all events, using data from all years, a sensitivity of 45–73 % was observed for EU-ADR and 77 % for FAERS. The specificity was high and similar for EU-ADR (82–96 %) and FAERS (98 %). EU-ADR data showed range of PPV (78–91 %) and accuracy (78–72 %) and FAERS data yielded a PPV of 97 % with 88 % accuracy. Conclusion Using all cumulative data, signal detection in SRS data achieved higher specificity and sensitivity than EHR data. However, when data were restricted to time prior to a regulatory action, performance characteristics changed in a manner consistent with both the type of data and nature of the ADR. Further research focusing on prospective validation of is necessary to learn more about the performance and utility of these databases in modern pharmacovigilance practice.
KeywordsAdverse drug reactions Data mining Electronic health records EU-ADR FAERS Performance metrics Pharmacovigilance Signal detection Spontaneous reporting systems
The authors would like to thank the EU-ADR consortium and the EU-ADR database owners for their contribution and support, particularly Rosa Gini (Agenzia Regionale Sanità Toscana, Florence, Italy), Ron Herings (PHARMO Institute, Utrecht, Netherlands), Giampiero Mazzaglia (Società Italiana Medicina Generale, Florence, Italy), Gino Picelli (Pedianet, Padova, Italy), Carla Fornari (Department of Statistics and Quantitative Methods, Università Milano-Bicocca, Milan, Italy), and Lars Pedersen (Aarhus University Hospital, Aarhus, Denmark) and Colette Saccomanno for editorial assistance.
This project has received funding from the European Commission’s Seventh Framework Programme (FP7/2007-2013) under Grant Agreement No 215847. The funding agency has not been involved in the collection of data, the analysis or interpretation of the data, or the decision to submit.
Conflicts of interest
The authors do not declare any conflict of interest.
- 1.Department of Health and Human Services (US), Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research. Guidance for Industry: premarketing Risk Assessment; March 2005.Google Scholar
- 2.International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use. Guideline for Good Clinical Practice E6(R1); June 1996.Google Scholar
- 3.Department of Health and Human Services (US), Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research. Guidance for Industry: E6 Good Clinical Practice: consolidated Guidance; April 1996.Google Scholar
- 4.Council for International Organizations of Medical Sciences. Management of Safety Information from Clinical Trials: report of CIOMS Working Group VI; CIOMS; April 2005.Google Scholar
- 5.Department of Health and Human Services (US), Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research. Good Pharmacovigilance Practices and Pharmacoepidemiologic Assessment; March 2005.Google Scholar
- 6.European Medicines Agency. Volume 9A: guidelines on Pharmacovigilance for Medicinal Products for Human Use. In: The Rules governing medicinal products in the European Union: London (UK); September 2008.Google Scholar
- 7.Council for International Organizations of Medical Sciences. Practical Aspects of Signal Detection in Pharmacovigilance: report of CIOMS Working Group VIII; CIOMS; September 2010.Google Scholar
- 19.FDAs Sentinel Initiative [Internet]. US Department of Health and Human Services, Food and Drug Administration; Jun 06 2014 [cited 17 Sept 2014]. http://www.fda.gov/Safety/FDAsSentinelinitiative/.
- 27.FDA Adverse Event Reporting System (FAERS) [Internet]. US Department of Health and Human Services, Food and Drug Administration; Sep 10 2012 [cited 9 May 2014]. http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Surveillance/AdverseDrugEffects.