Criteria Revision and Performance Comparison of Three Methods of Signal Detection Applied to the Spontaneous Reporting Database of a Pharmaceutical Manufacturer
- 178 Downloads
Background and objective: Several statistical methods exist for detecting signals of potential adverse drug reactions in spontaneous reporting databases. However, these signal-detection methods were developed using regulatory databases, which contain a far larger number of adverse event reports than the databases maintained by individual pharmaceutical manufacturers. Furthermore, the composition and quality of the spontaneous reporting databases differ between regulatory agencies and pharmaceutical companies. Thus, the signal-detection criteria proposed for regulatory use are considered to be inappropriate for pharmaceutical industry use without modification. The objective of this study was to revise the criteria for signal detection to make them suitable for use by pharmaceutical manufacturers.
Methods: A model comprising 40 drugs and 1000 adverse events was constructed based on a spontaneous reporting database provided by a pharmaceutical company and used in a simulation to investigate appropriate criteria for signal detection. In total, 1000 pseudo datasets were generated with this model, and three statistical methods (proportional reporting ratio [PRR], Bayesian Confidence Propagation Neural Network [BCPNN] and multi-item gamma Poisson shrinker [MGPS]) for signal detection were applied to each dataset. The sensitivity and specificity of each method were evaluated using these pseudo datasets. The optimum critical value for signal detection (i.e. the value that achieved the highest sensitivity with 95% specificity) was identified for each method. The optimum values were also examined with the adverse events classified into two categories according to frequency. The three original detection methods and their revised versions were applied to a real pharmaceutical company database to detect 173 known adverse reactions of four drugs.
Results:The 1000 pseudo datasets consisted of an average of 81 862 reports and 11 407 drug-event pairs, including 1192 adverse drug reactions. The sensitivities of PRR, BCPNN and MGPS methods were 49%, 45% and 26%, respectively, whereas their specificities were 95%, 99.6% and 99.99%, respectively; these sensitivities were unacceptably low for pharmaceutical manufacturers, whereas the specificities were acceptable. The highest sensitivity for each method, obtained by changing critical values and maintaining specificity at 95%, was 44%, 62% and 62%, respectively. When adverse events were classified into two categories, sensitivities as high as 75% for regular events and 39% for rare events were achieved with the revised BCPNN method. The critical values of the information component minus two standard deviations (IC —2SD) index of the revised BCPNN method were greater than −0.7 for regular events and greater than −0.6 for rare events. The revised BCPNN method yielded 51% sensitivity and 89% specificity for the real dataset.
Conclusion:A lower critical value may be needed when signal-detection methodology is applied to the spontaneous reporting databases of pharmaceutical manufacturers. For example, it is recommended that pharmaceutical manufacturers use the BCPNN method with IC —2SD criteria of greater than −0.7 for regular events and greater than −0.6 for rare events.
KeywordsAdverse Drug Reaction Receiver Operating Characteristic Curve Pharmaceutical Company Rare Event Real Dataset
The authors are grateful to Daiichi Sankyo Co., Ltd., for allowing access to its database. We would like to thank the two anonymous reviewers whose comments greatly improved this article.
Drs Chikuma Hamada and Isao Yoshimura have no conflicts of interest directly relevant to the content of this study. Yasuyuki Matsushita, Yasufumi Kuroda, Shinpei Niwa and Satoshi Sonehara are all employees of Daiichi Sankyo Co. Ltd.
No sources of funding were used to assist in the preparation of this study.
- 1.ICH-E2D-Expert Working Group. Post-approval safety data management: definitions and standards for expedited reporting [online]. Available from URL: http://www.ich.org/LOB/media/MEDIA631.pdf [Accessed 2006 Jul 28]
- 10.Follmann M, Michel A, Geyer C. Comparison of different methods for signal detection in the drug-safety database of a pharmaceutical company. In: the 20th International Conference of Pharmacoepidemiology (ICPE) & Therapeutic Risk Management; 2004 Aug 22–4; Bordeaux. Pharmacoepidemiol Drug Saf 2004; 13: S111CrossRefGoogle Scholar
- 13.DuMouchel W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. Am Statist 1999; 53(3): 177–202Google Scholar
- 14.DuMouchel W, Pregibon D. Empirical bayes screening for multi-item associations. Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2001 Aug 26–9; San Francisco, 67–76Google Scholar
- 16.Evans SJW. Statistics: analysis and presentation of safety data. In: Talbot J, Waller P, editors. Stephens’ detection of new adverse drug reactions. 5th ed. West Sussex: Wiley, 2004: 316–25Google Scholar
- 17.ICH-E2E-Expert Working Group. Pharmacovigilance planning [online]. Available from URL: http://www.ich.org/LOB/media/MEDIA1195.pdf [Accessed 2007 Jan 11]
- 20.Weber JCP. Epidemiology of adverse reactions to non-steroidal antiinflammatory drugs. Adv Inflam Res 1984; 6: 1–7Google Scholar