Drug Safety

, Volume 30, Issue 8, pp 715–726 | Cite as

Criteria Revision and Performance Comparison of Three Methods of Signal Detection Applied to the Spontaneous Reporting Database of a Pharmaceutical Manufacturer

  • Yasuyuki Matsushita
  • Yasufumi Kuroda
  • Shinpei Niwa
  • Satoshi Sonehara
  • Chikuma Hamada
  • Isao Yoshimura
Original Research Article


Background and objective: Several statistical methods exist for detecting signals of potential adverse drug reactions in spontaneous reporting databases. However, these signal-detection methods were developed using regulatory databases, which contain a far larger number of adverse event reports than the databases maintained by individual pharmaceutical manufacturers. Furthermore, the composition and quality of the spontaneous reporting databases differ between regulatory agencies and pharmaceutical companies. Thus, the signal-detection criteria proposed for regulatory use are considered to be inappropriate for pharmaceutical industry use without modification. The objective of this study was to revise the criteria for signal detection to make them suitable for use by pharmaceutical manufacturers.

Methods: A model comprising 40 drugs and 1000 adverse events was constructed based on a spontaneous reporting database provided by a pharmaceutical company and used in a simulation to investigate appropriate criteria for signal detection. In total, 1000 pseudo datasets were generated with this model, and three statistical methods (proportional reporting ratio [PRR], Bayesian Confidence Propagation Neural Network [BCPNN] and multi-item gamma Poisson shrinker [MGPS]) for signal detection were applied to each dataset. The sensitivity and specificity of each method were evaluated using these pseudo datasets. The optimum critical value for signal detection (i.e. the value that achieved the highest sensitivity with 95% specificity) was identified for each method. The optimum values were also examined with the adverse events classified into two categories according to frequency. The three original detection methods and their revised versions were applied to a real pharmaceutical company database to detect 173 known adverse reactions of four drugs.

Results:The 1000 pseudo datasets consisted of an average of 81 862 reports and 11 407 drug-event pairs, including 1192 adverse drug reactions. The sensitivities of PRR, BCPNN and MGPS methods were 49%, 45% and 26%, respectively, whereas their specificities were 95%, 99.6% and 99.99%, respectively; these sensitivities were unacceptably low for pharmaceutical manufacturers, whereas the specificities were acceptable. The highest sensitivity for each method, obtained by changing critical values and maintaining specificity at 95%, was 44%, 62% and 62%, respectively. When adverse events were classified into two categories, sensitivities as high as 75% for regular events and 39% for rare events were achieved with the revised BCPNN method. The critical values of the information component minus two standard deviations (IC —2SD) index of the revised BCPNN method were greater than −0.7 for regular events and greater than −0.6 for rare events. The revised BCPNN method yielded 51% sensitivity and 89% specificity for the real dataset.

Conclusion:A lower critical value may be needed when signal-detection methodology is applied to the spontaneous reporting databases of pharmaceutical manufacturers. For example, it is recommended that pharmaceutical manufacturers use the BCPNN method with IC —2SD criteria of greater than −0.7 for regular events and greater than −0.6 for rare events.


Adverse Drug Reaction Receiver Operating Characteristic Curve Pharmaceutical Company Rare Event Real Dataset 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The authors are grateful to Daiichi Sankyo Co., Ltd., for allowing access to its database. We would like to thank the two anonymous reviewers whose comments greatly improved this article.

Drs Chikuma Hamada and Isao Yoshimura have no conflicts of interest directly relevant to the content of this study. Yasuyuki Matsushita, Yasufumi Kuroda, Shinpei Niwa and Satoshi Sonehara are all employees of Daiichi Sankyo Co. Ltd.

No sources of funding were used to assist in the preparation of this study.


  1. 1.
    ICH-E2D-Expert Working Group. Post-approval safety data management: definitions and standards for expedited reporting [online]. Available from URL: [Accessed 2006 Jul 28]
  2. 2.
    Edwards IR, Biriell C. Harmonisation in pharmacovigilance. Drug Saf 1994; 10(2): 93–102PubMedCrossRefGoogle Scholar
  3. 3.
    Lindquist M, Edwards IR, Bate A, et al. From association to alert: a revised approach to international signal analysis. Pharmacoepidemiol Drug Saf 1999; 8: S15–25PubMedCrossRefGoogle Scholar
  4. 4.
    Evans SJW, Waller P, Davis S. Proportional reporting ratios: the uses of epidemiological methods of signal generation. Pharmacoepidemiol Drug Saf 1998; 7: S102CrossRefGoogle Scholar
  5. 5.
    Szarfman A, Machado SG, O’Neill RT. Use of screening algorithms and computer systems to efficiently signal higher-than-expected combinations of drugs and events in the US FDA’s spontaneous reports database. Drug Saf 2002; 25(6): 381–92 726PubMedCrossRefGoogle Scholar
  6. 6.
    Bate A, Lindquist M, Edwards IR, et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol 1998; 54: 315–21PubMedCrossRefGoogle Scholar
  7. 7.
    van Puijenbroek EP, Bate A, Leufkens HGM, et al. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol Drug Saf 2002; 11: 3–10PubMedCrossRefGoogle Scholar
  8. 8.
    Kubota K, Koide D, Hirai T. Comparison of data mining methodologies using Japanese spontaneous reports. Pharmacoepidemiol Drug Saf 2004; 13: 387–94PubMedCrossRefGoogle Scholar
  9. 9.
    Stahl M, Lindquist M, Edwards IR, et al. Introducing triage logic as a new strategy for the detection of signals in the WHO Drug Monitoring Database. Pharmacoipidemiol Drug Saf 2004; 13: 355–63CrossRefGoogle Scholar
  10. 10.
    Follmann M, Michel A, Geyer C. Comparison of different methods for signal detection in the drug-safety database of a pharmaceutical company. In: the 20th International Conference of Pharmacoepidemiology (ICPE) & Therapeutic Risk Management; 2004 Aug 22–4; Bordeaux. Pharmacoepidemiol Drug Saf 2004; 13: S111CrossRefGoogle Scholar
  11. 11.
    Roux E, Thiessard F, Fourrier-Reglat A, et al. Evaluation of Statistical Association Measures for the Automatic Signal Generation in Pharmacovigilance. IEEE Trans Inf Technol Biomed 2005; 9(4): 518–27PubMedCrossRefGoogle Scholar
  12. 12.
    Almenoff J, Tonning JM, Gould AL, et al. Perspectives on the use of data mining in pharmacovigilance. Drug Saf 2005; 28(11): 981–1007PubMedCrossRefGoogle Scholar
  13. 13.
    DuMouchel W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. Am Statist 1999; 53(3): 177–202Google Scholar
  14. 14.
    DuMouchel W, Pregibon D. Empirical bayes screening for multi-item associations. Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2001 Aug 26–9; San Francisco, 67–76Google Scholar
  15. 15.
    Lawson DH, O’Connor PC, Jick H. Drug attributed alterations in potassium handling in congestive cardiac failure. Eur J Clin Pharmacol 1982; 23: 21–5PubMedCrossRefGoogle Scholar
  16. 16.
    Evans SJW. Statistics: analysis and presentation of safety data. In: Talbot J, Waller P, editors. Stephens’ detection of new adverse drug reactions. 5th ed. West Sussex: Wiley, 2004: 316–25Google Scholar
  17. 17.
    ICH-E2E-Expert Working Group. Pharmacovigilance planning [online]. Available from URL: [Accessed 2007 Jan 11]
  18. 18.
    Waller P, Heeley E, Moseley J. Impact analysis of signal detected from spontaneous adverse drug reaction reporting data. Drug Saf 2005; 28(10): 843–50PubMedCrossRefGoogle Scholar
  19. 19.
    Hazell L, Shakir SAW. Under-reporting of adverse drug reactions: a systematic review. Drug Saf 2006; 29(5): 385–96PubMedCrossRefGoogle Scholar
  20. 20.
    Weber JCP. Epidemiology of adverse reactions to non-steroidal antiinflammatory drugs. Adv Inflam Res 1984; 6: 1–7Google Scholar

Copyright information

© Adis Data Information BV 2007

Authors and Affiliations

  • Yasuyuki Matsushita
    • 1
    • 2
  • Yasufumi Kuroda
    • 2
  • Shinpei Niwa
    • 2
  • Satoshi Sonehara
    • 2
  • Chikuma Hamada
    • 1
  • Isao Yoshimura
    • 1
  1. 1.Faculty of EngineeringTokyo University of ScienceTokyoJapan
  2. 2.Daiichi Sankyo Co. Ltd.TokyoJapan

Personalised recommendations