Skip to main content

Performance of Stratified and Subgrouped Disproportionality Analyses in Spontaneous Databases



Disproportionality analyses are used in many organisations to identify adverse drug reactions (ADRs) from spontaneous report data. Reporting patterns vary over time, with patient demographics, and between different geographical regions, and therefore subgroup analyses or adjustment by stratification may be beneficial.


The objective of this study was to evaluate the performance of subgroup and stratified disproportionality analyses for a number of key covariates within spontaneous report databases of differing sizes and characteristics.


Using a reference set of established ADRs, signal detection performance (sensitivity and precision) was compared for stratified, subgroup and crude (unadjusted) analyses within five spontaneous report databases (two company, one national and two international databases). Analyses were repeated for a range of covariates: age, sex, country/region of origin, calendar time period, event seriousness, vaccine/non-vaccine, reporter qualification and report source.


Subgroup analyses consistently performed better than stratified analyses in all databases. Subgroup analyses also showed benefits in both sensitivity and precision over crude analyses for the larger international databases, whilst for the smaller databases a gain in precision tended to result in some loss of sensitivity. Additionally, stratified analyses did not increase sensitivity or precision beyond that associated with analytical artefacts of the analysis. The most promising subgroup covariates were age and region/country of origin, although this varied between databases.


Subgroup analyses perform better than stratified analyses and should be considered over the latter in routine first-pass signal detection. Subgroup analyses are also clearly beneficial over crude analyses for larger databases, but further validation is required for smaller databases.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5


  1. 1.

    Rawlins MD. Spontaneous reporting of adverse drug reactions. II: uses. Br J Clin Pharmacol. 1988;26:7–11.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  2. 2.

    Evans SJW, Waller PC, Davis S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf. 2001;10(6):483–6.

    CAS  Article  PubMed  Google Scholar 

  3. 3.

    Bate A, Lindquist M, Edwards IR, Olsson S, Orre R, Lansner A, et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol. 1998;54(4):315–21.

    CAS  Article  PubMed  Google Scholar 

  4. 4.

    Van Puijenbroek EP, Diemont WL, Van Grootheest K. Application of quantitative signal detection in the Dutch spontaneous reporting system for adverse drug reactions. Drug Saf. 2003;26(5):293–301.

    Article  PubMed  Google Scholar 

  5. 5.

    Szarfman A, Machado SG, O’Neill RT. Use of screening algorithms and computer systems to efficiently signal higher-than-expected combinations of drugs and events in the US FDA’s spontaneous reports database. Drug Saf. 2002;25(6):381–92.

    CAS  Article  PubMed  Google Scholar 

  6. 6.

    Hauben M, Horn S, Reich L. Potential use of data-mining algorithms for the detection of ‘surprise’ adverse drug reactions. Drug Saf. 2007;30(2):143–55.

    Article  PubMed  Google Scholar 

  7. 7.

    Hopstadius J, Norén GN, Bate A, Edwards IR. Impact of stratification on adverse drug reaction surveillance. Drug Saf. 2008;31(11):1035–48.

    Article  PubMed  Google Scholar 

  8. 8.

    Gogolak VV. The effect of backgrounds in safety analysis: the impact of comparison cases on what you see. Pharmacoepidemiol Drug Saf. 2003;12(3):249–52.

    Article  PubMed  Google Scholar 

  9. 9.

    Almenoff J, Tonning JM, Gould AL, Szarfman A, Hauben M, Ouellet-Hellstrom R, et al. Perspectives on the use of data mining in pharmacovigilance. Drug Saf. 2005;28(11):981–1007.

    CAS  Article  PubMed  Google Scholar 

  10. 10.

    Gould AL. Practical pharmacovigilance analysis strategies. Pharmacoepidemiol Drug Saf. 2003;12:559–74.

    Article  PubMed  Google Scholar 

  11. 11.

    Hopstadius J, Norén GN. Robust discovery of local patterns: subsets and stratification in adverse drug reaction surveillance. Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium. New York: ACM; 2012. p. 265–73 (Abstract).

  12. 12.

    Caster O, Juhlin K, Watson S, Norén GN. Improved statistical signal detection in pharmacovigilance by combining multiple strength-of-evidence aspects in vigiRank. Drug Saf. 2014;37(8):617–28.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  13. 13.

    Van Holle L, Bauchau V. Optimization of a quantitative signal detection algorithm for spontaneous reports of adverse events post immunization. Pharmacoepidemiol Drug Saf. 2013;22(5):477–87.

    Article  PubMed  Google Scholar 

  14. 14.

    Woo EJ, Ball R, Burwen DR, et al. Effects of stratification on data mining in the US Vaccine Adverse Event Reporting System (VAERS). Drug Saf. 2008;31(8):667–74.

    Article  PubMed  Google Scholar 

  15. 15.

    Almenoff JS, LaCroix KK, Yuen NA, Fram D, DuMouchel W. Comparative performance of two quantitative safety signalling methods: implications for use in a pharmacovigilance department. Drug Saf. 2006;29(10):875–87.

    CAS  Article  PubMed  Google Scholar 

  16. 16.

    Zeinoun Z, Seifert H, Verstraeten T. Quantitative signal detection for vaccines: effects of stratification, background and masking on GlaxoSmithKline’s spontaneous reports database. Hum Vaccin. 2009;5(9):599–607.

    CAS  Article  PubMed  Google Scholar 

  17. 17.

    Grundmark B, Holmberg L, Garmo H, Zethelius B. Reducing the noise in signal detection of adverse drug reactions by standardizing the background: a pilot study on analyses of proportional reporting ratios-by-therapeutic area. Eur J Clin Pharmacol. 2014;70(5):627–35.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  18. 18.

    de Bie S, Verhamme KM, Straus SM, Stricker BH, Sturkenboom MC. Vaccine-based subgroup analysis in VigiBase: effect on sensitivity in paediatric signal detection. Drug Saf. 2012;35(4):335–46.

    Article  PubMed  Google Scholar 

  19. 19.

    Candore G, Juhlin K, Manlik K, Thakrar B, Quarcoo N, Seabroke S, et al. Comparison of statistical signal detection methods within and across databases. Drug Saf. 2015;38(6):577–87.

    CAS  Article  PubMed  Google Scholar 

  20. 20.

    van Puijenbroek EP, Bate A, Leufkens HGM, et al. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol Drug Saf. 2002;11:3–10.

    Article  PubMed  Google Scholar 

  21. 21.

    Norén GN, Hopstadius J, Bate A. Shrinkage observed-to-expected ratios for robust and transparent large-scale pattern discovery. Stat Methods Med Res. 2013;22(1):57–69.

    Article  PubMed  Google Scholar 

  22. 22.

    DuMouchel W. Bayesian data mining in large frequency tables with an application to the FDA spontaneous reporting system. Am Stat. 1999;53(3):177–90.

    Google Scholar 

  23. 23.

    Robins J, Greenland S, Breslow N. A general estimator for the variance of the Mantel–Haenszel odds ratio. Am J Epidemiol. 1986;124:719–23.

    CAS  PubMed  Google Scholar 

  24. 24.

    Alvarez Y, Hidalgo A, Maignen F, Slattery J. Validation of statistical signal detection procedures in EudraVigilance post-authorisation data: a retrospective evaluation of the potential for earlier signalling. Drug Saf. 2010;33(6):475–87.

    Article  PubMed  Google Scholar 

  25. 25.

    Bate A, Edwards IR, Lindquist M, Orre R. The authors’ reply [letter]. Drug Saf. 2003;26(5):364–6.

    Article  Google Scholar 

  26. 26.

    Hopstadius J, Norén GN, Bate A, Edwards IR. Stratification for spontaneous report databases. Drug Saf. 2008;31(12):1145–7.

    Article  PubMed  Google Scholar 

  27. 27.

    Norén GN, Caster O, Juhlin K, Lindquist M. Zoo or savannah? Choice of training ground for evidence-based pharmacovigilance. Drug Saf. 2014;37:655–9.

    Article  PubMed  Google Scholar 

Download references


The views expressed in this paper are those of the authors only and not of their respective organisations and do not reflect the official policy or position of the Innovative Medicines Initiative Joint Undertaking (IMI JU).

Author information



Corresponding author

Correspondence to Suzie Seabroke.

Ethics declarations


The research leading to these results was conducted as part of PROTECT (Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium;, which is a public–private partnership coordinated by the European Medicines Agency (EMA).

The PROTECT project has received support from the Innovative Medicines Initiative Joint Undertaking (IMI JU; under Grant Agreement No 115004, resources of which are composed of financial contribution from the European Union’s Seventh Framework Program (FP7/2007-2013) and companies of the European Federation of Pharmaceutical Industries and Associations (EFPIA) in-kind contribution.

Conflict of interest

Naashika Quarcoo and Jeffery Painter are employees of and hold shares in GlaxoSmithKline. Ramin Arani and Antoni Wisniewski are employees of and hold shares in AstraZeneca. Products from these companies were among those used to test the methodologies in this research. Suzie Seabroke, Gianmario Candore, Kristina Juhlin, Naashika Quarcoo, Antoni Wisniewski, Ramin Arani, Jeffery Painter, Philip Tregunno, Niklas Norén and Jim Slattery have no financial interest in any commercial signal detection software.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Seabroke, S., Candore, G., Juhlin, K. et al. Performance of Stratified and Subgrouped Disproportionality Analyses in Spontaneous Databases. Drug Saf 39, 355–364 (2016).

Download citation


  • Stratify Analysis
  • Crude Analysis
  • Reporting Odds Ratio
  • Disproportionality Analysis
  • Spontaneous Report Database