Drug Safety

, Volume 39, Issue 4, pp 355–364 | Cite as

Performance of Stratified and Subgrouped Disproportionality Analyses in Spontaneous Databases

  • Suzie SeabrokeEmail author
  • Gianmario Candore
  • Kristina Juhlin
  • Naashika Quarcoo
  • Antoni Wisniewski
  • Ramin Arani
  • Jeffery Painter
  • Philip Tregunno
  • G. Niklas Norén
  • Jim Slattery
Original Research Article



Disproportionality analyses are used in many organisations to identify adverse drug reactions (ADRs) from spontaneous report data. Reporting patterns vary over time, with patient demographics, and between different geographical regions, and therefore subgroup analyses or adjustment by stratification may be beneficial.


The objective of this study was to evaluate the performance of subgroup and stratified disproportionality analyses for a number of key covariates within spontaneous report databases of differing sizes and characteristics.


Using a reference set of established ADRs, signal detection performance (sensitivity and precision) was compared for stratified, subgroup and crude (unadjusted) analyses within five spontaneous report databases (two company, one national and two international databases). Analyses were repeated for a range of covariates: age, sex, country/region of origin, calendar time period, event seriousness, vaccine/non-vaccine, reporter qualification and report source.


Subgroup analyses consistently performed better than stratified analyses in all databases. Subgroup analyses also showed benefits in both sensitivity and precision over crude analyses for the larger international databases, whilst for the smaller databases a gain in precision tended to result in some loss of sensitivity. Additionally, stratified analyses did not increase sensitivity or precision beyond that associated with analytical artefacts of the analysis. The most promising subgroup covariates were age and region/country of origin, although this varied between databases.


Subgroup analyses perform better than stratified analyses and should be considered over the latter in routine first-pass signal detection. Subgroup analyses are also clearly beneficial over crude analyses for larger databases, but further validation is required for smaller databases.


Stratify Analysis Crude Analysis Reporting Odds Ratio Disproportionality Analysis Spontaneous Report Database 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The views expressed in this paper are those of the authors only and not of their respective organisations and do not reflect the official policy or position of the Innovative Medicines Initiative Joint Undertaking (IMI JU).

Compliance with Ethical Standards


The research leading to these results was conducted as part of PROTECT (Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium;, which is a public–private partnership coordinated by the European Medicines Agency (EMA).

The PROTECT project has received support from the Innovative Medicines Initiative Joint Undertaking (IMI JU; under Grant Agreement No 115004, resources of which are composed of financial contribution from the European Union’s Seventh Framework Program (FP7/2007-2013) and companies of the European Federation of Pharmaceutical Industries and Associations (EFPIA) in-kind contribution.

Conflict of interest

Naashika Quarcoo and Jeffery Painter are employees of and hold shares in GlaxoSmithKline. Ramin Arani and Antoni Wisniewski are employees of and hold shares in AstraZeneca. Products from these companies were among those used to test the methodologies in this research. Suzie Seabroke, Gianmario Candore, Kristina Juhlin, Naashika Quarcoo, Antoni Wisniewski, Ramin Arani, Jeffery Painter, Philip Tregunno, Niklas Norén and Jim Slattery have no financial interest in any commercial signal detection software.


  1. 1.
    Rawlins MD. Spontaneous reporting of adverse drug reactions. II: uses. Br J Clin Pharmacol. 1988;26:7–11.CrossRefPubMedPubMedCentralGoogle Scholar
  2. 2.
    Evans SJW, Waller PC, Davis S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf. 2001;10(6):483–6.CrossRefPubMedGoogle Scholar
  3. 3.
    Bate A, Lindquist M, Edwards IR, Olsson S, Orre R, Lansner A, et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol. 1998;54(4):315–21.CrossRefPubMedGoogle Scholar
  4. 4.
    Van Puijenbroek EP, Diemont WL, Van Grootheest K. Application of quantitative signal detection in the Dutch spontaneous reporting system for adverse drug reactions. Drug Saf. 2003;26(5):293–301.CrossRefPubMedGoogle Scholar
  5. 5.
    Szarfman A, Machado SG, O’Neill RT. Use of screening algorithms and computer systems to efficiently signal higher-than-expected combinations of drugs and events in the US FDA’s spontaneous reports database. Drug Saf. 2002;25(6):381–92.CrossRefPubMedGoogle Scholar
  6. 6.
    Hauben M, Horn S, Reich L. Potential use of data-mining algorithms for the detection of ‘surprise’ adverse drug reactions. Drug Saf. 2007;30(2):143–55.CrossRefPubMedGoogle Scholar
  7. 7.
    Hopstadius J, Norén GN, Bate A, Edwards IR. Impact of stratification on adverse drug reaction surveillance. Drug Saf. 2008;31(11):1035–48.CrossRefPubMedGoogle Scholar
  8. 8.
    Gogolak VV. The effect of backgrounds in safety analysis: the impact of comparison cases on what you see. Pharmacoepidemiol Drug Saf. 2003;12(3):249–52.CrossRefPubMedGoogle Scholar
  9. 9.
    Almenoff J, Tonning JM, Gould AL, Szarfman A, Hauben M, Ouellet-Hellstrom R, et al. Perspectives on the use of data mining in pharmacovigilance. Drug Saf. 2005;28(11):981–1007.CrossRefPubMedGoogle Scholar
  10. 10.
    Gould AL. Practical pharmacovigilance analysis strategies. Pharmacoepidemiol Drug Saf. 2003;12:559–74.CrossRefPubMedGoogle Scholar
  11. 11.
    Hopstadius J, Norén GN. Robust discovery of local patterns: subsets and stratification in adverse drug reaction surveillance. Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium. New York: ACM; 2012. p. 265–73 (Abstract).Google Scholar
  12. 12.
    Caster O, Juhlin K, Watson S, Norén GN. Improved statistical signal detection in pharmacovigilance by combining multiple strength-of-evidence aspects in vigiRank. Drug Saf. 2014;37(8):617–28.CrossRefPubMedPubMedCentralGoogle Scholar
  13. 13.
    Van Holle L, Bauchau V. Optimization of a quantitative signal detection algorithm for spontaneous reports of adverse events post immunization. Pharmacoepidemiol Drug Saf. 2013;22(5):477–87.CrossRefPubMedGoogle Scholar
  14. 14.
    Woo EJ, Ball R, Burwen DR, et al. Effects of stratification on data mining in the US Vaccine Adverse Event Reporting System (VAERS). Drug Saf. 2008;31(8):667–74.CrossRefPubMedGoogle Scholar
  15. 15.
    Almenoff JS, LaCroix KK, Yuen NA, Fram D, DuMouchel W. Comparative performance of two quantitative safety signalling methods: implications for use in a pharmacovigilance department. Drug Saf. 2006;29(10):875–87.CrossRefPubMedGoogle Scholar
  16. 16.
    Zeinoun Z, Seifert H, Verstraeten T. Quantitative signal detection for vaccines: effects of stratification, background and masking on GlaxoSmithKline’s spontaneous reports database. Hum Vaccin. 2009;5(9):599–607.CrossRefPubMedGoogle Scholar
  17. 17.
    Grundmark B, Holmberg L, Garmo H, Zethelius B. Reducing the noise in signal detection of adverse drug reactions by standardizing the background: a pilot study on analyses of proportional reporting ratios-by-therapeutic area. Eur J Clin Pharmacol. 2014;70(5):627–35.CrossRefPubMedPubMedCentralGoogle Scholar
  18. 18.
    de Bie S, Verhamme KM, Straus SM, Stricker BH, Sturkenboom MC. Vaccine-based subgroup analysis in VigiBase: effect on sensitivity in paediatric signal detection. Drug Saf. 2012;35(4):335–46.CrossRefPubMedGoogle Scholar
  19. 19.
    Candore G, Juhlin K, Manlik K, Thakrar B, Quarcoo N, Seabroke S, et al. Comparison of statistical signal detection methods within and across databases. Drug Saf. 2015;38(6):577–87.CrossRefPubMedGoogle Scholar
  20. 20.
    van Puijenbroek EP, Bate A, Leufkens HGM, et al. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol Drug Saf. 2002;11:3–10.CrossRefPubMedGoogle Scholar
  21. 21.
    Norén GN, Hopstadius J, Bate A. Shrinkage observed-to-expected ratios for robust and transparent large-scale pattern discovery. Stat Methods Med Res. 2013;22(1):57–69.CrossRefPubMedGoogle Scholar
  22. 22.
    DuMouchel W. Bayesian data mining in large frequency tables with an application to the FDA spontaneous reporting system. Am Stat. 1999;53(3):177–90.Google Scholar
  23. 23.
    Robins J, Greenland S, Breslow N. A general estimator for the variance of the Mantel–Haenszel odds ratio. Am J Epidemiol. 1986;124:719–23.PubMedGoogle Scholar
  24. 24.
    Alvarez Y, Hidalgo A, Maignen F, Slattery J. Validation of statistical signal detection procedures in EudraVigilance post-authorisation data: a retrospective evaluation of the potential for earlier signalling. Drug Saf. 2010;33(6):475–87.CrossRefPubMedGoogle Scholar
  25. 25.
    Bate A, Edwards IR, Lindquist M, Orre R. The authors’ reply [letter]. Drug Saf. 2003;26(5):364–6.CrossRefGoogle Scholar
  26. 26.
    Hopstadius J, Norén GN, Bate A, Edwards IR. Stratification for spontaneous report databases. Drug Saf. 2008;31(12):1145–7.CrossRefPubMedGoogle Scholar
  27. 27.
    Norén GN, Caster O, Juhlin K, Lindquist M. Zoo or savannah? Choice of training ground for evidence-based pharmacovigilance. Drug Saf. 2014;37:655–9.CrossRefPubMedGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Suzie Seabroke
    • 1
    Email author
  • Gianmario Candore
    • 2
  • Kristina Juhlin
    • 3
  • Naashika Quarcoo
    • 4
  • Antoni Wisniewski
    • 5
  • Ramin Arani
    • 5
  • Jeffery Painter
    • 4
  • Philip Tregunno
    • 1
  • G. Niklas Norén
    • 3
  • Jim Slattery
    • 2
  1. 1.UK Medicines and Healthcare Products Regulatory Agency (MHRA)LondonUK
  2. 2.European Medicines AgencyLondonUK
  3. 3.Uppsala Monitoring CentreUppsalaSweden
  4. 4.GlaxoSmithKlineLondonUK
  5. 5.AstraZenecaAlderley ParkUK

Personalised recommendations