Statistical Packages for Diagnostic Meta-Analysis and Their Application

  • Philipp DoeblerEmail author
  • Paul-Christian Bürkner
  • Gerta Rücker


The bivariate model has become a de facto standard in diagnostic meta-analysis. Complex iterative algorithms are needed to fit the model, and thus a meta-analysis of diagnostic accuracy data is much aided by appropriate software packages. Also, graphical methods ease exploration, interpretation, and communication in the context of a diagnostic meta-analysis. This chapter reviews existing software and discusses the relative merits of general packages and specialized packages for DTA meta-analysis. The use of software for diagnostic meta-analysis and especially fitting the bivariate model is illustrated with a sample workflow in the open-source statistical framework R. Some ways to extend the bivariate model and software for the case of multiple cutoff values per primary study are discussed.


Diagnostic test accuracy Meta-analysis Software Bivariate model Meta-regression SROC curve Descriptive statistics 


  1. 1.
    The Nordic Cochrane Centre. Review manager (RevMan) [Computer Program], version 5.3. The Cochrane Collaboration, Copenhagen; 2014.Google Scholar
  2. 2.
    Schwarzer G. meta: an R package for meta-analysis. R News. 2007;7:40–5.Google Scholar
  3. 3.
    Glas A, Lijmer J, Prins M, Bonsel G, Bossuyt P. The diagnostic odds ratio: a single indicator of test performance. J Clin Epidemiol. 2003;56:1129–35.CrossRefPubMedGoogle Scholar
  4. 4.
    Reitsma J, Glas A, Rutjes A, Scholten R, Bossuyt P, Zwinderman A. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol. 2005;58:982–90.CrossRefPubMedGoogle Scholar
  5. 5.
    Leeflang M, Deeks J, Gatsonis C, Bossuyt P. Systematic reviews of diagnostic test accuracy. Ann Intern Med. 2008;149:889–97.CrossRefPubMedPubMedCentralGoogle Scholar
  6. 6.
    Macaskill P, Gatsonis C, Deeks J, Harbord R, Takwoingi Y. Chapter 10: analysing and presenting results. In: Deeks J, Bossuyt P, Gatsonis C, editors. Cochrane handbook for systematic reviews of diagnostic test accuracy version 1.0. The Cochrane Collaboration; 2010. Available from Accessed 29 June 2018.
  7. 7.
    Rutter C, Gatsonis C. A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Stat Med. 2001;20:2865–84.CrossRefPubMedGoogle Scholar
  8. 8.
    Harbord R, Deeks J, Egger M, Whiting P, Sterne J. A unification of models for meta-analysis of diagnostic accuracy studies. Biostatistics. 2007;8:239–51.CrossRefPubMedGoogle Scholar
  9. 9.
    DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7:177–88.CrossRefGoogle Scholar
  10. 10.
    Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010;36:1–48. Available from Accessed 29 June 2018.
  11. 11.
    Gasparrini A, Armstrong B, Kenward M. Multivariate meta-analysis for non-linear and other multi-parameter associations. Stat Med. 2012;31:3821–39.CrossRefPubMedPubMedCentralGoogle Scholar
  12. 12.
    R Core Team. R: a language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2017. Available from Accessed 29 June 2018.
  13. 13.
    Schwarzer G, Carpenter J, Rücker G. Meta-analysis with r. New York: Springer; 2015. UseR!CrossRefGoogle Scholar
  14. 14.
    Liu Z, Yao Z, Li C, Liu X, Chen H, Gao C. A step-by-step guide to the systematic review and meta-analysis of diagnostic and prognostic test accuracy evaluations. Br J Cancer. 2013;108:2299–303.CrossRefPubMedPubMedCentralGoogle Scholar
  15. 15.
    Kim K, Lee J, Choi S, Huh J, Park S. Systematic review and meta-analysis of studies evaluating diagnostic test accuracy: a practical review for clinical researchers-part I. General guidance and tips. Korean J Radiol. 2015;16:1175–87.CrossRefPubMedPubMedCentralGoogle Scholar
  16. 16.
    Lee J, Kim K, Choi S, Huh J, Park S. Systematic review and meta-analysis of studies evaluating diagnostic test accuracy: a practical review for clinical researchers-part II. Statistical methods of meta-analysis. Korean J Radiol. 2015;16:1188–96.CrossRefPubMedPubMedCentralGoogle Scholar
  17. 17.
    Nikoloulopoulos A. CopulaREMADA: copula mixed effect models for bivariate and trivariate meta-analysis of diagnostic test accuracy studies. R package version 1.0. 2016. Available from Accessed 29 June 2018.
  18. 18.
    Schiller I, Dendukuri N. HSROC: joint meta-analysis of diagnostic test sensitivity and specificity with or without a gold standard reference test. R package version 2.1.8; 2015.Google Scholar
  19. 19.
    Verde PE. bamdit: bayesian meta-analysis of diagnostic test data. R package version 3.1.0. 2017. Available from Accessed 29 June 2018.
  20. 20.
    Lunn D, Thomas A, Best N, Spiegelhalter D. WinBUGS—a Bayesian modelling framework: concepts, structure, and extensibility. Stat Comput. 2000;10:325–37.CrossRefGoogle Scholar
  21. 21.
    Lunn D, Spiegelhalter D, Thomas A, Best N. The BUGS project: evolution, critique and future directions. Stat Med. 2009;28:3049–67.CrossRefPubMedGoogle Scholar
  22. 22.
    Plummer M. rjags: Bayesian graphical models using MCMC. R package version 4-6. 2016. Retrieved from Accessed 29 June 2018.
  23. 23.
    Bürkner P-C. brms: an R package for Bayesian multilevel models using Stan. J Stat Softw. 2017;80:1–28.CrossRefGoogle Scholar
  24. 24.
    Partlett C, Takwoingi Y. Meta-analysis of test accuracy studies in R: a summary of user-written programs and step-by-step guide to using glmer. Version 1.0. 2016. Available from Accessed 29 June 2018.
  25. 25.
    Doebler P. mada: meta-analysis of diagnostic accuracy. R package version 0.5.7. 2015. Available from Accessed 29 June 2018.
  26. 26.
    Guo J, Riebler A. meta4diag: meta-analysis for diagnostic test studies. R package version 2.0.5. 2016. Available from Accessed 29 June 2018.
  27. 27.
    Huang H. Metatron: meta-analysis for classification data and correction to imperfect reference. R package version 0.1-1. 2014. Available from Accessed 29 June 2018.
  28. 28.
    Botella J, Huang H, Suero M. Multinomial tree models for assessing the status of the reference in studies of the accuracy of tools for binary classification. Front Psychol. 2013;4:694.CrossRefPubMedPubMedCentralGoogle Scholar
  29. 29.
    Charlton C, Rasbash J, Browne W, Healy M, Cameron, B. Mlwin [computer program] version 3.00. Centre for Multilevel Modelling, University of Bristol; 2017.Google Scholar
  30. 30.
    Chu H, Cole S. Bivariate meta-analysis of sensitivity and specificity with sparse data: a generalized linear mixed model approach. J Clin Epidemiol. 2006;59:1331–2.CrossRefPubMedGoogle Scholar
  31. 31.
    Arends L, Hamza T, Van Houwelingen J, Heijenbrok-Kal M, Hunink M, Stijnen T. Bivariate random effects meta-analysis of ROC curves. Med Decis Mak. 2008;28:621–38.CrossRefGoogle Scholar
  32. 32.
    Menke J. Bivariate random-effects meta-analysis of sensitivity and specificity with sas proc glimmix. Methods Inf Med. 2010;49:54–64.PubMedGoogle Scholar
  33. 33.
    Takwoingi Y, Deeks J. METADAS: an SAS macro for meta-analysis of diagnostic accuracy studies, version 1.3. Computer program; 2011.Google Scholar
  34. 34.
    Rabe-Hesketh S, Skrondal A, Pickles A. Generalized multilevel structural equation modeling. Psychometrika. 2004;69:167–90.CrossRefGoogle Scholar
  35. 35.
    Harbord R, Whiting P. Metandi: meta-analysis of diagnostic accuracy using hierarchical logistic regression. Stata J. 2010;9:211–29.Google Scholar
  36. 36.
    Takwoingi Y. Meta-analysis of test accuracy studies in Stata: a bivariate model approach. Version 1.1. 2016. Available from Accessed 29 June 2018.
  37. 37.
    Dwamena B. midas: Stata module for meta-analytical integration of diagnostic accuracy studies. 2007. Available from: Accessed 29 June 2018.
  38. 38.
    Patrick D, Cheadle A, Thompson D, Diehr P, Koepsell T, Kinne S. The validity of self-reported smoking: a review and meta-analysis. Am J Public Health. 1994;84:1086–93.CrossRefPubMedPubMedCentralGoogle Scholar
  39. 39.
    Wickham H, Bryan J. readxl: read excel files. R package version 1.0.0. 2017. Available from Accessed 29 June 2018.
  40. 40.
    Phillips B, Stewart L, Sutton A. ‘cross hairs’ plots for diagnostic meta-analysis. Res Synth Methods. 2010;1:308–15.CrossRefPubMedGoogle Scholar
  41. 41.
    Vogelgesang F, Schlattmann P, Dewey M. The evaluation of bivariate mixed models in meta-analyses of diagnostic accuracy studies with SAS, Stata and R. Methods Inf Med. 2018;57:111–9.CrossRefPubMedGoogle Scholar
  42. 42.
    Doebler P, Holling H, Böhning D. A mixed model approach to meta-analysis of diagnostic studies with binary test outcome. Psychol Methods. 2012;17:418–36.CrossRefPubMedGoogle Scholar
  43. 43.
    Karrasch S, Linde K, Rücker G, Sommer H, Karsch-Völk M, Kleijnen J, et al. Accuracy of FENO for diagnosing asthma: a systematic review. Thorax. 2017;72:109–16.CrossRefPubMedGoogle Scholar
  44. 44.
    Brown H, Prescott R. Applied mixed models in medicine. 3rd ed. Hoboken: John Wiley & Sons; 2015.Google Scholar
  45. 45.
    Demidenko E. Mixed models: theory and applications. Hoboken: John Wiley & Sons; 2013.Google Scholar
  46. 46.
    Chu H, Nie L, Cole S, Poole C. Meta-analysis of diagnostic accuracy studies accounting for disease prevalence: alternative parameterizations and model selection. Stat Med. 2009;28:2384–99.CrossRefPubMedGoogle Scholar
  47. 47.
    Hoyer A, Kuss O. Meta-analysis of diagnostic tests accounting for disease prevalence: a new model using trivariate copulas. Stat Med. 2015;34:1912–24.CrossRefPubMedGoogle Scholar
  48. 48.
    Nikoloulopoulos A. A vine copula mixed effect model for trivariate meta-analysis of diagnostic test accuracy studies accounting for disease prevalence. Stat Methods Med Res. 2017;26:2270–86.CrossRefPubMedGoogle Scholar
  49. 49.
    Bates D, Mächler M, Bolker B, Walker S. Fitting linear mixed-effects models using lme4. J Stat Softw. 2015;67:1–48.CrossRefGoogle Scholar
  50. 50.
    Moses L, Shapiro D, Littenberg B. Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations. Stat Med. 1993;12:1293–316.CrossRefPubMedGoogle Scholar
  51. 51.
    Holling H, Böhning W, Böhning D. Meta-analysis of diagnostic studies based upon SROC-curves: a mixed model approach using the Lehmann family. Stat Model. 2012;12:347–75.CrossRefGoogle Scholar
  52. 52.
    Holling H, Böhning W, Böhning D. Likelihood-based clustering of meta-analytic SROC curves. Psychometrika. 2012;77:106–26.CrossRefGoogle Scholar
  53. 53.
    Schlattmann P, Höhne J, Verba M. CAMAN: finite mixture models and meta-analysis tools–based on C.A.MAN. R package version 0.74. 2016. Available from Accessed 29 June 2018.
  54. 54.
    Doebler P, Holling H. Meta-analysis of diagnostic accuracy and ROC curves with covariate adjusted semiparametric mixtures. Psychometrika. 2015;80:1084–104.CrossRefPubMedGoogle Scholar
  55. 55.
    Charoensawat S, Böhning W, Böhning D, Holling H. Meta-analysis and meta-modelling for diagnostic problems. BMC Med Res Methodol. 2014;14:56.CrossRefPubMedPubMedCentralGoogle Scholar
  56. 56.
    Rücker G, Schumacher M. Summary ROC curve based on a weighted Youden index for selecting an optimal cut point in meta-analysis of diagnostic accuracy. Stat Med. 2010;29:3069–78.CrossRefPubMedGoogle Scholar
  57. 57.
    Steinhauser S, Schumacher M, Rücker G. Modelling multiple thresholds in meta-analysis of diagnostic test accuracy studies. BMC Med Res Methodol. 2016;16:97.CrossRefPubMedPubMedCentralGoogle Scholar
  58. 58.
    Levis B, Benedetti A, Levis A, Ioannidis J, Shrier I, Cuijpers P, et al. Selective cutoff reporting in studies of diagnostic test accuracy: a comparison of conventional and individual-patient-data meta-analyses of the Patient Health Questionnaire-9 depression screening tool. Am J Epidemiol. 2017;185:954–64.CrossRefPubMedPubMedCentralGoogle Scholar
  59. 59.
    Dukic V, Gatsonis C. Meta-analysis of diagnostic test accuracy assessment studies with varying number of thresholds. Biometrics. 2003;59:936–46.CrossRefPubMedGoogle Scholar
  60. 60.
    Hamza T, Arends L, van Houwelingen H, Stijnen T. Multivariate random effects meta-analysis of diagnostic tests with multiple thresholds. BMC Med Res Methodol. 2009;9:73.CrossRefPubMedPubMedCentralGoogle Scholar
  61. 61.
    Putter H, Fiocco M, Stijnen T. Meta-analysis of diagnostic test accuracy studies with multiple thresholds using survival methods. Biom J. 2010;52:95–110.CrossRefPubMedGoogle Scholar
  62. 62.
    Simoneau G, Levis B, Cuijpers P, Ioannidis J, Patten S, Shrier I, et al. A comparison of bivariate, multivariate random-effects, and Poisson correlated gamma-frailty models to meta-analyze individual patient data of ordinal scale diagnostic tests. Biom J. 2017;59:1317.CrossRefPubMedGoogle Scholar
  63. 63.
    Riley R, Ahmed I, Ensor J, Takwoingi Y, Kirkham A, Morris R, et al. Meta-analysis of test accuracy studies: an exploratory method for investigating the impact of missing thresholds. Syst Rev. 2015;4:12.CrossRefPubMedPubMedCentralGoogle Scholar
  64. 64.
    Hoyer A, Hirt S, Kuss O. Meta-analysis of full ROC curves using bivariate time-to-event models for interval-censored data. Res Synth Methods. 2018;9:62–72.CrossRefPubMedGoogle Scholar
  65. 65.
    Martínez-Camblor P. Fully non-parametric receiver operating characteristic curve estimation for random-effects meta-analysis. Stat Methods Med Res. 2017;26:5–20.CrossRefPubMedGoogle Scholar
  66. 66.
    Riley R, Takwoingi Y, Trikalinos T, Guha A, Biswas A, Ensor J, et al. Meta-analysis of test accuracy studies with multiple and missing thresholds: a multivariate-normal model. J Biomet Biostat. 2014;5:196.CrossRefGoogle Scholar
  67. 67.
    Deeks J, Wisniewski S, Davenport C. Chapter 4: guide to the contents of a Cochrane Diagnostic Test Accuracy Protocol. In: Deeks J, Bossuyt P, Gatsonis C, editors Cochrane handbook for systematic reviews of diagnostic test accuracy version 1.0.0. 2013. Available from Accessed 29 June 2018.
  68. 68.
    Meyer P, Frings L, Rücker G, Hellwig S. 18F-FDG PET in parkinsonism: differential diagnosis and cognitive impairment in Parkinson’s disease. J Nucl Med. 2017;58:1888.CrossRefPubMedGoogle Scholar
  69. 69.
    Schlattmann P, Verba M, Dewey M, Walther M. Mixture models in diagnostic meta-analyses—clustering summary receiver operating characteristic curves accounted for heterogeneity and correlation. J Clin Epidemiol. 2015;68:61–72.CrossRefPubMedGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Philipp Doebler
    • 1
    Email author
  • Paul-Christian Bürkner
    • 2
  • Gerta Rücker
    • 3
  1. 1.Department of StatisticsTU Dortmund UniversityDortmundGermany
  2. 2.Institute of Psychology, Faculty of Psychology and Sport SciencesUniversity of MünsterMünsterGermany
  3. 3.Faculty of MedicineInstitute for Medical Biometry and Statistics, Medical Center – University of FreiburgFreiburg im BreisgauGermany

Personalised recommendations