Skip to main content
SpringerLink
Log in
Book cover

Methods of Clinical Epidemiology pp 213–228Cite as

Quality Assessment in Meta-analysis

Assessing the Validity of Study Outcomes

  • Chapter
  • First Online:

Part of the book series: Springer Series on Epidemiology and Public Health ((SSEH))

Abstract

Quality assessment of primary studies to evaluate the reliability of study results is an essential and mandatory part of meta-analyses. It refers to the internal validity of a study and is described more precisely as assessing the risk of bias. Potential biases derive from selection of participants, data collection, analysis and selective reporting of study results. Quality assessment tools systematically collect information about study characteristics that may lead to bias in order to estimate the overall risk of bias. There are numerous tools available; they can be classified into checklists, scales and component ratings. Focusing on tools for assessing randomized controlled studies, an overview of covered elements of six selected generic tools is given. The Cochrane Collaboration’s tool is described in more detail because it incorporates some important features. Practical aspects of conducting quality assessments are discussed including the meaning and importance of detailed and precise guidance.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Bibliography

  • Abraham NS, Moayyedi P, Daniels B, Veldhuyzen van Zanten SJ (2004) Systematic review: the methodological quality of trials affects estimates of treatment efficacy in functional (non-ulcer) dyspepsia. Aliment Pharmacol Ther 19:631–641

    Article  PubMed  CAS  Google Scholar 

  • Al Khalaf MM, Thalib L, Doi SA (2011) Combining heterogenous studies using the random-effects model is a mistake and leads to inconclusive meta-analyses. J Clin Epidemiol 64:119–123

    Article  PubMed  Google Scholar 

  • Armijo-Olivo S, Stiles CR, Hagen NA, Biondo PD, Cummings GG (2012) Assessment of study quality for systematic reviews: a comparison of the Cochrane collaboration risk of bias tool and the effective public health practice project quality assessment tool: methodological research. J Eval Clin Pract 18:12–18

    Article  PubMed  Google Scholar 

  • Balevi B (2011) Industry sponsored research may report more favourable outcomes. Evid Based Dent 12:5–6

    Article  PubMed  Google Scholar 

  • Barnes DE, Bero LA (1998) Why review articles on the health effects of passive smoking reach different conclusions. JAMA 279:1566–1570

    Article  PubMed  CAS  Google Scholar 

  • Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF (1996) Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 276:637–639

    Article  PubMed  CAS  Google Scholar 

  • Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP (2006) Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol 59:697–703

    Article  PubMed  Google Scholar 

  • Campbell MK, Elbourne DR, Altman DG (2004) CONSORT statement. Extension to cluster randomised trials. BMJ 328:702–708

    Article  PubMed  Google Scholar 

  • Chan AW, Krleza-Jeric K, Schmid I, Altman DG (2004) Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. Can Med Assoc J 171:735–740

    Article  Google Scholar 

  • Cho MK, Bero LA (1994) Instruments for assessing the quality of drug studies published in the medical literature. JAMA 272:101–104

    Article  PubMed  CAS  Google Scholar 

  • Clark HD, Wells GA, Huet C, McAlister FA, Salmi LR, Fergusson D, Laupacis A (1999) Assessing the quality of randomized trials. Reliability of the Jadad scale. Control Clin Trials 20:448–452

    Article  PubMed  CAS  Google Scholar 

  • Downs SH, Black N (1998) The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health 52:377–384

    Article  PubMed  CAS  Google Scholar 

  • Dreier M, Borutta B, Stahmeyer J, Krauth C, Walter U (2010) Comparison of tools for assessing the methodological quality of primary and secondary studies in health technology assessment reports in Germany. GMS Health Technol Assess 6:Doc07

    PubMed  Google Scholar 

  • Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, Decullier E, Easterbrook PJ, Von Elm E, Gamble C, Ghersi D, Ioannidis JP, Simes J, Williamson PR (2008) Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One 3:e3081

    Article  PubMed  Google Scholar 

  • Greenland S (1994) Invited commentary. A critical look at some popular meta-analytic methods. Am J Epidemiol 140:290–296

    PubMed  CAS  Google Scholar 

  • Haahr MT, Hrobjartsson A (2006) Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors. Clin Trials 3:360–365

    PubMed  Google Scholar 

  • Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Krebs SJ, Klassen TP (2009) Risk of bias versus quality assessment of randomised controlled trials. Cross sectional study. BMJ 339:b4012

    Article  PubMed  Google Scholar 

  • Herbison P, Hay-Smith J, Gillespie WJ (2006) Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol 59:1249–1256

    Article  PubMed  Google Scholar 

  • Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA (2011a) The Cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ 343:d5928

    Article  PubMed  Google Scholar 

  • Higgins JPT, Altman DG, Sterne JAC (2011) Assessing risk of bias in included studies. In: Higgins JPT, Green S (eds). Cochrane handbook for systematic reviews of interventions. Version 5.1.0 (updated March 2011). The Cochrane Collaboration. Available from http://www.cochrane-handbook.org

  • Huwiler-Müntener K, Jüni P, Junker C, Egger M (2002) Quality of reporting of randomized trials as a measure of methodologic quality. JAMA 287:2801–2804

    Article  PubMed  Google Scholar 

  • Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ (1996) Assessing the quality of reports of randomized clinical trials. Is blinding necessary? Control Clin Trials 17:1–12

    Article  PubMed  CAS  Google Scholar 

  • Juni P, Witschi A, Bloch R, Egger M (1999) The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 282:1054–1060

    Article  PubMed  CAS  Google Scholar 

  • Juni P, Altman DG, Egger M (2001) Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 323:42–46

    Article  PubMed  CAS  Google Scholar 

  • Kjaergaard LL, Villumsen J, Cluud C (2001) Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 135:982–989

    Article  Google Scholar 

  • Kunz R, Oxman AD (1998) The unpredictability paradox. Review of empirical comparisons of randomised and non-randomised clinical trials. BMJ 317:1185–1190

    Article  PubMed  CAS  Google Scholar 

  • Lexchin J, Bero LA, Djulbegovic B, Clark O (2003) Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 326:1167–1170

    Article  PubMed  Google Scholar 

  • Liddle J, Williamson M, Irwig L (1996) Method for evaluating research and guideline evidence. New South Wales Department of Health, Sydney

    Google Scholar 

  • Lohr KN, Carey TS (1999) Assessing “best evidence”. Issues in grading the quality of studies for systematic reviews. Jt Comm J Qual Improv 25:470–479

    PubMed  CAS  Google Scholar 

  • Ludwig-Boltzmann Institut HTA. (Internes) Manual–Abläufe und Methoden Tl. 2. 2007. Wien, Ludwig Bolzmann Institut, Health Technology Assessment. HTA-Projektbericht, Nr. 006

    Google Scholar 

  • Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P (2009) Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 302:977–984

    Article  PubMed  CAS  Google Scholar 

  • McGauran N, Wieseler B, Kreis J, Schuler YB, Kolsch H, Kaiser T (2010) Reporting bias in medical research: a narrative review. Trials 11:37

    Article  PubMed  Google Scholar 

  • Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S (1995) Assessing the quality of randomized controlled trials. An annotated bibliography of scales and checklists. Control Clin Trials 16:62–73

    Article  PubMed  CAS  Google Scholar 

  • Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP (1998) Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 352:609–613

    Article  PubMed  CAS  Google Scholar 

  • Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, Jones A, Pham B, Klassen TP (1999) Assessing the quality of reports of randomised trials. Implications for the conduct of meta-analyses. Health Technol Assess 3:1–98

    Google Scholar 

  • Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, Elbourne D, Egger M, Altman DG (2010) CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 340:c869

    Article  PubMed  Google Scholar 

  • National Health Service Public Health Resource Unit (2006) Critical appraisal skills programme: making sense of evidence. 10 questions to help you make sense of randomised controlled studies. Available from http://www.sph.nhs.uk/what-we-do/public-health-workforce/resources/critical-appraisals-skills-programme

  • Odgaard-Jensen J, Vist GE, Timmer A, Kunz R, Akl EA, Schünemann H, Briel M, Nordmann AJ, Pregno S, Oxman AD (2011) Randomisation to protect against selection bias in healthcare trials. Cochrane Database Syst Rev. doi:10.1002/14651858.MR000012.pub3

    PubMed  Google Scholar 

  • Oxman AD (1994) Checklists for review articles. BMJ 309:648–651

    Article  PubMed  CAS  Google Scholar 

  • Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJ (2006) Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA 295:1152–1160

    Article  PubMed  CAS  Google Scholar 

  • Pildal J, Hróbjartsson A, Jørgensen KJ, Hilden J, Altman DG, Gøtzsche PC (2007) Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 36:847–857

    Article  PubMed  CAS  Google Scholar 

  • Porta N, Bonet C, Cobo E (2007) Discordance between reported intention-to-treat and per protocol analyses. J Clin Epidemiol 60:663–669

    Article  PubMed  Google Scholar 

  • Scottish Intercollegiate Guidelines Network (SIGN) (2011) SIGN 50 – a guideline developer’s handbook. Available from http://www.sign.ac.uk/guidelines/fulltext/50/index.html

  • Thomas BH, Ciliska D, Dobbins M, Micucci S (2004) A process for systematically reviewing the literature. Providing the research evidence for public health nursing interventions. Worldviews Evid Based Nurs 1:176–184

    Article  PubMed  CAS  Google Scholar 

  • University of Oxford (2005) Centre of Evidence Based Medicine (CEBM). Critical appraisal for therapy articles. Available from http://www.cebm.net/index.aspx?o=1157

  • Verhagen AP, De Vet HCW, de Bie RA, Kessels AGH, Boers M, Bouter LM, Knipschild PG (1998) The Delphi list. A criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol 51:1235–1241

    Article  PubMed  CAS  Google Scholar 

  • Verhagen AP, de Bie RA, Lenssen AF, de Vet HC, Kessels AG, Boers M, van den Brandt PA (2000) Impact of quality items on study outcome. Treatments in acute lateral ankle sprains. Int J Technol Assess Health Care 16:1136–1146

    Article  PubMed  CAS  Google Scholar 

  • West S, King V, Carey TS, Lohr KN, McKoy N, Sutton SF, Lux L (2002) Systems to rate the strength of scientific evidence. Evid Rep Technol Assess 47:1–11

    Google Scholar 

  • Whiting P, Rutjes AW, Dinnes J, Reitsma J, Bossuyt PM, Kleijnen J (2004) Development and validation of methods for assessing the quality of diagnostic accuracy studies. Health Technol Assess 8:1–234

    Google Scholar 

  • Williamson PR, Gamble C (2005) Identification and impact of outcome selection bias in meta-analysis. Stat Med 24:1547–1561

    Article  PubMed  CAS  Google Scholar 

  • Wood L, Egger M, Gluud LL, Schulz K, Jüni P, Altman DG, Gluud C, Martin RM, Wood AJG, Sterne JAC (2008) Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 336:601–605

    Article  PubMed  Google Scholar 

  • Zaza S, Wright-De Aguero LK, Briss PA, Truman BI, Hopkins DP, Hennessy MH, Sosin DM, Anderson L, Carande-Kulis VG, Teutsch SM, Pappaioanou M (2000) Data collection instrument and procedure for systematic reviews in the guide to community preventive services. Task force on community preventive services. Am J Prev Med 18:44–74

    Article  PubMed  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Hannover Medical School, Institute for Epidemiology, Social Medicine and Health Systems Research, Carl-Neuberg-Str. 1, 30625, Hannover, Germany

    Maren Dreier

Authors
  1. Maren Dreier
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maren Dreier .

Editor information

Editors and Affiliations

  1. Clinical Epidemiology Unit, School of Population Health, University of Queensland, Herston Road, Herston, 4006, Queensland, Australia

    Suhail A. R. Doi

  2. Dept. of Epidemiology & Biostatistics, School of Population Health, University of Queensland, Herston Road, Herston, 4006, Queensland, Australia

    Gail M. Williams

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Dreier, M. (2013). Quality Assessment in Meta-analysis. In: Doi, S., Williams, G. (eds) Methods of Clinical Epidemiology. Springer Series on Epidemiology and Public Health. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-37131-8_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-37131-8_13

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-37130-1

  • Online ISBN: 978-3-642-37131-8

  • eBook Packages: MedicineMedicine (R0)

Publish with us

Policies and ethics

Search

Navigation