Are systematic reviews and meta-analyses still useful research? Yes
Systematic reviews (SR) are commonly defined as “a summary of studies addressing a clear question, using systematic and explicit methods to identify, select, and critically appraise relevant studies, and to collect and analyse data from them” . One of the essential tenets of evidence-based medicine is that optimal care requires up-to-date, rigorous summaries of evidence . Without such summaries, clinicians and patients are vulnerable to unrepresentative samples of the evidence, misinterpreted and biased estimates of benefits and harms of interventions. For instance, consider the evolution of expert views regarding thrombolysis for acute myocardial infarction (AMI) . An SR found 33 trials evaluating streptokinase in AMI patients, 25 suggesting mortality reduction and six reaching conventional levels of statistical significance. A cumulative meta-analysis found that by 1973—after eight trials (2432 patients)—results showed a mortality reduction with streptokinase. None of the subsequent trials changed the direction or magnitude of the odds ratio of dying. Yet, 34,542 AMI patients continued to be exposed to placebo in subsequent trials, including so-called mega-trials. The reason for the conduct of all the unnecessary trials was that experts, without the benefit of an SR and meta-analyses, could not grasp the big picture of the benefits of thrombolytic therapy, forcing trialists to conduct study after study until the message finally got through. In the meanwhile, patients died unnecessarily.
The fundamental standards for trustworthy clinical practice guidelines is that they rest on a foundation of a systematic summary of the highest quality evidence; among those offering standards for trustworthy guidelines, the need for such summaries is not in dispute [4, 5]. Moreover, those who have written compellingly regarding waste in medical research have emphasized the needless duplication and misguided research directions when carrying out new studies not preceded by systematic summaries of actual knowledge .
We consider these tenets indisputable: optimal clinical practice, optimal guidelines to inform clinical practice and avoiding research waste; all require systematic summaries of evidence. Indeed, though they were first introduced in medical science , other fields, including social  and business  sciences, have recognized the crucial role of SR.
The issue then arises of how to produce the most efficient, useful and informative SR. Arguing that the problems of duplication and poor conduct of SR cast doubt on their usefulness—indeed, their necessity—is as facile and imprudent as arguing that the profusion of poorly conducted randomized trials fundamentally challenges their usefulness.
In the 30 years since authorities suggested initial standards for the conduct of SR , their methodology has advanced enormously, including insights into the importance of appropriately chosen eligibility criteria and of partnership of methodologists and clinicians, greater sophistication in assessing risk of bias in individual studies, insights into use of optimal statistical models, and the advent of network meta-analysis for simultaneous consideration of multiple interventions. The GRADE system is the most recent and possibly most relevant development.
The GRADE methodology for rating the quality (otherwise known as certainty or confidence) of the evidence has been now adopted by over 100 organizations worldwide including the Cochrane Collaboration, the World Health Organization, UpToDate, and most critical societies. That methodology, extensively described for clinicians and other users , and for SR authors, guideline developers and health technology assessment practitioners , allows the classification of evidence as high, moderate, low or very low quality considering study design, risk of bias, precision, consistency, directness, publication bias and magnitude of effect. The detailed guidance for making quality judgements provides a transparent, straightforward approach that has facilitated a new rigour for the SR process. Not surprisingly, the widespread adoption of GRADE has not ushered in an era of uniformly exemplary SR, conducted efficiently and with successful efforts to avoid duplication. Systematic reviews continue to share a number of problems with other forms of research—poorly designed question, questionable population/interventions/outcomes, the conduct full of lapses in rigour both at the level of individual studies and of SR itself, and non-transparent academic and financial conflicts of interests.
We are grateful to Solène PITTION-ROSSILLON, department of education, university of Versailles SQY, for her help in drawing Figure 1.
- 1.Systematic review. GET-IT glossary. http://www.getitglossary.org/. Accessed 22 Jan 2018
- 2.Guyatt G, Jaeschke R, Wilson M, Montori V, Richardson S (2014) What is evidence-based medicine. In: Guyatt G, Meade MO, Cook DJ, Rennie D (eds) Users’ guides to the medical literature: a manual for evidence-based clinical practice, 3rd edn. McGraw-Hill, New YorkGoogle Scholar
- 4.Institute of Medicine (2011) Clinical practice guidelines we can trust. The National Academies Press, Washington, DC. https://doi.org/10.17226/13058
- 5.Brouwers M, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, Fervers B, Graham ID, Grimshaw J, Hanna S, Littlejohns P, Makarski J, Zitzelsberger L, For the AGREE Next Steps Consortium (2010) AGREE II: advancing guideline development, reporting and evaluation in healthcare. CMAJ 182:e839–e842CrossRefPubMedPubMedCentralGoogle Scholar
- 7.Cochrane AL (1979) 1931–1971: a critical review, with particular reference to the medical profession. Medicines for the year 2000. Office of Health Economics, London, pp 1–11Google Scholar
- 16.Murad MH, Montori VM, Ioannidis JP, Jaeschke R, Devereaux PJ, Prasad K, Neumann I, Carrasco-Labra A, Agoritsas T, Hatala R, Meade MO, Wyer P, Cook DJ, Guyatt G (2014) How to read a systematic review and meta-analysis and apply the results to patient care: users’ guides to the medical literature. JAMA 312:171–179CrossRefPubMedGoogle Scholar