Advertisement

BMC Medicine

, 14:59 | Cite as

Living cumulative network meta-analysis to reduce waste in research: A paradigmatic shift for systematic reviews?

  • Per Olav Vandvik
  • Romina Brignardello-Petersen
  • Gordon H. Guyatt
Open Access
Commentary

Abstract

In a recent research article in BMC Medicine, Créquit and colleagues demonstrate how published systematic reviews in lung cancer provide a fragmented, out-of-date picture of the evidence for all treatments. The results and conclusions drawn from this study, based on cumulative network meta-analyses (NMA) of evidence from randomized clinical trials over time, are quite compelling. The inherent waste of research resulting from incomplete evidence synthesis has wide-reaching implications for a range of target groups including developers of systematic reviews and guidelines and their end-users, health care professionals and patients at the point of care. Building on emerging concepts for living systematic reviews and NMA, the authors propose "living cumulative NMA" as a potential solution and paradigmatic shift. Here we describe how recent innovations within authoring, dissemination, and updating of systematic reviews and trustworthy guidelines may greatly facilitate the production of living NMA. Some additional challenges need to be solved for NMA in general, and for living cumulative NMA in particular, before a paradigmatic shift for systematic reviews can become reality.

Please see related research article: https://bmcmedicine.biomedcentral.com/articles/10.1186/s12916-016-0555-0

Keywords

Systematic reviews Network meta-analysis Evidence-based medicine Knowledge translation Clinical practice guidelines GRADE 

Background

Health care professionals and other decision makers responsible for providing patients with safe and high quality care need access to the best current research evidence about diagnosis, treatment and prognosis [1]. High quality systematic reviews synthesize the best available evidence regarding an intervention’s benefits and harms to answer a clinical question. Clinical practice guidelines combine this evidence with other relevant considerations such as patients’ values and preferences [2]. Together, reviews and guidelines constitute key tools to provide the best current evidence and optimal recommendations for choosing among alternative diagnostic and treatment strategies [3].

Over the past decades the world has seen large advances in standards, methods, and systems for conducting and reporting systematic reviews and trustworthy guidelines [4, 5, 6, 7]. These advances include systematic reviews that use network meta-analyses (NMA), the statistical technique that allows comparing multiple treatments at a time [8, 9], the methods of which are rapidly evolving [10].

In a study recently published in BMC Medicine, Créquit and colleagues illustrate how published systematic reviews in lung cancer provide a fragmented, out-of-date picture of the evidence for all treatments [11]. The evidence covered by existing systematic reviews for second-line treatments for advanced non-small cell lung cancer was incomplete, with 40 % or more of treatments, treatment comparisons, and trials missing. The incomplete evidence was detected through the construction of cumulative networks of evidence from randomized clinical trials over time and an evaluation of the proportion of trials, patients, treatments, and treatment comparisons not covered by systematic reviews on a yearly basis from 2009 onwards.

The study has limitations, including lack of consideration of the scope of the systematic reviews (limited scope may have been a legitimate reason to not include all available trials), and failure to address the possibility that it would have been inappropriate to pool together the results of all the randomized trials detected. Although these limitations may have resulted in an exaggeration of the magnitude of the problem, the results and conclusions are nevertheless compelling. As stated by the authors "available pairwise treatment comparisons do not allow for meeting clinicians’ and patients’ needs in decision making. Nor do they meet the needs of other target audiences such as developers of clinical practice guidelines, decision aids, funders and decision-makers in health care systems" [11].

Living cumulative NMA

Building on emerging concepts for living systematic reviews and network meta-analysis, the authors conclude that this waste of research might be reduced by "living cumulative NMA" (living NMA hereafter). With this approach, the focus would be on developing NMAs that include all available treatments (as opposed to a series of head-to-head comparisons), which are updated as soon as new trials become available. The authors propose methodological steps for living NMA and present a series of challenges and potential solutions for each step from creation to dissemination and updating. Their proposed solution represents an innovative input to existing challenges for systematic reviews and also relates to international calls for increased value and reduced waste in research, here within the context of evidence synthesis of available trials [12].

Although producing living NMAs is challenging, recent developments could greatly facilitate their production. Improved methods, strategies, and tools for more efficient authoring, dissemination, and updating of systematic reviews and trustworthy guidelines are emerging [13, 14, 15]. Tools, such as the Epistemonikos database, allow efficient and intuitive searching for systematic reviews and individual studies [16]. The use of GRADE combined with innovative health information technology (e.g., MAGICapp) facilitates dynamically updated digitally structured evidence summaries, guidelines, and decision aids that clinicians and patients can access on all devices in user-friendly formats [3, 14, 15, 17, 18]. Presentations on these devices include desirable and undesirable consequences of treatment alternatives customized for both shared decision-making and integration in electronic medical records linked to patient specific data [14, 19].

These innovations, however, have not fully addressed all the challenges related to NMA in general, and to living cumulative NMA in particular. We will now comment on some additional challenges and suggest potential solutions from the perspective of developers of systematic reviews and guidelines and their end-users, health care professionals and patients at the point of care.

Rating quality of evidence in NMA

Systematic reviews need to perform an appropriate assessment of the quality of the body of evidence (synonyms: certainty or confidence in the evidence) for patient-important outcomes [5, 7]. State of the art reviews now provide evidence summaries using GRADE Summary of Findings (SoF) tables that provide quality of evidence ratings to inform clinicians, patients, guideline developers, and policy decision-makers [20].

Créquit and co-authors point out that guideline developers and other decision-makers may further benefit from NMAs if they rate the quality of the body of evidence supporting treatment effect estimates for all patient-important outcomes. Rating quality of evidence in NMA presents additional challenges and represents a methodological frontier. Authors have already presented two systems based on GRADE principles for making quality of evidence ratings in NMAs [21, 22]. Much work remains in the testing, application, and refinement of these methods, currently being undertaken by (among others) the GRADE NMA project group [23].

Patient-important outcomes in NMA

Systematic reviews and trustworthy guidelines need to provide evidence for all outcomes to allow an assessment of the balance between the desirable and undesirable consequences of a course of action [4, 24]. The Créquit study focused on the availability of treatment comparisons for lung cancer rather than on the extent to which authors synthesized evidence across all patient-important outcomes. As acknowledged by the authors, the identification and reporting of all outcomes that are important to patients for decision-making in living NMA represents a particular challenge, including differential reporting across outcomes resulting in variation in the geometry of the network of trials. Further methodological work in NMA will be required to address these issues.

Dissemination of results from NMA to the point of care

Health care professionals and patients will, as outlined above, increasingly be able to access the best current evidence from systematic reviews and recommendations in guidelines at the point of care in new and user-friendly formats [3, 14, 19]. Interactive Summary of Findings (iSoF) tables represent one such innovation with new and improved presentation formats of evidence summaries linked to systematic reviews and guidelines. iSoF tables allow end-users to balance benefits and harms in absolute numbers, also taking into account the quality of the evidence [25].

Current SoF tables are structured only for paired comparisons, and so do not address the multiple comparisons of NMA. How to best present multiple comparisons for all patient-important outcomes to end-users remains uninvestigated. Members of the GRADE NMA project group are exploring both how to optimally present the results from NMA in SoF tables and how to translate relative estimates of effects (the output obtained from NMA) into absolute estimates of effect that facilitate the decision-making process [23].

Conclusions

The traditional methods of conducting systematic reviews are associated with a risk of presenting an incomplete picture of the relevant evidence addressing a clinical question. Living NMA represents a promising approach to providing a broad, complete, and updated presentation of the evidence regarding all the available management options. This might well represent a paradigmatic shift for systematic reviews. However, as is the case for any new technology, living NMA will require overcoming a number of challenges. Investigators are currently addressing these challenges which suggest that living NMA as a new tool for addressing clinical questions may, in the near future, become available.

Notes

Declarations

This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

References

  1. 1.
    Guyatt G, Rennie D, Meade M, Cook D. Users’ Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice. 3rd ed. New York, NY: McGraw-Hill Education, American Medical Association; 2015.Google Scholar
  2. 2.
    Andrews JC, Schunemann HJ, Oxman AD, et al. GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation's direction and strength. J Clin Epidemiol. 2013;66(7):726–35.CrossRefPubMedGoogle Scholar
  3. 3.
    Treweek S, Oxman AD, Alderson P, et al. Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6.CrossRefPubMedPubMedCentralGoogle Scholar
  4. 4.
    Laine C, Taichman DB, Mulrow C. Trustworthy clinical guidelines. Ann Intern Med. 2011;154(11):774–5.CrossRefPubMedGoogle Scholar
  5. 5.
    Guyatt GH, Oxman AD, Kunz R, et al. What is "quality of evidence" and why is it important to clinicians? BMJ. 2008;336(7651):995–8.CrossRefPubMedPubMedCentralGoogle Scholar
  6. 6.
    Guyatt GH, Oxman AD, Kunz R, et al. Going from evidence to recommendations. BMJ. 2008;336(7652):1049–51.CrossRefPubMedPubMedCentralGoogle Scholar
  7. 7.
    Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. 2011. http://www.cochrane.org/handbook. Accessed 5 Mar 2016.
  8. 8.
    Salanti G. Indirect and mixed-treatment comparison, network, or multiple-treatments meta-analysis: many names, many benefits, many concerns for the next generation evidence synthesis tool. Res Syn Meth. 2012;3:80–97.CrossRefGoogle Scholar
  9. 9.
    Lu G, Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Stat Med. 2004;23(20):3105–24.CrossRefPubMedGoogle Scholar
  10. 10.
    Efthimiou O, Debray TP, van Valkenhoef G, et al. GetReal in network meta-analysis: a review of the methodology. Res Syn Meth. 2016. doi: 10.1002/jrsm.Google Scholar
  11. 11.
    Crequit P, Trinquart L, Yavchitz A, Ravaud P. Wasted research when systematic reviews fail to provide a complete and up-to-date evidence synthesis: the example of lung cancer. BMC Med. 2016;14(1):8.CrossRefPubMedPubMedCentralGoogle Scholar
  12. 12.
    Macleod MR, Michie S, Roberts I, et al. Biomedical research: increasing value, reducing waste. Lancet. 2014;383(9912):101–4.CrossRefPubMedGoogle Scholar
  13. 13.
    Elliott JH, Turner T, Clavisi O, et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 2014;11(2), e1001603.CrossRefPubMedPubMedCentralGoogle Scholar
  14. 14.
    Vandvik PO, Brandt L, Alonso-Coello P, et al. Creating clinical practice guidelines we can trust, use, and share: a new era is imminent. Chest. 2013;144(2):381–9.CrossRefPubMedGoogle Scholar
  15. 15.
    MAGIC authoring and publication platform (MAGICapp) for improving patient care through guidelines, evidence summaries and decision aids that we can all trust, use and share. http://www.magicapp.org. Accessed 5 Mar 2016.
  16. 16.
    Epistemonikos. A Free, Relational, Collaborative, Multilingual Database of Health Evidence. http://www.epistemonikos.org Accessed 5 Mar 2016.
  17. 17.
    Agoritsas T, Heen AF, Brandt L, et al. Decision aids that really promote shared decision making: the pace quickens. BMJ. 2015;350:g7624.CrossRefPubMedPubMedCentralGoogle Scholar
  18. 18.
    Kristiansen A, Brandt L, Alonso-Coello P, et al. Development of a novel, multilayered presentation format for clinical practice guidelines. Chest. 2015;147(3):754–63.CrossRefPubMedGoogle Scholar
  19. 19.
    Elwyn G, Quinlan C, Mulley A, Agoritsas T, Vandvik PO, Guyatt G. Trustworthy guidelines - excellent; customized care tools - even better. BMC Med. 2015;13(1):199.CrossRefPubMedPubMedCentralGoogle Scholar
  20. 20.
    Guyatt G, Oxman AD, Akl EA, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94.CrossRefPubMedGoogle Scholar
  21. 21.
    Puhan MA, Schunemann HJ, Murad MH, et al. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ. 2014;349:g5630.CrossRefPubMedGoogle Scholar
  22. 22.
    Salanti G, Del Giovane C, Chaimani A, Caldwell DM, Higgins JP. Evaluating the quality of evidence from a network meta-analysis. PLoS One. 2014;9(7), e99682.CrossRefPubMedPubMedCentralGoogle Scholar
  23. 23.
    GRADE working group. http://www.gradeworkinggroup.org. Accessed 5 Mar 2016.
  24. 24.
    Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–6.CrossRefPubMedPubMedCentralGoogle Scholar
  25. 25.
    iSoF. The key information you need to understand the benefits and harms of treatments. http://isof.epistemonikos.org. Accessed 5 Mar 2016.

Copyright information

© Vandvik et al. 2016

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors and Affiliations

  • Per Olav Vandvik
    • 1
    • 2
  • Romina Brignardello-Petersen
    • 3
    • 4
  • Gordon H. Guyatt
    • 3
  1. 1.Department of MedicineInnlandet Hospital TrustGjøvikNorway
  2. 2.Norwegian Knowledge Centre for the Health ServicesUniversity of OsloOsloNorway
  3. 3.Department of Clinical Epidemiology & BiostatisticsMcMaster UniversityHamiltonCanada
  4. 4.Evidence-Based Dentistry UnitFaculty of Dentistry, Universidad de ChileSantiagoChile

Personalised recommendations