Appraising Evidence

  • Valentina Pecoraro


Appraising the quality of evidence is an essential step in the systematic review of diagnostic accuracy studies development. This procedure allows for appropriate interpretation of results and conclusions. Specific tools are available to report and evaluate the diagnostic accuracy studies. Once important aspect of methodology is the quality of reporting, that when inadequate restricts the comprehension of the study conduction and results. Whereby, the STARD checklist was developed to improve the quality of reporting. The quality assessment of studies included in a systematic review is required to identify possible source of bias and limit their effect on study results. The QUADAS-2 is the recommended and structured tool for diagnostic test accuracy studies. Likewise, method for assessing the quality of evidence is well recognized. The GRADE approach provides a transparent method for rating the quality of evidence. In this chapter, specific tolls were discussed to appraising evidence when performing systematic reviews on diagnostic accuracy studies.


Evidence Bias Critical appraisal Reporting Methodological quality 


  1. 1.
    de Groot JA, Bossuyt PM, Reitsma JB, Rutjes AW, Dendukuri N, Janssen KJ, Moons KG. Verification problems in diagnostic accuracy studies: consequences and solutions. BMJ. 2011;343:d4770.CrossRefPubMedGoogle Scholar
  2. 2.
    Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Bossuyt P, Chang S, Muti P, Jaeschke R, Guyatt GH. GRADE: assessing the quality of evidence for diagnostic recommendations. Evid Based Med. 2008;13:162–3.CrossRefPubMedGoogle Scholar
  3. 3.
    Schmidt RL, Factor RE. Understanding sources of bias in diagnostic accuracy studies. Arch Pathol Lab Med. 2013;137:558–65.CrossRefGoogle Scholar
  4. 4.
    Whiting P, Rutjes AW, Dinnes J, Reitsma JB, Bossuyt PM, Kleijnen J. A systematic review finds that diagnostic reviews fail to incorporate quality despite available tools. J Clin Epidemiol. 2005;58:1–12.CrossRefPubMedGoogle Scholar
  5. 5.
    Manchikanti L, Derby R, Wolfer L, Singh V, Datta S, Hirsch JA. Evidence-based medicine, systematic reviews, and guidelines in interventional pain management: part 5. Diagnostic accuracy studies. Pain Physician. 2009;12:517–40.Google Scholar
  6. 6.
    Reitsma JB, Moons KG, Bossuyt PM, Linnet K. Systematic reviews of studies quantifying the accuracy of diagnostic tests and markers. Clin Chem. 2012;58:1534–45.CrossRefPubMedGoogle Scholar
  7. 7.
    Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, Bossuyt PM. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999;282:1061–6.CrossRefPubMedGoogle Scholar
  8. 8.
    Rutjes AW, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, Bossuyt PM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ. 2006;174:469–76.Google Scholar
  9. 9.
    Mower WR. Evaluating bias and variability in diagnostic test reports. Ann Emerg Med. 1999;33:85–91.CrossRefPubMedGoogle Scholar
  10. 10.
    Whiting PF, Rutjes AW, Westwood ME, Mallett S, QUADAS-2 Steering Group. A systematic review classifies sources of bias and variation in diagnostic test accuracy studies. J Clin Epidemiol. 2013;66:1093–104.Google Scholar
  11. 11.
    Whiting P, Rutjes AW, Reitsma JB, Glas AS, Bossuyt PM, Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med. 2004;140:189–202.CrossRefPubMedGoogle Scholar
  12. 12.
    Cook C, Cleland J, Huijbregts P. Creation and critique of studies of diagnostic accuracy: use of the STARD and QUADAS methodological quality assessment tools. J Man Manip Ther. 2007;15:93–102.CrossRefGoogle Scholar
  13. 13.
  14. 14.
    Roever L. Types of bias in studies of diagnostic test accuracy. Evid Based Med Pract. 2016;1:e113.Google Scholar
  15. 15.
    Mulherin SA, Miller WC. Spectrum bias or spectrum effect? Subgroup variation in diagnostic test evaluation. Ann Intern Med. 2002;137:598–602.CrossRefPubMedGoogle Scholar
  16. 16.
    Willis BH. Spectrum bias—why clinicians need to be cautious when applying diagnostic test studies. Fam Pract. 2008;25:390–6.CrossRefPubMedGoogle Scholar
  17. 17.
    Kohn MA, Carpenter CR, Newman TB. Understanding the direction of bias in studies of diagnostic test accuracy. Acad Emerg Med. 2013;20:1194–206.CrossRefPubMedGoogle Scholar
  18. 18.
    Chien T, Malhotra R, Bhandari M. The 3-min appraisal of a diagnostic test. Indian J Orthop. 2011 Sep;45:389–91.CrossRefPubMedPubMedCentralGoogle Scholar
  19. 19.
    Reitsma JB, Rutjes AWS, Whiting P, Vlassov VV, Leeflang MMG, Deeks JJ. Chapter 9: assessing methodological quality. In: Deeks JJ, Bossuyt PM, Gatsonis C, editors. Cochrane handbook for systematic reviews of diagnostic test accuracy version 1.0.0: The Cochrane Collaboration; 2009. Available from: Accessed 28 June 2018.
  20. 20.
    Manikandan R, Dorairajan LN. How to appraise a diagnostic test. Indian J Urol. 2011;27:513–9.CrossRefPubMedPubMedCentralGoogle Scholar
  21. 21.
  22. 22.
    Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, Michie S, Moher D, Wager E. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383:267–76.CrossRefGoogle Scholar
  23. 23.
    Bossuyt PM. The quality of reporting in diagnostic test research: getting better, still not optimal. Clin Chem. 2004;50:465–6.CrossRefPubMedGoogle Scholar
  24. 24.
    Cohen JF, Korevaar DA, Altman DG, Bruns DE, Gatsonis CA, Hooft L, Irwig L, Levine D, Reitsma JB, de Vet HC, Bossuyt PM. STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration. BMJ Open. 2016;6:e012799.CrossRefPubMedPubMedCentralGoogle Scholar
  25. 25.
    Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, Lijmer JG, Moher D, Rennie D, de Vet HC, Kressel HY, Rifai N, Golub RM, Altman DG, Hooft L, Korevaar DA, Cohen JF, STARD Group. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527.CrossRefPubMedPubMedCentralGoogle Scholar
  26. 26.
    Ochodo EA, Bossuyt PM. Reporting the accuracy of diagnostic tests: the STARD initiative 10 years on. Clin Chem. 2013;59:917–9.CrossRefPubMedGoogle Scholar
  27. 27.
    Korevaar DA, Wang J, van Enst WA, Leeflang MM, Hooft L, Smidt N, Bossuyt PM. Reporting diagnostic accuracy studies: some improvements after 10 years of STARD. Radiology. 2015;274:781–9.CrossRefPubMedGoogle Scholar
  28. 28.
    Pecoraro V, Banzi R, Trenti T. Quality of reporting of diagnostic test accuracy studies in medical laboratory journals. Clin Chem Lab Med. 2016;54:e319–21.Google Scholar
  29. 29.
    Leeflang MM, Deeks JJ, Gatsonis C, Bossuyt PM, Cochrane Diagnostic Test Accuracy Working Group. Systematic reviews of diagnostic test accuracy. Ann Intern Med. 2008;149(12):889–97.CrossRefPubMedPubMedCentralGoogle Scholar
  30. 30.
    Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, Leeflang MM, Sterne JA, Bossuyt PM, QUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155:529–36.CrossRefPubMedGoogle Scholar
  31. 31.
    Whiting P, Rutjes AW, Reitsma JB, Bossuyt PM, Kleijnen J. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol. 2003;3:25.CrossRefPubMedPubMedCentralGoogle Scholar
  32. 32.
    Whiting PF, Weswood ME, Rutjes AW, Reitsma JB, Bossuyt PN, Kleijnen J. Evaluation of QUADAS, a tool for the quality assessment of diagnostic accuracy studies. BMC Med Res Methodol. 2006;6:9.CrossRefPubMedPubMedCentralGoogle Scholar
  33. 33.
    Whiting P, Harbord R, Kleijnen J. No role for quality scores in systematic reviews of diagnostic accuracy studies. BMC Med Res Methodol. 2005;5:19.CrossRefPubMedPubMedCentralGoogle Scholar
  34. 34.
    Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Jaeschke R, Vist GE, Williams JW Jr, Kunz R, Craig J, Montori VM, Bossuyt P, Guyatt GH, GRADE Working Group. Grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ. 2008;336:1106–10.CrossRefPubMedPubMedCentralGoogle Scholar
  35. 35.
    Gopalakrishna G, Mustafa RA, Davenport C, Scholten RJ, Hyde C, Brozek J, Schünemann HJ, Bossuyt PM, Leeflang MM, Langendam MW. Applying Grading of Recommendations Assessment, Development and Evaluation (GRADE) to diagnostic tests was challenging but doable. J Clin Epidemiol. 2014;67:760–8.CrossRefPubMedGoogle Scholar
  36. 36.
    Brozek JL, Akl EA, Jaeschke R, Lang DM, Bossuyt P, Glasziou P, Helfand M, Ueffing E, Alonso-Coello P, Meerpohl J, Phillips B, Horvath AR, Bousquet J, Guyatt GH, Schünemann HJ, GRADE Working Group. Grading quality of evidence and strength of recommendations in clinical practice guidelines: Part 2 of 3. The GRADE approach to grading quality of evidence about diagnostic tests and strategies. Allergy. 2009;64:1109–16.CrossRefPubMedGoogle Scholar
  37. 37.
    Balshem H, Helfand M, Schünemann HJ, Oxman AD, Kunz R, Brozek J, Vist GE, Falck-Ytter Y, Meerpohl J, Norris S, Guyatt GH. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64:401–6.CrossRefPubMedGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Valentina Pecoraro
    • 1
  1. 1.Laboratory of Toxicology, Department of Laboratory Medicine and Pathological AnatomyAzienda USL of ModenaModenaItaly

Personalised recommendations