Barriers to Evidence Synthesis

  • Kamran Ahmed
  • Nick Sevdalis
  • Ara Darzi
  • Thanos Athanasiou


In the hierarchy of research designs, randomised controlled trials and meta-analyses of randomised controlled trials are considered to be the highest level of evidence. They have been established as essential areas of research since their introduction into clinical sciences. Research in the interventional disciplines such as surgery, rely mostly on observational studies. Therefore, the quality and quantity of randomised trials with regards to interventions remain limited. Researchers in these disciplines face various obstacles during building, assessment or implementation of evidence. This chapter aims to provide a critical overview of the obstacles to randomised trials and meta-analyses. It also proposes solution to these problems.


Randomise Trial Publication Bias Bile Duct Injury Evidence Synthesis Everyday Clinical Practice 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342:1887-1892.CrossRefPubMedGoogle Scholar
  2. 2.
    MRC_Investigation. Streptomycin treatment of pulmonary tuberculosis: a Medical Research Council investigation. BMJ. 1948;2:769-782.CrossRefGoogle Scholar
  3. 3.
    Egger M, Smith GD, Phillips AN. Meta-analysis: principles and procedures. BMJ. 1997;315:1533-1537.PubMedGoogle Scholar
  4. 4.
    Huque MF. Experiences with meta-analysis in NDA submissions. Proc Biopharm Sect Am Stat Assoc. 1988;2:28-33.Google Scholar
  5. 5.
    Sacks H, Chalmers TC, Smith H Jr. Randomized versus historical controls for clinical trials. Am J Med. 1982;72:233-240.CrossRefPubMedGoogle Scholar
  6. 6.
    Farrokhyar F, Karanicolas PJ, Thoma A, et al. Randomized controlled trials of surgical interventions. Ann Surg. 2010;251:409-416.CrossRefPubMedGoogle Scholar
  7. 7.
    Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ. 1996;312:1215-1218.PubMedGoogle Scholar
  8. 8.
    Solomon MJ, McLeod RS. Should we be performing more randomized controlled trials evaluating surgical operations? Surgery. 1995;118:459-467.CrossRefPubMedGoogle Scholar
  9. 9.
    Shaw LW, Chalmers TC. Ethics in cooperative clinical trials. Ann NY Acad Sci. 1970;169:487-495.CrossRefPubMedGoogle Scholar
  10. 10.
    Weijer C, Shapiro SH, Cranley Glass K. For and against: clinical equipoise and not the uncertainty principle is the moral underpinning of the randomised controlled trial. BMJ. 2000;321:756-758.CrossRefPubMedGoogle Scholar
  11. 11.
    McCulloch P, Taylor I, Sasako M, Lovett B, Griffin D. Randomised trials in surgery: problems and possible solutions. BMJ. 2002;324:1448-1451.CrossRefPubMedGoogle Scholar
  12. 12.
    Barkun JS, Barkun AN, Sampalis JS, et al. Randomised controlled trial of laparoscopic versus mini cholecystectomy. The McGill Gallstone Treatment Group. Lancet. 1992;340:1116-1119.CrossRefPubMedGoogle Scholar
  13. 13.
    McMahon AJ, Russell IT, Baxter JN, et al. Laparoscopic versus minilaparotomy cholecystectomy: a randomised trial. Lancet. 1994;343:135-138.CrossRefPubMedGoogle Scholar
  14. 14.
    Neugebauer E, Troidl H, Kum CK, Eypasch E, Miserez M, Paul A. The E.A.E.S. Consensus development conferences on laparoscopic cholecystectomy, appendectomy, and hernia repair. Consensus statements – September 1994. The Educational Committee of the European Association for Endoscopic Surgery. Surg Endosc. 1995;9:550-563.PubMedGoogle Scholar
  15. 15.
    Ahmed K, Khan MS, Vats A, et al. Current status of robotic assisted pelvic surgery and future developments. Int J Surg. 2009;7:431-440.CrossRefPubMedGoogle Scholar
  16. 16.
    Nix J, Smith A, Kurpad R, Nielsen ME, Wallen EM, Pruthi RS. Prospective randomized controlled trial of robotic versus open radical cystectomy for bladder cancer: perioperative and pathologic results. Eur Urol. 2010;57:196-201.CrossRefPubMedGoogle Scholar
  17. 17.
    Ouriel K. The PIVOTAL study: a randomized comparison of endovascular repair versus surveillance in patients with smaller abdominal aortic aneurysms. J Vasc Surg. 2009;49:266-269.CrossRefPubMedGoogle Scholar
  18. 18.
    Solomon MJ, Laxamana A, Devore L, McLeod RS. Randomized controlled trials in surgery. Surgery. 1994;115:707-712.PubMedGoogle Scholar
  19. 19.
    Solomon MJ, McLeod RS. Surgery and the randomised controlled trial: past, present and future. Med J Aust. 1998;169:380-383.PubMedGoogle Scholar
  20. 20.
    Hall JC, Mills B, Nguyen H, Hall JL. Methodologic standards in surgical trials. Surgery. 1996;119:466-472.CrossRefPubMedGoogle Scholar
  21. 21.
    Diamond GA, Forrester JS. Clinical trials and statistical verdicts: probable grounds for appeal. Ann Intern Med. 1983;98:385-394.PubMedGoogle Scholar
  22. 22.
    Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet. 1991;337:867-872.CrossRefPubMedGoogle Scholar
  23. 23.
    Hochman JS, Sleeper LA, Godfrey E, et al. Should we emergently revascularize Occluded Coronaries for cardiogenic shock: an international randomized trial of emergency PTCA/CABG-trial design. The SHOCK Trial Study Group. Am Heart J. 1999;137:313-321.CrossRefPubMedGoogle Scholar
  24. 24.
    Hochman JS, Sleeper LA, White HD, et al. One-year survival following early revascularization for cardiogenic shock. JAMA. 2001;285:190-192.CrossRefPubMedGoogle Scholar
  25. 25.
    van der Linden W. Pitfalls in randomized surgical trials. Surgery. 1980;87:258-262.PubMedGoogle Scholar
  26. 26.
    Chalmers TC, Celano P, Sacks HS, Smith H Jr. Bias in treatment assignment in controlled clinical trials. N Engl J Med. 1983;309:1358-1361.CrossRefPubMedGoogle Scholar
  27. 27.
    McCulloch P. Developing appropriate methodology for the study of surgical techniques. J R Soc Med. 2009;102:51-55.CrossRefPubMedGoogle Scholar
  28. 28.
    Parikh D, Johnson M, Chagla L, Lowe D, McCulloch P. D2 gastrectomy: lessons from a prospective audit of the learning curve. Br J Surg. 1996;83:1595-1599.CrossRefPubMedGoogle Scholar
  29. 29.
    Testori A, Bartolomei M, Grana C, et al. Sentinel node localization in primary melanoma: learning curve and results. Melanoma Res. 1999;9:587-593.CrossRefPubMedGoogle Scholar
  30. 30.
    Bonenkamp JJ, Songun I, Hermans J, et al. Randomised comparison of morbidity after D1 and D2 dissection for gastric cancer in 996 Dutch patients. Lancet. 1995;345:745-748.CrossRefPubMedGoogle Scholar
  31. 31.
    Bonenkamp JJ, Hermans J, Sasako M, et al. Extended lymph-node dissection for gastric cancer. N Engl J Med. 1999;340:908-914.CrossRefPubMedGoogle Scholar
  32. 32.
    McCulloch P. D1 versus D2 dissection for gastric cancer. Lancet. 1995;345:1516-1517. author reply 1517–1518.PubMedGoogle Scholar
  33. 33.
    Sauerland S, Seiler CM. Role of systematic reviews and meta-analysis in evidence-based medicine. World J Surg. 2005;29:582-587.CrossRefPubMedGoogle Scholar
  34. 34.
    Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315:629-634.PubMedGoogle Scholar
  35. 35.
    Slim K, Raspado O, Brugere C, Lanay-Savary MV, Chipponi J. Failure of a meta-analysis on the role of elective surgery for left colonic diverticulitis in young patients. Int J Colorectal Dis. 2008;23:665-667.CrossRefPubMedGoogle Scholar
  36. 36.
    Hernandez AV, Walker E, Ioannidis JP, Kattan MW. Challenges in meta-analysis of randomized clinical trials for rare harmful cardiovascular events: the case of rosiglitazone. Am Heart J. 2008;156:23-30.CrossRefPubMedGoogle Scholar
  37. 37.
    Borzak S, Ridker PM. Discordance between meta-analyses and large-scale randomized, controlled trials. Examples from the management of acute myocardial infarction. Ann Intern Med. 1995;123:873-877.PubMedGoogle Scholar
  38. 38.
    Cappelleri JC, Ioannidis JP, Schmid CH, et al. Large trials vs meta-analysis of smaller trials: how do their results compare? JAMA. 1996;276:1332-1338.CrossRefPubMedGoogle Scholar
  39. 39.
    LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med. 1997;337:536-542.CrossRefPubMedGoogle Scholar
  40. 40.
    Bailar JC 3rd. The promise and problems of meta-analysis. N Engl J Med. 1997;337:559-561.CrossRefPubMedGoogle Scholar
  41. 41.
    Lau J, Ioannidis JP, Schmid CH. Summing up evidence: one answer is not always enough. Lancet. 1998;351:123-127.CrossRefPubMedGoogle Scholar
  42. 42.
    Lau J, Antman EM, Jimenez-Silva J, Kupelnick B, Mosteller F, Chalmers TC. Cumulative meta-analysis of therapeutic trials for myocardial infarction. N Engl J Med. 1992;327:248-254.CrossRefPubMedGoogle Scholar
  43. 43.
    Huston P, Naylor CD. Health services research: reporting on studies using secondary data sources. CMAJ. 1996;155:1697-1709.PubMedGoogle Scholar
  44. 44.
    Wachter KW. Disturbed by meta-analysis? Science. 1988;241:1407-1408.CrossRefPubMedGoogle Scholar
  45. 45.
    Abrams KR, Gillies CL, Lambert PC. Meta-analysis of heterogeneously reported trials assessing change from baseline. Stat Med. 2005;24:3823-3844.CrossRefPubMedGoogle Scholar
  46. 46.
    Ioannidis JP, Patsopoulos NA, Evangelou E. Uncertainty in heterogeneity estimates in meta-analyses. BMJ. 2007;335:914-916.CrossRefPubMedGoogle Scholar
  47. 47.
    Davey Smith G, Egger M, Phillips AN. Meta-analysis. Beyond the grand mean? BMJ. 1997;315:1610-1614.PubMedGoogle Scholar
  48. 48.
    Rosenberg MS. The file-drawer problem revisited: a general weighted method for calculating fail-safe numbers in meta-analysis. Evolution. 2005;59:464-468.PubMedGoogle Scholar
  49. 49.
    Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of two methods to detect publication bias in meta-analysis. JAMA. 2006;295:676-680.CrossRefPubMedGoogle Scholar
  50. 50.
    Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17:1-12.CrossRefPubMedGoogle Scholar
  51. 51.
    Oxman AD. Checklists for review articles. BMJ. 1994;309:648-651.PubMedGoogle Scholar
  52. 52.
    Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354:1896-1900.CrossRefPubMedGoogle Scholar
  53. 53.
    Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000;283:2008-2012.CrossRefPubMedGoogle Scholar
  54. 54.
    Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.CrossRefPubMedGoogle Scholar
  55. 55.
    Juni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282:1054-1060.CrossRefPubMedGoogle Scholar
  56. 56.
    Huedo-Medina TB, Sanchez-Meca J, Marin-Martinez F, Botella J. Assessing heterogeneity in meta-analysis: Q statistic or I2 index? Psychol Methods. 2006;11:193-206.CrossRefPubMedGoogle Scholar
  57. 57.
    Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21:1539-1558.CrossRefPubMedGoogle Scholar
  58. 58.
    Bailey KR. Inter-study differences: how should they influence the interpretation and analysis of results? Stat Med. 1987;6:351-360.CrossRefPubMedGoogle Scholar
  59. 59.
    Sevdalis N, Jacklin R. Interaction effects and subgroup analyses in clinical trials: more than meets the eye? J Eval Clin Pract. 2008;14:919-922.PubMedGoogle Scholar
  60. 60.
    Cook DI, Gebski VJ, Keech AC. Subgroup analysis in clinical trials. Med J Aust. 2004;180:289-291.PubMedGoogle Scholar
  61. 61.
    Mayer EK, Bottle A, Rao C, Darzi AW, Athanasiou T. Funnel plots and their emerging application in surgery. Ann Surg. 2009;249:376-383.CrossRefPubMedGoogle Scholar
  62. 62.
    Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. 2000;56:455-463.CrossRefPubMedGoogle Scholar
  63. 63.
    Gray SM. Knowledge management: a core skill for surgeons who manage. Surg Clin North Am. 2006;86:17-39. vii-viii.CrossRefPubMedGoogle Scholar
  64. 64.
    Howes N, Chagla L, Thorpe M, McCulloch P. Surgical practice is evidence based. Br J Surg. 1997;84:1220-1223.CrossRefPubMedGoogle Scholar
  65. 65.
    Kingston R, Barry M, Tierney S, Drumm J, Grace P. Treatment of surgical patients is evidence-based. Eur J Surg. 2001;167:324-330.CrossRefPubMedGoogle Scholar
  66. 66.
    Meakins JL. Evidence-based surgery. Surg Clin North Am. 2006;86:1-16. vii.CrossRefPubMedGoogle Scholar
  67. 67.
    Ergina PL, Cook JA, Blazeby JM, et al. Challenges in evaluating surgical innovation. Lancet. 2009;374:1097-1104.CrossRefPubMedGoogle Scholar
  68. 68.
    Knight T, Brice A. Librarians, surgeons, and knowledge. Surg Clin North Am. 2006;86:71-90. viii–ix.CrossRefPubMedGoogle Scholar
  69. 69.
    Glasziou P. Managing the evidence flood. Surg Clin North Am. 2006;86:193-199. xi.CrossRefPubMedGoogle Scholar
  70. 70.
    Lee MJ. Evidence-based surgery: creating the culture. Surg Clin North Am. 2006;86:91-100. ix.CrossRefPubMedGoogle Scholar
  71. 71.
    McCulloch P, Badenoch D. Finding and appraising evidence. Surg Clin North Am. 2006;86:41-57. viii.CrossRefPubMedGoogle Scholar
  72. 72.
    Ahmed K, Ashrafian H. Life-long learning for physicians. Science. 2009;326:227.CrossRefPubMedGoogle Scholar
  73. 73.
    Ahmed K, Jawad M, Dasgupta P, Darzi A, Athanasiou T, Khan MS. Assessment and maintenance of competence in urology. Nat Rev Urol. 2010;7:403-413.CrossRefPubMedGoogle Scholar
  74. 74.
    Arora S, Sevdalis N, Suliman I, Athanasiou T, Kneebone R, Darzi A. What makes a competent surgeon?: experts’ and trainees’ perceptions of the roles of a surgeon. Am J Surg. 2009;198:726-732.CrossRefPubMedGoogle Scholar
  75. 75.
    Satava RM, Gallagher AG, Pellegrini CA. Surgical competence and surgical proficiency: definitions, taxonomy, and metrics. J Am Coll Surg. 2003;196:933-937.CrossRefPubMedGoogle Scholar
  76. 76.
    Rothwell PM. Factors that can affect the external validity of randomised controlled trials. PLoS Clin Trials. 2006;1:e9.CrossRefPubMedGoogle Scholar
  77. 77.
    Marshall JC. Surgical decision-making: integrating evidence, inference, and experience. Surg Clin North Am. 2006;86:201-215. xii.CrossRefPubMedGoogle Scholar
  78. 78.
    Kraemer HC, Wilson GT, Fairburn CG, Agras WS. Mediators and moderators of treatment effects in randomized clinical trials. Arch Gen Psychiatry. 2002;59:877-883.CrossRefPubMedGoogle Scholar
  79. 79.
    Sevdalis N, McCulloch P. Teaching evidence-based decision-making. Surg Clin North Am. 2006;86:59-70. viii.CrossRefPubMedGoogle Scholar

Copyright information

© Springer London 2011

Authors and Affiliations

  • Kamran Ahmed
    • 1
  • Nick Sevdalis
  • Ara Darzi
  • Thanos Athanasiou
  1. 1.Department of Surgery and CancerImperial College LondonLondonUK

Personalised recommendations