An ill-supported but commonly bandied-about claim holds that fewer than 20% of common medical interventions are supported by scientific evidence. Although seemingly reputable sources continue to repeat this old chestnut [7], the data behind the original claim reside in a congressional report that is nearly 40 years old [2], and newer scholarship suggests that things have improved considerably in this regard [3]. All the same, ancient observations about gaps in our knowledge base—such as that offered by William Osler more than 100 years ago—still seem relevant: “The greater the ignorance, the greater the dogmatism” [11].

figure 1

David Ring MD, PhD

What really do surgeons know, and how do we know it? As importantly, how do we react when we do not know the answers? David Ring MD, PhD—who just moved from Harvard University to become Associate Dean for Comprehensive Care at the new Dell Medical School in Austin, Texas, USA—surveyed the membership of the Science of Variation Group (SOVG) to find out. If you are not familiar with this group’s work, seek it out; its investigators ask big, fun, thought-provoking questions. Recent manuscripts using SOVG data have included “Do Surgeons Treat Patients Like They Would Treat Themselves” [5] and “How Prevalent are Hazardous Attitudes Among Orthopaedic Surgeons” [1], the latter having been covered with an interview in this space last year [8]. In this latest offering, Dr. Ring takes up Osler’s ignorance/dogmatism equation, by looking at factors associated with what the investigators called “overconfidence bias.”

This study evaluated a host of factors from years in practice to the depth of a surgeon’s religious conviction (if any) and created a model that accounted for nearly one-third of the observed variation in a surgeon’s reaction to uncertainty in clinical practice. Surgeons in multispecialty groups were somewhat less likely to recognize uncertainty compared to those in academic practices, as were surgeons who endorsed some measure of religious faith (compared to self-identified atheists). Interestingly “faith” in the evidence base (to be fair, they called it “trust”) also was associated with less recognition of uncertainty, while greater statistical expertise was more likely to be seen among surgeons who are more sensitive to uncertainty.

As a journal editor, my work exposes me to the occupational hazard of firsthand exposure to raw data without protective equipment, which is not something I necessarily would wish on others. In this instance, though, a deeper dive into the supplemental material rewards the effort: The authors found, for example, that 20% of those surveyed believed that they were in the top 5% of their specialties in terms of diagnostic skills, 83% believed they were above average, and none self-identified as below average. Lake Wobegon, where “all the women are strong, all the men are good looking, and all the children are above average” [6], has nothing on this crowd.

Of course, naming a problem—particularly one that William Osler pointed to in 1902—is not the same as solving it, and if this detailed model explained 29% of the variance among providers, then 71% remains unexplained. But this paper is as rigorous an approach to a critically important topic as I have read, and this topic is as important as any in contemporary medicine. Join me in the Take-5 interview that follows, where we go behind the discovery with Dr. Ring, the senior author of “Do Orthopaedic Surgeons Acknowledge Uncertainty?”

Take Five Interview with David Ring MD PhD, senior author of “Do Orthopaedic Surgeons Acknowledge Uncertainty?”

Seth S. Leopold MD: Congratulations on this fascinating paper. Before we go further, though, please explain to readers why you think the findings you observed represent actual bias. After all, the surgeons from the SOVG who participated in your survey are experts by any definition, and even the least experienced among them have been through a decade of education and training after graduating from university.

David Ring MD, PhD: Overconfidence bias (when one’s subjective confidence in judgments is greater than the objective accuracy of those judgments) is one of the most important human biases. If we assume that SOVG members are more aware of the best available evidence in orthopaedic surgery, we might expect them to be less certain. It is often stated that wisdom is knowing what you do not know. As Ioannidis [4] has pointed out, the accumulation of more experience and better evidence usually leads us to the conclusion that treatments do not work as well as we earlier had thought. In my own personal growth as a surgeon-scientist, I have found that many, if not most, of the variations, debates, and conundrums in orthopaedic surgery can be understood on the basis of the inherent uncertainties and probabilities in medicine. These include the lack of a consensus reference standard for many common diseases, the magnified influence of imperfect diagnostic performance characteristics in low prevalence settings, and the myriad explanations for an apparent treatment effect (regression to the mean, the self-limiting course of many illnesses, and the placebo effect all being towards the top of that list).

Dr. Leopold: Each factor in your model accounted for a relatively small portion of the observed relationship; it would not seem correct for a patient to say, apply a litmus test about religious conviction to a provider based on your findings. How do you see this?

Dr. Ring: My research collaborations with economics colleagues have taught me to value any signal in the noise. Given the complexity of human uncertainty and confidence, statistical models that account for 29% of the variation in fact are quite compelling. I might not emphasize a particular factor that accounted for just a few percentage points of the variation in uncertainty, but it seems safe to conclude that—collectively—things like confidence bias, faith, and less statistical sophistication are associated with less recognition of uncertainty. In the quality and safety world this might be framed as: Less recognition that “to err is human” creates greater risk that mistakes will lead to harm. This is the essence of the scientific method, which humans invented to avoid fooling ourselves and to avoid being fooled by others. Put simply: We should cultivate our curiosity.

Dr. Leopold: Ironically, “faith” (or, as you called it, “trust”) in the evidence base behaved somewhat similarly in your model. Why is this finding important, and how might gender or culture factor in here? It is said that all men believe they are better-than-average drivers and better-than-average (ahem) kissers. More than 90% of your respondents were men (and, interestingly, none believed themselves below-average diagnosticians). Likewise, your sample evaluated only western-trained surgeons.

Dr. Ring: The answer to a survey question about trust in the evidence base may be measuring faith and confidence in authority more so than it measures whether or not a surgeon does or does not use treatments that are unsupported by evidence (such as corticosteroid injection for tennis elbow). And while there likely are some variations by race, gender, and culture—such as those that you and others [9, 10] have observed—I believe it is safe to assume that the associations we observed represent basic human traits that would remain relatively consistent in other settings.

Dr. Leopold: If you had a health condition, how would your findings influence how you might converse with the physician who is caring for you?

Dr. Ring: It is no surprise that the art of medicine has a lot to do with the preferences and values of the physician. And I would be curious about how the physician caring for me evaluates the evidence and the uncertainty in it. But in the end, I would want my preferences and values to determine management. I am a fan of attempts to provide complete, balanced, dispassionate, and hopeful information to patients in the form of a decision aid. As a patient, I see a decision aid as a way of getting multiple opinions all at once, in language that I can understand, in a form that I can review repeatedly, with language that anticipates my vulnerabilities as a patient, and with the primary goal of helping me determine my preferences based on current best evidence and the range of available options and opinions.

Dr. Leopold: How will you approach resident education differently based on your findings, and, on a more “macro” level (but still as specifically as you can), how might our specialty deal with your discoveries to mitigate overconfidence bias among orthopaedic surgeons?

Dr. Ring: As a science-based profession, orthopaedic surgeons should be encouraged and trained to value and recognize uncertainty. We should build checklists and systems to catch errors before they cause harm; expect or our impressions (and those of the patient) to sometimes be contradicted by objective, reproducible evidence; and work as teams on difficult diagnostic and treatment dilemmas.