Introduction

Much of the decision-making in orthopaedics rests on uncertain evidence. Well-designed randomized controlled trials frequently show no difference or a small and possibly unimportant differences between two treatments [8, 19]. Clinical Evidence [3] currently classifies 50% of 3000 common medical treatments as of “unknown effectiveness” and only 11% as proven beneficial (of the remainder 24% are likely to be beneficial, 7% a tradeoff between benefits and harms, 5% unlikely to be beneficial, and 3% likely to be ineffective or harmful). Although a dearth of evidence seems part of everyday practice, physician uncertainty regarding treatment could diminish patients’ health, because many come to their provider for confident guidance. The degree to which providers perceive uncertainty about what is and is not actually known about matters relevant to everyday orthopaedic practice has not, to our knowledge, been measured.

As surgeons gain experience, it is plausible that what they previously considered to be uncertain becomes less so. Recognition of uncertainty could be influenced by many other personal factors as well. There are some reports suggesting religion [1, 21, 24] and overconfidence are important factors [7, 25, 26]. As evidence might replace uncertainty, confidence in, and understanding of, what is published might also influence the recognition of uncertainty.

We therefore asked: (1) Does uncertainty and overconfidence bias decrease with years of practice? (2) What sociodemographic factors are independently associated with less recognition of uncertainty, in particular belief in God or other deity or deities, and how is atheism associated with recognition of uncertainty? (3) Do overconfidence bias, degree of trust in the orthopaedic evidence, and degree of statistical sophistication correlate independently with recognition of uncertainty?

Materials and Methods

Study Design

We approached all 706 participants of the Science of Variation Group (SOVG; an international collaboration of upper extremity surgeons) to complete the survey on the recognition of uncertainty of whom seven participants opted out from the SOVG (Appendix 1 [Supplemental material is available with the online version of CORR®.]). The SOVG is a collaboration that aims to study variation in the definition and treatment of human illness without financial incentives. The remaining 699 members represent mainly Western orthopaedic surgeons (86% [599] practice in Europe or North America), who are involved in teaching (80% [560]). Seventy percent specializes in orthopaedic traumatology (33% [229]) or hand and wrist surgery (37% [257]). Experience is relatively equally distributed, because half of the group has more than 10 years of experience (47% [329]). On joining, the SOVG participants gave approval to be approached for questionnaire studies. Participation in our study was optional; therefore, we did not acquire additional institutional review board approval. The first invitation was sent in the beginning of May 2015 and we sent reminders at 2 and 4 weeks.

Questionnaire Development

The survey was developed by us after a collaborative discussion about how uncertainty is treated in daily clinical orthopaedics. Our aim was to evaluate factors that we predicted would be associated with more or less uncertainty. The selected questions were the more relevant and provocative of a much larger group that we chose from. The survey was then reviewed by the American Association for Hand Surgery Research Listserv, a collaboration of hand surgeons who help each other with the design of research protocols, without financial incentives. After baseline characteristics, four questions were used to establish an overall recognition of uncertainty score. Subsequently, four questions determined the level of trust in the orthopaedic evidence base and three questions determined the level of confidence bias (confidence that one’s skill is greater than it actually is). Answers were provided on an ordinal scales; this way we could normalize final trait scores to a 0 to 10 score for comparison with 0 being the lowest and 10 being the highest possible score for each trait. Six multiple-choice questions gauged the respondent’s level of statistical understanding. This trait was also scored on a 0 to 10 scale with 0 meaning no correct answers and 10 answering all six questions correctly. After completion of the study we analyzed item correlations in their domain (convergent validity) and with other domains (divergent validity). No items had higher divergent than convergent validity. This means each item measures predominantly its own domain and does not more accurately capture any of the other domains. We also measured the Cronbach alpha of each domain. This is a measure of internal consistency; in other words, how closely related a set of items are as a group. If items are more closely related, they probably measure the same construct. Recognition of uncertainty (0.70) and confidence bias (0.75) had relatively high Cronbach alpha levels. The alpha levels were lower for statistical understanding (0.48) and trust in the orthopaedic evidence base (0.37) (Table 1). The survey was concluded by four additional questions regarding uncertainty in daily practice, of which the final open question asked participants why their confidence changed with time. These answers were categorized by two investigators (TT, SJ). Consensus on the categories was reached by discussion after independently analyzing and assigning themes to a subset of 50 suggestions. After this consensus, both investigators (TT, SJ) analyzed the remaining suggestions. In the final analysis we grouped improved diagnostic, surgical, and communication skills under experience because those are expected to improve with practice. Statements similar to “knowledge of what works in my hands” were categorized as anecdotal outcomes. We made a distinction between statements similar to “acquiring more knowledge” and “learned that there is an absence of knowledge.” Each participant’s answer could pertain to three categories.

Table 1 Assessing convergent and divergent validity of individual items with the five domains of our questionnaire
Full size table

Study Population

Of the 706 approached participants, 242 responded (34%). This does not represent a response rate per se, because many of the surgeons we email do not regularly participate, and the email addresses have not been confirmed. After excluding seven participants opting out from the SOVG, we found no difference in sex, practice years, or specialization between the remaining responders and nonresponders (n = 457). Responders were more likely to be European (nonresponders 23% [107 of 457] versus responders 37% [90 of 242], p < 0.001) and supervise trainees (nonresponders 76% [346 of 457] versus responders 88% [214 of 242] (Appendix 2 [Supplemental materials are available with the online version of CORR®.]).

Two hundred thirty-four (97%) completed the survey; incomplete surveys were excluded from multivariable analysis. Ninety-two percent (222) were men, and 51% (123) worked predominantly in academic practice. Approximately half (52% [126]) worked in North America followed by 37% (90) working in Europe. Sixty percent (146) were politically liberal or moderately liberal, and 32% (76) were conservative or very conservative. Sixty percent (146) believed in God, 8% (20) had no opinion, 17% (40) were agnostic, and 15% (36) were atheists. Only 8% (17) thought their confidence had decreased since graduate training (Table 2).

Table 2 Participants’ demographics and other questions.
Full size table

Statistical Analysis

We used frequencies to describe discrete variables; continuous variables are reported as means and SDs. For statistical analysis we grouped separately the affirming and disaffirming answers to “do you believe in God, any other deity/deities?” and “would you consider yourself a religious person?”

Fisher’s exact test was used to determine the differences between categorical variables. Unpaired Student’s t-test and one-way analysis of variance were used to determine the differences between continuous and dichotomous variables and Pearson correlations were used for two continuous variables (Table 3). All variables with p < 0.10 on explanatory bivariate analysis were entered in a multivariable linear regression model. We regarded a two-tailed p value < 0.05 to be significant.

Table 3 Uncertainty and explanatory variables
Full size table

A priori power analysis indicated that 185 participants would provide 0.80 power to detect a variable explaining 3% of the variability in recognition of uncertainty, assuming our full model with five predictors would explain 30% of the variability and alpha set at 0.05.

Results

Recognition of uncertainty in daily practice did not vary by years in practice (0–5 years, 3.2 ± 1.3; 6–10 years, 2.9 ± 1.3; 11–20 years, 3.2 ± 1.4; 21–30 years, 3.3 ± 1.6 years; p = 0.51) (Table 3); however, confidence bias did increase by years of practice (0–5 years, 6.2 ± 1.4; 6–10 years, 7.1 ± 1.3; 11–20 years, 7.4 ± 1.4; 21–30 years, 7.1 ± 1.2; p < 0.001). To the question of how confidence changed after graduate training, the majority who answered “decreased” were in the 21 to 30 years of practice group (0–5 years, 5% [four of 83]; 6–10 years, 6% [three of 53]; 11–20 years, 6% [four of 73]; 21–30 years, 19% [six of 33]; p = 0.038) (Appendix 3 [Supplemental material is available with the online version of CORR®.]). Of all participants, the majority stated that their current level of confidence was reached because of increased experience (58% [173 of 297]) and more knowledge (20% [58 of 297]) (Fig. 1). When assessing only the 21 statements of participants whose confidence had decreased, 43% (nine of 21) ascribed this to recognizing the limits of the orthopaedic evidence base (Fig. 2).

Fig. 1
figure 1

Participants provided 297 reasons for their current confidence level. These were categorized into eight groups.

Full size image
Fig. 2
figure 2

Participants with decreased confidence provided 21 reasons for this.

Full size image

After accounting for a potential interaction of confounding variables, we found that working in a multispecialty group, compared with academic practice, was weakly associated with less recognition of uncertainty (β regression coefficient, −0.53; 95% confidence interval [CI], −1.0 to −0.055; partial R2, 0.021; p = 0.029).

Belief in God or any other deity or deities also was weakly associated with less recognition of uncertainty (β, −0.57; 95% CI, −1.0 to −0.11; partial R2, 0.026; p = 0.015). Additional analysis showed statistical understanding, belief in God, and regarding oneself to be a religious person to be associated (multicollinear). On excluding those variables, we found that being an atheist was independently, weakly associated with greater recognition of uncertainty (β, 0.65; 95% CI, 0.20–1.11; partial R2, 0.035; p= 0.005) (Appendix 4 [Supplemental material is available with the online version of CORR®.]).

Greater confidence bias was somewhat correlated with less recognition of uncertainty (β, −0.26; 95% CI, −0.37 to −0.14; partial R2, 0.084; p < 0.001).

Greater trust in the orthopaedic evidence base also was weakly correlated with less recognition of uncertainty (β, −0.16; 95% CI, −0.26 to −0.058; partial R2, 0.040; p = 0.002).

Better statistical understanding was more strongly correlated with greater recognition of uncertainty (β, 0.25; 95% CI, 0.17–0.34; partial R2, 0.13; p < 0.001).

Our full model accounted for 29% of the variability in recognition of uncertainty (adjusted R2, 0.29) (Table 4).

Table 4 Multivariable analysis of factors associated with greater uncertainty
Full size table

Discussion

A lack of evidence seems to be part of everyday practice, affecting approximately 50% of common conditions in medicine [3]. Physician uncertainty regarding treatment may adversely affect patients’ health. The aim of our study was to evaluate the acknowledgment of uncertainty by orthopaedic surgeons and to establish factors that might affect how surgeons regard uncertainty in daily orthopaedic practice. Working in a multispecialty group, belief in God or any other deity or deities, greater confidence bias, and greater trust in the orthopaedic evidence base were weakly associated with less recognition of uncertainty. Greater statistical understanding was more strongly associated with greater recognition of uncertainty.

This study has some limitations. First, surgeons of the SOVG are a subgroup in the community of orthopaedic surgeons. They represent Western orthopaedic surgeons of whom the majority supervises trainees. Using them as study subjects may have resulted in selection bias, because non-Western orthopaedic surgeons, or surgeons not involved in teaching, are not represented. This limits the generalizability of our results. Second, only 34% of the SOVG members filled out our survey. This is not a response rate per se, because many of the surgeons we email do not regularly participate, and the email addresses have not been confirmed. Also, we found no gross difference between responders and nonresponders. Third, when we gauged the reliability of our nonvalidated survey, we found that each survey item measured its own trait better than any of the other traits. Also recognition of uncertainty and confidence bias had relatively high Cronbach alpha levels; in other words, the questions making up these traits are closely related and probably measure the same construct. However, this was lower for statistical understanding and trust in the orthopaedic evidence base. There is no benchmark score for recognition of uncertainty, trust in the orthopaedic evidence base, confidence bias, and statistical understanding in the orthopaedic literature. Interpretation of these scores is a matter of preferences and values until evidence suggests that specific opinions or attitudes are beneficial.

The degree to which surgeons regard uncertainty does not seem to change with greater experience. We found a pervasive overconfidence bias (confidence that one’s skill is greater than it actually is) among orthopaedic surgeons, because 83% of the group considered themselves to be above average diagnosticians, and none regarded themselves as below average. Similarly, 74% of our group regarded themselves as above average surgeons and 25% regarded themselves as being in the top 5%. Although it is possible that this is accurate—maybe the surgeons in this study group are above average—this seems more likely attributable to overconfidence bias. There is evidence that confidence and accuracy are at odds in medicine. For example, radiologists who performed less well were highly confident that they were accurate [20]; a survey of 100 internal medicine physicians found only a very small difference in confidence in diagnostic accuracy between very difficult and simple clinical cases, whereas there was a large difference in actual diagnostic accuracy [16], and surgical residents were confident they would recognize different distal radius fractures 68% of the time while actually identifying only 33% correctly [18]. Overconfidence bias can lead to other biases such as the availability heuristic (considering only the first thing that comes to mind) and confirmation bias, where a person notices only the things that agree with his or her point of view and is less attentive to support for alternative viewpoints [15]. Limiting confirmation bias in medical decision-making requires an effort to seek disconfirming evidence, a characteristic of type 2 (analytical, reflective, slow) rather than type 1 (fast, intuitive, heuristic) thinking disposition [7, 25, 26]. Uncertainty-intolerant physicians might be less likely to use analytical thinking, contributing to their sense of overconfidence.

Surgeons working in academic centers had the greatest awareness of uncertainty, especially compared with those working in multispecialty groups. This might be because academicians are more exposed to teaching, writing papers, and complex cases and as such might be more likely to be required to identify the foundations in evidence that underlie their propositions. In that search, they may develop a more acute awareness of the many gaps in our shared knowledge base [3, 17].

One of the more interesting findings in our study was that we found that theists (believers in God or another deity) showed less recognition of uncertainty in daily orthopaedic practice. Conversely, being an atheist was independently associated with greater recognition of uncertainty. The effect of theistic belief on uncertainty was as strong as the other effects we studied. Although at first glance a physician’s personal view on God and religion might seem to be irrelevant to how medical evidence is regarded, our data suggest that an association may exist. A plausible explanation could be that the highly theistic desire certainty and that desire is fulfilled with their belief and that same desire for certainty could carry over to a desire for certainty in medical decision-making. There are some reports supporting the contention that intolerance of uncertainty is a quality of people who are highly religious [1, 21, 24]. There are also abundant explanations given in the popular literature relating theism to certainty [4, 5]. Furthermore, it is argued that believers are more likely to see uncertainty as a failure of knowledge rather than as an intrinsic fact of the world [9]. On these views, theism as a denial of uncertainty is in contradistinction to a view in which truth is only approximated and all knowledge and theories are subject to revision based on additional evidence.

Greater confidence bias was correlated with less uncertainty. Orthopaedic surgeons in this study group showed a confidence bias, which seems discordant with the paucity of available evidence. The way surgeons process areas of debate and uncertainty can affect their justification for medical decisions and recommendations. A study describing factors influencing recommendations for surgery in scenarios where evidence for a optimal treatment is limited found that justifications such as “works in my hands” and “familiarity with the treatment” are prevalent [12]. The Dartmouth Atlas [23] showed rates of procedures that are associated with more uncertainty vary more than procedures that have more evidential support in the literature. Physicians, and surgeons in particular may naturally have difficulty with uncertainty because of the need to satisfy patient questions and because many patients find confidence comforting. Patients likely would prefer and probably ethically should be aware of the boundaries of medicine’s knowledge as part of informed decision-making. For this to happen physicians need to be aware of and should be able to concisely articulate these boundaries to patients with various levels of health literacy. Balanced and dispassionate patient information tools such as decision aids might help surgeons inform their patients. Previous authors reported that expressing uncertainty does not diminish patient satisfaction [11], and patients provided with decision aids often are more satisfied with their care [22].

Greater trust in the orthopaedic evidence base was associated with less uncertainty. To some, trust in the orthopaedic evidence base may fulfill a role in health care similar to that of religion in everyday life. Published studies serve as an anchor to reduce some of the experienced uncertainty. However, a large part of what is published may not be true [14], and even highly cited studies often are refuted with time [13]. Having unwaivering faith in the orthopaedic evidence therefore is not warranted and a healthy skepticism seems a more appropriate attitude, something journal clubs on how to critique the literature might help to achieve.

We found greater recognition of uncertainty in those who scored higher on statistical sophistication. Conceivably, subjects who are more versed in statistical understanding also are more exposed to the concept of uncertainty. However, even among subjects who are more cognizant of uncertainty, the overall level of statistical understanding was low, a finding commonly noted in other studies [2, 6, 27]. Physicians’ lack of understanding of health statistics is thought by some to be one of the major impediments to healthcare efficiency [10]. Differences in tolerance for uncertainty and deficient understanding of statistics might have measurable effects on care. For example, those less tolerant of uncertainty may be more inclined to order more diagnostic tests, raising costs, but also unwittingly introducing more false-positive and -negative results. The added uncertainty that each test brings might lead to erroneous decisions. Future research might address this possibility.

Our data suggest the possibility that intrinsic and extrinsic factors outside our immediate awareness may affect how surgeons regard uncertainty. Our aim is to draw attention to this and to show that there are variations in how aware we are of the prevalence of uncertainty in our daily lives as surgeons. Recognition of our tolerance or intolerance of uncertainty can influence how we communicate with our patients. A deficiency in recognizing uncertainty and a propensity toward overconfidence might impede adequate informed consent and incorporation of patients’ preferences. Furthermore, this intolerance of uncertainty could lead to ordering more diagnostic tests, thereby raising costs and increasing the risk of unhelpful treatments, a thesis that merits additional study. Perhaps improved statistical teaching in residency, journal clubs to improve the critique of evidence and awareness of bias, and acknowledgment of knowledge gaps at courses and conferences might create awareness about existing uncertainties.