In a study 1000 people were tested. Among the participants there were 5 engineers and 995 lawyers. Jack is a randomly chosen participant of this study.

Jack is 36 years old. He is not married and is somewhat introverted. He likes to spend his free time reading science fiction and writing computer programs.

What is the likelihood that Jack is a Lawyer?

How much influence does personality have on the types of careers people pursue in a given society? In a highly free society, where individuals are left to choose their own career paths, one could imagine that personality directly causes certain individuals to choose certain occupations. In the problem which opened this article, if having to make a wager as to whether a randomly sampled individual is a lawyer or engineer, learning about that person’s personality may be highly informative. Of course, this must be taken in the context that out of 1000 people sampled, 995 of them are lawyers. The degree to which these pieces of information are informative depends heavily on the underlying causal mappings between personality and occupation that operate in a given society. If personality causes a person to have a particular job, it is extremely informative, but if there is no causal link between personality and occupation, it may be completely uninformative.

Like the example above, classic base-rate problems are thought to capture the influence of individuating information on statistical reasoning (Tversky & Kahneman, 1974, 1982; Bar-Hillel, 1980). For these types of problems, base-rate neglect is the degree to which people tend to ignore or underweight statistical information in favour of individuating information. For those prone to base-rate neglect, the fact that 995 members of a study population are engineers is given relatively little weight in deciding which job Jack holds, while the personality description is treated as if it is wholly informative (Bar-Hillel, 1980; Kahneman & Tversky, 1973). A popular explanation for the neglect of statistical information has been the notion of causality. That is, a random sample did not cause Jack to be an engineer, while his personality leading him to have interests in computers may have. To better understand the role of causality in these problems, consider a new version of the lawyer-engineer problem. “Jane is randomly selected from a group of 995 men and 5 women. Jane is pregnant. How likely is Jane to be a woman?”. Obviously, in this extreme example, the base-rate is completely irrelevant, since only biological females can be pregnant, hence, Jane has to be a woman regardless of the proportions of the sample. In this example, there is a strong causal mapping between the state of pregnancy and the qualitative information concerning Jane.

Multiple models with roles for causality of information have been forwarded in the literature. One is the explanation view, which posits that causal information is given special status in making judgements. That is, people make use of a causality heuristic, which holds that an event is more probable if a clear cause can be identified (Ajzen, 1977). Base-rate neglect emerges in this view, because people underemphasize base-rate information because they judge it as being strictly non-causal. If such base-rate information can be presented such that it appears to be causal, people stop neglecting it (Ajzen, 1977). It is an open question whether non-causal quantitative information is ever integrated in the presence of qualitative descriptions. A model as proposed by Ajzen (1977) predicts that if no qualitative information is present, or if that information is of low causal relevance, quantitative information will instead be used. An alternative view in the literature would hold that even in the absence of causally-relevant qualitative information, non-causal base-rates will still not be used (Bes, Sloman, Lucas, & Raufaste, 2012). A final view in causality and judgment posits that the integration of qualitative and quantitative information occurs in three steps: the first is to a construct a causal model, second is to assign parameters, and finally to infer probabilities. This model in particular should be sensitive to manipulations that target underlying causal relations between personality and occupation (Krynski & Tenenbaum, 2007). Using a manipulation involving participants imagining societies with very different causal mappings between personality and occupation, we test these models of a causality driven base-rate neglect.

Study 1

Method

Participants

A sample of 300 participants was collected from Amazon’s Mechanical Turk with each participant receiving $1.25 upon completion of a 10-minute online questionnaire. Participants were recruited under the condition that they be U.S. residents and possess a Mechanical Turk HIT approval rate greater than or equal to 95%. Our participants were on average 36 years old (SD = 11), male (59%), white (83%) and completed some amount of post-secondary education (89%).

Procedure

Participants were randomly assigned to one of three between-subject conditions. These were: 1) No Instruction Condition: participants were provided no additional information about what type of society they were making their judgments in. 2) Causal Personality Environment Condition: participants were told that they were making a decision in a society where the government assigns jobs based on the results of a personality assessment. 3) Non-Causal Personality Environment Condition: participants were told they were making a decision in a society where the government assigns jobs randomly without any regard for an individual’s personality.

All participants were presented with six base-rate problems (adapted from Obrecht & Chesney, 2016; see supplementary materials). Each participant judged six scenarios, an example of which can be found at the opening of this article. In three of the scenarios, the stereotype and base-rate information were congruent (i.e., Hank is selected from 997 writers and the stereotype also suggests that he is a writer). In the other three, stereotype and base-rate information were incongruent (i.e., Hank is selected from 997 construction workers while the stereotype suggests that he is a writer), in all cases the base-rate information was non-causal in nature. Dependent on condition, as described above, participants may have received an additional statement describing what sort of society they were to make their judgments in. Participants submitted a judgment on a six-point scale indicating which occupation group they believed the subject in the scenario (e.g., Hank) belonged to. These options ranged from “very confident [Hank] is a [construction worker]” to “very confident [Hank] is a [writer]”. Responses were interpreted as being in line with the individuating information or base-rate respectively if they chose any of the three options toward that end of the scale. Scenario order was randomized between subjects. After all of the assigned scenarios were completed, participants responded to the question: “In the society you live, to what extent do you think a person’s personality determines the sort of job they will hold?” on a five point scale ranging from “no impact” to “completely determines their job” (henceforth referred to as “Occupation-Personality Correspondence”). This question was designed to assess participants’ general belief in the correspondence between personality and job type in their own society. Furthermore, we assessed Numeracy using a scale taken from Weller, Dieckmann, Tusler, Mertz, Burns, and Peters (2013; see supplement for Numeracy analyses). Lastly, participants answered several demographic questions.

Results & Discussion

As expected, participants responded more in line with base-rates during congruent trials compared to incongruent trials, F(1, 297) = 279.20, p < .001, ηp2 = .490. Additionally, base-rate use differed as a function of instruction condition, F(2, 297) = 7.32, p = .001, ηp2 = .047. Further, we observed an instruction and problem type interaction, F(2, 297) = 17.83, p < .001, ηp2 = .11, such that instructional manipulations only affected incongruent problems. For trials in which the stereotype information and base-rate were incongruent, the number of responses in line with the base-rate differed across conditions (see Table 1). Responding in line with the base-rate was highest in the Non-Causal Personality Environment, where participants were told to make the decision in a society where personality had a low impact on the types of occupations people hold. Conversely, base-rate responding was lowest in the Causal Personality Environment Condition, where participants were told to make the decision about a society where personality solely determined what occupations people hold. In the No Instruction Condition, where no additional information was provided as to the type of society participants were making their decisions in, responses in line with the base-rate landed between the two environmental conditions (see Figure 1). Importantly, there were no differences between conditions for trials in which base-rate and stereotype information were congruent.

Table 1 Comparisons between the 3 relevancy conditions for average responding in line with the base-rate for conflict problems.
Full size table
Fig. 1
figure 1

Mean number of times participants responded in line with the base-rate based on condition and base-rate-stereotype congruency. Error-bars: ± 1 SE.

Full size image

For incongruent problem types, a negative correlation was observed in the No Instruction Condition between responding in line with the base-rate and Occupation-Personality Correspondence, r(101) = -.26, p = .007. This suggests that participants used their beliefs about their own society when responding to base-rate problems when no societal information was provided. This finding is consistent with work on other classes of base-rate problems which demonstrate the impact worldly beliefs have on Bayesian reasoning (Cohen, Sidlowski, & Staub, 2017). In the current study, participants appear to be relying on the causal link between qualitative information and life outcomes that they perceive in their own world in answering base-rate problems. No associations between base-rate responses and Occupation-Personality Correspondence were detected in either our Non-Causal or Causal Personality Environment conditions (all ps > .099), suggesting that participants’ beliefs about their own society may not have impacted their judgments when they were asked to reason about hypothetical societies clearly unlike their own.

When thinking about an environment where qualitative information was causally relevant, participants responded in line with qualitative information. Conversely, when thinking about an environment where qualitative information was of low causal relevance, participants instead more heavily emphasised base-rate statistics. Thus, people appear to be sensitive to the causal relevancy of qualitative information in the environment. We show here that, under the right circumstances, people are capable of deemphasising this information in favour of base-rate statistics.

While we believe the effects observed in Study 1 were due to participants being in tune with the causal relevance of qualitative information in their environment, an alternative explanation would be that our manipulation of the society type was too transparent, and resulted in demand characteristics driving the effects. Specifically, when submitting a judgement on incongruent trials, participants may have been able to guess what we (the experimenters) wanted and responded as such without honestly considering the problem. To rule out this possibility, we conducted a second study.

Study 2

Study 2 included two new instructional conditions designed to be explicitly demanding, as a comparison against the instructions used in Study 1. We also attempted to measure participants’ experience of demand directly and use this to predict responses in line with the base-rate across all conditions. If demand is driving the effects in Study 1, a correlation should be observed between responses in line with the base-rate and perceptions of demand.

Method

Participants

A sample of 499 participants was collected from Amazon’s Mechanical Turk with the exact same criteria as Study 1. Our participants were on average 38 years old (SD = 12), male (53%), white, (74%) and completed some amount of post-secondary education (80%). They could not have participated in Study 1 to be eligible.

Procedure

Study 2 was nearly identical to Study 1 with the exception of two alterations. In addition to the Causal Personality Environment, Non-Causal Personality Environment, and No Instruction conditions, participants could have also been assigned to a Use Statistics or Use Belief condition. These two new conditions (adapted from Pennycook, Trippas, Handley & Thompson, 2014) were chosen because we judged these instructions to be heavy-handed, pressuring participants to respond in a certain manner. These likely-to-be demanding instructions provide a clear comparison to the original three conditions in Study 1. In the Use Statistics condition, participants were asked to answer according to statistics for all problems, while in the Use Belief condition, participants were instructed to answer according to their beliefs about the world. All participants were presented with six base-rate problems as in Study 1.

The second alteration was that after the base-rate problems, participants were asked a question probing their experience of demand. In a procedure adapted from Lord, Lepper and Preston (1984), participants were given a sample conflict base-rate problem with the full set of problem instructions they had received earlier. Participants answered the following: “Did the wording of the problem or instructions cause you to feel any pressure to choose one of the options? (For example, if you thought the researchers wanted you to choose one over the other)”. They provided a rating on a scale ranging from -3 (A lot of pressure to choose [stereotype option] to 3 (A lot of pressure to choose [base-rate option]). After completing the demand rating task, participants responded to the same Occupation-Personality Correspondence, Numeracy, and Demographics questions as in Study 1.

Results & Discussion

Participants responded more in line with the base-rate during congruent trials compared to incongruent trials, F(1, 494) = 450.01, p < .001, ηp2 = .477. Additionally, base-rate use differed as a function of instruction condition F(1, 494) =10.01, p < .001, ηp2 = .075. Again as in Study 1, problem type and instructions condition interacted, F(4, 494) = 8.65, p < .001. Specifically, when responding to congruent trials, participants responded in a similar manner regardless of their instruction condition, F(4, 498) = 1.06, p = .378. For incongruent problems, responses in line with the base-rate differed based on instruction condition, F(4, 498) = 12.01, p < .001, a pattern of results that is consistent with Study 1. This effect was followed up with a Student-Newman-Keuls post-hoc test which demonstrated a difference between the Causal Personality Environment and Non-Causal Personality Environment conditions (p < .05) but did not detect a difference between either of these two conditions and the No Instruction condition (Causal Personality Environment vs No Instruction, p = .108; Non-Causal Personality Environment vs No Instruction, p =.138). Responses in line with the base-rate in the Use Statistics condition were different from all other conditions (p < .05), while the Use Belief condition was different from the Non-Causal Personality Environment condition but not detectably different from the Causal Personality Environment nor No Instruction conditions (see Figure 2). This indicates participants responded in line with the base-rate the most in the Use Statistics condition, and the least in the Non-Causal Environment as well as Use Belief conditions. The Causal Environment condition had the second most responses in line with the base-rate, while the neutral No Instruction condition fell between the Causal and Non-Causal Environment conditions. Consistent with Study 1, we observe a significant correlation between Occupation-Personality Correspondence belief and responses in line with the base-rate for the No Instruction condition r(99) = .20, p = .041. This again demonstrates that participants may use beliefs about their own society when approaching base-rate problems in the absence of additional manipulations.

Fig. 2
figure 2

Average number of times participants responded in line with the base-rate based on condition and base-rate-stereotype congruency. Error-bars: ± 1 SE

Full size image

Demand analysis

For incongruent trials, a significant difference in demand ratings based on instruction condition was detected, F(4, 498) =12.31, p < .001. A Student-Newman-Keuls test placed only the Use Statistics condition in its own subset (p < .05), all other conditions were placed within the same category as the neutral, No Instruction condition suggesting that only the Use Statistics condition differed in the amount of perceived demand compared to when no instructional manipulation was provided (see Table 2). Bivariate correlations were computed for each Instruction condition testing the degree to which demand predicted responses in line with the base-rate. A demand account of our results requires that if subjects were feeling pressured by their instruction-set to respond in a certain manner then a significant correlation between demand and base-rate responses ought to exist. However, it was only in the Use Statistics condition that demand was predictive of participants’ responses, r(97) = -.45, p < .001. In other words, in none of our original experimental conditions did participants’ experienced demand predict their responding.

Table 2 Summary statistics for base-rate use and perceived demand ratings in Study 2. Pearson r values represent the correlation between base-rate use and perceived demand ratings
Full size table

In Summary, Study 2 replicated the overall pattern of results between our three main conditions of interest. Importantly, these results did not appear to be caused by a demand effect as experienced demand was not predictive of responding in line with the base-rate for any of our main conditions of interest.

General Discussion

Across two experiments we demonstrate that people’s use of quantitative information (e.g., base-rates) when making a judgment varies as the causal link of qualitative information (e.g., stereotypes) changes. That is, when a clear causal link for stereotypes is provided, people make judgments that are far more in line with them. When the causal link is diminished, people readily incorporate non-causal base-rates into their judgments. This supports the view that reasoners, as a rule, do not assume that all qualitative information is automatically causal in nature, and will deemphasise it when it is non-causal. Further, while our results support the overall claim that causality is a fundamental aspect in how people make judgments, our results challenge the notion that quantitative data is only used when such data is casually relevant during the judgment process. We suggest, rather, that people will use and integrate all of the information that is provided to them to make a judgment (Białek, 2017; Pennycook, Trippas, Handley, & Thompson, 2014), though they will strongly prioritize information with clear causal relevance.

The causality of information has current widespread acceptance for its role as the primary factor during judgment, yet its exact function during the process is debated (e.g., Ajzen, 1977; Bes, Sloman, Lucas, Raufaste, 2012; Krynski & Tenenbaum, 2007). One model suggests that people first assess the causal relevance of available information, assign probabilities weighted by this causal relevance, and then finally enter these adjusted probabilities into a Bayesian framework (Krynski & Tenenbaum, 2007; cf. Hayes, Ngo, Hawkins, & Newell, 2018). Ajzen (1977), suggests that people use a causality heuristic, such that information that is causally relevant is preferred whenever available. In particular, he suggests that qualitative information will tend to be preferred in the absence of causal quantitative information due to qualitative information being perceived to carry a degree of causality. If quantitative information can be made to appear causal, reasoners will begin to make use of it. Bes et al. (2012) agree regarding the existence of a causality heuristic, but additionally suggest that people may not make use of quantitative data at all, even if casually relevant. Our data are compatible with these models but challenge the notion surrounding quantitative data. In particular, we provide direct evidence that people will incorporate non-causal quantitative information into their responses when the alternative is low-causal qualitative data.

When approaching reasoning problems, people tend to have their decisions influenced by their prior beliefs about the world. During syllogistic reasoning for example, belief bias occurs when one’s prior beliefs about the world affect one’s ability to judge logical validity (Klauer, Musch, & Naumer, 2000; Newstead, Pollard, Evans, & Allen, 1992). Likewise, when a person’s prior beliefs match up with normative Bayesian prescriptions, they can experience an advantage on Bayesian reasoning problems such as judging the probability that a person has breast cancer (Cohen, Sidlowski, & Staub, 2017). Consistent with this, we found that Personality-Society Correspondence responses were predictive of performance in base-rate problems. This suggests that people do not complete these problems in a naïve way. Instead, it appears that participants rely on already constructed casual mappings between personalities and occupations that are generated by their real-world experiences (Tentori, Crupi, & Russo, 2013; Tentori & Crupi, 2012). Thus, we support the notion that one reason why base-rate neglect occurs is because people’s decisions are significantly impacted by the prior beliefs they hold, which they tend not to discard in answering constructed hypothetical reasoning problems (also see, Benjamin, Bodoh-Creed & Rabin, 2019; Grether, 1992; Welsh & Navarro, 2012). What we show, is that if a human were transplanted into a brand new environment with different causal rules from our own, their behaviour would be radically different. So perhaps base-rate neglect is less of a fundamental bias of our decision making processes and instead is the output of a flexible system sensitive to the underlying causal structures of the environment it inhabits.

Open Science Statement

The data and materials for the experiments are available at https://osf.io/ugba8/ .