We will introduce our model in terms of the example elaborated in the previous section. Suppose we wanted to know the income of a particular household in late nineteenth century Philadelphia. Following Du Bois we distinguish four possible answers by introducing income brackets — less than $ 5 (per week), between $ 5 and $ 10, between $ 10 and $ 15, or more than $ 15. We also assume that one answer is in some (epistemic) sense superior to the others (call this the “correct” answer). In our example we will suppose the correct answer is “between $ 5 and $ 10”.
Three purist scholars set out to investigate the matter. One goes door to door asking whoever opens the door to report the household’s income. Another estimates the household’s income based on the profession(s) of those members of the household who work. And a third estimates their income based on the appearance of the house and its occupants.
First suppose that each of these methods has some positive connection with the correct answer. Say each method has, independently of the other methods, a 1 / 3 probability of yielding the answer “between $ 5 and $ 10”, and only a 2 / 9 probability each for each of the other three answers.
Now we introduce a final actor, the triangulator (modeled on Du Bois, except without giving favor to any particular method), who runs no investigation of her own, but adopts the strategy: pick whatever answer is triangulated upon, otherwise guess between any of the answers selected by at least one method. In this example, the triangulator has a 29 / 81 probability of getting the answer “between $ 5 and $ 10”. Since \(29/81 > 1/3\), the triangulator has a better chance of settling on the right answer than the purists.
It might be thought that this result is an artifact of the particular numbers we chose. Theorem 1 shows this suspicion to be mistaken. In order to state the theorem, we need a little more notation.
Suppose there are m methods \(a_1,\ldots ,a_m\) available to address a given question. The question has n possible answers \(b_1,\ldots ,b_n\), one of which is “correct”. Without loss of generality, suppose the correct answer is \(b_1\).
Each method, independently from the others, yields upon application one answer it endorses (we will call this the answer “picked” by that method). A method picks answer \(b_j\) with probability \(r_j\). The positive connection to the correct answer is represented by the assumption that \(r_1 > r_j\) for all \(j\ne 1\). So each method is more likely to pick the correct answer than it is to pick any given incorrect answer.
A purist picks a single method and always believes the answer picked by that method to be the correct answer. By assumption, then, the purist’s belief is correct with probability \(r_1\). A triangulator looks at the answers picked by all the methods available to her, and believes the answer picked by the greatest number of methods to be the correct one (if multiple answers are tied for being picked the most times, she picks a random answer among the tied ones to believe). Let \(p_j\) denote the probability that the triangulator ends up believing answer \(b_j\).
\(p_1 \ge r_1\) for all n and m. The inequality is strict whenever \(m\ge 3\) and \(n\ge 2\). Moreover, \(p_1\) is increasing in m.
This is a slightly strengthened version of List and Goodin (2001, proposition 1). A proof is available from the authors upon request.
So not only does a triangulator do better than a purist, a triangulator with more methods available also does better than a triangulator with less methods available. In fact, as the number of methods increases, it becomes virtually certain that the triangulator will get it right: \(p_1 \rightarrow 1\) as \(m\rightarrow \infty \) (List and Goodin 2001, proposition 2).
The above result arguably captures what Du Bois had in mind. Each method yields some evidence. Perhaps this evidence is not particularly strong on its own, but taken together the various methods can support a conclusion quite strongly. However, from the purist’s perspective it may seem that our analysis is rigged: we assumed that each method has some probabilistic connection to the correct answer (“the reliability of all the processes”, in Hudson’s terminology), whereas in reality (according to the purist) only the purist’s preferred method does. So let us now turn to that scenario.
As it turns out, suppose, asking people directly to report their income really is The One True Method, sure to give the correct answer (that the income is between $ 5 and $ 10), and the other two methodologies are more or less glorified guesswork (probability 1 / 4 of yielding each of the four possible answers).
Note that “guesswork” is the weakest possible assumption we can make about a method, as it entails that the results of this method provide no information whatsoever. If we made the “weaker” assumption of a negative connection with the correct answer (probability less than 1 / 4 of yielding the answer “between $ 5 and $ 10”) the method actually becomes potentially more useful: an “anti-triangulator” could use such a method to determine which answers are likely to be incorrect. We take the worst case scenario for a method to be that it is never more informative than guesswork. Further, since no opponent of triangulation has proposed using methods to knock out potential answers we assume guesswork is what they have in mind when they say other methods are bad.
In this case the triangulator has a 9 / 16 probability of settling on the answer “between $ 5 and $ 10”. She is doing worse than the purist who asks people to report their income directly (this purist gets the correct answer with probability 1) but better than the other two purists (who get the correct answer with probability 1 / 4).
What should we conclude from this? Obviously the triangulator is not doing as well as the first purist. So if we know that asking people to report their own income is The One True Method there is no reason to use methodological triangulation. This, we note, is consistent with Lahno (2014), who argues that if one is in certain kinds of evidential states one may do better by avoiding answers that have been triangulated upon. Similarly, in our model there are occasions where one does better not to use triangulation. But to know one is in the case Lahno discusses one has to have a good understanding of how well one’s methods respond to evidence of various sorts. Whereas we take it that if one was in a position to know exactly how it is one’s methods were responding to evidence, one would not be in a state of Du Boisian diffidence about them, and may even know which is The One True Method. As we shall now argue, it is when one is not sure about how one’s methods are responding to evidence that one should use triangulation.
For, if we are in a case of Du Boisian diffidence things are different. Even if we know that there is a true method and the other two are just guesswork, it is good to be a triangulator: the triangulator gets it right 9 out of 16 times, whereas guessing what the right method is and sticking with that one only gets it right 8 out of 16 times (\(1\cdot 1/3 + 1/4\cdot 2/3 = 1/2\)). Triangulation is a sensible response to ignorance about the performance of one’s own methods.
Here again one might worry that the result is a numerical artifact, but once again we can assuage this worry. Consider the same setup as before, except now there is a special \(m+1\)-st method (call it \(a_0\)) which always picks the correct answer (answer \(b_1\)), while the other m methods pick any answer with probability 1 / n.
The purist chooses a method at random; this reflects Du Boisian diffidence: the purist does not know which method is The One True Method. The purist then believes whatever answer that method picks to be the correct one. Not only does this guessing at the correct method represent a high degree of uncertainty, or Du Boisian diffidence, it also captures something about the present state of social scientific inquiry. In fields which are largely pre-paradigm there will be competing “schools”, and attendant competing methodologies. Plausibly this is the case in most of the social sciences. What method a scholar ends up using is largely determined by which school they get educated into, and this itself will be a function of choices they made as an undergrad and before, at points when they had no idea about the relative merits of competing schools and methodologies. This is effectively a kind of randomization, or at least may reasonably be modeled as such. While methodological purists may not consciously randomize between potential approaches, at least in the social sciences we think they very often are de facto choosing at random among the methods.
The triangulator, as before, believes whatever answer is picked by the most methods (randomizing in case of ties). Let \(p_j\) and \(q_j\) denote the probabilities of believing answer \(b_j\) for the triangulator and the purist respectively.
\(p_1 \ge q_1\) for all n and m. The inequality is strict whenever \(m\ge 2\) and \(n\ge 2\).
This result and Theorem 3 are proved in the Appendix.
We believe the above scenario is the most favorable possible scenario for the methodological purist, because it assumes that the purist’s preferred method is as good as it could possibly be and the other methods are as bad as they could possibly be. We hence think that showing that methodological triangulation can be valuable in this scenario is our strongest argument in triangulation’s favor. But it might still be objected that it is unrealistic that The One True Method delivers the correct answer with probability 1.
So now consider a case in which asking people to report their income directly (The One True Method) yields the answer “between $ 5 and $ 10” with probability 1 / 3 (2 / 9 each for the other three possible answers) while the other methods are random (1 / 4 for each answer). In this case the triangulator gets the answer “between $ 5 and $ 10” with probability 41 / 144. The triangulator does worse than the first purist (\(41/144 < 1/3\)) but better than the other two (\(1/4 < 41/144\)). Just as before, if a scientist is subject to Du Boisian diffidence triangulation is the way to go. In particular, triangulation does better than guessing a method and being a purist about that method (\(1/3 \cdot 1/3 + 1/4\cdot 2/3 = 40/144 < 41/144\)).
More generally, suppose that method \(a_0\) picks answer \(b_j\) with probability \(r_j\) and assume that \(r_1 > 1/n\) (so \(a_0\) favors \(b_1\) more than chance, although another answer might be favored even more). As before, the other methods pick randomly: any answer \(b_j\) has a 1 / n chance of being picked. \(p_j\) and \(q_j\) are defined as above.
\(p_1 \ge q_1\) for all n and m. The inequality is strict whenever \(m\ge 2\) and \(n\ge 2\).