Vindicating methodological triangulation

Abstract

Social scientists use many different methods, and there are often substantial disagreements about which method is appropriate for a given research question. In response to this uncertainty about the relative merits of different methods, W. E. B. Du Bois advocated for and applied “methodological triangulation”. This is to use multiple methods simultaneously in the belief that, where one is uncertain about the reliability of any given method, if multiple methods yield the same answer that answer is confirmed more strongly than it could have been by any single method. Against this, methodological purists believe that one should choose a single appropriate method and stick with it. Using tools from voting theory, we show Du Boisian methodological triangulation to be more likely to yield the correct answer than purism, assuming the scientist is subject to some degree of diffidence about the relative merits of the various methods. This holds even when in fact only one of the methods is appropriate for the given research question.

Introduction

Methodological pluralism is an entrenched fact of life for the working social scientist. There exist a variety of methods of carrying out social scientific work that are actually applied in the course of various research projects. While the contrast between quantitative and qualitative methods is the most striking, one can find methodological difference within those categories as well. For instance, ethnographic participant observation and hermeneutic textual analysis are distinct yet equally qualitative methods, whereas Bayesian and frequentist statistics provide different methods of running quantitative analysis.

It is not clear whether the fact of methodological pluralism is beneficial to social science. One optimistic response, which we shall defend in this paper, is to devise strategies for exploiting methodological pluralism to bolster the reliability of results obtained in the social sciences. Such strategies have come to be called methodological triangulation. The idea behind methodological triangulation is that the convergence of multiple methods upon a single conclusion better supports that conclusion than just one of those methods arriving at the conclusion. Against this, however, pessimists might think that methodological pluralism is both a result and a source of confusion in the social sciences, and thus be unmoved by the advocacy of triangulation. After all, somebody who deduces that $2 + 2 = 4$ need not have their confidence bolstered by the fact that somebody who says that the sum of any two numbers is 4 has converged on the same answer as them in this case. Nor should they be concerned by their lack of triangulation with the person who always says “5”. To somebody who sees methodological pluralism as arising from widespread methodological error, it may therefore be unclear why methodological triangulation should be beneficial.

There is indeed a persistent vein of skepticism about methodological triangulation running through the literature. A class of theorists we term “methodological purists” argue that in order to understand any given phenomenon there is one method that should be used at the exclusion of others (McEvoy and Richards 2006, p. 68). There are, typically, two sorts of arguments for this. The first is that different methods are often based on such wildly different presuppositions that any attempt to combine them can only lead to mischief or confusion. Kelle summarized this view as follows: “[r]esearch methods are often developed within differing research traditions carrying varying epistemological and theoretical assumptions with them. Thus the combination of methods...[will] not lead to more valid results” (Kelle 2005, p. 99; see also Blaikie 1991, p. 115; Sim and Sharp 1998, p. 27). Sim and Sharp (1998, p. 26) claim that to avoid issues such as this one would have to decide in favor of one method and its accompanying theory. Since the fact of methodological pluralism in the social sciences is partially the result of theorists being unable to decide which paradigm to adopt, this would not bode well for methodological triangulation. Contrary to this, our argument for methodological triangulation shall not depend on methods having shared presuppositions beyond assuming that they can be addressed to the same questions.

The second sort of argument rests upon the sheer difficulty of actually simultaneously running multiple methodologies (cf. Farmer et al. 2006). This has led some to go so far as to argue that “using several different methods can actually increase the chance of error” (Kelle 2005, p. 99), since overtaxed scholars will be more haphazard in their work. Contrary to this, our argument for methodological triangulation shall not require that one individual is doing the work of applying the various methods, instead adopting the position of an agent able to survey the results of multiple lines of inquiry, which may have been carried out by separate research teams.

Recently, Hudson has offered a book-length critique of the idea that methodological triangulation can be exploited to increase the reliability of results obtained in scientific research (Hudson 2014). Especially relevant to our purposes in this paper are two arguments he develops therein. First, he argues that a number of purported cases of methodological triangulation being used to great success in the sciences are not in fact examples thereof, and so cannot be used as empirical evidence for the success of the strategy. Second, he argues that probabilistic arguments for methodological triangulation work by effectively arguing that the causal process underlying the various methods triangulating upon a given result are independent and as such that their convergence can only be explained by “the reliability of all the processes that generate this report, along with the presumption that the report is true” (Hudson 2014, p. 24).

Our argument responds to both of these. First, as will be made clear in Sect. 2, our model of methodological triangulation is closely based on an actual instance of methodological triangulation being deployed in a classic work of social science, one not considered in the previous literature on methodological triangulation that Hudson responds to in his book. Second, the results we prove very explicitly do not rely on any notion of independence that can be leveraged into an argument for the reliability of the underlying methods triangulating upon the result in question. We will show that even assuming the opposite of this (explicitly granting that some methods are essentially randomization devices) one can still sometimes do better through triangulation.

Stegenga (2012) has criticized defenders of triangulation for cherry-picking from the history of science. He points out that when multiple methods are addressed to the same question this frequently yields “discordant” evidence: different answers are supported by different methods. Defenders of triangulation have only addressed cases of “concordant” evidence in which all methods agree on the answer, according to Stegenga. In contrast, the results we prove apply both to cases of concordant and discordant evidence, so our argument avoids this objection.

Finally, note that another motivation for methodological purism would be the conviction that one’s favored method is simply epistemically superior, or, at least, epistemically superior when applied to some particular class of problems. Of course, while that may motivate methodological purism, it is unlikely by itself to persuade those of different methodological predilections. Hence, one rarely finds the conviction expressed in so naked a form in the literature. That said, it is not difficult to find works by partisans of qualitative versus quantitative methodology, or vice versa, in which they argue for their preferred style of research (see Bryman 1984 for a review and Tewksbury 2009 for a recent example). Hence it is worth explicitly noting this source of support for methodological purism, as sheer preference for one method over another is plausibly what motivates many in their methodological purism. Against this, we shall argue that the recognition that one method is superior should not by itself motivate methodological purism.

In order to respond to this skepticism about the merits of triangulation we outline a formal model of methodological triangulation in Sect. 3. This model is designed to be an abstraction from an actual use of methodological triangulation by Du Bois (1996 [1899], to be described in detail in Sect. 2), while at the same time remaining maximally generous to the opponent of methodological triangulation. Within our model there are multiple methods being run simultaneously to ascertain which of several propositions ought to be believed. We then show that under a variety of scenarios favorable to the purist, including scenarios more pessimistic in their appraisal of rival methods than any actual purists are likely to countenance, methodological triangulation still provides a good guide to truth, providing one exhibits what we call Du Boisian diffidence, as discussed below. That is to say, there are reasons for an observer of a process of inquiry who is not sure which method to trust to none the less assent to the proposition which has been endorsed by multiple methods. The formal tools we use for this investigation are borrowed from voting theory, and more particularly the literature surrounding Condorcet’s Jury Theorem (Grofman et al. 1983; List and Goodin 2001). We rely on some existing results and prove some new ones. We conclude in Sect. 4 by suggesting lines of future research.

There have long been practicing social scientists who have thought that methodological pluralism was an exploitable resource. W. E. B. Du Bois, writing in the 1890s, is perhaps the earliest example of a scholar advocating methodological triangulation (Wortham 2005). He claimed that pluralism could be exploited to overcome the fact that “the methods of social research are at present so liable to inaccuracies that the careful student discloses the results of individual research with diffidence” (Du Bois 1996 [1899], p. 2). We therefore say that a scholar is in a state of Du Boisian diffidence just in case they are not confident which (if any) of various competing methodologies to trust. Du Bois thought that the use of multiple methods to study the same problem “may perhaps have corrected to some extent the errors of each” (Du Bois 1996 [1899], p. 3), but he did not outline why this should be. We take ourselves to be providing the mathematical foundations for Du Bois’ insight, explaining why triangulation works in the type of situation he found himself in.

Other social scientists have followed Du Bois in making use of methodological triangulation in their work (e.g., Farrall et al. 1997; Cunningham et al. 2000; Mangan et al. 2004; Jack and Raturi 2006). Furthermore, discussions in the philosophy of climate science (e.g., Parker 2011) and philosophy of biology (e.g., Weisberg and Reisman 2008) suggest that it is not just social scientists who make use of triangulation in their work. Although the focus of our argument here is on the social sciences, triangulation may be beneficial in other fields for the same reasons.

The social scientific literature by itself now contains a multitude of types of “methodological triangulation”, each with their own rationale (for review see Thurmond 2001). Hence, although triangulation has been criticized in ways we mentioned above, we are certainly not the first to argue that triangulation “allows researchers to be more confident of their results” (Jick 1979, p. 608). Except in so far as they explicitly deny the ability of triangulation to provide additional confirmatory support for a hypothesis, we do not consider our arguments in tension with these alternate accounts of the benefits of triangulation. We are thus open to the possibility that there are additional benefits to methodological triangulation.

The tradition of work closest to ours in defending methodological triangulation is that which has implicitly or explicitly appealed to confirmation theory. At least as far back as Hempel, confirmation theorists have acknowledged that “the confirmation of a hypothesis depends not only on the quantity of the favorable evidence available, but also on its variety: the greater the variety, the stronger the resulting support” (Hempel 1966, p. 34). Further, while philosophers dispute the concept’s precise meaning, some scholars who discuss Whewell’s notion of “consilience” interpret this in line with the idea that triangulation increases confirmatory support (Laudan 1971; Fisch 1985; Snyder 2005; for application see Leung and van de Vijver 2008). More recently, Fredericks and Miller (1988, p. 350) argue that Carnappian confirmation theory explains how it is that triangulation upon a proposition serves to increase one’s rational degree of confidence in that proposition. Risjord et al. (2001, 2002) have even argued in the other direction, using the phenomenon of methodological triangulation to support a coherentist theory of confirmation. Finally, Bayesian theorists have developed results within their framework about the benefits of independent sources of evidence which are closely related to our discussion (e.g., Fitelson 2001; Claveau 2013). We advance on this previous work by providing a formal argument in favor of methodological triangulation which does not rely on any specifically Bayesian assumptions, thus avoiding the critiques that have been leveled at such assumptions (Stegenga 2012, Appendix).

Du Bois’ use of triangulation

The model we develop in the next section provides a mathematical foundation for, and generalization of, the form of methodological triangulation actually deployed by Du Bois. To evince this claim, and to illustrate methodological triangulation at work in a piece of classic social scientific research, we discuss an especially explicit example of methodological triangulation at work in Du Bois’ The Philadelphia Negro.

Du Bois had carried out an exhaustive series of door-to-door surveys and interviews with all (or almost all) households in the predominantly Negro Seventh Ward of Philadelphia (to avoid anachronism, we follow Du Bois’ terminology in using “Negro” rather than the more contemporary “African American”). With this information in hand, he asked “What do Negroes earn?” In particular, Du Bois was attempting to discern how many Negro households fall within various income brackets. Immediately upon raising the question he conceded “Such a question is difficult to answer with anything like accuracy. Only returns based on actual written accounts would furnish thoroughly reliable statistics; such accounts cannot be had in this case” (Du Bois 1996 [1899], p. 168). Instead, Du Bois had available to him four methods: (1) direct estimations of income offered by families during interviews, (2) information based on combined average income for the professions represented in a given household, (3) family members’ estimations of time lost to work, given their occupation, and (4) the apparent circumstances of the family judging from the appearance of the home and occupants, rent paid, presence of lodgers, etc. (Du Bois 1996 [1899], p. 169). However, doubts and reservations are expressed about the reliability of all four of these methods of discerning household income (Du Bois 1996 [1899], pp. 169–170). As such, Du Bois makes it explicit that he is in a situation of what we call Du Boisian diffidence: he does not know which of the available methods will yield reliable answers to the question he is asking.

Faced with this problem, the procedure Du Bois adopted was as follows. For each household, he deployed all four methods, and gave an estimation of income based on “three or more” of the four methods just described (Du Bois 1996 [1899], p. 169). That is to say, Du Bois used the methods as providing a kind of vote on a household’s income bracket, and where a strict majority of methods agreed he placed the household in the agreed-upon bracket. We note that where Du Bois says that “in most cases, the first item was given greatest weight in settling the matter; but was modified by the others” (Du Bois 1996 [1899], p. 169), we interpret that as meaning that this was a weighted voting procedure, with favor given to method (1).

This, then, was Du Bois’ application of methodological triangulation in a clear case of Du Boisian diffidence. Two weaknesses stand out in this procedure. First, Du Bois never gives any good argument that the agreement of a majority of methods, each admitted to be of dubious reliability, is any reason to be confident in a given income bracket classification. For all Du Bois said, it is not obvious that this procedure of triangulation actually helps given the epistemic situation he faced.

We note that this is not the only occasion on which Du Bois appealed to triangulation when in a situation of Du Boisian diffidence. In Du Bois (2000, a 1905 essay which did not appear in print until 2000) much time is spent laying out the difficulties human free will creates for discovering and confirming the existence of sociological laws, hence giving us cause for some Du Boisian diffidence in sociology. In response to this, Du Bois again praises a multi-method approach to studying human society, saying that “our knowledge of human life has been vastly increased by Statisticians, Ethnologists, Political Scientists, Economists, Students of Finance and Philanthropy, Criminologists, Educators, Moral Philosophers, and critics of art and literature” (Du Bois 2000, p. 42). In fact, his critique of these studies was that there has not been enough attempt at triangulation between them, as he bemoaned the “lack of adequate recognition of the essential unity in the various studies of human activity, and of effort to discover and express that unity” (Du Bois 2000, p. 43). Once again, however, while Du Bois plainly does think people should try and connect up the results of various approaches to sociological inquiry, he does not clearly state what the advantage in doing so would be. A consistent feature of his work thus seems to be that Du Bois advocated triangulation as a methodological response to diffidence, but did not offer clear or explicit argument in favor of this response.

The second weakness that stands out in Du Bois’s use of triangulation is that it is not clear how to generalize it. Suppose that of his four methods, (2) and (4) suggested the income of a given household was between $ 10 and $ 15 (per week), (1) suggested the income of said household was between $ 5 and $ 10, and (3) put the income at more than $ 15. On our reading of the text, which is admittedly unclear on this point, Du Bois would have two options available to him in such a scenario. Given his stated policy of giving greater weight to method (1), he could put the household’s income down as “between $ 5 and $ 10”, despite the other methods agreeing that the household is not in this bracket. But then why not generally just deploy method (1), since it is apparently trusted enough to overrule a unanimous judgment of all other methods (that the income is greater than $ 10)? The other response available to Du Bois is to say that here the procedure simply fails to give an answer as to the household’s income, and Du Bois must throw away the data point.

From the text it is not clear if Du Bois ever faced such scenarios nor, if he did, how he responded. The model we develop in the next section solves both these problems, and illustrates that Du Bois’ procedure is capable of being placed on secure foundations, while also yielding a general response to the situation of the diffident inquirer.

The model

We will introduce our model in terms of the example elaborated in the previous section. Suppose we wanted to know the income of a particular household in late nineteenth century Philadelphia. Following Du Bois we distinguish four possible answers by introducing income brackets — less than $ 5 (per week), between $ 5 and $ 10, between $ 10 and $ 15, or more than $ 15. We also assume that one answer is in some (epistemic) sense superior to the others (call this the “correct” answer). In our example we will suppose the correct answer is “between $ 5 and $ 10”.

Three purist scholars set out to investigate the matter. One goes door to door asking whoever opens the door to report the household’s income. Another estimates the household’s income based on the profession(s) of those members of the household who work. And a third estimates their income based on the appearance of the house and its occupants.

First suppose that each of these methods has some positive connection with the correct answer. Say each method has, independently of the other methods, a 1 / 3 probability of yielding the answer “between $ 5 and $ 10”, and only a 2 / 9 probability each for each of the other three answers.

Now we introduce a final actor, the triangulator (modeled on Du Bois, except without giving favor to any particular method), who runs no investigation of her own, but adopts the strategy: pick whatever answer is triangulated upon, otherwise guess between any of the answers selected by at least one method. In this example, the triangulator has a 29 / 81 probability of getting the answer “between $ 5 and $ 10”. Since $29/81 > 1/3$, the triangulator has a better chance of settling on the right answer than the purists.

It might be thought that this result is an artifact of the particular numbers we chose. Theorem 1 shows this suspicion to be mistaken. In order to state the theorem, we need a little more notation.

Suppose there are m methods $a_1,\ldots ,a_m$ available to address a given question. The question has n possible answers $b_1,\ldots ,b_n$, one of which is “correct”. Without loss of generality, suppose the correct answer is $b_1$.

Each method, independently from the others, yields upon application one answer it endorses (we will call this the answer “picked” by that method). A method picks answer $b_j$ with probability $r_j$. The positive connection to the correct answer is represented by the assumption that $r_1 > r_j$ for all $j\ne 1$. So each method is more likely to pick the correct answer than it is to pick any given incorrect answer.

A purist picks a single method and always believes the answer picked by that method to be the correct answer. By assumption, then, the purist’s belief is correct with probability $r_1$. A triangulator looks at the answers picked by all the methods available to her, and believes the answer picked by the greatest number of methods to be the correct one (if multiple answers are tied for being picked the most times, she picks a random answer among the tied ones to believe). Let $p_j$ denote the probability that the triangulator ends up believing answer $b_j$.

Theorem 1

$p_1 \ge r_1$ for all n and m. The inequality is strict whenever $m\ge 3$ and $n\ge 2$. Moreover, $p_1$ is increasing in m.

This is a slightly strengthened version of List and Goodin (2001, proposition 1). A proof is available from the authors upon request.

So not only does a triangulator do better than a purist, a triangulator with more methods available also does better than a triangulator with less methods available. In fact, as the number of methods increases, it becomes virtually certain that the triangulator will get it right: $p_1 \rightarrow 1$ as $m\rightarrow \infty$ (List and Goodin 2001, proposition 2).

The above result arguably captures what Du Bois had in mind. Each method yields some evidence. Perhaps this evidence is not particularly strong on its own, but taken together the various methods can support a conclusion quite strongly. However, from the purist’s perspective it may seem that our analysis is rigged: we assumed that each method has some probabilistic connection to the correct answer (“the reliability of all the processes”, in Hudson’s terminology), whereas in reality (according to the purist) only the purist’s preferred method does. So let us now turn to that scenario.

As it turns out, suppose, asking people directly to report their income really is The One True Method, sure to give the correct answer (that the income is between $ 5 and $ 10), and the other two methodologies are more or less glorified guesswork (probability 1 / 4 of yielding each of the four possible answers).

Note that “guesswork” is the weakest possible assumption we can make about a method, as it entails that the results of this method provide no information whatsoever. If we made the “weaker” assumption of a negative connection with the correct answer (probability less than 1 / 4 of yielding the answer “between $ 5 and $ 10”) the method actually becomes potentially more useful: an “anti-triangulator” could use such a method to determine which answers are likely to be incorrect. We take the worst case scenario for a method to be that it is never more informative than guesswork. Further, since no opponent of triangulation has proposed using methods to knock out potential answers we assume guesswork is what they have in mind when they say other methods are bad.

In this case the triangulator has a 9 / 16 probability of settling on the answer “between $ 5 and $ 10”. She is doing worse than the purist who asks people to report their income directly (this purist gets the correct answer with probability 1) but better than the other two purists (who get the correct answer with probability 1 / 4).

What should we conclude from this? Obviously the triangulator is not doing as well as the first purist. So if we know that asking people to report their own income is The One True Method there is no reason to use methodological triangulation. This, we note, is consistent with Lahno (2014), who argues that if one is in certain kinds of evidential states one may do better by avoiding answers that have been triangulated upon. Similarly, in our model there are occasions where one does better not to use triangulation. But to know one is in the case Lahno discusses one has to have a good understanding of how well one’s methods respond to evidence of various sorts. Whereas we take it that if one was in a position to know exactly how it is one’s methods were responding to evidence, one would not be in a state of Du Boisian diffidence about them, and may even know which is The One True Method. As we shall now argue, it is when one is not sure about how one’s methods are responding to evidence that one should use triangulation.

For, if we are in a case of Du Boisian diffidence things are different. Even if we know that there is a true method and the other two are just guesswork, it is good to be a triangulator: the triangulator gets it right 9 out of 16 times, whereas guessing what the right method is and sticking with that one only gets it right 8 out of 16 times ($1\cdot 1/3 + 1/4\cdot 2/3 = 1/2$). Triangulation is a sensible response to ignorance about the performance of one’s own methods.

Here again one might worry that the result is a numerical artifact, but once again we can assuage this worry. Consider the same setup as before, except now there is a special $m+1$-st method (call it $a_0$) which always picks the correct answer (answer $b_1$), while the other m methods pick any answer with probability 1 / n.

The purist chooses a method at random; this reflects Du Boisian diffidence: the purist does not know which method is The One True Method. The purist then believes whatever answer that method picks to be the correct one. Not only does this guessing at the correct method represent a high degree of uncertainty, or Du Boisian diffidence, it also captures something about the present state of social scientific inquiry. In fields which are largely pre-paradigm there will be competing “schools”, and attendant competing methodologies. Plausibly this is the case in most of the social sciences. What method a scholar ends up using is largely determined by which school they get educated into, and this itself will be a function of choices they made as an undergrad and before, at points when they had no idea about the relative merits of competing schools and methodologies. This is effectively a kind of randomization, or at least may reasonably be modeled as such. While methodological purists may not consciously randomize between potential approaches, at least in the social sciences we think they very often are de facto choosing at random among the methods.

The triangulator, as before, believes whatever answer is picked by the most methods (randomizing in case of ties). Let $p_j$ and $q_j$ denote the probabilities of believing answer $b_j$ for the triangulator and the purist respectively.

Theorem 2

$p_1 \ge q_1$ for all n and m. The inequality is strict whenever $m\ge 2$ and $n\ge 2$.

This result and Theorem 3 are proved in the Appendix.

We believe the above scenario is the most favorable possible scenario for the methodological purist, because it assumes that the purist’s preferred method is as good as it could possibly be and the other methods are as bad as they could possibly be. We hence think that showing that methodological triangulation can be valuable in this scenario is our strongest argument in triangulation’s favor. But it might still be objected that it is unrealistic that The One True Method delivers the correct answer with probability 1.

So now consider a case in which asking people to report their income directly (The One True Method) yields the answer “between $ 5 and $ 10” with probability 1 / 3 (2 / 9 each for the other three possible answers) while the other methods are random (1 / 4 for each answer). In this case the triangulator gets the answer “between $ 5 and $ 10” with probability 41 / 144. The triangulator does worse than the first purist ($41/144 < 1/3$) but better than the other two ($1/4 < 41/144$). Just as before, if a scientist is subject to Du Boisian diffidence triangulation is the way to go. In particular, triangulation does better than guessing a method and being a purist about that method ($1/3 \cdot 1/3 + 1/4\cdot 2/3 = 40/144 < 41/144$).

More generally, suppose that method $a_0$ picks answer $b_j$ with probability $r_j$ and assume that $r_1 > 1/n$ (so $a_0$ favors $b_1$ more than chance, although another answer might be favored even more). As before, the other methods pick randomly: any answer $b_j$ has a 1 / n chance of being picked. $p_j$ and $q_j$ are defined as above.

Theorem 3

$p_1 \ge q_1$ for all n and m. The inequality is strict whenever $m\ge 2$ and $n\ge 2$.

Conclusion

Some social scientists have attempted to exploit the fact of methodological pluralism by claiming that where triangulation can be achieved this provides more support for the point triangulated upon than any method considered individually could. Though confirmation theorists seemed generally sympathetic to the idea, and saw links between points of interest to them and methodological triangulation, what demonstrations they did produce tended to make heavy use of explicitly Bayesian assumptions. Further, other social scientists expressed skepticism about the benefits of triangulation.

Our model has vindicated individuals’ use of methodological triangulation, and thus also the instincts of the confirmation theorists, without the Bayesian baggage. In line with Du Bois’ methodological advice and scientific practice, triangulation does provide confirmatory support—-and, in particular, it does so even if one is not sure which of one’s available methods can actually be relied upon.

Since we were following Du Bois in this we take ourselves to have supplied underpinnings for at least some of the actual social scientific rationale for methodological triangulation. In particular, our model closely mirrors an explicit deployment of methodological triangulation by Du Bois (1996 [1899]) in his scientific work. We therefore take our model to represent a mathematical foundation for a practically viable procedure for deploying methodological triangulation in the social sciences. Even if it is true that, at present, most scientists do not in practice run elections among methods to exploit methodological pluralism and thereby boost the reliability of their results, Du Bois did. Since what is actual is possible, others could too, and our model suggests they may benefit from doing so.

The net effect of our arguments is to give those scholars who feel some degree of Du Boisian diffidence about the available methods in the social sciences reason to be happy about the fact of methodological pluralism. The various epistemological and methodological battles that have plagued the social sciences need not be resolved before one can proceed. Nor does the proliferation of methods necessarily need to be viewed as unfortunate.

Rather, we find that the tolerance of methodological pluralism does the diffident individual benefit, by allowing them to exploit triangulation in order to better arrive at the truth. We accept that to those who feel no degree of diffidence, our arguments may be less moving. In particular, to those who feel that the one true method in the social sciences should be qualitative, these arguments may all seem question begging. But in our experience some degree of Du Boisian diffidence is the typical state of the scholar, and thus we take our results to be of interest to a broad range of people.

We end by suggesting three additional lines of research that build on the present work.

One source of anti-confirmationist skepticism we have not addressed is the worry that there is widespread correlated error. We can distinguish two questions here. The first question is concerned with individuating methods. For example, when we have a mathematical model (say, an economists’ rational choice model) the parameters of which can be fitted based on data, does that count as one method or is each parametrization of the model a different method? Under a coarse-grained approach there will be fewer methods but they are less likely to display correlated error, whereas under a fine-grained approach there will be more methods but these methods are more likely to be correlated. To put the point more positively, if we suspect that two approaches we have so far counted as different methods (almost) always give the same answer, then we should count these as only one method for the purpose of triangulation. In practice it may be difficult to identify whether methods are independent in the relevant sense (Stegenga 2012; Schupbach 2015). But Kuorikoski and Marchionni (2016) provide evidence that such independence holds in at least some cases. Future work could fruitfully explore the question of how to individuate methods in more detail.

Second, we may ask how highly correlated methods need to be before the results reported in Sect. 3 no longer hold. We leave this question for future work. The formal apparatus deployed here makes it possible to explore the circumstances in which correlated error will undo the advantages of triangulation. We may expect to find results similar in spirit to those that have been found in the previously mentioned Bayesian tradition (Fitelson 2001; Claveau 2013).

Finally, our arguments were markedly about the benefits of methodological triangulation for the diffident individual. However, work in social epistemology implies that what may be a rational strategy for an individual inquirer may be disadvantageous for the community as a whole if generally adopted (Mayo-Wilson et al. 2011). Hence, while our model vindicates individuals in exploiting methodological triangulation, it does not show that science would be better off if all scientists were triangulators. Future work in this field could thus profitably explore a game-theoretic (or otherwise social) model of the operation of methodological triangulation. We hope the work we have done here shall provide a useful foundation for further work in the field.

Appendix: Proofs

For ease of exposition, we prove our results in the terminology of voting theory. The methods are the voters and the possible answers to the research question are the candidates.

Consider elections of the following form: there are n candidates $b_1,\ldots ,b_n$, and $m+1$ voters $a_1,\ldots ,a_m$, and $a_0$ (the reason we single out $a_0$ will be explained shortly). Formally, we can describe a vote as a function $v: \{a_1,\ldots ,a_m, a_0\}\rightarrow \{b_1,\ldots ,b_n\}$. Each vote v induces a probability measure $\mu _v$ on the set of candidates defined as follows:

• $\mu _v(b_k) = 1$ iff $|v^{-1}(b_k)|> |v^{-1}(b_j)|$ whenever $j\ne k$, i.e. candidate $b_k$ receives the most votes outright.

• $\mu _v(b_k) = 1/\ell$ iff $|v^{-1}(b_k)| \ge |v^{-1}(b_j)|$ for every $1\le j\le n$, and there are $\ell$ candidates, including $b_k$, who receive the maximum number of votes.

Additionally, we may suppose that $\pi$ is a probability measure on the space X of all possible votes. We define the overall probability$p_k$ that $b_k$ wins to be the quantity:

$$\begin{aligned} \int _X \mu _v(b_k) d\pi = \sum _{v\in X} \mu _v(b_k)\pi (v) \end{aligned}$$

We will consider the following two procedures for choosing a candidate using a vote:

• 1. Choose candidate $b_k$ with probability $p_k$ (i.e. choose the winner of the vote). This is the triangulator’s procedure.

• 2. Choose a voter $y\in \{a_1,\ldots ,a_m, a_0\}$ randomly and uniformly, then choose the candidate chosen by voter y, i.e. choose candidate $b_k$ with probability $\int _X \delta (b_k, v(y)) d\pi = \sum _{v\in X} \delta (b_k, v(y))\pi (v)$. This is the purist’s procedure.

Here $\delta (a,b) = 1$ if $a = b$ and $\delta (a,b) = 0$ otherwise. We see that in procedure 2, we hire candidate $b_k$ with probability $q_k$ defined as follows:

$$\begin{aligned} q_k = \frac{1}{m+1}\sum _{0\le i \le m} \sum _{v\in X} \delta (b_k, v(a_i))\pi (v) \end{aligned}$$

Lemma 4

Let $A_v$ be the random variable $|v^{-1}(b_1)|$. Then $q_1 = {\mathbb {E}}(A_v)/(m+1)$.

Proof

For any fixed $v\in X$, we have

$$\begin{aligned} |v^{-1}(b_1)| = \sum _{0\le i\le m} \delta (b_1, v(a_i)). \end{aligned}$$

It follows that

$$\begin{aligned} {\mathbb {E}}(A_v) = \sum _{v\in X}\sum _{0\le i\le m} \delta (b_1, a_i)\pi (v). \end{aligned}$$

$\square$

We now focus on the special case where $a_0$ votes for $b_1$ and $a_1,\ldots ,a_m$ vote randomly, uniformly, and independently. Set $Y = \{v\in X: v(a_0) = b_1\}$.

Lemma 5

Let $B_v$ be the random variable ${\mathrm {max}}(|v^{-1}(b_j)|: 1\le j\le n)$. Additionally, assume that $\pi$ is supported on Y and uniform on Y. Then $p_1 = {\mathbb {E}}(B_v)/(m+1)$.

Proof

Fix a vote v. Notice first that

$$\begin{aligned} B_v = \sum _{0\le i\le m} \mu _v(v(a_i)). \end{aligned}$$

Now the assumption that $\pi$ is uniform is equivalent to asserting that voter $a_0$ votes for $b_1$ while voters $a_1,\ldots ,a_k$ each pick a candidate randomly, uniformly, and independently. In particular, we have for any $0\le i\le m$ that

$$\begin{aligned} p_1 = \sum _{v\in Y} \mu _v(v(a_0))\pi (v) = \sum _{v\in Y} \mu _v(v(a_i))\pi (v). \end{aligned}$$

$\square$

Remark

Note that ${\mathbb {E}}(B_v)$ is the same for any $\pi$ where voters $a_1,\ldots ,a_m$ vote randomly, uniformly, and independently; we will use this later to consider changing the manner in which $a_0$ votes.

Theorem 2

If $\pi$ is supported on Y and uniform on Y, then $p_1 \ge q_1$. The inequality is strict whenever $m\ge 2$ and $n\ge 2$.

Proof

The first assertion is immediate from Lemmas 4 and 5 as $B_v \ge A_v$. If $m\ge 2$ and $n\ge 2$, there is some $v\in Y$ with $B_v > A_v$ (any vote where there is some outright winner who is not $b_1$ works). $\square$

Remark

Notice that in the setting of Theorem 2, we have $q_1 = (m+n)/(n(m+1))$. In particular, for any $\pi$ in which $a_1,\ldots ,a_m$ vote randomly, uniformly, and independently, we have ${\mathbb {E}}(B_v) \ge (m+n)/n$, with strict inequality for $m,n\ge 2$. We will use this in the proof of Theorem 3.

We now prove the same result for $\pi$ in which $a_0$ votes for $b_i$ with probability $r_i$, where $r_1 > 1/n$.

Theorem 3

Suppose $\pi$ is a measure where $a_1,\ldots ,a_m$ vote randomly, uniformly, and independently (and independently of $a_0$), and suppose $a_0$ votes for $b_i$ with probability $r_i$, where $r_1 > 1/n$. Then $p_1 \ge q_1$, with strict inequality whenever $m,n \ge 2$.

Proof

In the proof of Lemma 5 (and in the remark after), we saw that the probability that $a_0$ votes for the winner of v is exactly ${\mathbb {E}}(B_v)/(m+1)$. So to compute $p_1$ in terms of ${\mathbb {E}}(B_v)$, we need to consider two cases: if $a_0$ does vote for $b_1$, we want to count the probability that $a_0$ voted for the winner, whereas if $a_0$ votes for $b_2,\ldots ,b_n$, we want to count the probability that $a_0$ does not vote for the winner and that $b_1$ did in fact win. We see that:

$$\begin{aligned} p_1&= r_1\left( \frac{\mathbb {E}(B_v)}{m+1}\right) + (1-r_1)\left( \frac{m+1 - {\mathbb {E}}(B_v)}{(m+1)(n-1)}\right) \\&= r_1\left( \frac{\mathbb {E}(B_v)(n-1)}{(m+1)(n-1)}\right) + (1-r_1)\left( \frac{m+1-{\mathbb {E}}(B_v)}{(m+1)(n-1)}\right) \\&= \frac{(1-r_1)(m+1)+{\mathbb {E}}(B_v)(r_1n-1)}{(m+1)(n-1)}\\&\ge \frac{(1-r_1)(m+1)n+ (m+n)(r_1n-1)}{(m+1)(n-1)n} \\&= \frac{(m+r_1n)(n-1)}{(m+1)(n-1)n} \end{aligned}$$

Now $q_1$ is just given by Lemma 4:

$$\begin{aligned} q_1&= {\mathbb {E}}(A_v)/(m+1) \\&= \frac{m+r_1n}{n(m+1)} \end{aligned}$$

$\square$