Doing the Survey Two-Step: The Effects of Reticence on Estimates of Corruption in Two-Stage Survey Questions

Abstract

This paper develops a structural approach for modeling how respondents answer survey questions and uses it to estimate the proportion of respondents who are reticent in answering corruption questions, as well as the extent to which reticent behavior biases conventional estimates of corruption downwards. The context is a common two-step question, first inquiring whether a government official visited a business, and then asking about bribery if a visit was acknowledged. Reticence is a concern for both steps, since denying a visit side-steps the bribe question. This paper considers two alternative models of how reticence affects responses to two-step questions, with differing assumptions on how reticence affects the first question about visits. Maximum-likelihood estimates are obtained for seven countries using data on interactions with tax officials. Different models work best in different countries, but cross-country comparisons are still valid because both models use the same structural parameters. On average 40% of corruption questions are answered reticently, with much variation across countries. A statistic reflecting how much standard measures underestimate the proportion of all respondents who had a bribe interaction is developed. The downward bias in standard measures is highly statistically significant in all countries, varying from 12% in Nigeria to 90% in Turkey. The source of bias varies widely across countries, between denying a visit and denying a bribe after admitting a visit.

Economics studies facts, and seeks to arrange the facts in such ways as make it possible to draw conclusions from them. As always, it is the arrangement which is the delicate operation. Facts, arranged in the right way, speak for themselves; unarranged they are as dead as mutton.

John Hicks, The Social Framework, 1950

The authors can be contacted at nkaralashvili@worldbank.org, akraay@worldbank.org, or murrell@econ.umd.edu We thank discussants of a first version of this paper, presented at the Conference on “Ethics and Corporate Malfeasance: Interdisciplinary Perspectives”, organized by the Center for the Study of Business Ethics, Regulation, and Crime (C-BERC), University of Maryland, September 12, 2014. We are grateful to our discussant, Joao de Mello, and to participants in the IEA Roundtable in Montevideo, for helpful comments, and to the Enterprise Survey Team at the World Bank for their collaboration. We are grateful to Patricia Funk for providing data on Swiss referenda. Financial support from the Knowledge for Change Program of the World Bank is gratefully acknowledged. The views expressed here are the authors’, and do not reflect those of the World Bank, its Executive Directors, or the countries they represent.

Appendices

Appendix A: The Survey Questions and the Data

The Two-Step Conventional Question

• 1. A professional surveyor read the following to the respondent: “Over the last year, was this establishment visited or inspected by tax officials?” Respondents could either answer “Yes”, or “No”, or “Don’t know” (DK) or refuse (R) to answer. Respondents answering DK or R were dropped from the sample. The incidence of DK and R responses to these questions is given in Table 11.12.

• 2. We set D = 1 if the respondent did not acknowledge that the visit occurred, that is if the respondent answered “No”.

• 3. If the above question was answered with a “Yes”, then the interviewer read the following to the respondent²⁴: “In any of these inspections or meetings was a gift or informal payment expected or requested?” Respondents could either answer “Yes”, or “No”, or DK or R.²⁵ Respondents who answered DK or R were dropped from the sample. The incidence of DK and R responses to these questions is given in Table 11.12.

• 4. We set D = 2 if the respondent said “No” to the inquiry about the bribe request, and we set D = 3 if the bribe request is acknowledged.

The Random-Response Questions

• 1. A professional surveyor read the following to the respondent: “We have designed an alternative experiment which provides the opportunity to answer questions based on the outcome of a coin toss. Before you answer each question, please toss this coin and do not show me the result. If the coin comes up heads, please answer ‘yes’ to the question regardless of the question asked. If the coin comes up tails, please answer in accordance with your experience. Since I do not know the result of the coin toss, I cannot know whether your response is based on your experience or by chance.”

• 2. The ten sensitive questions used in this battery of questions are given in Table 11.2. Respondents who refused to respond or responded “Don’t know” were dropped from the sample.

• 3. The variable X used in the analysis is equal to the number of the seven bolded questions in Table 11.2 for which the respondent answers “Yes”.

The incidence of DK and R responses to these questions is given in Table 11.12.

Cleaning the Data for Interviewer Effects

The RRQ battery is a key ingredient in our methodology, and therefore it is important to ensure that this unusual and cumbersome-to-administer procedure was implemented as designed. Enumerators received specific training on the RRQ methodology. As part of this training, they learned how the RRQ methodology is supposed to provide greater anonymity for respondents. However, they were not briefed on our intention to use the RRQ battery to make inferences about reticence.

Despite these precautions we do find some evidence of interviewer effects in the data that might indicate variation across interviewers in the implementation of the RRQs. In all countries we have information on the identity of the interviewer for each respondent.²⁶ For each interviewer, we calculated the proportion of respondents with seven “No” responses on the RRQs. For most interviewers in most countries, we found rates of seven “No” responses that were not too different from the corresponding country averages. However, we did find some interviewers with implausibly high rates of seven “No” responses. We speculate that this may reflect differences across interviewers in how the RRQ was implemented. One possibility is that the interviewer incorrectly had the respondent toss the coin only once and let a single outcome govern the responses to all questions in the RRQ battery. This could lead to an upward bias in our estimates of the prevalence of reticent behavior. To avoid such a possibility, we drop all interviews performed by interviewers whose interviewer-specific rate of seven “No” responses on the sensitive RRQ questions was more than five standard deviations above the corresponding country average.²⁷ Combining all surveys except India, we drop 2% of interviewers who accounted for just under one tenth of all respondents who answered “No” seven times on the RRQ battery. Including India, we drop a total of 4% of interviewers, together accounting for over third of all of the respondents who answered “No” seven times.

This process is necessary in order to pursue the objective of focusing solely on the effects of respondent reticence. Our goal is not to evaluate the properties of survey data as a whole, but rather to investigate the effect of reticent behavior on the CQ as a possible source of bias in estimates of corruption. The goal is furthered by focusing on a subset of the data where one can be most sure that interview procedures were followed faithfully. We also note that while dropping these interviewers naturally increases the rate of “Yes” responses on the RRQ, it only has small effects on the rate of “Yes” responses on the CQ. This treatment of interviewers did not result in any changes in the data from Turkey and Nigeria. It had the biggest effect on the rate of “Yes” responses on the second part of the CQ in the data from India where the rate increased from 9.1% to 11.9%. In Peru, Bangladesh 2011, and Bangladesh 2013, this rate increased by less than one percentage point, and in Ukraine it decreased by less than 0.1 percentage point. This suggests that our concerns about the dropped interviewers applies only to their administration of the RRQs, except in India, perhaps.

Appendix B: The Assumption That Reticence on the CQ and the RRQs Is the Same

Our methodology assumes that rates of reticence on the CQ are the same as on the RRQs. This appears to be a strong assumption because the RRQ was developed with the exact purpose of reducing respondent reticence relative to that on CQs. In this Appendix we justify our assumption in two ways. First, we show that the assumption is reasonable given current evidence in the survey-research literature. Second, we show that one of our major conclusions—the underestimation of corruption—is robust when this assumption is relaxed, that is, assuming the RRQ does reduce respondent reticence.

In a meta-analysis, Lensvelt-Mulders et al. (2005) examined the few studies where RRQs and CQs were used and external validation of survey responses was possible. They found that on average RRQs had 90% of the reticence of conventional face-to-face interview questions (CQs). Holbrook and Krosnick (2010) and Wolter and Preisendörfer (2013) cite a large number of studies of the effects of RRQs versus CQs and both conclude that there are reasons to doubt the efficacy of RRQs. After conducting their own study showing that the use of RRQs actually increased estimates of voter turnout to impossible levels, Holbrook and Krosnick (2010, p. 336) conclude that “… among the few studies that have compared RRT and direct self-report estimates of socially admirable attributes, none yielded consistent evidence that the RRT significantly reduced reported rates … This calls into question interpretations of all past RRT studies and raises serious questions about whether the RRT has practical value for increasing survey reporting accuracy.”²⁸

Coutts and Jann (2011) used exactly the technique that we used in our study—forced-response, manual-coin toss RRQ—to examine six sensitive topics. They find that admission rates for RRQs are much lower than for CQs for not buying a ticket on public transport, shoplifting, marijuana use, driving under the influence (DUI), and infidelity, while higher only for keeping extra change when too much was given in a transaction. They attribute their results on RRQs as reflecting the fact that a forced-yes response can feel like an admission of guilt. (Indeed, a yes response should mean that the Bayesian posterior probability of guilt is higher than the prior for anybody but the respondent, such as a judgmental interviewer.) Wolter and Preisendörfer (2013) also compared a CQ to a forced-response RRQ, questioning a sample of known convicted criminals on whether they had committed an offense. Whereas 100% of the sample were guilty, 57.5% acknowledged this in a CQ and 59.6% in an RRQ, a trivial increase in candor. As the qualitative study of Lensvelt-Mulders and Boeije (2007) shows, the forced-response of yes after a coin toss is highly unpopular among respondents, thus suggesting the reason why RRQs might not produce their desired effect.

One can also address the issue analytically. Suppose that the world is such that reticence on the RRQ is less than on the CQ, that is the RRQ has some of the effect that its proponents hoped for. In terms of the parameters of our models, there are now two values of q, one for the CQ and one for the RRQs, and q ^CQ > q ^RRQ. One can then ask what the biases in our estimation procedure would be given that our procedure embodies the assumption that reticence is the same on the two types of questions. This is easy to answer analytically in one case, when there is a one-step CQ and k = 1. (This is equivalent to Model A with k =1, since in that model respondents are always candid about visits). Suppose that our ML estimate of average guilt is denoted g ^e and the true value of average guilt is g ^a. Then, it is straightforward to show that $\frac{g^e}{\left(1+{g}^e\right)}$ consistently estimates $\frac{g^a}{\left(1+{g}^a\right)}\cdot \frac{\left(1-{rq}^{CQ}\right)}{\left(1-{rq}^{RRQ}\right)}$.²⁹ Thus our procedures underestimate the actual rate of guilt in this special case.

When we turn to model B or instances where k < 1, or both, the analytics is not as straightforward. Thus, we use simulations for the analysis. A single simulation is as follows. We generate a data set of 10,000 observations using one of our models, for example model B, and parameter values that appear in Table 11.11 for a particular country for which that model is preferred, for example India. However, when we generate the observations we make one variation on the model: we assume reticence on the RRQ is less than on the CQ. That is, q ^CQ is set at the value of q in Table 11.11, but q ^RRQ = 0.8 × q ^CQ. Then when we estimate the model we incorrectly assume that the simulated world is one where q ^CQ = q ^RRQ, that is, estimation is as described in section “Estimation and Definitions of Composite Parameters”.

Table 11.13 Simulation results

The results of the simulations appear in the table immediately above. Because the results are so consistent, and consistent with the analytics for the simple case above, six simulations are sufficient, each one matching our preferred model for a country. In all cases, our procedures severely underestimate the true rates of effective corruption when the world is one in which q ^CQ = 0.8 × q ^RRQ and estimation incorrectly imposes the assumption that q ^CQ = q ^RRQ. The degree of underestimation varies between 3 standard deviations (Peru) and 43 (Ukraine). Thus, our conclusion that standard estimates of corruption are significantly underestimated is robust to the criticism that we have incorrectly assumed that the RRQ has no affect in diminishing respondent reticence (Table 11.13).