Is the List Experiment Doing its Job?

Inconclusive Evidence!
  • Stefanie GosenEmail author
  • Peter Schmidt
  • Stefan Thörner
  • Jürgen Leibold


This paper sheds new light on the unobtrusive measure known as the list experiment or unmatched count technique. Proponents of this method claim that it detects social desirability bias in responses to sensitive questions in surveys. The logic of this method is quite straightforward. After a critical overview of the theory, logic, and empirical results of this type of measure, we present the results of a series of three studies. While the first study yielded promising results, the replication of the outcome pattern in Study 2 failed completely. The third study, based on longitudinal data, delivers indications for the systematic inconsistency of the claimed logic of the list experiment. First, significant mean differences between baseline and test condition occur even if the additional item is non-sensitive and has an agreement rate of about two percent in direct questioning. Second, test-retest reliability shows fluctuating results depending on the sensitivity and number of items included in the experiment. Implications for theory and practice in measuring social desirability by unobtrusive measures are discussed.


social desirability indirect survey techniques sensitive question Islamophobia anti-Semitism 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Ahart, A. M., and P. R Sackett. 2004. A new method of examining relationships between individual difference measures and sensitive behavior criteria: Evaluating the unmatched count technique. Organizational Research Methods 7: 101–114.
  2. Auspurg, K., B. Jann, I. Krumpal, and H. von Hermanni. 2012, June. Randomized-Response-Technik: Hope or Hype? Eine Meta-Analyse unter Berücksichtigung von Publication-Bias [Randomized-response-technique: Hope or hype? A meta-analysis taking into account publication bias]. Paper presented at the First Mini-Conference of the Center of Quantitative Methods of the University of Leipzig. Asking Sensitive Questions: Theory and Data Collection Methods, Leipzig, Germany.Google Scholar
  3. Blair, G., and K. Imai. 2012. Statistical analysis of list experiments. Political Analysis 20: 47–77.
  4. Chaudhuri, A., and T. C. Christofides. 2007. Item count technique in estimating the proportion of people with a sensitive feature. Journal of Statistical Planning and Inference 137(2): 589–593.
  5. Coutts, E., and B. Jann. 2011. Sensitive questions in online surveys: Experimental results for the randomized response technique (RRT) and the unmatched count technique (UCT). Sociological Methods & Research 40: 169–193.
  6. Creighton, M., D. Zavala-Rojas, P. Schmidt, and P. Brenner. 2018. Analysis of List Experiments. In: Experimental Methods in Survey Research: Techniques that Combine Random Sampling with Random Assignment, eds. P .J. Lavrakas, M. W. Traugott, C. Kennedy, A. Holbrook, E. D. de Leeuw, and B. T. West. New York: John Wiley & Sons.Google Scholar
  7. Crowne, D. P., and D. Marlowe. 1960. A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology 24: 349–354.
  8. Dalton, D. R., J. C. Wimbush, and C. M. Daily. 1994. Using the unmatched count technique (UCT) to estimate base rates for sensitive behavior. Personnel Psychology 47(4): 817–828.
  9. Davidov, E., S. Thörner, P. Schmidt, S. Gosen, and C. Wolf. 2011. Level and change of group-focused enmity in Germany: Unconditional and conditional latent growth curve models with four panel waves. AStA Advances in Statistical Analysis 95(4): 481–500.
  10. Dovidio, J. F., and S. L. Gaertner. 1986. Prejudice, Discrimination, and Racism: Historical Trends and Contemporary Approaches. In Prejudice, Discrimination, and Racism, eds. J. F. Dovidio and S. L. Gaertner, 1–34. San Diego, CA: Academic Press.Google Scholar
  11. Droitcour, J., R. A. Caspar, M. L. Hubbard, T. L. Parsley, W. Visscher, and T. M. Ezzati. 1991. The Item Count Technique as a Method of Indirect Questioning: A Review of its Development and a Case Study Application. In Measurement Errors in Surveys, eds. P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, and S. Sudman, 185–210. New York: Wiley.
  12. Gilens, M., P. M. Sniderman, and J. H. Kuklinski. 1998. Affirmative action and the politics of realignment. British Journal of Political Science 28: 159–183.
  13. Glynn, A. 2013. What can we learn with statistical truth serum? Design and analysis of the list experiment. Public Opinion Quarterly 77(S1): 159–172.
  14. Heitmeyer, W. 2002. Deutsche Zustände, Folge 1 [German Conditions, Vol. 1]. Frankfurt am Main: Suhrkamp.Google Scholar
  15. Himmelfarb, S., and C. Lickteig. 1982. Social desirability and the randomized response technique. Journal of Personality and Social Psychology 43(4): 710–717.
  16. Holbrook, A. L., and J. A. Krosnick. 2010a. Measuring voter turnout by using the randomized response technique: Evidence calling into question the method’s validity. Public Opinion Quarterly 74(2): 328–343.
  17. Holbrook, A. L., and J. A. Krosnick. 2010b. Social desirability bias in voter turnout reports: Tests using the item count technique. Public Opinion Quarterly 74(1): 37–67.
  18. Huddy, L., and S. Feldman. 2009. On assessing the political effects of racial prejudice. Annual Review of Political Science 12(1): 423–447.
  19. Imai, K. 2011. Multivariate regression analysis for the item count technique. Journal of the American Statistical Association 106(494): 407–416.
  20. Janus, A. L. 2010. The influence of social desirability Pressures on expressed immigration attitudes. Social Science Quarterly 91(4): 928–946.
  21. Jones, E. E. and H. Sigall. 1971. The bogus pipeline: A new paradigm for measuring affect and attitude. Psychological Bulletin 76: 349–364.
  22. Kane, J. G., S. C. Craig, and K. D. Wald. 2004. Religion and presidential politics in Florida: A list experiment. Social Science Quarterly 85(2): 281–293.
  23. Karlan, D., and J. Zinman. 2011. List randomization for sensitive behavior: An application for measuring use of loan proceeds. Working Paper. Retrieved from
  24. Kuklinski, J. H., M. D. Cobb, and M. Gilens. 1997. Racial attitudes and the “new South.” The Journal of Politics 59(2): 323–349.
  25. Kuklinski, J. H., P. M. Sniderman, K. Knight, T. Piazza, P. E. Tetlock, G. R. Lawrence, and B. Mellers. 1997. Racial prejudice and attitudes toward affirmative action. American Journal of Political Science 41(2): 402–419.
  26. Krumpal, I. 2011. Determinants of social desirability bias in sensitive surveys: A literature review. Quality and Quantity 47(4): 2025–2047.
  27. LaBrie, J. W., and M. Earleywine. 2000. Sexual risk behaviors and alcohol: Higher base rates revealed using the unmatched‐count technique. Journal of Sex Research 37(4): 321–326.
  28. Lensvelt-Mulders, G. J. L. M., J. J. Hox, and P. G. M. van der Heijden. 2005. Meta-analysis of randomized response research: 35 years of validation. Sociological Methods & Research 33(3): 319–348.
  29. Miller, J. D. 1984. A New Survey Technique for Studying Deviant Behavior (Doctoral dissertation). George Washington University, Washington, DC.Google Scholar
  30. Nickel, B., M. Berger, P. Schmidt, and K. Plies. 1995. Qualitative sampling in a multi-method survey. Quality & Quantity 29(3): 223–240.
  31. Oberski, D., W. Weber, and M. Révilla. 2012. The Effect of Individual Characteristics on Reports of Socially Desirable Attitudes towards Immigration. In Methods, Theories, and Empirical Applications in the Social Sciences. Festschrift for Peter Schmidt, eds. S. Salzborn, E. Davidov, and J. Reinecke, 151–157. Wiesbaden: Springer VS.
  32. Papastefanou, G., and M. Wiedenbeck. 1998. Singuläre und multiple Imputation fehlender Einkommenswerte: ein empirischer Vergleich [Singular and multiple imputation of missing income data: An empirical comparison]. ZUMA-Nachrichten 22(43): 73–89.Google Scholar
  33. Paulhus, D. L. 1984. Two-component models of socially desirable responding. Journal of Personality and Social Psychology 46(3): 598–609.
  34. Paulhus, D. L., and D. B. Reid. 1991. Enhancement and denial in socially desirable responding. Journal of Personality and Social Psychology 60(2): 307–317.
  35. Raghavarao, D., and W. T. Federer. 1979. Block total response as an alternative to the randomized response method in surveys. Journal of the Royal Statistical Society. Series B (Methodological) 41(1): 40–45.Google Scholar
  36. Salzborn, S. 2010. The politics of antisemitism. Journal for the Study of Antisemitism 2(1): 89–114.Google Scholar
  37. Schuessler, K. F. 1982. Measuring Social Life Feelings (1st ed.). San Francisco: Jossey-Bass.Google Scholar
  38. Schwarz, N., and S. Sudman. 1996. Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research (1st ed.). San Francisco: Jossey-Bass.Google Scholar
  39. Sniderman, P. M., and D. B. Grob. 1996. Innovations in experimental design in attitude surveys. Annual Review of Sociology 22(1): 377–399.
  40. Streb, M. J., B. Burrell, B. Frederick, and M. A. Genovese. 2008. Social desirability effects and support for a female American president. Public Opinion Quarterly 72(1): 76–89.
  41. Tourangeau, R., L. J. Rips, and K. A. Rasinski. 2000. The Psychology of Survey Response. New York: Cambridge University Press.Google Scholar
  42. Tourangeau, R., and T. W. Smith. 1996. Asking sensitive questions: The impact of data collection mode, question format, and question context. Public Opinion Quarterly 60(2): 275–304.
  43. Tourangeau, R., and T. Yan. 2007. Sensitive questions in surveys. Psychological Bulletin 133(5): 859–883.
  44. Tsuchiya, T., Y. Hirai, and S. Ono. 2007. A study of the properties of the item count technique. Public Opinion Quarterly 71(2): 253–272.
  45. Tsuchiya, T., and Y. Hirai. 2010. Elaborate item count questioning: Why do people underreport in item count responses? Survey Research Methods 4(3): 139–149.
  46. Warner, S. L. 1965. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association 60: 63–69.
  47. Weisband, S., and S. Kiesler. 1996. Self-disclosure on computer forms: Meta-analysis and implications. In Conference Companion on Human Factors in Computing Systems – CHI´96, ed. M. Tauber, 3–10. Vancouver: ACM Press.
  48. Zick, A., C. Wolf, B. Küpper, E. Davidov, P. Schmidt, and W. Heitmeyer. 2008. The syndrome of group-focused enmity: The interrelation of prejudices tested with multiple cross-sectional and panel data. Journal of Social Issues 64(2): 363–383.
  49. Zigerell, L. J. 2011. You wouldn’t like me when I’m angry: List experiment misreporting. Social Science Quarterly 92(2): 552–562.
  50. Zimmerman, R. S., and L. M. Langer. 1995. Improving estimates of prevalence rates of sensitive behaviors: The randomized lists technique and consideration of self‐reported honesty. Journal of Sex Research 32(2): 107–111.

Copyright information

© Springer Fachmedien Wiesbaden GmbH, ein Teil von Springer Nature 2019

Authors and Affiliations

  • Stefanie Gosen
    • 1
    Email author
  • Peter Schmidt
    • 1
  • Stefan Thörner
    • 1
  • Jürgen Leibold
    • 1
  1. 1.GießenDeutschland

Personalised recommendations