Advertisement

Survey-software implicit association tests: A methodological and empirical analysis

  • Thomas P. CarpenterEmail author
  • Ruth Pogacar
  • Chris Pullig
  • Michal Kouril
  • Stephen Aguilar
  • Jordan LaBouff
  • Naomi Isenberg
  • Alek Chakroff
Article

Abstract

The implicit association test (IAT) is widely used in psychology. Unfortunately, the IAT cannot be run within online surveys, requiring researchers who conduct online surveys to rely on third-party tools. We introduce a novel method for constructing IATs using online survey software (Qualtrics); we then empirically assess its validity. Study 1 (student n = 239) revealed good psychometric properties, expected IAT effects, and expected correlations with explicit measures for survey-software IATs. Study 2 (MTurk n = 818) showed predicted IAT effects across four survey-software IATs (ds = 0.82 [Black–White IAT] to 2.13 [insect–flower IAT]). Study 3 (MTurk n = 270) compared survey-software IATs and IATs run via Inquisit, yielding nearly identical results and intercorrelations that would be expected for identical IATs. Survey-software IATs appear to be reliable and valid, offer numerous advantages, and make IATs accessible for researchers who use survey software to conduct online research. We present all the materials, links to tutorials, and an open-source tool that rapidly automates survey-software IAT construction and analysis.

Keywords

Implicit association test Online research Implicit measures 

Notes

References

  1. Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N., & Evershed J. K. (2019). Gorilla in our midst: An online behavioral experiment builder. Behavior Research Methods. Advance online publication.  https://doi.org/10.3758/s13428-019-01237-x
  2. Back, M. D., Schmukle, S. C., & Egloff, B. (2005). Measuring task-switching ability in the Implicit Association Test. Experimental Psychology, 52, 167–179.  https://doi.org/10.1027/1618-3169.52.3.167 CrossRefGoogle Scholar
  3. Bluemke, M., & Friese, M. (2006). Do features of stimuli IAT effects? Journal of Experimental Social Psychology, 42, 163–176.  https://doi.org/10.1016/j.jesp.2005.03.004 CrossRefGoogle Scholar
  4. Bluemke, M., & Friese, M. (2008). Reliability and validity of the Single-Target IAT (ST-IAT): Assessing automatic affect towards multiple attitude objects. European Journal of Social Psychology, 38, 977–997.  https://doi.org/10.1002/ejsp.487 CrossRefGoogle Scholar
  5. Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018). An evaluation of Amazon’s Mechanical Turk, its rapid rise, and its effective use. Perspectives on Psychological Science, 13, 149–154.  https://doi.org/10.1177/1745691617706516 CrossRefGoogle Scholar
  6. Capozza, D., Vezzali, L., Trifiletti, E. T., Falvo, R., & Favara, I. (2010). Improving intergroup relationships within and outside the contact situation: The role of common in-group identity and emotions of empathy and anxiety. Testing, Psychometrics, Methodology in Applied Psychology, 17, 17–36.Google Scholar
  7. Carpenter, T. P. (2015). Enjoy! A dual-process examination of product-pleasure associations and preferences (Unpublished doctoral dissertation). Baylor University, Waco, TX.Google Scholar
  8. Cohen, T. R., Wolf, S. T., Panter, A. T., & Insko, C. A. (2011). Introducing the GASP scale: A new measure of guilt and shame proneness. Journal of Personality and Social Psychology, 100, 947.  https://doi.org/10.1037/a0022641 CrossRefGoogle Scholar
  9. De Houwer, J., & De Bruycker, E. (2007). The Implicit Association Test outperforms the extrinsic affective Simon task as an implicit measure of inter-individual differences in attitudes. British Journal of Social Psychology, 46, 401–421.  https://doi.org/10.1348/014466606X130346 CrossRefGoogle Scholar
  10. de Leeuw, J. R. (2015). jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior Research Methods, 47, 1–12.  https://doi.org/10.3758/s13428-014-0458-y CrossRefGoogle Scholar
  11. Duckitt, J., & Bizumic, B. (2013). Multidimensionality of right-wing authoritarian attitudes: Authoritarianism-Conservatism-Traditionalism. Political Psychology, 34, 841–862.CrossRefGoogle Scholar
  12. Fatfouta, R., & Schröder-Abé, M. (2018). Agentic to the core? Facets of narcissism and positive implicit self-views in the agentic domain. Journal of Research in Personality, 74, 78–82.  https://doi.org/10.1016/j.jrp.2018.02.006 CrossRefGoogle Scholar
  13. Fazio, R. H., & Olson, M. A. (2003). Implicit measures in social cognition research: Their meaning and use. Annual Review of Psychology, 54, 297–327.  https://doi.org/10.1146/annurev.psych.54.101601.145225 CrossRefGoogle Scholar
  14. Friese, M., Wänke, M., & Plessner, H. (2006). Implicit consumer preferences and their influence on product choice. Psychology & Marketing, 23, 727–740.  https://doi.org/10.1002/mar.20126 CrossRefGoogle Scholar
  15. Gattol, V., Sääksjärvi, M., & Carbon, C.-C. (2011). Extending the Implicit Association Test (IAT): Assessing consumer attitudes based on multi-dimensional implicit associations. PLoS ONE, 6, e15849.  https://doi.org/10.1371/journal.pone.0015849 CrossRefGoogle Scholar
  16. Gawronski, B. (2002). What does the implicit association test measure? A test of the convergent and discriminant validity of prejudice-related IATs. Experimental Psychology, 49, 171–180.  https://doi.org/10.1026/1618-3169.49.3.171 CrossRefGoogle Scholar
  17. Gawronski, B., Morrison, M., Phills, C. E., & Galdi, S. (2017). Temporal stability of implicit and explicit measures: A longitudinal analysis. Personality and Social Psychology Bulletin, 43, 300–312.  https://doi.org/10.1177/0146167216684131 CrossRefGoogle Scholar
  18. Gervais, W. M., Shariff, A. F., & Norenzayan, A. (2011). Do you believe in atheists? Distrust is central to anti-atheist prejudice. Journal of Personality and Social Psychology, 101, 1189–1206.  https://doi.org/10.1037/a0025882 CrossRefGoogle Scholar
  19. Gervais W. M., Xygalatas, D., McKay, R. T., van Elk, M., Buchtel, E. E., Aveyard, M., … Bulbulia, J. (2017). Global evidence of extreme intuitive moral prejudice against atheists. Nature Human Behavior, 1, 0151.CrossRefGoogle Scholar
  20. Gosling, S. D., & Mason, W. (2015). Internet research in psychology. Annual Review of Psychology, 66, 877–902.  https://doi.org/10.1146/annurev-psych-010814-015321 CrossRefGoogle Scholar
  21. Greenwald, A. G., & Banaji, M. R. (2017). The implicit revolution: Reconceiving the relation between conscious and unconscious. American Psychologist, 72, 861–871.  https://doi.org/10.1037/amp0000238 CrossRefGoogle Scholar
  22. Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998). Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology, 74, 1464–1480.  https://doi.org/10.1037/0022-3514.74.6.1464 CrossRefGoogle Scholar
  23. Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003). Understanding and using the Implicit Association Test: I. An improved scoring algorithm. Journal of Personality and Social Psychology, 85, 197–216.  https://doi.org/10.1037/0022-3514.85.2.197 CrossRefGoogle Scholar
  24. Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., & Banaji, M. R. (2009). Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity. Journal of Personality and Social Psychology, 97, 17–41.  https://doi.org/10.1037/a0015575 CrossRefGoogle Scholar
  25. Hittner, J. B., May, K., & Silver, N. C. (2003). A Monte Carlo evaluation of tests for comparing dependent correlations. Journal of General Psychology, 130, 149–168.  https://doi.org/10.1080/00221300309601282 CrossRefGoogle Scholar
  26. Hofmann, W., Gawronski, B., Gschwendner, T., Le, H., & Schmitt, M. (2005). A meta-analysis on the correlation between the Implicit Association Test and explicit self-report measures. Personality and Social Psychology Bulletin, 31, 1369–1385.  https://doi.org/10.1177/0146167205275613 CrossRefGoogle Scholar
  27. Instituto Nacional Electoral (2005). Estadísticas Lista Nominal y Padrón Electoral. Retrieved from http://portalanterior.ine.mx/archivos3/portal/historico/contenido/Estadisticas_Lista_Nominal_y_Padron_Electoral/
  28. Karpinski, A., & Steinman, R. B. (2006). The Single Category Implicit Association Test as a measure of implicit social cognition. Journal of Personality and Social Psychology, 91, 16–32.  https://doi.org/10.1037/0022-3514.91.1.16 CrossRefGoogle Scholar
  29. Kirk, R. E. (2008). Statistics: An introduction (5th ed.). Belmont, CA: Thomson Wadsworth.Google Scholar
  30. Kuchenbrandt, D., Eyssel, F., & Seidel, S. K. (2013). Cooperation makes it happen: Imagined intergroup cooperation enhances the positive effects of imagined contact. Group Processes and Intergroup Relations, 16, 635–647.  https://doi.org/10.1177/1368430212470172 CrossRefGoogle Scholar
  31. LaBouff, J. P., & Ledoux, A. M. (2016). Imagining atheists: Reducing fundamental distrust in atheist intergroup attitudes. Psychology of Religion and Spirituality, 8, 330–340.  https://doi.org/10.1037/rel0000066 CrossRefGoogle Scholar
  32. Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8, 355–362.  https://doi.org/10.1177/1948550617697177 CrossRefGoogle Scholar
  33. Lane, K. A., Banaji, M. R., Nosek, B. A., & Greenwald, A. G. (2007). Understanding and using the Implicit Association Test: IV. What we know (so far) about the method. In B. Wittenbrink & N. Schwarz (Eds.), Implicit measures of attitudes (pp. 59–102). New York, NY: Guilford Press.Google Scholar
  34. Marquardt, N., & Hoeger, R. (2009). The effect of implicit moral attitudes on managerial decision-making: An implicit social cognition approach. Journal of Business Ethics, 85, 157–171.  https://doi.org/10.1007/s10551-008-9754-8 CrossRefGoogle Scholar
  35. Nock, M. K., Park, J. M., Finn, C. T., Deliberto, T. L., Dour, H. J., & Banaji, M. R. (2010). Measuring the suicidal mind: implicit cognition predicts suicidal behavior. Psychological Science, 21, 511–517.  https://doi.org/10.1177/0956797610364762 CrossRefGoogle Scholar
  36. Nosek, B. A., & Banaji, M. R. (2001). The go/no-go association task. Social Cognition, 19, 625–666.CrossRefGoogle Scholar
  37. Nosek, B. A., Banaji, M., & Greenwald, A. G. (2002). Harvesting implicit group attitudes and beliefs from a demonstration web site. Group Dynamics: Theory, Research, and Practice, 6, 101–115.  https://doi.org/10.1037/1089-2699.6.1.101 CrossRefGoogle Scholar
  38. Nosek, B. A., Greenwald, A. G., & Banaji, M. R. (2005). Understanding and using the implicit association test: II. Method variables and construct validity. Personality and Social Psychology Bulletin, 31, 166–180.  https://doi.org/10.1177/0146167204271418 CrossRefGoogle Scholar
  39. Nosek, B. A., Smyth, F. L., Hansen, J. J., Devos, T., Lindner, N. M., Ranganath, K. A., Banaji, M. R. (2007). Pervasiveness and correlates of implicit attitudes and stereotypes. European Review of Social Psychology, 18, 36–88.  https://doi.org/10.1080/10463280701489053.
  40. Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184–188.  https://doi.org/10.1177/0963721414531598 CrossRefGoogle Scholar
  41. Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023–1031.  https://doi.org/10.3758/s13428-013-0434-y CrossRefGoogle Scholar
  42. Penke, L., Eichstaedt, J., & Asendorpf, J. B. (2006). Single-Attribute Implicit Association Tests (SA-IAT) for the assessment of unipolar constructs: The case of sociosexuality. Experimental Psychology, 53, 283–291.  https://doi.org/10.1027/1618-3169.53.4.283 CrossRefGoogle Scholar
  43. Pittinsky, T. L., Rosenthal, S. A., & Montoya, R. M. (2011). Liking is not the opposite of disliking: The functional separability of positive and negative attitudes toward minority groups. Cultural Diversity and Ethnic Minority Psychology, 17, 134–143.  https://doi.org/10.1037/a0023806 CrossRefGoogle Scholar
  44. Pogacar, R., Kouril, M., Carpenter, T. P., & Kellaris, J. J. (2018). Implicit and explicit preferences for brand name sounds. Marketing Letters, 29, 241–259.  https://doi.org/10.1007/s11002-018-9456-7 CrossRefGoogle Scholar
  45. Reimers, S., & Stewart, N. (2015). Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. Behavior Research Methods, 47, 309–327.  https://doi.org/10.3758/s13428-014-0471-1 CrossRefGoogle Scholar
  46. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237.  https://doi.org/10.3758/PBR.16.2.225 CrossRefGoogle Scholar
  47. Rudman, L. A. (2011). Implicit measures for social and personality psychology. Thousand Oaks, CA: Sage.CrossRefGoogle Scholar
  48. Schnabel, K., Asendorpf, J. B., & Greenwald, A. G. (2008). Using Implicit Association Tests for the Assessment of Implicit Personality Self-Concept. In SAGE handbook of personality theory and assessment: Vol. 2. Personality measurement and testing (pp. 508–528).  https://doi.org/10.4135/9781849200479.n24
  49. Schnabel, K., Banse, R., & Asendorpf, J. B. (2006). Assessment of implicit personality self-concept using the Implicit Association Test (IAT): Concurrent assessment of anxiousness and angriness. British Journal of Social Psychology, 45, 373–396.  https://doi.org/10.1348/014466605X49159 CrossRefGoogle Scholar
  50. Smith, A. G., & Winter, D. G. (2002). Right-wing authoritarianism, party identification, and attitudes toward feminism in student evaluations of the Clinton–Lewinsky story. Political Psychology, 23, 355–383.  https://doi.org/10.1111/0162-895X.00285 CrossRefGoogle Scholar
  51. Sriram, N., & Greenwald, A. G. (2009). The Brief Implicit Association Test. Experimental Psychology, 56, 283–294.CrossRefGoogle Scholar
  52. Stephan, W. G., Boniecki, K. A., Ybarra, O., Bettencourt, A., Ervin, K. S., Jackson, L. A., … Renfro, C. L. (2002). The role of threats in the racial attitudes of Blacks and White. Personality and Social Psychology Bulletin, 28, 1242–1254.  https://doi.org/10.1177/01461672022812009 CrossRefGoogle Scholar
  53. Szucs, D., & Ioannidis, J. P. A. (2017). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biology, 15, e2000797.  https://doi.org/10.1371/journal.pbio.2000797 CrossRefGoogle Scholar
  54. United States Census Bureau. (2010). Frequently occurring surnames in the 2010 census. Retrieved from https://www.census.gov/library/visualizations/2016/comm/cb16-tps154_surnames_top15.html
  55. Vezzali, L., Capozza, D., Giovannini, D., & Stathi, S. (2012). Improving implicit and explicit intergroup attitudes using imagined contact: An experimental intervention with elementary school children. Group Processes & Intergroup Relations, 15, 203–212.  https://doi.org/10.1177/1368430211424920 CrossRefGoogle Scholar
  56. Wang-Jones, T. S., Alhassoon, O. M., Hattrup, K., Ferdman, B. M., & Lowman, R. L. (2017). Development of gender identity implicit association tests to assess attitudes toward transmen and transwomen. Psychology of Sexual Orientation and Gender Diversity, 4, 169–183.  https://doi.org/10.1037/sgd0000218 CrossRefGoogle Scholar

Copyright information

© The Psychonomic Society, Inc. 2019

Authors and Affiliations

  1. 1.Seattle Pacific UniversitySeattleUSA
  2. 2.University of CalgaryCalgaryCanada
  3. 3.Baylor UniversityWacoUSA
  4. 4.Cincinnati Children’s Hospital Medical CenterCincinnatiUSA
  5. 5.University of Cincinnati College of MedicineCincinnatiUSA
  6. 6.University of Southern CaliforniaLos AngelesUSA
  7. 7.University of MaineOronoUSA
  8. 8.Harvard UniversityCambridgeUSA

Personalised recommendations