Abstract
In audits, as in all experiments, researchers are confronted with choices about whether to collect and analyze repeated measures on the unit of analysis. In typical social science practice, this decision often involves consideration of whether to send single or multiple auditors to test for discrimination at a site that represents the unit of analysis, such as employers, landlords, or schools. In this chapter, we provide tools for researchers considering the statistical and substantive implications of this decision. For the former, we show how sample size and statistical efficiency questions hinge in large part on the expected concordance of outcomes when testers are sent to the same unit or site. For the latter, we encourage researchers to think carefully about what is gained and lost via matched and non-matched designs, particularly regarding the finite nature of certain populations, resource constraints, and the likelihood of detection in the field. For both approaches, we make recommendations for the appropriate statistical analysis in light of the given design and direct readers to software and code that may be helpful in informing design choices.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Berk, R. A. (2005). Randomized experiments as the bronze standard. Journal of Experimental Criminology, 1, 417–433.
Bertrand, M., & Mullainathan, S. (2004). Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. The American Economic Review, 94, 991–1013.
Chen, D., & Peace, K. E. (2011). Clinical trial data analysis using R. Boca Raton: CRC Press.
Cochran, W. G. (1950). The comparison of percentages in matched samples. Biometrika, 37, 256–266.
Correll, S. J., Benard, S., & Paik, I. (2007). Getting a job: Is there a motherhood penalty? The American Journal of Sociology, 112, 1297–1339.
Dalgaard, P. (2008). Introductory statistics with R (2nd ed.). New York: Springer.
Decker, S. H., Ortiz, N., Spohn, C., et al. (2015). Criminal stigma, race, and ethnicity: The consequences of imprisonment for employment. Journal of Criminal Justice, 43, 108–121.
Desmond, M. (2016). Evicted: Poverty and profit in the American city. New York: Crown.
Donner, A., & Li, K. Y. R. (1990). The relationship between chi-square statistics from matched and unmatched analyses. Journal of Clinical Epidemiology, 43, 827–831.
Evans, D. N., & Porter, J. R. (2015). Criminal history and landlord rental decisions: A New York quasi-experiment study. Journal of Experimental Criminology, 11, 21–42.
Gaddis, S. M. (2015). Discrimination in the credential society: An audit study of race and college selectivity in the labor market. Social Forces, 93, 1451–1479.
Gaddis, S. M. (2018). An introduction to audit studies in the social sciences. In S. M. Gaddis (Ed.), Audit studies: Behind the scenes with theory, method, and nuance. Cham: Springer International Publishing.
Gaddis, S. M., & Ghoshal, R. (2015). Arab American housing discrimination, ethnic competition, and the contact hypothesis. The Annals of the American Academy of Political and Social Science, 660, 282–299.
Ghoshal, R., & Gaddis, S. M. (2015). Finding a roommate on craigslist: Racial discrimination and residential segregation. Available at SSRN: https://ssrn.com/abstract=2605853
Halaby, C. N. (2004). Panel models in sociological research: Theory into practice. Annual Review of Sociology, 30, 507–544.
Hedberg, E. C., & Ayers, S. (2015). The power of a paired t-test with a covariate. Social Science Research, 50, 277–291.
Hipes, C., Lucas, J., Phelan, J. C., et al. (2016). The stigma of mental illness in the labor market. Social Science Research, 56, 16–25.
Hogan, B., & Berry, B. (2011). Racial and ethnic biases in rental housing: An audit study of online apartment listings. City Community, 10, 351–372.
Kramer, M. S. (1991). Clinical biostatistics: An overview. In H. Troidl, W. O. Spitzer, D. S. Mulder, et al. (Eds.), Principles and practice of research: Strategies for surgical investigators (2nd ed., pp. 126–143). New York: Springer.
Kugelmass, H. (2016). ‘Sorry, I’m not accepting new patients’: an audit study of access to mental health care. Journal of Health and Social Behavior, 57, 168–183.
Lahey, J., & Beasley, R. (2018). Technical aspects of correspondence studies. In S. M. Gaddis (Ed.), Audit studies: Behind the scenes with theory, method, and nuance. Cham: Springer International Publishing.
Lauster, N., & Easterbrook, A. (2011). No room for new families? A field experiment measuring rental discrimination against same-sex couples and single parents. Social Problems, 58, 389–409.
McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12, 153–157.
Michel, E. (2016). Discrimination against queer women in the U.S. workforce: A resume audit study. Socius, 2, 1–13.
Pager, D. (2003). The mark of a criminal record. The American Journal of Sociology, 108, 937–975.
Pager, D. (2007). The use of field experiments for studies of employment discrimination: Contributions, critiques, and directions for the future. The Annals of the American Academy of Political and Social Science, 609, 104–133.
Pager, D., Western, B., & Bonikowski, B. (2009). Discrimination in a low-wage labor market a field experiment. American Sociological Review, 74, 777–799.
Pedulla, D. S. (2016). Penalized or protected? Gender and the consequences of nonstandard and mismatched employment histories. American Sociological Review, 81, 262–289.
Quillian, L. (2006). New approaches to understanding racial prejudice and discrimination. Annual Review of Sociology, 32, 299–328.
R Core Team. (2015). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna. Available via http://www.R-project.org
Rivera, L. A., & Tilcsik, A. (2016). Class advantage, commitment penalty: The gendered effect of social class signals in an elite labor market. American Sociological Review, 81, 1097–1131.
Rosner, B. (2011). Fundamentals of biostatistics (7th ed.). Boston: Brooks/Cole.
Schwartz, R. D., & Skolnick, J. H. (1962). Two studies of legal stigma. Social Problems, 10, 133–142.
Shih, W. J., & Aisner, J. (2016). Statistical design and analysis of clinical trials: Principles and methods. New York: CRC Press.
Stewart, R., & Uggen, C. (2016, November 17). A modified experimental audit of criminal records and college admissions. Paper presented at the American Society of Criminology Meetings, New Orleans.
Tilcsik, A. (2011). Pride and prejudice: Employment discrimination against openly gay men in the United States. The American Journal of Sociology, 117, 586–626.
Uggen, C., Vuolo, M., Lageson, S., et al. (2014). The edge of stigma: An experimental audit of the effects of low-level criminal records on employment. Criminology, 52, 627–654.
Vuolo, M., Uggen, C., & Lageson, S. (2016). Statistical power in experimental audit studies: Cautions and calculations for matched tests with nominal outcomes. Sociological Methods & Research, 45, 260–303.
Vuolo, M., Uggen, C., & Lageson, S. (2017). Race, recession, and social closure in the low wage labor market: Experimental and observational evidence. Research in the Sociology of Work, 30, 141–183.
Wallace, M., Wright, B. R. E., & Hyde, A. (2014). Religious affiliation and hiring discrimination in the American south: A field experiment. Social Currents, 1, 189–207.
Widner, D., & Chicoine, S. (2011). It’s all in the name: Employment discrimination against Arab Americans. Sociological Forum, 26, 806–823.
Winship, C., & Morgan, S. L. (1999). The estimation of causal effects from observational data. Annual Review of Sociology, 25, 659–706.
Wright, B. R. E., Wallace, M., Bailey, J., et al. (2013). Religious affiliation and hiring discrimination in New England: A field experiment. Research in Social Stratification and Mobility, 34, 111–126.
Wright, B. R. E., Wallace, M., Wisnesky, A. S., et al. (2015). Religion, race, and discrimination: A field experiment of how American churches welcome newcomers. Journal for the Scientific Study of Religion, 54, 185–204.
Acknowledgments
The illustrative empirical data used in this article come from a study conducted in partnership with the Council on Crime and Justice and supported by the JEHT Foundation and the National Institute of Justice [grant number 2007-IJ-CX-0042]. We are grateful to Laura DeMarco for the example of a landlord audit and Rob Stewart for the example of college admissions, with each coming from their respective dissertations. The R functions referenced herein, including instructions for use, are publicly available on the first author’s website.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this chapter
Cite this chapter
Vuolo, M., Uggen, C., Lageson, S. (2018). To Match or Not to Match? Statistical and Substantive Considerations in Audit Design and Analysis. In: Gaddis, S. (eds) Audit Studies: Behind the Scenes with Theory, Method, and Nuance. Methodos Series, vol 14. Springer, Cham. https://doi.org/10.1007/978-3-319-71153-9_6
Download citation
DOI: https://doi.org/10.1007/978-3-319-71153-9_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-71152-2
Online ISBN: 978-3-319-71153-9
eBook Packages: Social SciencesSocial Sciences (R0)