Because of the prominent role played by surveys in higher education research and assessment, declining response rates are a source of concern. Low response rates increase the likelihood that estimators of population parameters will be both imprecise and systematically biased. This chapter describes four approaches that can be used to adjust for nonresponse: population weighting, sample weighting, raking ratio estimation, and response-propensity weighting. Although these methods can adjust for nonresponse, their effectiveness is limited by four factors: (1) weighting will not allow inferences to be made about a population based on data from a convenience sample; (2) weighting will not compensate for nonresponse when there are differences between respondents and nonrespondents on survey variables; (3) gains in the reduction of bias will be at least partly offset by loss of precision; and (4) researchers who use weighting adjustments must also use variance estimates that are appropriate for the weighting method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Astin, A. W. (1993). What matters in college? Four critical years revisited. San Francisco: Jossey-Bass.
Astin, A. W., & Molm, L. D. (1972). Correcting for nonresponse bias in follow-up surveys. Unpublished manuscript, American Council on Education, Washington, DC.
Bethlehem, J. G. (2002). Weighting nonresponse adjustments based on auxiliary information. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 275–287). New York: Wiley.
Chapman, D. W. (1976). A survey of nonresponse imputation procedures. Proceedings of the Social Statistics Section, American Statistical Association, 1976(1), 245–251.
Curtin, R., Presser, S., & Singer, E. (2000). The effects of response rate changes on the index of consumer sentiment. Public Opinion Quarterly, 64, 413–428.
de Leeuw, E., & de Heer, W. (2002). Trends in household survey nonresponse: A longitudinal and international comparison. In R. M. Groves, D. A. Dillman,J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 41–54). New York: Wiley.
Deming, W. E. (1943). Statistical adjustment of data. New York: Wiley.
Deming, W. E., & Stephan, F. F. (1940). On a least squares adjustment of a sample frequency table when the expected marginal totals are known. Annals of Mathematical Statistics, 11, 427–444.
Dey, E. L. (1997). Working with low survey response rates: The efficacy of weighting adjustments. Research in Higher Education, 38,215–227.
Dey, E. L., Hurtado, S., Rhee, B., Inkelas, K. K., Wimsatt, L. A., & Guan, F. (1997). Improving research on postsecondary student outcomes: A review of the strengths and limitations of national data resources. Stanford, CA: National Center for Postsecondary Improvement.
Dillman, D. A. (2000). Mail and internet surveys: The tailored design method (2nd ed.). New York: Wiley.
Dillman, D. A., Eltinge, J. L., Groves, R. M., & Little, R. J. A. (2002). Survey nonresponse in design, data collection, and analysis. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 3–26). New York: Wiley.
Drew, J. H., & Fuller, W. A. (1980). Modeling nonresponse in surveys with callbacks. Proceedings of the Section on Survey Research Methods, American Statistical Association, 1980,639–642.
Drew, J. H., & Fuller, W. A. (1981). Nonresponse in complex multiphase surveys. Proceedings of the Section on Survey Research Methods, American Statistical Association, 1981,623–628.
DuMouchel, W. H., & Duncan, G. J. (1983). Using survey weights in multiple regression analyses of stratified samples. Journal of the American Statistical Association, 78,535–543.
Gelman, A., & Carlin, J. B. (2002). Poststratification and weighting adjustments. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 289–302). New York: Wiley.
Goyder, J. (1987). The silent majority: Nonrespondents on sample surveys. Boulder, CO: Westview Press.
Goyder, J., Warriner, K., & Miller, S. (2002). Evaluating socio-economic status (SES) bias in survey nonresponse. Journal of Official Statistics, 18, 1–11.
Groves, R. M. (1989). Survey errors and survey costs. New York: Wiley.
Groves, R. M., Cialdini, R. B., & Couper, M. P. (1992). Understanding the decision to participate in a survey. Public Opinion Quarterly, 56, 475–495.
Groves, R. M., & Couper, M. P. (1998). Nonresponse in household interview surveys. New York: Wiley.
Groves, R. M., Presser, S., & Dipko, S. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68, 2–31.
Groves, R. M., Singer, E., & Corning, A. (2000). Leverage-salience theory of survey participation: Description and an illustration. Public Opinion Quarterly, 64, 299–308.
Holt, D., & Smith, T. M. F. (1979). Post stratification. Journal of the Royal Statistical Society, Series A (General), 142(1), 33–46.
Hutchinson, J., Tollefson, N., & Wigington, H. (1987). Response bias in college freshmen’s responses to mail surveys. Research in Higher Education, 26, 99–106.
Jones, J. (1996). The effects of non-response on statistical inference. Journal of health and Social Policy, 8, 49–62.
Kalton, G. (1983a). Compensating for missing survey data. Ann Arbor: University of Michigan Institute for Social Research.
Kalton, G. (1983b). Introduction to survey sampling (Sage University Paper series on Quantitative Applications in the Social Sciences, 07-035). Beverly Hills, CA: Sage.
Kalton, G., & Kasprzyk, D. (1986). The treatment of missing survey data. Survey Methodology, 12, 1–16.
Keeter, S., Miller, C., Kohut, A., Groves, R. M., & Presser, S. (2000). Consequences of reducing nonresponse in a national telephone survey. Public Opinion Quarterly, 64, 125–148.
Kish, L. (1965). Survey sampling. New York: Wiley.
Kuh, G. D. (2001). The National Survey of Student Engagement: Conceptual framework and overview of psychometric properties. Bloomington, IN: Indiana University Center for Postsecondary Research. Retrieved February 24, 2006 from the national Survey of Student Engagement Web site: http://nsse.iub.edu/html/psychometric_framework_ 2002.cfm.
Kuh, G. D., Hayek, J. C., Carini, R. M., Ouimet, J. A., Gonyea, R. M., & Kennedy, J. (2001). NSSE technical and norms report. Bloomington, IN: Indiana University Center for Postsecondary Research.
Levy, P. S., & Lemeshow, S. (1999). Sampling of populations: Methods and applications (3rd ed.). New York: Wiley.
Lindholm, J. A., Szelényi, K., Hurtado, S., & Korn, W. S. (2005). The American college teacher: national norms for the 2004–2005 HERI faculty survey. Los Angeles, CA: University of California, Los Angeles Higher Education Research Institute.
Little, R. J. A. (1986). Survey nonresponse adjustments. International Statistical Review, 54, 139–157.
Little, R. J. A. (1993). Post-stratification: A modeler’s perspective. Journal of the American Statistical Association, 88, 1001–1012.
Merkle, D. M., & Edelman, M. (2002). Nonresponse in exit polls: A comprehensive analysis. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 243–257). New York: Wiley.
Marti, C. N. (n. d.). Overview of the CCSSE instrument and psychometric properties. Retrieved February 2, 2006 from the Community College Survey of Student Engagement Web site: http://www.ccsse.org/aboutsurvey/psychometrics.pdf.
Moore, D. L., & Tarnai, J. (2002). Evaluating nonresponse error in mail surveys. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 197–211). New York: Wiley.
Muthén, B. O., & Satorra, A. (1995). Complex sample data in structural equation modeling. In P. V. Marsden (Ed.), Sociological methodology 1995 (Vol. 25, pp. 267–316). Boston: Blackwell.
National Center for Education Statistics. (1997, October). National Postsecondary Student Aid Study, 1995–96 (NPSAS:96) methodology report. NCES technical report 98-073. Washington, DC: U. S. Department of Education, Office of Educational Research and Improvement.
National Center for Education Statistics. (2002a, June). 1999 National Study of Postsecondary Faculty (NSOPF:99) methodology report. NCES technical report 2002-154. Washington, DC: U. S. Department of Education, Office of Educational Research and Improvement.
National Center for Education Statistics. (2002b). Statistical standards. Retrieved January 3, 2006 from the NCES Web site: http://nces.ed.gov/statprog/2002/ stdintro.asp.
National Survey of Student Engagement (2005). 2005 revisions to the NSSE benchmarks. Retrieved January 20, 2006 from the NSSE Web site: http://webdb.iu.edu/Nsse/NSSE_2005_Annual_Report/benchmarks_weighting.cfm? subtab=Weighting&tab=Benchmarks.
Oh, H. L., & Scheuren, F. (1983). Weighting and adjustment for unit nonresponse. In W. G. Madow, I. Olkin, & D. B. Rubin (Eds.), Incomplete data in sample surveys, volume 2, theory and bibliographies (pp. 143–184). New York: Academic Press.
Pascarella, E. T., Whitt, E. J., Nora, A., Edison, M., Hagedorn, L. S., & Terenzini, P. T. (1996). What have we learned from the first year of the national Study of Student learning? Journal of College Student Development, 37, 182–192.
Peterson, M. W., Einarson, M. K., Augustine, C. H., & Vaughan, D. S. (1999). Institutional support for student assessment: Methodology and results of a national survey. Stanford, CA: National Center for Postsecondary Improvement.
Pfeffermann, D. (1993). The role of sampling weights when modeling survey data. International Statistical Review, 61, 317–337.
Pfeffermann, D. (1996). The use of sampling weights for survey data analysis. Statistical Methods in Medical Research, 5, 239–261.
Politz, A., & Simmons, W. (1949a). An attempt to get the ‘not at homes’ into the sample without callbacks. Journal of the American Statistical Association, 44, 9–16.
Politz, A., & Simmons, W. (1949b). Further theoretical considerations regarding the plan for eliminating callbacks. Journal of the American Statistical Association, 44, 17–31.
Politz, A., & Simmons, W. (1950). Note on an attempt to get the ‘not at homes’ into the sample without callbacks. Journal of the American Statistical Association, 45, 136–137.
Porter, S. R., & Umbach, P. D. (2006). Student survey response rates across institutions: Why do they vary? Research in Higher Education, 47, 229–247.
Porter, S. R., & Whitcomb, M. E. (2005). Non-response in student surveys: The role of demographics, engagement, and personality. Research in Higher Education, 46, 127–152.
Pryor, J. H., Hurtado, S., Saenz, V. B., Lindholm, J. A., Korn, W. S., & Mahoney, K. M. (2005). The American freshman: National norms for fall 2005. Los Angeles, CA: University of California, Los Angeles Higher Education Research Institute.
Redline, C. D., & Dillman, D. A. (2002). The influence of alternative visual designs on respondents’ performance with branching instructions in self-administered questionnaires. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 179–193). New York: Wiley.
Thomas, S. L. (2006). Sampling: Rationale and rigor in choosing what to observe. In C. F. Conrad & R. C. Serlin (Eds.), The SAGE handbook for research in education: Engaging ideas and enriching inquiry (pp. 393–404). Thousand Oaks, CA: Sage.
Thomas, S. L., & Heck, R. H. (2001) Analysis of large-scale secondary data in higher education research: Potential perils associated with complex sampling designs. Research in Higher Education, 42, 517–540.
Thomas, S. L., Heck, R. H., & Bauer, K. W. (2005). Weighting and adjusting for design effects in secondary data analyses. In P. D. Umbach (Ed.), Survey research: Emerging issues (New Directions for Institutional Research Series, no. 127, pp. 51–72). San Francisco: Jossey-Bass.
Thomsen, I. (1973). A note on the efficiency of weighting subclass means to reduce the effects of non-response when analyzing survey data. Statistisk Tidskrift, 11, 278–283.
Tremblay, V. (1986). Practical criteria for definition of weighting classes. Survey Methodology, 12, 85–97.
Vartivarian, S. L. (2004). On the formation of weighting class adjustments for unit nonresponse in sample surveys. (Doctoral dissertation, University of Michigan, 2004). Ann Arbor, MI: ProQuest. (UMI No. 3137952).
Warwick, D. P., & Lininger, C. A. (1975). The sample survey: Theory and practice. New York: McGraw-Hill.
Winship, C., & Radbill, L. (1994). Sampling weights and regression analysis. Sociological methods and Research, 23, 230–257.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2007 Springer
About this chapter
Cite this chapter
Gary, P.R. (2007). Adjusting for Nonresponse in Surveys. In: Smart, J.C. (eds) Higher Education: Handbook of Theory and Research. Higher Education: Handbook of Theory and Research, vol 22. Springer, Dordrecht. https://doi.org/10.1007/978-1-4020-5666-6_8
Download citation
DOI: https://doi.org/10.1007/978-1-4020-5666-6_8
Publisher Name: Springer, Dordrecht
Print ISBN: 978-1-4020-5665-9
Online ISBN: 978-1-4020-5666-6
eBook Packages: Humanities, Social Sciences and LawEducation (R0)