Skip to main content

Advertisement

SpringerLink
Log in

Research participant selection in non-English language questionnaire pretesting: findings from Chinese and Korean cognitive interviews

  • Published:
Quality & Quantity Aims and scope Submit manuscript

Abstract

In survey research, cognitive interviewing has been widely used as a tool for pretesting to improve the quality of questionnaires. However, the cognitive interviewing method relies on interview participants’ ability to articulate their thought process by answering the probe questions asked during the interview. Including participants who exhibit difficulty in understanding the probe question as intended by the researcher and difficulty in providing the appropriate answer pertinent to the probes affects the quality of cognitive interview outcome. Using main interview and its paradata from 258 non-English language (Chinese and Korean) cognitive interviews which pretest the American Community Survey questionnaire, this paper examines how levels of participant difficulty exhibited during the non-English cognitive interview affect the interview outcome. Furthermore, this paper provides practical guidelines about participant selection to balance interview quality and cost for pretesting translated survey questionnaires. We found that interviews with participants who exhibited greater difficulty were lengthier and did not add more value to the findings; in addition, those participants were less likely to complete any questionnaires in the future.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

Notes

  1. We did not analyze the effect of acculturation separately in this study because all of the research participants included in the dataset were unacculturated immigrants—they were non-English speakers (an eligibility criterion to participate). Since they are the intended users of the translations, they should be selected regardless of their levels of interview difficulty to elicit directly relevant information to improve survey translation.

References

  • Ackermann, A.C., Blair, J.: Efficient sample selection. In: Proceedings of the Section on Survey Research Methods, Annual Meetings of the American Statistical Association, pp. 3997–4004 (2006)

  • Alwin, D.F., McCammon, R.J.: Aging, cohorts, and verbal ability. J. Gerontol. Soc. Sci. 56B, S151–S161 (2001)

    Article  Google Scholar 

  • Barnes, L.L., Wilson, R.S., Hebert, L.E., Scherr, P.A., Evans, D.A., de Leon, C.F.M.: Racial differences in the association of education with physical and cognitive function in older blacks and whites. J. Gerontol. Ser. B 66(3), 354–363 (2011)

    Article  Google Scholar 

  • Beatty, P., Willis, G.: Research synthesis: the practice of cognitive interviewing. Public Opin. Q. 71(2), 287–311 (2007)

    Article  Google Scholar 

  • Blair, J., Conrad, F.: Sample size for cognitive interview pretesting. Public Opin. Q. 75(4), 636–658 (2011)

    Article  Google Scholar 

  • Borgers, N., deLeeuw, E., Hox, J.: Children as respondents in survey research: cognitive development and response quality. Bull. Methodol. Sociol. 66, 60–75 (2000)

    Article  Google Scholar 

  • Borgers, N., Hox, J.: Item nonresponse in questionnaire research with children. J. Off. Stat. 17(2), 321–335 (2001)

    Google Scholar 

  • Boone, K.B., Victor, T.L., Wen, J., Razini, J., Ponton, M.: The association between neuropsychological scores and ethnicity, language, and acculturation variables in a large patient population. Arch. Clin. Neuropsychol. 22, 355–365 (2007)

    Article  Google Scholar 

  • Brim Jr, O.G., Kagan, J.: Constancy and Change in Human Development. Harvard University Press, Cambridge (1980)

    Google Scholar 

  • Choi, H.: Discourse characteristics in healthy elderly: effects of aging, gender and education. J. Korean Soc. Speech Sci. 4(2), 135–143 (2012)

    Article  Google Scholar 

  • Fortuny, L., Heaton, R.K., Hermosillo, D.: Neuropsychological comparisons of Spanish-speaking participants from the US-Mexico border region versus Spain. J. Int. Neuropsychol. Soc. 4(4), 363–379 (1998)

    Google Scholar 

  • Glymour, M.M., Kawachi, I., Jencks, C.S., Berkman, L.F.: Does childhood schooling affect old age memory or mental status? Using state schooling laws as natural experiments. J. Epidemiol. Community Health 62(6), 532–537 (2008)

    Article  Google Scholar 

  • Goerman, P., Caspar, R., Sha, M., McAvinchey, Quiroz, R.: Census Bilingual Questionnaire Research Final Round 2 Report, Census Bureau SRD Report Series Survey Methodology #2007-27. U.S. Census Bureau, Washington D.C (2007)

    Google Scholar 

  • Gonzales, R., Roll, S.: Relationship between acculturation, cognitive-style, and intelligence—a cross-sectional study. J. Cross Cult. Psychol. 16(2), 190–205 (1985)

    Article  Google Scholar 

  • Hamby, S., Finkelhor, D.: The Victimization of children: recommendations for assessment and instrument development. J. Am. Acad. Child Adolesc. Psychiatry 39(7), 829–840 (2000)

    Article  Google Scholar 

  • Huppert, F.A., Johnson, T., Nickson, J.: High prevalence of prospective memory impairment in the elderly and in early-stage dementia: findings from a population based study. Appl. Cogn. Psychol. 14, 1–19 (2001)

    Google Scholar 

  • Kim, S., Choi, H.: Characteristics of verbal definitions in normal elderly Koreans. Korean J. Commun. Disord. 17, 107–117 (2012)

    Google Scholar 

  • Lee, T.M.C., Yuen, K.S.L., Chan, G.G.H.: Normative Data for neuropsychological measures of fluency, attention, and memory measures for Hong Kong Chinese. J. Clin. Exp. Neuropsychol. 24, 615–632 (2002)

    Article  Google Scholar 

  • Lin, H., Chan, R.C.K., Zheng, L., Yang, T., Wang, Y.: Executive functioning in healthy elderly Chinese people. Arch. Clin. Neuropsychol. 22(4), 501–511 (2007)

    Article  Google Scholar 

  • Marmot, M., Banks, J., Blundell, R., Lessof, C., Nazroo, J.: Health, Wealth and Lifestyles of Older Population in England: The 2002 English Longitudinal Studies of Ageing. Institute for Fiscal Studies, London (2003)

    Google Scholar 

  • Mazaux, J.M., Dartigues, J.F., Letenneur, L., Darriet, D., Wiart, L., Gagnon, M., Commenges, D., Boller, F.: Visuospatial attention and psychomotor performance in elderly community residents—effects of age, gender and education. J. Clin. Exp. Neuropsychol. 17(1), 71–81 (1995)

    Article  Google Scholar 

  • Mejia, S., Gutierrez, L.M., Villa, A.R., Ostrosky-Solis, F.: Cognition, functional status, education, and the diagnosis of dementia and mild cognitive impairment in Spanish-speaking elderly. Appl. Neuropsychol. 11(4), 196–203 (2004)

    Article  Google Scholar 

  • Myerson, J., Rank, M.R., Rainees, F.Q., Schnitzler, M.A.: Race and general cognitive skills: the myth of diminishing returns to education. Psychol. Sci. 9(2), 139–142 (1998)

    Article  Google Scholar 

  • Nicholas, L.E., Brookshire, R.H.: A system for quantifying the informativeness and efficiency of the connected speech of adults with aphasia. J. Speech Hear. Res. 36, 338–350 (1993)

    Article  Google Scholar 

  • Pan, Y.: Cognitive interviews in languages other than English: methodological and research issues. In: Proceedings of the Section on Survey Research Methods, Annual Meetings of the American Statistical Association, pp. 4859–4865 (2004)

  • Pan, Y., Fond, M.: Evaluating Multilingual Questionnaires: A Sociolinguistic Perspective. Statistical Research Division Research Report Series (Survey Methodology #2012-4). U.S. Census Bureau, Washington, D.C (2012)

    Google Scholar 

  • Park, H., Sha, M.: Evaluating the efficiency of methods to recruit Asian research participants. J. Off. Stat. 30(2), 335–354 (2014)

    Google Scholar 

  • Raji, T., Tenneti, Johnson, D., Goldenberg, L., Parker, R.A., Huppert, F.A.: Towards a capabilities database to inform inclusive design: experimental investigation of effective survey-based predictors of human-product interaction. Appl. Ergon. 43, 713–726 (2012)

    Article  Google Scholar 

  • Razani, J., Burciaga, J., Madore, M., Wong, J.: Effects of culture on tests of attention and information processing in an ethnically diverse group. Arch. Clin. Neuropsychol. 22, 333–341 (2007)

    Article  Google Scholar 

  • Ridolfo, H., Schoua-Glusberg, A.: Analyzing cognitive interview data using the constant comparative method of analysis to understand cross-cultural patterns in survey data. Field Methods 23(4), 420–438 (2011)

    Article  Google Scholar 

  • Schaie, K.W.: The hazards of cognitive ageing. Gerontologist 29, 484–493 (1989)

    Article  Google Scholar 

  • Sha, M., Park, H., Pan, Y. Adapting translation of the American Community Survey in Chinese and Korean. Presented at American Association for Public Opinion Research Conference, Boston (2013)

  • Snitz, B.E., Unverzagt, F.W., Chang, C.H., Vander Bilt, J., Gao, S.J., Saxton, J., Hall, K.S., Gonguli, M.: Effects of age, gender, education and race on two tests of language ability in community-based older adults. Int. Psychogeriatr. 21(6), 1051–1062 (2009)

    Article  Google Scholar 

  • Sohn, E., Kang, S.: A study on story retelling in the adults. J. Speech Lang. Hear. Disord. 15(1), 71–85 (2006)

    Google Scholar 

  • Sung, J., Kim, J.: Differential performance on generative naming and typicality rating between young and older adults. J. Speech Lang. Hear. Disord. 20(1), 31–50 (2011)

    Article  Google Scholar 

  • Tate, R., Smeeth, L., Evans, J., Fletcher, A.: The prevalence of visual impairment in the UK. RNIBlind Web http://www.rnib.org.uk/xpedio/groups/public/documents/PublicWebsite/public_prevalencereport.doc (2005). Accessed 30 March 2013

  • Varni, J.W., Limbers, C.A., Burwinkle, T.M.: How young can children reliably and validly self-report their health-related quality of life: an analysis of 8,591 children across age subgroups with the PedsQL (TM) 4.0 Generic Core Scales. Health Qual. Life Outcomes (2007). doi:10.1186/1477-7525-5-1

    Google Scholar 

  • Weber, L., Miracle, A., Skehan, T.: Interviewing early adolescents: some methodological considerations. Hum. Organ. 53(1), 42–47 (1994)

    Article  Google Scholar 

  • Willis, G.B.: Cognitive interviewing: A Tool for Improving Questionnaire Design. Sage, Thousand Oaks (2005)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. RTI International, 351 California Street, Suite 500, San Francisco, CA, 94104, USA

    Hyunjoo Park

  2. RTI International, 230 W. Monroe St Suite 2100, Chicago, IL, 60606, USA

    M. Mandy Sha

  3. RTI International, 3040 East Cornwallis Road, Post Office Box 12194, Research Triangle Park, NC, 27709, USA

    Murrey Olmsted

Authors
  1. Hyunjoo Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. Mandy Sha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Murrey Olmsted
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hyunjoo Park.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Park, H., Sha, M.M. & Olmsted, M. Research participant selection in non-English language questionnaire pretesting: findings from Chinese and Korean cognitive interviews. Qual Quant 50, 1385–1398 (2016). https://doi.org/10.1007/s11135-015-0211-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11135-015-0211-3

Keywords

Search

Navigation