Quality & Quantity

, Volume 50, Issue 3, pp 1385–1398 | Cite as

Research participant selection in non-English language questionnaire pretesting: findings from Chinese and Korean cognitive interviews

  • Hyunjoo Park
  • M. Mandy Sha
  • Murrey Olmsted


In survey research, cognitive interviewing has been widely used as a tool for pretesting to improve the quality of questionnaires. However, the cognitive interviewing method relies on interview participants’ ability to articulate their thought process by answering the probe questions asked during the interview. Including participants who exhibit difficulty in understanding the probe question as intended by the researcher and difficulty in providing the appropriate answer pertinent to the probes affects the quality of cognitive interview outcome. Using main interview and its paradata from 258 non-English language (Chinese and Korean) cognitive interviews which pretest the American Community Survey questionnaire, this paper examines how levels of participant difficulty exhibited during the non-English cognitive interview affect the interview outcome. Furthermore, this paper provides practical guidelines about participant selection to balance interview quality and cost for pretesting translated survey questionnaires. We found that interviews with participants who exhibited greater difficulty were lengthier and did not add more value to the findings; in addition, those participants were less likely to complete any questionnaires in the future.


Cognitive interview Non-English questionnaire pretesting Participant difficulty Asians 


  1. Ackermann, A.C., Blair, J.: Efficient sample selection. In: Proceedings of the Section on Survey Research Methods, Annual Meetings of the American Statistical Association, pp. 3997–4004 (2006)Google Scholar
  2. Alwin, D.F., McCammon, R.J.: Aging, cohorts, and verbal ability. J. Gerontol. Soc. Sci. 56B, S151–S161 (2001)CrossRefGoogle Scholar
  3. Barnes, L.L., Wilson, R.S., Hebert, L.E., Scherr, P.A., Evans, D.A., de Leon, C.F.M.: Racial differences in the association of education with physical and cognitive function in older blacks and whites. J. Gerontol. Ser. B 66(3), 354–363 (2011)CrossRefGoogle Scholar
  4. Beatty, P., Willis, G.: Research synthesis: the practice of cognitive interviewing. Public Opin. Q. 71(2), 287–311 (2007)CrossRefGoogle Scholar
  5. Blair, J., Conrad, F.: Sample size for cognitive interview pretesting. Public Opin. Q. 75(4), 636–658 (2011)CrossRefGoogle Scholar
  6. Borgers, N., deLeeuw, E., Hox, J.: Children as respondents in survey research: cognitive development and response quality. Bull. Methodol. Sociol. 66, 60–75 (2000)CrossRefGoogle Scholar
  7. Borgers, N., Hox, J.: Item nonresponse in questionnaire research with children. J. Off. Stat. 17(2), 321–335 (2001)Google Scholar
  8. Boone, K.B., Victor, T.L., Wen, J., Razini, J., Ponton, M.: The association between neuropsychological scores and ethnicity, language, and acculturation variables in a large patient population. Arch. Clin. Neuropsychol. 22, 355–365 (2007)CrossRefGoogle Scholar
  9. Brim Jr, O.G., Kagan, J.: Constancy and Change in Human Development. Harvard University Press, Cambridge (1980)Google Scholar
  10. Choi, H.: Discourse characteristics in healthy elderly: effects of aging, gender and education. J. Korean Soc. Speech Sci. 4(2), 135–143 (2012)CrossRefGoogle Scholar
  11. Fortuny, L., Heaton, R.K., Hermosillo, D.: Neuropsychological comparisons of Spanish-speaking participants from the US-Mexico border region versus Spain. J. Int. Neuropsychol. Soc. 4(4), 363–379 (1998)Google Scholar
  12. Glymour, M.M., Kawachi, I., Jencks, C.S., Berkman, L.F.: Does childhood schooling affect old age memory or mental status? Using state schooling laws as natural experiments. J. Epidemiol. Community Health 62(6), 532–537 (2008)CrossRefGoogle Scholar
  13. Goerman, P., Caspar, R., Sha, M., McAvinchey, Quiroz, R.: Census Bilingual Questionnaire Research Final Round 2 Report, Census Bureau SRD Report Series Survey Methodology #2007-27. U.S. Census Bureau, Washington D.C (2007)Google Scholar
  14. Gonzales, R., Roll, S.: Relationship between acculturation, cognitive-style, and intelligence—a cross-sectional study. J. Cross Cult. Psychol. 16(2), 190–205 (1985)CrossRefGoogle Scholar
  15. Hamby, S., Finkelhor, D.: The Victimization of children: recommendations for assessment and instrument development. J. Am. Acad. Child Adolesc. Psychiatry 39(7), 829–840 (2000)CrossRefGoogle Scholar
  16. Huppert, F.A., Johnson, T., Nickson, J.: High prevalence of prospective memory impairment in the elderly and in early-stage dementia: findings from a population based study. Appl. Cogn. Psychol. 14, 1–19 (2001)Google Scholar
  17. Kim, S., Choi, H.: Characteristics of verbal definitions in normal elderly Koreans. Korean J. Commun. Disord. 17, 107–117 (2012)Google Scholar
  18. Lee, T.M.C., Yuen, K.S.L., Chan, G.G.H.: Normative Data for neuropsychological measures of fluency, attention, and memory measures for Hong Kong Chinese. J. Clin. Exp. Neuropsychol. 24, 615–632 (2002)CrossRefGoogle Scholar
  19. Lin, H., Chan, R.C.K., Zheng, L., Yang, T., Wang, Y.: Executive functioning in healthy elderly Chinese people. Arch. Clin. Neuropsychol. 22(4), 501–511 (2007)CrossRefGoogle Scholar
  20. Marmot, M., Banks, J., Blundell, R., Lessof, C., Nazroo, J.: Health, Wealth and Lifestyles of Older Population in England: The 2002 English Longitudinal Studies of Ageing. Institute for Fiscal Studies, London (2003)Google Scholar
  21. Mazaux, J.M., Dartigues, J.F., Letenneur, L., Darriet, D., Wiart, L., Gagnon, M., Commenges, D., Boller, F.: Visuospatial attention and psychomotor performance in elderly community residents—effects of age, gender and education. J. Clin. Exp. Neuropsychol. 17(1), 71–81 (1995)CrossRefGoogle Scholar
  22. Mejia, S., Gutierrez, L.M., Villa, A.R., Ostrosky-Solis, F.: Cognition, functional status, education, and the diagnosis of dementia and mild cognitive impairment in Spanish-speaking elderly. Appl. Neuropsychol. 11(4), 196–203 (2004)CrossRefGoogle Scholar
  23. Myerson, J., Rank, M.R., Rainees, F.Q., Schnitzler, M.A.: Race and general cognitive skills: the myth of diminishing returns to education. Psychol. Sci. 9(2), 139–142 (1998)CrossRefGoogle Scholar
  24. Nicholas, L.E., Brookshire, R.H.: A system for quantifying the informativeness and efficiency of the connected speech of adults with aphasia. J. Speech Hear. Res. 36, 338–350 (1993)CrossRefGoogle Scholar
  25. Pan, Y.: Cognitive interviews in languages other than English: methodological and research issues. In: Proceedings of the Section on Survey Research Methods, Annual Meetings of the American Statistical Association, pp. 4859–4865 (2004)Google Scholar
  26. Pan, Y., Fond, M.: Evaluating Multilingual Questionnaires: A Sociolinguistic Perspective. Statistical Research Division Research Report Series (Survey Methodology #2012-4). U.S. Census Bureau, Washington, D.C (2012)Google Scholar
  27. Park, H., Sha, M.: Evaluating the efficiency of methods to recruit Asian research participants. J. Off. Stat. 30(2), 335–354 (2014)Google Scholar
  28. Raji, T., Tenneti, Johnson, D., Goldenberg, L., Parker, R.A., Huppert, F.A.: Towards a capabilities database to inform inclusive design: experimental investigation of effective survey-based predictors of human-product interaction. Appl. Ergon. 43, 713–726 (2012)CrossRefGoogle Scholar
  29. Razani, J., Burciaga, J., Madore, M., Wong, J.: Effects of culture on tests of attention and information processing in an ethnically diverse group. Arch. Clin. Neuropsychol. 22, 333–341 (2007)CrossRefGoogle Scholar
  30. Ridolfo, H., Schoua-Glusberg, A.: Analyzing cognitive interview data using the constant comparative method of analysis to understand cross-cultural patterns in survey data. Field Methods 23(4), 420–438 (2011)CrossRefGoogle Scholar
  31. Schaie, K.W.: The hazards of cognitive ageing. Gerontologist 29, 484–493 (1989)CrossRefGoogle Scholar
  32. Sha, M., Park, H., Pan, Y. Adapting translation of the American Community Survey in Chinese and Korean. Presented at American Association for Public Opinion Research Conference, Boston (2013)Google Scholar
  33. Snitz, B.E., Unverzagt, F.W., Chang, C.H., Vander Bilt, J., Gao, S.J., Saxton, J., Hall, K.S., Gonguli, M.: Effects of age, gender, education and race on two tests of language ability in community-based older adults. Int. Psychogeriatr. 21(6), 1051–1062 (2009)CrossRefGoogle Scholar
  34. Sohn, E., Kang, S.: A study on story retelling in the adults. J. Speech Lang. Hear. Disord. 15(1), 71–85 (2006)Google Scholar
  35. Sung, J., Kim, J.: Differential performance on generative naming and typicality rating between young and older adults. J. Speech Lang. Hear. Disord. 20(1), 31–50 (2011)CrossRefGoogle Scholar
  36. Tate, R., Smeeth, L., Evans, J., Fletcher, A.: The prevalence of visual impairment in the UK. RNIBlind Web (2005). Accessed 30 March 2013
  37. Varni, J.W., Limbers, C.A., Burwinkle, T.M.: How young can children reliably and validly self-report their health-related quality of life: an analysis of 8,591 children across age subgroups with the PedsQL (TM) 4.0 Generic Core Scales. Health Qual. Life Outcomes (2007). doi: 10.1186/1477-7525-5-1 Google Scholar
  38. Weber, L., Miracle, A., Skehan, T.: Interviewing early adolescents: some methodological considerations. Hum. Organ. 53(1), 42–47 (1994)CrossRefGoogle Scholar
  39. Willis, G.B.: Cognitive interviewing: A Tool for Improving Questionnaire Design. Sage, Thousand Oaks (2005)Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  1. 1.RTI InternationalSan FranciscoUSA
  2. 2.RTI InternationalChicagoUSA
  3. 3.RTI InternationalResearch Triangle ParkUSA

Personalised recommendations