Quality of Life Research

, Volume 27, Issue 4, pp 1077–1088 | Cite as

Careless responding in internet-based quality of life assessments

Article

Abstract

Purpose

Quality of life (QoL) measurement relies upon participants providing meaningful responses, but not all respondents may pay sufficient attention when completing self-reported QoL measures. This study examined the impact of careless responding on the reliability and validity of Internet-based QoL assessments.

Methods

Internet panelists (n = 2000) completed Patient-Reported Outcomes Measurement Information System (PROMIS®) short-forms (depression, fatigue, pain impact, applied cognitive abilities) and single-item QoL measures (global health, pain intensity) as part of a larger survey that included multiple checks of whether participants paid attention to the items. Latent class analysis was used to identify groups of non-careless and careless responders from the attentiveness checks. Analyses compared psychometric properties of the QoL measures (reliability of PROMIS short-forms, correlations among QoL scores, “known-groups” validity) between non-careless and careless responder groups. Whether person-fit statistics derived from PROMIS measures accurately discriminated careless and non-careless responders was also examined.

Results

About 7.4% of participants were classified as careless responders. No substantial differences in the reliability of PROMIS measures between non-careless and careless responder groups were observed. However, careless responding meaningfully and significantly affected the correlations among QoL domains, as well as the magnitude of differences in QoL between medical and disability groups (presence or absence of disability, depression diagnosis, chronic pain diagnosis). Person-fit statistics significantly and moderately distinguished between non-careless and careless responders.

Conclusions

The results support the importance of identifying and screening out careless responders to ensure high-quality self-report data in Internet-based QoL research.

Keywords

Quality of life Patient-reported outcomes Careless responding Inattentive responding Person-fit statistics 

Notes

Acknowledgements

We would like to thank Margaret Gatz, PhD, and Doerte U. Junghaenel, PhD, for their comments on the study design and helpful discussions in preparation of this manuscript.

Compliance with ethical standards

Conflict of interest

A.A.S. is a Senior Scientist with the Gallup Organization and a consultant with Adelphi Values, inc. S.S. and M.M. declare that they have no conflict of interest.

Ethical approval

The study was approved by the University of Southern California Institutional Review Board. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

References

  1. 1.
    Cella, D., Gershon, R., Lai, J.-S., & Choi, S. (2007). The future of outcomes measurement: Item banking, tailored short-forms, and computerized adaptive assessment. Quality of Life Research, 16, 133–141.CrossRefPubMedGoogle Scholar
  2. 2.
    Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S., et al. (2010). The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology, 63, 1179–1194.CrossRefPubMedCentralPubMedGoogle Scholar
  3. 3.
    Eysenbach, G., & Wyatt, J. (2002). Using the Internet for surveys and health research. Journal of Medical Internet Research, 4, e13.CrossRefPubMedCentralPubMedGoogle Scholar
  4. 4.
    Liu, H., Cella, D., Gershon, R., Shen, J., Morales, L. S., Riley, W., et al. (2010). Representativeness of the patient-reported outcomes measurement information system internet panel. Journal of Clinical Epidemiology, 63, 1169–1178.CrossRefPubMedCentralPubMedGoogle Scholar
  5. 5.
    Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63, 539–569.CrossRefPubMedGoogle Scholar
  6. 6.
    Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213–236.CrossRefGoogle Scholar
  7. 7.
    Johnson, J. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103–129.CrossRefGoogle Scholar
  8. 8.
    Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.CrossRefPubMedGoogle Scholar
  9. 9.
    Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19.CrossRefGoogle Scholar
  10. 10.
    Godinho, A., Kushnir, V., & Cunningham, J. A. (2016). Unfaithful findings: Identifying careless responding in addictions research. Addiction, 111, 955–956.CrossRefPubMedGoogle Scholar
  11. 11.
    Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872.CrossRefGoogle Scholar
  12. 12.
    Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99–114.CrossRefGoogle Scholar
  13. 13.
    Maniaci, M. R., & Rogge, R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61–83.CrossRefGoogle Scholar
  14. 14.
    McGrath, R. E., Mitchell, M., Kim, B. H., & Hough, L. (2010). Evidence for response bias as a source of error variance in applied assessment. Psychological Bulletin, 136, 450–470.CrossRefPubMedGoogle Scholar
  15. 15.
    Piedmont, R. L., McCrae, R. R., Riemann, R., & Angleitner, A. (2000). On the invalidity of validity scales: Evidence from self-reports and observer ratings in volunteer samples. Journal of Personality and Social Psychology, 78, 582–593.CrossRefPubMedGoogle Scholar
  16. 16.
    Osborne, J. W., & Blanchard, M. R. (2010). Random responding from participants is a threat to the validity of social science research results. Frontiers in Psychology, 1, 220.PubMedGoogle Scholar
  17. 17.
    Miura, A., & Kobayashi, T. (2016). Survey satisficing inflates stereotypical responses in online experiment: The case of immigration study. Frontiers in Psychology, 7, 1563.PubMedCentralPubMedGoogle Scholar
  18. 18.
    Ward, M. K., & Pond, S. B. (2015). Using virtual presence and survey instructions to minimize careless responding on internet-based surveys. Computers in Human Behavior, 48, 554–568.CrossRefGoogle Scholar
  19. 19.
    Fervaha, G., & Remington, G. (2013). Invalid responding in questionnaire-based research: Implications for the study of schizotypy. Psychological Assessment, 25, 1355–1360.CrossRefPubMedGoogle Scholar
  20. 20.
    Reeve, B. B., Hays, R. D., Bjorner, J. B., Cook, K. F., Crane, P. K., Teresi, J. A., et al. (2007). Psychometric evaluation and calibration of health-related quality of life item banks - Plans for the patient-reported outcomes measurement information system (PROMIS). Medical Care, 45, S22–S31.CrossRefGoogle Scholar
  21. 21.
    Zhao, Y. (2017). Impact of IRT item misfit on score estimates and severity classifications: An examination of PROMIS depression and pain interference item banks. Quality of Life Research, 26, 555–564.CrossRefPubMedGoogle Scholar
  22. 22.
    Mokkink, L. B., Terwee, C. B., Stratford, P. W., Alonso, J., Patrick, D. L., Riphagen, I., et al. (2009). Evaluation of the methodological quality of systematic reviews of health status measurement instruments. Quality of Life Research, 18, 313–333.CrossRefPubMedGoogle Scholar
  23. 23.
    Emons, W. H. (2008). Nonparametric person-fit analysis of polytomous item scores. Applied Psychological Measurement, 32, 224–247.CrossRefGoogle Scholar
  24. 24.
    Woods, C. M., Oltmanns, T. F., & Turkheimer, E. (2008). Detection of aberrant responding on a personality scale in a military sample: An application of evaluating person fit with two-level logistic regression. Psychological Assessment, 20, 159–168.CrossRefPubMedCentralPubMedGoogle Scholar
  25. 25.
    Schneider, S. (2017). Careless responding. osf.io/um9d3.Google Scholar
  26. 26.
    Ekstrom, R. B., French, J. W., Harman, H. H., & Dermen, D. (1976). Manual for kit of factor-referenced cognitive tests. Princeton, NJ: Educational Testing Service.Google Scholar
  27. 27.
    Lindenberger, U., Mayr, U., & Kliegl, R. (1993). Speed and intelligence in old age. Psychology and Aging, 8, 207–220.CrossRefPubMedGoogle Scholar
  28. 28.
    Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 163–183.CrossRefGoogle Scholar
  29. 29.
    Pilkonis, P. A., Choi, S. W., Reise, S. P., Stover, A. M., Riley, W. T., Cella, D., et al. (2011). Item banks for measuring emotional distress from the Patient-Reported Outcomes Measurement Information System (PROMIS (R)): Depression, anxiety, and anger. Assessment, 18, 263–283.CrossRefPubMedCentralPubMedGoogle Scholar
  30. 30.
    Lai, J. S., Cella, D., Choi, S., Junghaenel, D. U., Christodoulou, C., Gershon, R., et al. (2011). How item banks and their application can influence measurement practice in rehabilitation medicine: A PROMIS fatigue item bank example. Archives of Physical Medicine and Rehabilitation, 92, S20–S27.CrossRefPubMedCentralPubMedGoogle Scholar
  31. 31.
    Amtmann, D., Cook, K. F., Jensen, M. P., Chen, W. H., Choi, S., Revicki, D., et al. (2010). Development of a PROMIS item bank to measure pain interference. Pain, 150, 173–182.CrossRefPubMedCentralPubMedGoogle Scholar
  32. 32.
    Becker, H., Stuifbergen, A., Lee, H., & Kullberg, V. (2014). Reliability and validity of PROMIS cognitive abilities and cognitive concerns scales among people with multiple sclerosis. International Journal of MS Care, 16, 1–8.CrossRefPubMedCentralPubMedGoogle Scholar
  33. 33.
    Ware, J. E. Jr., & Sherbourne, C. D. (1992). The MOS 36-item short-form health survey (SF-36): I. Conceptual framework and item selection. Medical Care, 30, 473–483.CrossRefPubMedGoogle Scholar
  34. 34.
    Cleeland, C. (1994). Pain assessment: Global use of the Brief Pain Inventory. Annals of Academic Medicine Singapore, 23, 129–138.Google Scholar
  35. 35.
    Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55.CrossRefGoogle Scholar
  36. 36.
    Muthén, B. (2004). Latent variable analysis: Growth mixture modeling and related techniques for longitudinal data. In D. Kaplan (Ed.), The Sage handbook of quantitative methodology for the social sciences (pp. 345–369). Thousand Oaks, CA: Sage.Google Scholar
  37. 37.
    Nylund, K. L., Asparoutiov, T., & Muthen, B. O. (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling-A Multidisciplinary Journal, 14, 535–569.CrossRefGoogle Scholar
  38. 38.
    Muthén, B. (2003). Statistical and substantive checking in growth mixture modeling: Comment on Bauer and Curran (2003). Psychological Methods, 8, 369–377.CrossRefPubMedGoogle Scholar
  39. 39.
    Muthén, L. K., & Muthén, B. O. (2017). Mplus user’s guide (7th edn.). Los Angeles, CA: Muthén & Muthén.Google Scholar
  40. 40.
    Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.Google Scholar
  41. 41.
    Cohen, J. (1988). Statistical power analysis for the behavioral sciences. New York, NY: Erlbaum.Google Scholar
  42. 42.
    Tendeiro, J. N., Meijer, R. R., & Niessen, A. S. M. (2015). PerFit: An R package for person-fit analysis in IRT. Journal of Statistical Software, 74, 1–27.Google Scholar
  43. 43.
    Meijer, R. R., & Sijtsma, K. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107–135.CrossRefGoogle Scholar
  44. 44.
    Greiner, M., Pfeiffer, D., & Smith, R. (2000). Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests. Preventive Veterinary Medicine, 45, 23–41.CrossRefPubMedGoogle Scholar
  45. 45.
    Swets, J. A. (1988). Measuring the accuracy of diagnostic systems. Science, 240, 1285–1293.CrossRefPubMedGoogle Scholar
  46. 46.
    Celeux, G., & Soromenho, G. (1996). An entropy criterion for assessing the number of clusters in a mixture model. Journal of Classification, 13, 195–212.CrossRefGoogle Scholar
  47. 47.
    Asparouhov, T., & Muthén, B. (2014). Variable-specific entropy contribution. Retrieved June 30, 2017, from https://www.statmodel.com/download/UnivariateEntropy.pdf.
  48. 48.
    Cook, K. F., Kallen, M. A., & Amtmann, D. (2009). Having a fit: Impact of number of items and distribution of data on traditional criteria for assessing IRT’s unidimensionality assumption. Quality of Life Research, 18, 447–460.CrossRefPubMedCentralPubMedGoogle Scholar
  49. 49.
    DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44, 837–845.CrossRefPubMedGoogle Scholar
  50. 50.
    Credé, M. (2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70, 596–612.CrossRefGoogle Scholar
  51. 51.
    Van Vaerenbergh, Y., & Thomas, T. D. (2013). Response styles in survey research: A literature review of antecedents, consequences, and remedies. International Journal of Public Opinion Research, 25, 195–217.CrossRefGoogle Scholar
  52. 52.
    Schneider, S. (2018). Extracting response style bias from measures of positive and negative affect in aging research. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences, 73, 64–74.Google Scholar
  53. 53.
    Fong, D. Y., Ho, S., & Lam, T. (2010). Evaluation of internal reliability in the presence of inconsistent responses. Health and Quality of Life Outcomes, 8, 27.CrossRefPubMedCentralPubMedGoogle Scholar
  54. 54.
    Callegaro, M., Villar, A., Krosnick, J., & Yeager, D. (2014). A critical review of studies investigating the quality of data obtained with online panels. In M. Callegaro, R. Baker, J. Bethlehem, A. S. Göritz, J. A. Krosnick & P. J. Lavrakas (Eds.), Online panel research: A data quality perspective. Hoboken, NJ: Wiley.CrossRefGoogle Scholar
  55. 55.
    Bowling, N. A., Huang, J. L., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111, 218–229.CrossRefPubMedGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2017

Authors and Affiliations

  1. 1.University of Southern CaliforniaLos AngelesUSA
  2. 2.Dornsife Center for Self-Report Science and Center for Economic & Social ResearchUniversity of Southern CaliforniaLos AngelesUSA

Personalised recommendations