Skip to main content
SpringerLink
Log in

Detection of Rater Errors on Cognitive Instruments in a Clinical Trial Setting

  • Original Research
  • Published:
The Journal of Prevention of Alzheimer's Disease Aims and scope Submit manuscript

Abstract

Objectives

This study examines errors committed by raters in a clinical trial of a memory enhancement compound.

Background

Findings of clinical trials are directly dependent on the quality of the data obtained but there is little literature on rates or nature of rater errors on cognitive instruments in a multi-site setting.

Design

Double-blind placebo-controlled study.

Setting

21 clinical sites in North America.

Participants

Two hundred seventy-five participants.

Measurements

MMSE, WMS-R Logical Memory I & II, WMS-R Verbal Paired Associates I, WASi Vocabulary, WASi Matrix Reasoning, GDS and MAC-Q.

Results

The WMS-R Logical Memory I & II and WASi Vocabulary tests were found to have the greatest number of scoring errors. Few substantive errors were detected on source document review of the MMSE, GDS, MAC-Q and WMS-R Verbal Paired Associates I. Some additional administration and scoring issues were identified during feedback sessions with the raters.

Conclusions

Cognitive measures used in clinical trials are prone to errors which can be detected with proper monitoring. Some instruments are particularly prone to inter-rater variably and should therefore be targets for focused training and ongoing monitoring. Areas in need of further investigation to help inform and optimize quality of clinical trial data are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Hartling L, Hamm M, Milne A, Vandermeer B, et al. Validity and inter-rater reliability testing of quality assessment instruments. AHRQ Publication No. 12-EHC039-EF 2012. www.effectivehealthcare.ahrq.gov/reports/final.cfm. Accessed 6 April 2018

    Google Scholar 

  2. Schafer K, De Santi S, Schneider LS. Errors in ADAS-Cog Administration and Scoring May Undermine Clinical Trials Results. Curr Alz Res 2010;6:S496–S497.

    Google Scholar 

  3. Connor DJ, Sabbagh MN. Administration and Scoring Variance on the ADASCog. J Alzheimers Dis 2008;15:461–464.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Connor DJ, Sabbagh MN, Cummings JL. Comment on administration and scoring of the Neuropsychiatric Inventory (NPI) in clinical trials. Alz Dem 2008;4:390–394.

    Article  Google Scholar 

  5. Petersen RC, Thomas RG, Aisen PS, et al. Randomized controlled trials in mild cognitive impairment: Sources of variability. Neurology 2017;88:1751–1758.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Hanninen T, Soininen H. Age-assoicated memory impairment. Normal ageing or warning of dementia? Drugs Aging 1997;11:480–9.

    Article  PubMed  CAS  Google Scholar 

  7. Crook, T, Bartus RT, Ferris SH, et al. Age associated memory impairment: Proposed diagnostic criteria and measures of clinical change-Report of a National Institute of Mental Health work group. Dev Neuropsych 1986;2:261–276.

    Article  Google Scholar 

  8. Cummings J, Gould H, Zhong, K. Advances in designs of Alzheimer’s disease clinical trials. Am J Neurodegener Dis 2012;1:205–216.

    PubMed  PubMed Central  Google Scholar 

  9. Zygouris S, Tsolaki M. Computerized cognitive testing for older adults: a review. Am J Alzheimers Dis Other Demen 2015;30:13–28.

    Article  PubMed  Google Scholar 

  10. Gates NJ, Kochan NA. Computerized and on-line neuropsychological testing for late-life cognition and neurocognitive disorders: are we there yet? Curr Opin Psychiatry 2015;28:165–172.

    PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Consultants in Cognitive and Clinical Trials, PO Box 33724, San Diego, CA, 92163, USA

    Donald J. Connor

  2. Alzheimer’s Therapeutic Research Institute and Department of Neurology, Keck School of Medicine of University of Southern California, San Diego, CA, 92121, USA

    C. W. Jenkins

  3. Dart Neuroscience, San Diego, CA, 92131, USA

    D. Carpenter, R. Crean & P. Perera

Authors
  1. Donald J. Connor
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. C. W. Jenkins
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. D. Carpenter
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. R. Crean
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. P. Perera
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Donald J. Connor.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Connor, D.J., Jenkins, C.W., Carpenter, D. et al. Detection of Rater Errors on Cognitive Instruments in a Clinical Trial Setting. J Prev Alzheimers Dis 5, 188–196 (2018). https://doi.org/10.14283/jpad.2018.20

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.14283/jpad.2018.20

Search

Navigation