Advertisement

Reliability and Validity in Neuropsychology

  • Elisabeth M. S. Sherman
  • Brian L. Brooks
  • Grant L. Iverson
  • Daniel J. Slick
  • Esther Strauss
Chapter

Abstract

There are now literally hundreds of neuropsychological tests designed for evaluating cognitive abilities in children, adolescents, adults, and older adults. Given this vast library of instruments, how do test users decide which neuropsychological tests to choose? Like most decisions, choosing a test relies on a careful weighing of the relative balance of strengths and weaknesses. Two critical sources of information for making that decision are evidence of a test’s reliability and validity. Carefully examining these will help the user make an informed decision as to whether the test is appropriate for a particular purpose, a particular examinee, and a particular setting. This seems like a straightforward task for most neuropsychologists, who have typically covered basic concepts of reliability and validity during undergraduate or graduate training. Yet, a common mistake is to ask an all-or-none question, such as “is this test reliable?” or “has this test been validated?” Reliability and validity often appear deceptively simple, but continue to be complex topics to master.

The goal of this chapter is to facilitate the process of assessing the reliability and validity of tests for clinical use. We will provide an overview of reliability, including different types of reliability, methods for determining reliability, factors that affect reliability, and limits to reliability. We will also cover basic concepts relating to validity, including specific kinds of evidence contributing to validity, ways of evaluating validity, and basic guidelines for interpreting validity. We will do this while keeping the context focused as much as possible on everyday clinical practice.

Keywords

Test Score Interrater Reliability Internal Reliability Practice Effect Validity Evidence 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.Google Scholar
  2. Anastasi, A., & Urbina, S. (1997). Psychological testing (7th edition) Upper Saddle River, New Jersey, Prentice-Hall.Google Scholar
  3. Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet, i, 307–310.CrossRefGoogle Scholar
  4. Bornstein, R. F. (1996). Face validity in psychological assessment: Implications for a unified model of validity. American Psychologist, 51(9), 983–984.CrossRefGoogle Scholar
  5. Brooks, B. L., Iverson, G. L., & White, T. (2009). Advanced interpretation of the Neuropsychological Assessment Battery (NAB) with older adults: Base rate analyses, discrepancy scores, and interpreting change. Unpublished manuscript.Google Scholar
  6. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290.CrossRefGoogle Scholar
  7. Cicchetti, D. V., & Sparrow, S. S. (1981). Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86, 127–137.PubMedGoogle Scholar
  8. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302.PubMedCrossRefGoogle Scholar
  9. Culbertson, W. C., & Zillmer, E. A. (2005). Tower of London – Drexel University (2nd ed.). North Tonawanda: MHS.Google Scholar
  10. Delis, D. C., Kaplan, E., & Kramer, J. H. (2001). Delis Kaplan executive function system technical manual. San Antonio: The Psychological Corporation.Google Scholar
  11. Fastenau, P. S., Bennett, J. M., & Denburg, N. L. (1996). Application of psychometric standards to scoring system evaluation: Is “new” necessarily “improved”? Journal of Clinical and Experimental Neuropsychology, 18(3), 462–472.PubMedCrossRefGoogle Scholar
  12. Franzen, M. D. (1989). Reliability and validity in neuropsychological assessment. New York: Plenum Press.Google Scholar
  13. Franzen, M. D. (2000). Reliability and validity in neurological assessment (2nd ed.). New York: Kluwer Academic/Plenum Press.Google Scholar
  14. Heaton, R. K., Chelune, G. J., Talley, J. L., Kay, G. G., & Curtis, G. (1993). Wisconsin card sorting task (WCST) manual, revised and expanded. Odessa: Psychological Assessment Resources.Google Scholar
  15. Lineweaver, T. T., & Chelune, G. J. (2003). Use of the WAIS-III and WMS-III in the context of serial assessments: Interpreting reliable and meaningful change. In D. S. Tulsky, D. H. Saklofske, G. J. Chelune, R. K. Heaton, R. Ivnik, R. Bornstein, A. Prifitera, & M. F. Ledbetter (Eds.), Clinical interpretation of the WAIS-III and WMS-III (pp. 303–337). New York: Academic press.CrossRefGoogle Scholar
  16. McGrew, K. S., & Woodcock, R. W. (2001). Woodcock-Johnson III technical manual. Itasca: Riverside Publishing.Google Scholar
  17. Messick, S. (1996). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.CrossRefGoogle Scholar
  18. Mitrushina, M. N., Boone, K. B., Razani, J., & D’Elia, L. F. (2005). Handbook of normative data for neuropsychological assessment (2nd ed.). New York: Oxford University Press.Google Scholar
  19. Nevo, B. (1985). Face validity revisited. Journal of Educational Measurement, 22, 287–293.CrossRefGoogle Scholar
  20. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill, Inc.Google Scholar
  21. Psychological Corporation. (2002). WAIS-III/WMS-III technical manual. San Antonio: Psychological Corporation.Google Scholar
  22. Rapport, L. J., Brines, D. B., & Axelrod, B. N. (1997). Full scale IQ as a mediator of practice effects: The rich get richer. Clinical Neuropsychologist, 11(4), 375–380.CrossRefGoogle Scholar
  23. Reynolds, C. R. (2002). Comprehensive trail making test. Austin: PRO-ED.Google Scholar
  24. Ruff, R. M., & Hibbard, K. M. (2003). Ruff neurobehavioral inventory. Lutz: Psychological Assessment Resources, Inc.Google Scholar
  25. Sattler, J. M. (2001). Assessment of children: Cognitive applications (4th ed.). San Diego: Jerome M. Sattler Publisher, Inc.Google Scholar
  26. Sjögren, P., Thomsen, A., & Olsen, A. (2000). Impaired neuropsychological performance in chronic nonmalignant pain patients receiving long-term oral opioid therapy. Journal of Pain and Symptom Management, 19(2), 100–108.PubMedCrossRefGoogle Scholar
  27. Stern, R. A., & White, T. (2003). Neuropsychological assessment battery (NAB). Lutz: Psychological Assessment Resources.Google Scholar
  28. Strauss, E., Sherman, E. M. S., & Spreen, O. (2006). A compendium of neuropsychological tests (3rd ed.). New York: Oxford University Press.Google Scholar
  29. Urbina, S. (2004). Essentials of psychological testing. Hoboken: John Wiley & Sons.Google Scholar
  30. Wechsler, D. (2004). Wechsler intelligence scale for children (Integrated technical and interpretive manual 4th ed.). San Antonio: The Psychological Corporation.Google Scholar
  31. Wechsler, D., Coalson, D. L., & Raiford, S. E. (2008). Wechsler adult intelligence scale (Technical and interpretive manual 4th ed.). San Antonio: NCS Pearson, Inc.Google Scholar
  32. Wechsler, D., Holdnack, J. A., & Drozdick, L. W. (2009). Wechsler memory scale (Technical and interpretive manual 4th ed.). San Antonio: NCS Pearson, Inc.Google Scholar
  33. White, T., & Stern, R. A. (2003). Neuropsychological assessment battery (NAB): Psychometric and technical manual. Lutz: Psychological Assessment Resources.Google Scholar
  34. Yun, J., & Ulrich, D. A. (2002). Estimating measurement validity: A tutorial. Adapted physical activity quarterly, 19, 32–47.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Elisabeth M. S. Sherman
    • 1
  • Brian L. Brooks
  • Grant L. Iverson
  • Daniel J. Slick
  • Esther Strauss
  1. 1.Alberta Children’s HospitalUniversity of CalgaryCalgaryCanada

Personalised recommendations