Abstract
Background
To optimally care for concussed individuals, a multi-dimensional approach is critical and a key component of this assessment in the athletic environment is computer-based neurocognitive testing. However, there continues to be concerns about the reliability and validity of these testing tools. The purpose of this study was to determine the sensitivity and specificity of three common computer-based neurocognitive tests (Immediate Post-Concussion Assessment and Cognitive Testing [ImPACT], CNS Vital Signs, and CogState Computerized Assessment Tool [CCAT]), to provide guidance on their clinical utility.
Methods
This study analyzed assessments from a cohort of collegiate athletes and non-varsity cadets from the NCAA-DoD CARE Consortium. The data were collected from 2014–2018. Study participants were divided into two testing groups [concussed, n = 1414 (baseline/24–48 h) and healthy, n = 8305 (baseline/baseline)]. For each test type, change scores were calculated for the components of interest. Then, the Normative Change method, which used normative data published in a similar cohort, and the Reliable Change Index (RCI) method were used to determine if the change scores were significant.
Results
Using the Normative Change method, ImPACT performed best with an 87.5%-confidence interval and 1 number of components failed (NCF; sensitivity = 0.583, specificity = 0.625, F1 = 0.308). CNS Vital Signs performed best with a 90%-confidence interval and 1 NCF (sensitivity = 0.587, specificity = 0.532, F1 = 0.314). CCAT performed best when using a 75%-confidence interval and 2 NCF (sensitivity = 0.513, specificity = 0.715, F1 = 0.290). When using the RCI method, ImPACT performed best with an 87.5%-confidence interval and 1 NCF (sensitivity = 0.626, specificity = 0.559, F1 = 0.297).
Conclusion
When considering all three computer-based neurocognitive tests, the overall low sensitivity and specificity results provide additional evidence for the use of a multi-dimensional assessment for concussion diagnosis, including symptom evaluation, postural control assessment, neuropsychological status, and other functional assessments.
Similar content being viewed by others
References
About the Cogstate CCAT. Retrieved September 8, 2019, from http://axonsports.ca/index.cfm?pid=65
Aoullay A. What's WRONG with Metrics? Retrieved February 2, 2020. (2018). https://towardsdatascience.com/choosing-the-right-metric-is-a-huge-issue-99ccbe73de61
Aubry M, Cantu R, Dvorak J, Graf-Baumann T, Johnston K, Kelly J, et al. Summary and agreement statement of the first international conference on concussion in sport, Vienna 2001. Br J Sports Med. 2002;36(1):6–10. https://doi.org/10.1136/bjsm.36.1.6.
Broglio SP, Ferrara MS, Macciocchi SN, Baumgartner TA, Elliott R. Test-retest reliability of computerized concussion assessment programs. J Athl Train. 2007;42(4):509–14.
Broglio SP, Harezlak J, Katz B, Zhao S, McAllister T, McCrea M, Investigators CC. Acute sport concussion assessment optimization: a prospective assessment from the CARE Consortium. Am J Sports Med. 2019;49(12):1977–87. https://doi.org/10.1007/s40279-019-01155-0.
Broglio SP, Katz BP, Zhao S, McCrea M, McAllister T, CARE Consortium Investigators. Test–retest reliability and interpretation of common concussion assessment tools: findings from the NCAA-DoD CARE consortium. Am J Sports Med. 2018;48(5):1255–68. https://doi.org/10.1007/s40279-017-0813-0.
Broglio SP, Macciocchi SN, Ferrara MS. Sensitivity of the concussion assessment battery. J Neurosurg. 2007;60(6):1050–8. https://doi.org/10.1227/01.NEU.0000255479.90999.C0.
Broglio SP, McCrea M, McAllister T, Harezlak J, Katz B, Hack D, et al. A National Study on the Effects of Concussion in Collegiate Athletes and US Military Service Academy Members: The NCAA–DoD Concussion Assessment, Research and Education (CARE) Consortium Structure and Methods. Am J Sports Med. 2017;47(7):1437–51. https://doi.org/10.1007/s40279-017-0707-1.
Brooks BL, Iverson GL, Sherman EMS, Roberge M. Identifying cognitive problems in children and adolescents with depression using computerized neuropsychological testing. Appl Neuropsychol. 2010;17(1):37–43. https://doi.org/10.1080/09084280903526083.
Bruce J, Echemendia R, Meeuwisse W, Comper P, Sisco A. 1 year test–retest reliability of ImPACT in professional ice hockey players. Clin Neuropsychol. 2014;28(1):14–25. https://doi.org/10.1080/13854046.2013.866272.
Carney N, Ghajar J, Jagoda A, Bedrick S, Davis-OʼReilly C, du Coudray H, Hack D, Helfand N, Huddleston A, Nettleton T, Riggio S. Concussion guidelines step 1: systematic review of prevalent indicators. J Neurosurg. 2014;75:S3–15. https://doi.org/10.1227/NEU.0000000000000433.
Cassidy JD, Carroll LJ, Peloso PM, Borg J, von Holst H, Holm L, et al. Incidence, risk factors and prevention of mild traumatic brain injury: Results of the WHO Collaborating Centre Task Force on Mild Traumatic Brain Injury. J Rehabil Med. 2004;43(Suppl):28–60. https://doi.org/10.1080/16501960410023732.
Chin EY, Nelson LD, Barr WB, McCrory P, McCrea MA. Reliability and validity of the sport concussion assessment tool-3 (SCAT3) in high school and collegiate athletes. Am J Sports Med. 2016;44(9):2276–85. https://doi.org/10.1177/0363546516648141.
CNS Vital Signs Interpretation Guide. (2014). CNS Vital Signs.
Collins MW, Grindel SH, Lovell MR, Dede DE, Moser DJ, Phalin BR, et al. Relationship between concussion and neuropsychological performance in college football players. JAMA. 1999;282(10):964–70. https://doi.org/10.1001/jama.282.10.964.
Covassin T, Moran T, Wilhelm K. Concussion symptoms and neurocognitive performance of high school and college athletes who incur multiple concussions. Am J Sports Med. 2013;41(12):2885–9. https://doi.org/10.1177/0363546513499230.
Downey RI, Hutchison MG, Comper P. Determining sensitivity and specificity of the Sport Concussion Assessment Tool 3 (SCAT3) components in university athletes. Brain Inj. 2018;32(11):1345–52. https://doi.org/10.1080/02699052.2018.1484166.
Echemendia RJ, Bruce JM, Bailey CM, Sanders JF, Arnett P, Vargas G. The utility of post-concussion neuropsychological data in identifying cognitive change following sports-related MTBI in the absence of baseline data. Clin Neuropsychol. 2012;26(7):1077–91. https://doi.org/10.1080/13854046.2012.721006.
Echemandia RJ, Iverson GL, McCrea M, Macciocchi SN, Gioia GA, Putukian M, Comper P. Advances in neuropsychological assessment of sport-related concussion. Br J Sports Med. 2013;47(5):294–8. https://doi.org/10.1136/bjsports-2013-092186.
Elbin RJ, Kontos AP, Kegel N, Johnson E, Burkhart S, Schatz P. Individual and combined effects of LD and ADHD on computerized neurocognitive concussion test performance: evidence for separate norms. Arch Clin Neuropsychol. 2013;28(5):476–84. https://doi.org/10.1093/arclin/act024.
Garcia G-GP, Broglio SP, Lavieri MS, McCrea M, McAllister T, McAllister T. Quantifying the value of multidimensional assessment models for acute concussion: an analysis of data from the NCAA-DoD care consortium. Am J Sports Med. 2018;48(7):1739–49. https://doi.org/10.1007/s40279-018-0880-x.
Garcia G-GP, Yang J, Lavieri MS, McAllister TW, McCrea MA, Broglio SP. Optimizing components of the sport concussion assessment tool for acute concussion assessment. J Neurosurg. 2020. https://doi.org/10.1093/neuros/nyaa150.
Gualtieri CT, Johnson LG. Reliability and validity of a computerized neurocognitive test battery, CNS Vital Signs. Arch Clin Neuropsychol. 2006;21:623–43. https://doi.org/10.1016/j.acn.2006.05.007.
Guskiewicz KM, McCrea M, Marshall SW, Cantu RC, Randolph C, Barr W, et al. Cumulative effects associated with recurrent concussion in collegiate football players: The NCAA Concussion Study. JAMA. 2003;290(19):2549–55. https://doi.org/10.1001/jama.290.19.2549.
Hinton-Bayre AD. Choice of reliable change model can alter decisions regarding neuropsychological impairment after sports-related concussion. Clin J Sport Med. 2012;22(2):105–8. https://doi.org/10.1097/JSM.0b013e318248a526.
Houck ZM, Asken BM, Bauer RM, Kontos AP, McCrea MA, McAllister TW, Broglio SP, Clugston JR, Care Consortium Investigators. Multivariate base rates of low scores and reliable decline on ImPACT in healthy collegiate athletes using CARE consortium norms. J Int Neuropsychol Soc. 2019;25(09):961–71. https://doi.org/10.1017/S1355617719000729.
ImPACT Administration and Interpretation Manual. ImPACT Applications, Inc. (2016).
James G, Witten D, Hastie T, Tibshirani R. An introduction to statistical learning, vol. 103. New York : Springer; 2013. https://doi.org/10.1007/978-1-4614-7138-7.
Langlois J, Rutland-Brown W, Wald M. The epidemiology and impact of traumatic brain injury: a brief overview. J Head Trauma Rehabil. 2006;21(5):375–8.
Lempke LB, Schmidt JD, Lynall RC. Athletic trainers’ concussion-assessment and concussion-management practices: an update. J Athl Train. 2020;55(1):17–26. https://doi.org/10.4085/1062-6050-322-18.
Louey AG, Cromer JA, Schembri AJ, Darby DG, Maruff P, Makdissi M, McCrory P. Detecting cognitive impairment after concussion: sensitivity of change from baseline and normative data methods using the CogSport/Axon Cognitive Test Battery. Arch Clin Neuropsychol. 2014;29(5):432–41. https://doi.org/10.1093/arclin/acu020.
Lovell MR, Collins MW, Podell K, Powell J, Maroon J. ImPACT: Immediate post-concussion assessment and cognitive testing. Pittsburgh: NeuroHealth Systems LLC; 2000.
Lucini FR, Fogliatto FS, da Silveira GJC, Neyeloff JL, Anzanello MJ, Kuchenbecker RS, Schaan BD. Text mining approach to predict hospital admissions using early medical records from the emergency department. Int J Med Inform. 2017;100:1–8. https://doi.org/10.1016/j.ijmedinf.2017.01.001.
Maruff P, Thomas E, Cysique L, Brew B, Collie A, Snyder P, Pietrzak RH. Validity of the CogState Brief Battery: relationship to standardized tests and sensitivity to cognitive impairment in mild traumatic brain injury, schizophrenia, and AIDS dementia complex. Arch Clin Neuropsychol. 2009;24(2):165–78. https://doi.org/10.1093/arclin/acp010.
McAllister T, McCrea M. Long-term cognitive and neuropsychiatric consequences of repetitive concussion and head-impact exposure. J Athl Train. 2017;52(3):309–17. https://doi.org/10.4085/1062-6050-52.1.14.
McCrory P, Meeuwisse WH, Aubry M, Cantu RC, Dvorak J, Echemendia RJ, et al. Consensus statement on concussion in sport—the 4th international conference on concussion in sport held in Zurich, November 2012. PM&R. 2013;5(4):255–79. https://doi.org/10.1016/j.pmrj.2013.02.012.
McCrory P, Meeuwisse W, Dvorak J, Aubry M, Bailes J, Broglio S, Cantu RC, Cassidy D, Echemendia RJ, Castellani RJ, Davis GA, Ellenbogen R, Emery C, Engebretsen L, Feddermann-Demont N, Giza CC, Guskiewicz KM, Herring S, Iverson GL, et al. Consensus statement on concussion in sport—The 5th international conference on concussion in sport held in Berlin, October 2016. Br J Sports Med. 2017. https://doi.org/10.1136/bjsports-2017-097699.
Meehan WP, d’Hemecourt P, Collins CL, Taylor AM, Comstock RD. Computerized neurocognitive testing for the management of sport-related concussions. J Pediatr. 2012;129(1):38–44. https://doi.org/10.1542/peds.2011-1972.
Nelson LD, LaRoche AA, Pfaller AY, Lerner EB, Hammeke TA, Randolph C, et al. Prospective, head-to-head study of three computerized neurocognitive assessment tools (CNTs): reliability and validity for the assessment of sport-related concussion. J Int Neuropsychol Soc. 2016;22:24–37. https://doi.org/10.1017/S1355617715001101.
Putukian M, Echemendia R, Dettwiler-Danspeckgruber A, Duliba T, Bruce J, Furtado JL, Murugavel M. Prospective clinical assessment using sideline concussion assessment tool-2 testing in the evaluation of sport-related concussion in college athletes. Clin J Sport Med. 2015;25(1):7.
Rabinowitz AR, Levin HS. Cognitive sequelae of traumatic brain injury. Psychiatr Clin N Am. 2014;37(1):1–11. https://doi.org/10.1016/j.psc.2013.11.004.
Rahman-Filipiak AAM, Woodard JL. Administration and environment considerations in computer-based sports-concussion assessment. Neuropsychol Rev. 2013;23(4):314–34. https://doi.org/10.1007/s11065-013-9241-6.
Randolph C, McCrea M, Barr WB. Is neuropsychological testing useful in the management of sport-related concussion? J Athl Train. 2005;40(3):139–52.
Randolph C, Millis S, Barr WB, McCrea M, Guskiewicz KM, Hammeke TA, Kelly JP. Concussion Symptom Inventory: an empirically derived scale for monitoring resolution of symptoms following sport-related concussion. Arch Clin Neuropsychol. 2009;24(3):219–29. https://doi.org/10.1093/arclin/acp025.
Reeves DL, Winter KP, Bleiberg J, Kane RL. ANAM® genogram: historical perspectives, description, and current endeavors. Arch Clin Neuropsychol. 2007;22(1):15–37. https://doi.org/10.1016/j.acn.2006.10.013.
Register-Mihalik JK, Guskiewicz KM, Mihalik JP, Schmidt JD, Kerr ZY, McCrea MA. Reliable change, sensitivity, and specificity of a multidimensional concussion assessment battery: implication for caution in clinical practice. J Head Trauma Rehabil. 2013;28(4):274–83. https://doi.org/10.1097/HTR.0b013e3182585d37.
Resch JE, Brown CN, Schmidt J, Macciocchi SN, Blueitt D, Cullum CM, Ferrara MS. The sensitivity and specificity of clinical measures of sport concussion: three tests are better than one. Med BMJ Open Sport Exerc. 2016. https://doi.org/10.1136/bmjsem-2015-000012.
Resch JE, McCrea MA, Munro CC. Computerized neurocognitive testing in the management of sport-related concussion: an update. Neuropsychol Rev. 2013;23(4):335–49. https://doi.org/10.1007/211065-013-9242-5.
Robin X. Package “pROC.” Retrieved from https://cran.r-project.org/web/packages/pROC/pROC.pdf. (2019).
Schatz P, Robertshaw S. Comparing post-concussive neurocognitive test data to normative data presents risks for under-classifying “above average” athletes. Arch Clin Neuropsychol. 2014;29(7):625–32. https://doi.org/10.1093/arclin/acu041.
Schatz P, Sandel N. Sensitivity and specificity of the online version of ImPACT in high school and collegiate athletes. Am J Sports Med. 2013;41(2):321–6. https://doi.org/10.1177/0363546512466038.
Schmidt JD, Register-Mihalik JK, Mihalik JP, Kerr ZY, Guskiewicz KM. Identifying impairments after concussion: normative data versus individualized baselines. Med Sci Sports Exerc. 2012;44(9):1621–8. https://doi.org/10.1249/MSS.0b013e318258a9fb.
Thoma RJ, Cook JA, McGrew C, King JH, Pulsipher DT, Yeo RA, et al. Convergent and discriminant validity of the ImPACT with traditional neuropsychological measures. Cogent Psychol. 2018;5(1):1430199. https://doi.org/10.1080/23311908.2018.1430199.
Walpole RE, Myers RH, Myers SL, Ye K. Probability & statistics for engineers & scientists. 9th ed. Upper Saddle River: Prentice Hall; 2012.
Wood TA, Hsieh KL, An R, Ballard RA, Sosnoff JJ. Balance and gait alterations observed more than 2 weeks after concussion: a systematic review and meta-analysis. Am J Phys Med Rehabil. 2019;98(7):566–76. https://doi.org/10.1097/PHM.0000000000001152.
Acknowledgements
Alphabetically by last name, the CARE Consortium investigators include: Scott A. Anderson, ATC (University of Oklahoma, USA); Holly J. Benjamin, MD, FAAP, FACSM (University of Chicago, USA); Alison Brooks, MD, MPH (University of Wisconsin-Madison, USA); Thomas Buckley, EdD, ATC (University of Delaware, USA); Kenneth L. Cameron, PhD, MPH, ATC, FNATA (United States Military Academy, USA); Sara PD Chrisman, MD, MPH (University of Washington, USA); James R. Clugston, MD, MS, CAQSM (University of Florida, USA); Stefan M. Duma, PhD (Virginia Tech, USA); James Eckner, MD, MS (University of Michigan, USA); Luis A. Feigenbaum, PT, DPT, ATC/L (University of Miami, USA); Joshua T. Goldman, MD, MBA (University of California, Los Angeles, USA); Joseph B. Hazzard Jr., ATC (Bloomsburg University, USA); Megan N. Houston, PhD, ATC (United States Military Academy West Point, USA); April Hoy, MS, ATC, CSCS (Azusa Pacific University, USA); Thomas W. Kaminski, PhD, ATC, FNAK, FNATA, FACSM, RFSA (University of Delaware, USA); Louise A. Kelly, PhD (California Lutheran University, USA); Anthony P. Kontos, PhD (University of Pittsburgh, USA); Laura Lintner, DO (Wake Forest University School of Medicine, USA); Christina L. Master, MD, FAAP, CAQSM, FACSM (University of Pennsylvania, USA); Jane McDevitt, PhD, LAT, ATC, CSCS (Temple University, USA); Gerald McGinty, PT, DPT (United States Air Force Academy, USA); Jessica Dysart Miles, LAT, ATC (University of North Georgia, USA); Chris Miles, MD (Wake Forest University, USA); Justus Ortega, PhD (Humboldt State University, USA); Nicholas Port, PhD (Indiana University, USA); Margot Putukian, MD, FACSM, FAMSSM (Princeton University, USA); Steve Rowson, PhD (Virginia Tech, USA); Julianne Schmidt, PhD, ATC (University of Georgia, USA); Joel D. Stitzel, PhD (Wake Forest University, USA); Adam James Susmarski, DO, CDR, MC, USN (United States Naval Academy, USA); Christopher T. Whitlow, MD, PhD, MHA (Wake Forest University, USA). We would also like to thank Jody Harland, Janetta Matesan, and Larry Riggen (Indiana University), Ashley Rettmann, Nicole L'Heureux (University of Michigan), Melissa Koschnitzke (Medical College of Wisconsin), Michael Jarrett, Vibeke Brinck, and Bianca Byrne (Quesgen), Thomas Dompier, Christy Collins, Melissa Niceley Baker, and Sara Dalton (Datalys Center for Sports Injury Research and Prevention), and the research and medical staff at each of the participating sites.
Author information
Authors and Affiliations
Consortia
Corresponding author
Ethics declarations
Funding
This publication was made possible, in part, by support from the Grand Alliance CARE Consortium, funded by the NCAA and the Department of Defense. The US Army Medical Research Acquisition Activity, 820 Chandler Street, Fort Detrick, MD 21702-5014, USA, is the awarding and administering acquisition office. This work was supported by the Office of the Assistant Secretary of Defense for Health Affairs through the Psychological Health and Traumatic Brain Injury Program under Award no. W81XWH-14-2-0151. Opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the Department of Defense (Defense Health Program funds).
Conflict of interest
Lauren L. Czerniak, Spencer W. Liebel, Gian-Gabriel P. Garcia, and Mariel S. Lavieri have no competing interests directly relevant to the content of this study. Michael A. McCrea, Thomas W. McAllister, and Steven P. Broglio received research support from the NCAA and the U.S. Department of Defense.
Ethics approval
This research study follows the standards of ethics stated in the Declaration of Helsinki.
Consent to participate
All individuals provided written informed consent which was approved by the local institution and the US Army Human Research Protection Office.
Consent for publication
CARE Consortium Review Board as well as the US Army Human Research Protection Office.
Availability of data and material
The NCAA-DoD CARE dataset is available in the FITBIR database (https://fitbir.nih.gov/).
Code availability
The R code is accessible by contacting the corresponding author.
Author contributions
LLC performed all data analysis, created and formatted tables and figures, and was a major contributor in writing the manuscript. SWL was a major contributor in writing the manuscript. GPG contributed greatly in data acquisition, data analysis, and writing the manuscript. MSL contributed to the design of the statistical analysis and writing of the manuscript. MAM and TM helped with data collection, securing funding, and final approval of the manuscript. SPB helped with the study design, securing funding, data interpretation, drafting and final approval of the manuscript.
Additional information
This article is part of a Topical Collection on The NCAA-DoD Concussion Assessment, Research and Education (CARE) Consortium.
The members of the CARE Consortium Investigators are listed in acknowledgements.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Czerniak, L.L., Liebel, S.W., Garcia, GG.P. et al. Sensitivity and Specificity of Computer-Based Neurocognitive Tests in Sport-Related Concussion: Findings from the NCAA-DoD CARE Consortium. Sports Med 51, 351–365 (2021). https://doi.org/10.1007/s40279-020-01393-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40279-020-01393-7