Data Quality in Clinical Research

  • Meredith Nahm ZozusEmail author
  • Michael G. Kahn
  • Nicole G. Weiskopf
Part of the Health Informatics book series (HI)


Every scientist knows that research results are only as good as the data upon which the conclusions were formed. However, most scientists receive no training in methods for achieving, assessing, or controlling the quality of research data—topics central to clinical research informatics. This chapter covers the basics of acquiring or collecting and processing data for research given the available data sources, systems, and people. Data quality dimensions specific to the clinical research context are used, and a framework for data quality practice and planning is developed. Available research is summarized, providing estimates of data quality capability for common clinical research data collection and processing methods. This chapter provides researchers, informaticists, and clinical research data managers basic tools to assure, assess, and control the quality of data for research.


Clinical research data Data quality Research data collection Processing methods Informatics Management of clinical data Data accuracy Secondary use 


  1. 1.
    Davis JR, Nolan VP, Woodcock J, Estabrook EW, editors. Assuring data quality and validity in clinical trials for regulatory decision making, Institute of Medicine Workshop report. Roundtable on research and development of drugs, biologics, and medical devices. Washington, DC: National Academy Press; 1999. Accessed 6 July 2009.Google Scholar
  2. 2.
    Deming WE, Geoffrey L. On sample inspection in the processing of census returns. J Am Stat Assoc. 1941;36:351–60.CrossRefGoogle Scholar
  3. 3.
    Deming WE, Tepping BJ, Geoffrey L. Errors in card punching. J Am Stat Assoc. 1942;37:525–36.CrossRefGoogle Scholar
  4. 4.
    Donabedian A. A guide to medical care administration, Medical care appraisal – quality and utilization, vol. 2. New York: American Public Health Association; 1969. p. 176.Google Scholar
  5. 5.
    Arndt S, Tyrell G, Woolson RF, Flaum M, Andreasen NC. Effects of errors in a multicenter medical study: preventing misinterpreted data. J Psychiatr Res. 1994;28:447–59.CrossRefGoogle Scholar
  6. 6.
    Lee YW, Pipino LL, Wang RY, Funk JD. Journey to data quality. Reprint ed. Cambridge, MA: MIT Press; 2009.Google Scholar
  7. 7.
    Weber GM, Mandl KD, Kohane IS. Finding the missing link for big biomedical data. JAMA. 2014;311(24):2479–80.PubMedGoogle Scholar
  8. 8.
    Steinhubl SR, Muse ED, Topol EJ. The emerging field of mobile health. Sci Transl Med. 2015;7(283):283rv3.CrossRefGoogle Scholar
  9. 9.
    Friedman CP. A “fundamental theorem” of biomedical informatics. J Am Med Inform Assoc. 2009;16(2):169–70. Epub 2008 Dec 11.CrossRefPubMedPubMedCentralGoogle Scholar
  10. 10.
    United States Department of Health and Human Services (HHS), E6(R2) Good Clinical Practice: Integrated Addendum to ICH E6(R1) Guidance for Industry, OMB Control No. 0910-0843 March 2018. Available from:.
  11. 11.
    International Organization for Standardization (ISO). Data quality – Part 2: Vocabulary ISO 8000-2:2017.Google Scholar
  12. 12.
    Reprinted with permission from Data Gone Awry, DataBasics, vol 13, no 3, Fall. 2007. Society for Clinical Data Management. Available from
  13. 13.
    Nagurney JT, Brown DF, Sane S, Weiner JB, Wang AC, Chang Y. The accuracy and completeness of data collected by prospective and retrospective methods. Acad Emerg Med. 2005;12:884–95.CrossRefGoogle Scholar
  14. 14.
    Feinstein AR, Pritchett JA, Schimpff CR. The epidemiology of cancer therapy. 3. The management of imperfect data. Arch Intern Med. 1969;123:448–61.CrossRefGoogle Scholar
  15. 15.
    Reason J. Human error. Cambridge, UK: Cambridge University Press; 1990.CrossRefGoogle Scholar
  16. 16.
    Nahm M, Dziem G, Fendt K, Freeman L, Masi J, Ponce Z. Data quality survey results. Data Basics. 2004;10:7.Google Scholar
  17. 17.
    Schuyl ML, Engel T. A review of the source document verification process in clinical trials. Drug Info J. 1999;33:789–97.CrossRefGoogle Scholar
  18. 18.
    Batini C, Catarci T, Scannapieco M. A survey of data quality issues in cooperative information systems. In: 23rd international conference on conceptual modeling (ER 2004), Shanghai; 2004.Google Scholar
  19. 19.
    Tayi GK, Ballou DP. Examining data quality. Commun ACM. 1998;41:4.CrossRefGoogle Scholar
  20. 20.
    Redman TC. Data quality for the information age. Boston: Artech House; 1996.Google Scholar
  21. 21.
    Wand Y, Wang R. Anchoring data quality dimensions in ontological foundations. Commun ACM. 1996;39:10.CrossRefGoogle Scholar
  22. 22.
    Wang R, Strong D. Beyond accuracy: what data quality means to data consumers. J Manag Inf Syst. 1996;12:30.Google Scholar
  23. 23.
    Batini C, Scannapieco M. Data quality concepts, methodologies and techniques. Berlin: Springer; 2006.Google Scholar
  24. 24.
    Wyatt J. Acquisition and use of clinical data for audit and research. J Eval Clin Pract. 1995;1:15–27.CrossRefGoogle Scholar
  25. 25.
    U.S. Food and Drug Administration. In: Services DoHaH, editor. Guidance for industry. Computerized systems used in clinical trials. Rockville: U.S. Food and Drug Administration; 2007.Google Scholar
  26. 26.
    Arts DG, De Keizer NF, Scheffer GJ. Defining and improving data quality in medical registries: a literature review, case study, and generic framework. J Am Med Inform Assoc. 2002;9:600–11.CrossRefGoogle Scholar
  27. 27.
    Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc. 2013;20:144–51. Scholar
  28. 28.
    GCP Inspectors Working Group European Medicines Agency (EMA). Reflection paper on expectations for electronic source data and data transcribed to electronic data collection tools in clinical trials. EMA/INS/GCP 454280/2010, 9 June 2010.Google Scholar
  29. 29.
    Kahn MG, Callahan TJ, Barnard J, Bauck AE, Brown J, Davidson BN, Estiri H, Goerg C, Holve E, Johnson SG, Liaw S-T, Hamilton-Lopez M, Meeker D, Ong TC, Ryan P, Shang N, Weiskopf NG, Weng C, Zozus MN, Schilling L. A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. eGEMs (Generating Evid Methods Improve Patient Outcomes) [Internet]. 2016;4(1):1244. Sep 11 [cited 2016 Sep 12]. Available from: Scholar
  30. 30.
    Callahan TJ, Bauck AE, Bertoch D, Brown J, Khare R, Ryan PB, Staab J, Zozus MN, Kahn MG. A comparison of data quality assessment checks in six data sharing networks. eGEMs (Generating Evid Methods Improve Patient Outcomes) [Internet]. 2017;5(1):8. Jun 12 [cited 2017 Jun 15]. Available from: Scholar
  31. 31.
    Estiri H, Stephens K. DQe-v: a database-agnostic framework for exploring variability in electronic health record data across time and site location. eGEMs (Generating Evid Methods Improve Patient Outcomes) [Internet]. 2017;5(1):3. May 10 [cited 2017 Jul 30]. Available from: Scholar
  32. 32.
    Kahn MG, Brown JS, Chun AT, Davidson BN, Meeker D, Ryan PB, Schilling LM, Weiskopf NG, Williams AE, Zozus MN. Transparent reporting of data quality in distributed data networks. eGEMs (Generating Evid Methods Improve Patient Outcomes). 2015;3(1):7. Available at: Scholar
  33. 33.
    Zozus MN, Lazarov A, Smith L, Breen T, Krikorian S, Zbyszewski P, Knoll K, Jendrasek D, Perrin D, Zambas D, Williams T, Pieper C. Analysis of professional competencies for the clinical research data management profession: implications for training and professional certification. JAMIA. 2017;24:737–45.PubMedGoogle Scholar
  34. 34.
    (CDISC) CDISC. The protocol representation model version 1.0 draft for public comment: CDISC; 2009. p. 96. Available from
  35. 35.
    Jacobs M, Studer L. Forms design II: the course for paper and electronic forms. Cleveland: Ameritype & Art; 1991.Google Scholar
  36. 36.
    Eisenstein EL, Lemons PW, Tardiff BE, Schulman KA, Jolly MK, Califf RM. Reducing the costs of phase III cardiovascular clinical trials. Am Heart J. 2005;9:482–8.CrossRefGoogle Scholar
  37. 37.
    Eisenstein EL, Collins R, Cracknell BS, et al. Sensible approaches for reducing clinical trial costs. Clin Trials. 2008;5:75–84.CrossRefGoogle Scholar
  38. 38.
    Galešic M. Effects of questionnaire length on response rates: review of findings and guidelines for future research. 2002. Accessed 29 Dec 2009.
  39. 39.
    Roszkowski MJ, Bean AG. Believe it or not! Longer questionnaires have lower response rates. J Bus Psychol. 1990;4:495–509.CrossRefGoogle Scholar
  40. 40.
    Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R, Kwan I. Increasing response rates to postal questionnaires systematic review. Br Med J. 2002;324:1183.CrossRefGoogle Scholar
  41. 41.
    Wickens CD, Hollands JG, Parasuraman R. Engineering psychology and human performance. 4th ed. New York: Routledge; 2016.Google Scholar
  42. 42.
    Stevens SS. On the theory of scales of measurement. Science. 1946;103:677–80.CrossRefGoogle Scholar
  43. 43.
    Allison JJ, Wall TC, Spettell CM, et al. The art and science of chart review. Jt Comm J Qual Improv. 2000;26:115–36.PubMedGoogle Scholar
  44. 44.
    Banks NJ. Designing medical record abstraction forms. Int J Qual Health Care. 1998;10:163–7.CrossRefGoogle Scholar
  45. 45.
    Engel L, Henderson C, Fergenbaum J, Interrater A. Reliability of abstracting medical-related information medical record review conduction model for improving. Eval Health Prof. 2009;32:281.CrossRefGoogle Scholar
  46. 46.
    Cunningham R, Sarfati D, Hill S, Kenwright D. An audit of colon cancer data on the New Zealand cancer registry. N Z Med J. 2008;121(1279):46–56.PubMedGoogle Scholar
  47. 47.
    Fritz A. The SEER program’s commitment to data quality. J Registry Manag. 2001;28(1):35–40.Google Scholar
  48. 48.
    German RR, Wike JM, Wolf HJ, et al. Quality of cancer registry data: findings from CDC-NPCR’s breast, colon, and prostate cancer data quality and patterns of care study. J Registry Manag. 2008;35(2):67–74.Google Scholar
  49. 49.
    Herrmann N, Cayten CG, Senior J, Staroscik R, Walsh S, Woll M. Interobserver and intraobserver reliability in the collection of emergency medical services data. Health Serv Res. 1980;15(2):127–43.PubMedPubMedCentralGoogle Scholar
  50. 50.
    Pan L, Fergusson D, Schweitzer I, Hebert PC. Ensuring high accuracy of data abstracted from patient charts: the use of a standardized medical record as a training tool. J Clin Epidemiol. 2005;58(9):918–23.CrossRefGoogle Scholar
  51. 51.
    Reeves MJ, Mullard AJ, Wehner S. Inter-rater reliability of data elements from a prototype of the Paul Coverdell National Acute Stroke Registry. BMC Neurol. 2008;8:19.CrossRefGoogle Scholar
  52. 52.
    Scherer R, Zhu Q, Langenberg P, Feldon S, Kelman S, Dickersin K. Comparison of information obtained by operative note abstraction with that recorded on a standardized data collection form. Surgery. 2003;133(3):324–30.CrossRefGoogle Scholar
  53. 53.
    Stange KC, Zyzanski SJ, Smith TF, et al. How valid are medical records and patient questionnaires for physician profiling and health services research? A comparison with direct observation of patients visits. Med Care. 1998;36(6):851–67.CrossRefGoogle Scholar
  54. 54.
    Thoburn KK, German RR, Lewis M, Nichols PJ, Ahmed F, Jackson-Thompson J. Case completeness and data accuracy in the centers for disease control and prevention’s national program of cancer registries. Cancer. 2007;109(8):1607–16.CrossRefGoogle Scholar
  55. 55.
    To T, Estrabillo E, Wang C, Cicutto L. Examining intra-rater and inter-rater response agreement: a medical chart abstraction study of a community-based asthma care program. BMC Med Res Methodol. 2008;8:29.CrossRefGoogle Scholar
  56. 56.
    Yawn BP, Wollan P. Interrater reliability: completing the methods description in medical records review studies. Am J Epidemiol. 2005;161(10):974–7.CrossRefGoogle Scholar
  57. 57.
    La France BH, Heisel AD, Beatty MJ. A test of the cognitive load hypothesis: investigating the impact of number of nonverbal cues coded and length of coding session on observer accuracy. Commun Rep. 2007;20:11–23.CrossRefGoogle Scholar
  58. 58.
    Zozus MN. The data book: collection and management of research data. Taylor & Francis/CRC Press Catalog #: K26788, ISBN: 978-1-4987-4224-5.Google Scholar
  59. 59.
    Helms R. Redundancy: an important data forms/design data collection principle. In: Proceedings Stat computing section, Alexandria; 1981. p. 233–7.Google Scholar
  60. 60.
    Helms R. Data quality issues in electronic data capture. Drug Inf J. 2001;35:827–37.CrossRefGoogle Scholar
  61. 61.
    U.S. Food and Drug Administration regulations. Title 21 CFR Part 58. 2011. Available from Accessed Aug 2011.
  62. 62.
    Nahm ML, Pieper CF, Cunningham MM. Quantifying data quality for clinical trials using electronic data capture. PLoS One. 2008;3(8):e3049.CrossRefGoogle Scholar
  63. 63.
    Winchell T. The mystery of source documentation. SOCRA Source 62. 2009. Available from
  64. 64.
    Nahm M. Data accuracy in medical record abstraction. Doctoral Dissertation, University of Texas at Houston, School of Biomedical Informatics, Houston, May 6, 2010.Google Scholar
  65. 65.
    Zozus MN, Pieper C, Johnson CM, Johnson TR, Franklin A, Smith J, et al. Factors affecting accuracy of data abstracted from medical records. PLoS One. 2015;10(10):e0138649.CrossRefGoogle Scholar
  66. 66.
    SCDM. Good clinical data management practices. Society for Clinical Data Management; 2010. Available from
  67. 67.
    Rostami R, Nahm M, Pieper CF. What can we learn from a decade of database audits? The Duke Clinical Research Institute experience, 1997–2006. Clin Trials. 2009;6(2):141–50.CrossRefGoogle Scholar
  68. 68.
    Stellman SD. The case of the missing eights an object lesson in data quality assurance. Am J Epidemiol. 1989;129(4):857–60. Scholar
  69. 69.
    Hogan WR1, Wagner MM. Accuracy of data in computer-based patient records. J Am Med Inform Assoc. 1997;4(5):342–55.CrossRefGoogle Scholar
  70. 70.
    Thiru K, Hassey A, Sullivan F. Systematic review of scope and quality of electronic patient record data in primary care. BMJ. 2003;326(7398):1070. Review.CrossRefGoogle Scholar
  71. 71.
    Chan KS, Fowles JB, Weiner JP. Review: electronic health records and the reliability and validity of quality measures: a review of the literature. Med Care Res Rev. 2010;67(5):503–27. Scholar
  72. 72.
    Observational Health Data Sciences and Informatics. OHDSI Observational Medical Outcomes Partnership (OMOP) Common Data Model. Accessed 29 May 2018.
  73. 73.
    The National Patient-Centered Clinical Research Network (PCORnet). Common data model v3.0. Accessed 1 Feb 2016.
  74. 74.
    Kahn MG, Raebel MA, Glanz JM, Riedlinger K, Steiner JF. A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research. Med Care. 2012;50(suppl):S21–9. Scholar
  75. 75.
    Weiskopf NG, Hripcsak G, Swaminathan S, Weng C. Defining and measuring completeness of electronic health records for secondary use. J Biomed Inform. 2013;46:830–6. Scholar
  76. 76.
    Svolba G, Bauer P. Statistical quality control in clinical trials. Control Clin Trials. 1999;20(6):519–30.CrossRefGoogle Scholar
  77. 77.
    Chilappagari S, Kulkarni A, Bolick-Aldrich S, Huang Y, Aldrich TE. A statistical process control method to monitor completeness of central cancer registry reporting data. J Registry Manag. 2002;29(4):121–7.Google Scholar
  78. 78.
    Chiu D, Guillaud M, Cox D, Follen M, MacAulay C. Quality assurance system using statistical process control: an implementation for image cytometry. Cell Oncol. 2004;26(3):101–17.PubMedPubMedCentralGoogle Scholar
  79. 79.
    McNees P, Dow KH, Loerzel VW. Application of the CuSum technique to evaluate changes in recruitment strategies. Nurs Res. 2005;54(6):399–405.CrossRefGoogle Scholar
  80. 80.
    Baigent C, Harrell FE, Buyse M, Emberson JR, Altman DG. Ensuring trial validity by data quality assurance and diversification of monitoring methods. Clin Trials. 2008;5(1):49–55.CrossRefGoogle Scholar
  81. 81.
    Matheny ME, Morrow DA, Ohno-Machado L, Cannon CP, Sabatine MS, Resnic FS. Validation of an automated safety surveillance system with prospective, randomized trial data. Med Decis Mak. 2009;29(2):247–56.CrossRefGoogle Scholar
  82. 82.
    McGilvray D. Executing data quality projects: ten steps to quality data and trusted information. 1st ed. Amsterdam: Morgan Kaufmann; 2008. 352 p.Google Scholar
  83. 83.
    Ladley J. Data governance: how to design, deploy and sustain an effective data governance program. 1st ed. Waltham: Morgan Kaufmann; 2012. 264 p.Google Scholar
  84. 84.
    Loshin D. The practitioner’s guide to data quality improvement. 1st ed. Burlington: Morgan Kaufmann; 2010. 432 p.Google Scholar
  85. 85.
    Baskarada S. IQM-CMM: information quality management capability maturity model. Germany: Vieweg and Teubner; 2010.Google Scholar
  86. 86.
    Capability Maturity Model Integration (CMMITM) Institute, Data Management maturity model, CMMI Institute 2014.Google Scholar
  87. 87.
    Stanford University. Stanford data governance maturity model. Accessed 12 May 2018. Available from
  88. 88.
    Williams M, Bagwell J, Zozus M. Data management plans, the missing perspective. J Biomed Inform. 2017;71:130–42.CrossRefGoogle Scholar
  89. 89.
    Freedman LS, Schatzkin A, Wax Y. The impact of dietary measurement error on planning sample size required in a cohort study. Am J Epidemiol. 1990;132:1185–95.CrossRefGoogle Scholar
  90. 90.
    Perkins DO, Wyatt RJ, Bartko JJ. Penny-wise and pound-foolish: the impact of measurement error on sample size requirements in clinical trials. Biol Psychiatry. 2007;47:762–6.CrossRefGoogle Scholar
  91. 91.
    Mullooly JP. The effects of data entry error: an analysis of partial verification. Comput Biomed Res. 1990;23:259–67.CrossRefGoogle Scholar
  92. 92.
    Liu K. Measurement error and its impact on partial correlation and multiple linear regression analyses. Am J Epidemiol. 1988;127:864–74.CrossRefGoogle Scholar
  93. 93.
    Stepnowsky CJ Jr, Berry C, Dimsdale JE. The effect of measurement unreliability on sleep and respiratory variables. Sleep. 2004;27:990–5.CrossRefGoogle Scholar
  94. 94.
    Myer L, Morroni C, Link BG. Impact of measurement error in the study of sexually transmitted infections. Sex Transm Infect. 2004;80(318–323):328.Google Scholar
  95. 95.
    Williams SC, Watt A, Schmaltz SP, Koss RG, Loeb JM. Assessing the reliability of standardized performance indicators. Int J Qual Health Care. 2006;18:246–55.CrossRefGoogle Scholar
  96. 96.
    Watt A, Williams S, Lee K, Robertson J, Koss RG, Loeb JM. Keen eye on core measures. Joint commission data quality study offers insights into data collection, abstracting processes. J AHIMA. 2003;74:20–5; quiz 27–8.PubMedGoogle Scholar
  97. 97.
    US Government Accountability Office. Hospital quality data: CMS needs more rigorous methods to ensure reliability of publicly released data. In: Office UGA, editor. Washington, DC; 2006.
  98. 98.
    Braun BI, Kritchevsky SB, Kusek L, et al. Comparing bloodstream infection rates: the effect of indicator specifications in the evaluation of processes and indicators in infection control (EPIC) study. Infect Control Hosp Epidemiol. 2006;27:14–22.CrossRefGoogle Scholar
  99. 99.
    Jacobs R, Goddard M, Smith PC. How robust are hospital ranks based on composite performance measures? Med Care. 2005;43:1177–84.CrossRefGoogle Scholar
  100. 100.
    Pagel C, Gallivan S. Exploring consequences on mortality estimates of errors in clinical databases. IMA J Manag Math. 2008;20(4):385–93. Scholar
  101. 101.
    Goldhill DR, Sumner A. APACHE II, data accuracy and outcome prediction. Anaesthesia. 1998;53:937–43.CrossRefGoogle Scholar

Copyright information

© Springer International Publishing 2019

Authors and Affiliations

  • Meredith Nahm Zozus
    • 1
    Email author
  • Michael G. Kahn
    • 2
  • Nicole G. Weiskopf
    • 3
  1. 1.Department of Biomedical Informatics, College of MedicineUniversity of Arkansas for Medical SciencesLittle RockUSA
  2. 2.Department of Pediatrics and the Colorado Clinical and Translational Sciences InstituteUniversity of Colorado Anschutz Medical CampusAuroraUSA
  3. 3.Department of Medical Informatics and Clinical Epidemiology, School of MedicineOregon Health & Science UniversityPortlandUSA

Personalised recommendations