Abstract
This chapter introduces the purposes and many of the complexities of conducting evaluation studies in biomedical and health informatics. It describes the work of informatics and how evaluation in general supports that work. It distinguishes evaluation from research, offers specific purposes that motivate evaluation studies, and describes several challenges that add complexity to informatics evaluations.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Readers familiar with methods of epidemiology may recognize the “evaluation machine” as an informal way of portraying the counterfactual approach to the study of cause and effect. More details about this approach may be found in a paper by Maldonado (Maldonado 2016).
- 2.
These questions were given to the authors in a personal communication on December 8, 1995.
- 3.
The authors use the term “bioinformatics” to refer to the use of information resources in support of biological research, and more specifically molecular biology and genomics.
- 4.
“NP” stands for “Nondeterministic Polynomial-time” and is a term used in computational complexity theory.
References
Abroms LC, Boal AL, Simmens SJ, Mendel JA, Windsor RA. A randomized trial of Text2Quit: a text messaging program for smoking cessation. Am J Prev Med. 2014;47:242–50.
Birkmeyer JD, Reames BN, McCulloch P, Carr AJ, Campbell WB, Wennberg JE. Understanding of regional variation in the use of surgery. Lancet. 2013;382:1121–9.
Elstein A, Shulman L, Sprafka S. Medical problem solving: an analysis of clinical reasoning. Cambridge, MA: Harvard University Press; 1978.
Federal food, drug, and cosmetic act, Amended. 21 USC 301, Amended through PL 117-11 (2021). Available from: https://www.govinfo.gov/content/pkg/COMPS-973/pdf/COMPS-973.pdf. Accessed 9 Jun 2021.
Florance V. Training for research careers in biomedical informatics and data science supported by the National Library of Medicine. In: Berner ES, editor. Informatics education in healthcare: lessons learned. 2nd ed. Cham: Springer Nature; 2020. p. 13–22. https://doi.org/10.1007/978-3-030-53813-2_2.
Friedman CP. Where’s the science in medical informatics? J Am Med Inform Assoc. 1995;2:65–7.
Friedman CP. A “fundamental theorem” of biomedical informatics. J Am Med Inform Assoc. 2009 Mar;16:169–70.
Graber ML. The incidence of diagnostic error in medicine. BMJ Qual Saf. 2013;22:21–7.
Haneuse S, Arterburn D, Daniels MJ. Assessing missing data assumptions in EHR-based studies: A complex & underappreciated task. JAMA Netw Open. 2021;4:e210184.
Kasperbauer TJ, Wright DE. Expanded FDA regulation of health and wellness apps. Bioethics. 2020;34:235–41.
Kwan JL, Lo L, Ferguson J, Goldberg H, Diaz-Martinez JP, Tomlinson G, et al. Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials. BMJ. 2020;370:3216. https://doi.org/10.1136/bmj.m3216.
Maldonado G. The role of counterfactual theory in causal reasoning. Ann Epidemiol. 2016;26:681–2.
McDonald CJ, Hui SL, Smith DM, Tierney WM, Cohen SJ, Weinberger M, et al. Reminders to physicians from an introspective computer medical record: a two-year randomized trial. Ann Intern Med. 1984;100:130–8.
McGinnis JM, Aisner D, Olsen L, editors. The learning healthcare system: workshop summary. Institute of Medicine (IOM). Washington, DC: The National Academies Press; 2007.
Mun YY, Jackson JD, Park JS, Probst JC. Understanding information technology acceptance by individual professionals: toward an integrative view. Inf Manag. 2006;43:350–63.
Musen MA, Middleton B, Greenes RA. Clinical decision-support systems. In: Shortliffe EH, Cimino JJ, editors. Biomedical informatics: computer applications in health care and biomedicine. 5th ed. Cham: Springer Nature; 2021. p. 795–840. https://doi.org/10.1007/978-3-030-58721-5_24.
National Library of Medicine (NLM). MEDLINE citation counts by year of publication (as of January 2021). Bethesda, MD: United States National Institutes of Health (NIH), National Library of Medicine (NLM); 2021. Available from: https://www.nlm.nih.gov/bsd/medline_cit_counts_yr_pub.html. Accessed 19 Jun 2021
Norman GR, Eva KW. Diagnostic error and clinical reasoning. Med Educ. 2010;44:94–100.
O’Connor M. An end to preemptively limiting the scope of a manufacturer’s duty: why the Arizona Court of Appeals was right in striking down the learned intermediary doctrine. Ariz State Law J. 2017;49:607.
Rafferty AM, Ball J, Aiken LH. Are teamwork and professional autonomy compatible, and do they result in improved hospital care? BMJ Qual Saf. 2001;10:32–7.
Ratwani RM, Hodgkins M, Bates DW. Improving electronic health record usability and safety requires transparency. JAMA. 2018;320:2533–4.
Roethligsburger F, Dickson W. Management and the worker. Cambridge, MA: Harvard University Press; 1939.
Shortliffe EH, Cimino JJ, editors. Biomedical informatics: computer applications in health care and biomedicine. 4th ed. London: Springer; 2014.
Somerville I. Software engineering. 10th ed. London: Pearson; 2016.
Stead W, Haynes RB, Fuller S, Friedman CP, Travis LE, Beck JR, et al. Designing medical informatics research and library-resource projects to increase what is learned. J Am Med Inform Assoc. 1994;1:28–33.
The New York Times. Alice Toklas, 89, is dead in Paris; literary figure Ran Salon with Gertrude Stein. Archive of The New York Times. 1967 Mar 8; Sect A:45. Available from: https://www.nytimes.com/1967/03/08/archives/alice-toklas-89-is-dead-in-paris-literary-figure-ran-salon-with.html. Accessed 9 Jun 2021.
Tierney WM, Miller ME, Overhage JM, McDonald CJ. Physician inpatient order writing on microcomputer workstations: effects on resource utilization. JAMA. 1993;269:379–83.
United Kingdom Medicines and Healthcare products Regulatory Agency (MHRA). Guidance: medical devices: how to comply with the legal requirements in Great Britain. London: United Kingdom Medicines and Healthcare products Regulatory Agency (MHRA); 2020. Available from: https://www.gov.uk/guidance/medical-devices-how-to-comply-with-the-legal-requirements. Accessed 9 Jun 2021
United States Food and Drug Administration (FDA). How to determine if your product is a medical device. Silver Spring, MD: United States Food and Drug Administration (FDA); 2021. Available from: https://www.fda.gov/medical-devices/classify-your-medical-device/how-determine-if-your-product-medical-device. Accessed 9 Jun 2021
Wyatt JC. Clinical data systems, part 1: data and medical records. Lancet. 1994;344:1543–7.
Wyatt J, Spiegelhalter D. Evaluating medical expert systems: what to test, and how? In: Talmon JL, Fox J, editors. Knowledge based systems in medicine: methods, applications and evaluation: proceedings of the workshop “system engineering in medicine”, Maastricht, March 16-18, 1989. Lecture notes in medical informatics. Heidelberg: Springer; 1991. https://doi.org/10.1007/978-3-662-08131-0_22.
Zhang Y, Baicker K, Newhouse JP. Geographic variation in the quality of prescribing. N Engl J Med. 2010;363:1985–8.
Author information
Authors and Affiliations
Corresponding author
Answers to Self-Tests
Answers to Self-Tests
1.1 Self-Test 1.1
-
1.
Personal health
-
2.
Public/population health
-
3.
Research
-
4.
Health care
-
5.
Education
-
6.
Public/population health
1.2 Self-Test 1.2
-
1.
Pragmatic
-
2.
Promotional
-
3.
Managerial
-
4.
Scholarly
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Friedman, C.P., Wyatt, J.C., Ash, J.S. (2022). What Is Evaluation and Why Is It Challenging?. In: Evaluation Methods in Biomedical and Health Informatics. Health Informatics. Springer, Cham. https://doi.org/10.1007/978-3-030-86453-8_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-86453-8_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86452-1
Online ISBN: 978-3-030-86453-8
eBook Packages: MedicineMedicine (R0)