Testing System Accuracy

  • Eta S. Berner
Part of the Health Informatics book series (HI)

Abstract

Evaluation is a crucial component in the development of any clinical diagnostic decision support system (CDDSS). Much of it takes place informally as part of the development process and is used by the CDDSS developers for system improvement. Once a system is sufficiently mature, more formal evaluation studies should be done, initially of system accuracy and later, of system impact. A wide range of study design choices can be appropriate for assessing accuracy, but once the CDDSS appears to be ready for use in practice, there is a need for more rigorous evaluation. Most published evaluation studies have focused on the issue of system accuracy, with few studies evaluating the impact of using a CDDSS on clinical care. This chapter will address issues involved in assessing the accuracy of CDDSS. Key results from research or evaluation studies of system accuracy will be summarized and discussed. The reader who is interested in the details of individual studies should read the references at the end of this chapter.

Keywords

Decision Support System Primary User Clinical Decision Support System System Accuracy Proc Amia 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bleich HL. Computer evaluation of acid-base disorders. J Clin Invest 1969; 48:1689–1696.PubMedCrossRefGoogle Scholar
  2. 2.
    Willems JL, Abreu-Lima C, Arnaud P et al. The diagnostic performance of computer programs for the interpretation of electrocardiograms. N Engl J Med 1991; 325:1767–1773.PubMedCrossRefGoogle Scholar
  3. 3.
    de Dombal, FT. The diagnosis of acute abdominal pain with computer assistance: worldwide perspective. Ann Chir 1991; 45:273–277.PubMedGoogle Scholar
  4. 4.
    Adams ID, Chan M, Clifford PC et al. Computer aided diagnosis of acute abdominal pain: a multicentre study. Br Med J (Clin Res Ed) 1986; 293:800–804.CrossRefGoogle Scholar
  5. 5.
    Sutton GC. How accurate is computer-aided diagnosis? Lancet 1989; 2:905–908.PubMedCrossRefGoogle Scholar
  6. 6.
    Gough IR. Computer assisted diagnosis of the acute abdomen. Aust N Z J Surg 1993; 63:699–702.PubMedCrossRefGoogle Scholar
  7. 7.
    Barnett GO, Cimino JJ, Hupp JA et al. DXplain—an evolving diagnostic decision-support system. JAMA 1987; 258:67–74.PubMedCrossRefGoogle Scholar
  8. 8.
    Warner HR Jr. Iliad: moving medical decision-making into new frontiers. Methods Inf Med 1989; 28:370–372.PubMedGoogle Scholar
  9. 9.
    Waxman HS, Worley WE. Computer-assisted adult medical diagnosis: subject review and evaluation of a new microcomputer-based system. Medicine 1990; 69:125–136.PubMedCrossRefGoogle Scholar
  10. 10.
    Weed LL. Knowledge Coupling: New Premises and New Tools for Medical Care and Education. New York, NY: Springer-Verlag, 1991.CrossRefGoogle Scholar
  11. 11.
    Miller R, Masarie FE, Myers J. Quick Medical Reference (QMR) for diagnostic assistance. MD Comput 1986; 3:34–48.PubMedGoogle Scholar
  12. 12.
    Miller RA. Medical diagnostic decision support systems—past, present, and future: a threaded bibliography and commentary. JAMIA 1994; 1:8–27.PubMedCrossRefGoogle Scholar
  13. 13.
    Georgakis DC, Trace DA, Naeymi-Rad F et al. A statistical evaluation of the diagnostic performance of MEDAS—the medical emergency decision assistance system. Proc Annu Symp Comput Appl Med Care 1990:815–819.Google Scholar
  14. 14.
    Nelson SJ, Blois MS, Tuttle MS et al. Evaluating RECONSIDER: a computer program for diagnostic prompting. J Med Sys 1985; 9:379–388.CrossRefGoogle Scholar
  15. 15.
    Hammersley JR, Cooney K. Evaluating the utility of available differential diagnosis systems. Proc Annu Symp Comput Appl Med Care 1988:229–231.Google Scholar
  16. 16.
    Feldman MJ, Barnett GO. An approach to evaluating the accuracy of DXplain. Comput Methods Programs Biomed 1991; 35:261–266.PubMedCrossRefGoogle Scholar
  17. 17.
    Heckerling PS, Elstein AS, Terzian CG et al. The effect of incomplete knowledge on the diagnosis of a computer consultant system. Med Inf 1991; 16:363–370.CrossRefGoogle Scholar
  18. 18.
    Lau LM, Warner HR. Performance of a diagnostic system (Iliad) as a tool for quality assurance. Comput Biomed Res 1992; 25:314–323.PubMedCrossRefGoogle Scholar
  19. 19.
    Barness LA, Tunnessen WW Jr, Worley WE et al. Computer-assisted diagnosis in pediatrics. Am J Dis Child 1974; 127:852–858.PubMedGoogle Scholar
  20. 20.
    O’Shea JS. Computer-assisted pediatric diagnosis. Am J Dis Child 1975; 129:199–202.PubMedGoogle Scholar
  21. 21.
    Swender PT, Tunnessen WW Jr, Oski FA. Computer-assisted diagnosis. Am J Dis Child 1974; 127:859–861.PubMedGoogle Scholar
  22. 22.
    Wexler JR, Swender PT, Tunnessen WW Jr et al. Impact of a system of computer-assisted diagnosis. Initial evaluation of the hospitalized patient. Am J Dis Child 1975; 129:203–205.PubMedGoogle Scholar
  23. 23.
    Bankowitz RA, Lave JR, McNeil MA. A method for assessing the impact of a computer-based decision support system on health care outcomes. Methods Inf Med 1992; 31:3–11.PubMedGoogle Scholar
  24. 24.
    Bankowitz RA, McNeil MA, Challinor SM et al. A computer-assisted medical diagnostic consultation service: implementation and prospective evaluation of a prototype. Ann Intern Med 1989; 110:824–832.PubMedCrossRefGoogle Scholar
  25. 25.
    Bankowitz RA, McNeil MA, Challinor SM et al. Effect of a computer-assisted general medicine diagnostic consultation service on housestaff diagnostic strategy. Methods Inf Med 1989; 28:352–356.PubMedGoogle Scholar
  26. 26.
    Berman L, Miller RA. Problem area formation as an element of computer aided diagnosis: a comparison of two strategies within quick medical reference (QMR). Methods Inf Med 1991; 30:90–95.PubMedGoogle Scholar
  27. 27.
    Middleton B, Shwe MA, Heckerman DE et al. Probabilistic diagnosis using a reformulation of the Internist-1/QMR knowledge base. II. Evaluation of diagnostic performance. Methods Inf Med 1991; 30:256–267.PubMedGoogle Scholar
  28. 28.
    Miller RA, Pople HE Jr, Myers J. Internist-I, an experimental computer-based diagnostic consultant for general internal medicine. N Engl J Med 1982; 307:468–476.PubMedCrossRefGoogle Scholar
  29. 29.
    Miller RA, Masarie FE Jr. The quick medical reference (QMR) relationships function: description and evaluation of a simple, efficient “multiple diagnoses” algorithm. Medinfo 1992:512–518.Google Scholar
  30. 30.
    Miller RA, McNeil MA, Challinor S et al. Status Report: The Internist-1 / Quick Medical Reference project. West J Med 1986; 145:816–822.PubMedGoogle Scholar
  31. 31.
    Berner ES, Webster GD, Shugerman AA et al. Performance of four computer-based diagnostic systems. N Engl J Med 1994; 330: 1792–1796.PubMedCrossRefGoogle Scholar
  32. 32.
    Berner ES, Jackson JR, Algina J. Relationships among performance scores of four diagnostic decision support systems. JAMIA 1996; 3:208–215.PubMedCrossRefGoogle Scholar
  33. 33.
    Bankowitz, RA. The effectiveness of QMR in medical decision support. Executive summary and final report. Springfield, VA: U.S. Department of Commerce, National Technical Information Service, 1994.Google Scholar
  34. 34.
    Murphy GC, Friedman CP, Elstein AS. The influence of a decision support system on the differential diagnosis of medical practitioners at three levels of training. Proc AMIA Fall Symp Comput 1996: 219–223.Google Scholar
  35. 35.
    Elstein AS, Friedman CP, Wolf FM et al. Effects of a decision support system on the diagnostic accuracy of users: a preliminary report. JAMIA 1996; 3:422–428.PubMedCrossRefGoogle Scholar
  36. 36.
    Innis MD. Medisets. Computer-assisted diagnosis using a modified set theory. Proc Second Natl Health Conf, Health Informatics Conf, ’94, Gold Coast Australia, 1994, 286–291.Google Scholar
  37. 37.
    Bacchus CM, Quinton C, O’Rourke K et al. A ramdomized crossover trial of quick medical reference (QMR) as a teaching tool for medical interns. J Gen Intern Med 1994; 9:616–621.PubMedCrossRefGoogle Scholar
  38. 38.
    Schoolman HM. Obligations of the expert system builder: meeting the needs of the user. MD Comput 1991; 8:316–321.PubMedGoogle Scholar
  39. 39.
    Diamond LW. A different view of ILIAD. MD Comput 1992; 9:76–77.PubMedGoogle Scholar
  40. 40.
    Waxman HS, Worley WE. Computer-assisted diagnosis. Ann Intern Med 1990; 113:561.CrossRefGoogle Scholar
  41. 41.
    Berner ES. The problem with software reviews of decision support systems. MD Comput 1993; 10:8–12.PubMedGoogle Scholar
  42. 42.
    Kassirer JP. A report card on computer-assisted diagnosis—the grade: C. N Engl J Med 1994; 330:1824–1825.CrossRefGoogle Scholar
  43. 43.
    de Dombal FT. Computer-assisted diagnosis in Europe. N Engl J Med 1994; 331:1238.PubMedCrossRefGoogle Scholar
  44. 44.
    Anderson RE, Hill RB, Key CR. The sensitivity and specificity of clinical diagnostics during five decades. JAMA 1989; 261:1610–1617.PubMedCrossRefGoogle Scholar
  45. 45.
    Chimowitz MI, Logigian EL, Caplan LR. The accuracy of bedside neurological diagnoses. Ann Neurol 1990; 28:78–87.PubMedCrossRefGoogle Scholar
  46. 46.
    Elmore JG, Wells CK, Lee CH et al. Variability in radiologists’ interpretation of mammograms. N Engl J Med 1994; 331:1493–1499.PubMedCrossRefGoogle Scholar
  47. 47.
    Mazoue JG. Diagnosis without doctors. J Med Philos 1990; 15: 559–579.PubMedCrossRefGoogle Scholar
  48. 48.
    Yolton RL. Computer-based diagnostic systems. N Engl J Med 1994; 331:1023.PubMedGoogle Scholar
  49. 49.
    Miller RA, Masarie FE Jr. The demise of the “Greek Oracle” model for medical diagnosis systems. Methods Inf Med 1990; 29:1–2.PubMedGoogle Scholar
  50. 50.
    Miller RA. Why the standard view is standard: people, not machines, understand patients’ problems. J Med Philos 1990; 15:581–591.PubMedCrossRefGoogle Scholar
  51. 51.
    Weed LL. Problem-knowledge couplers: Philosophy, use and interpretation. PKC Corporation 1982; pgs. 2–22.Google Scholar
  52. 52.
    Weed LL. Reengineering medicine: questions and answers. Federation Bull 1995; 82:24–36.Google Scholar
  53. 53.
    Heathfield HA, Wyatt J. Philosophies for the design and development of clinical decision-support systems. Methods Inf Med 1993; 32:1–8.PubMedGoogle Scholar
  54. 54.
    Johnston ME, Langton KB, Haynes RB et al. Effects of computer-based clinical decision support systems on clinician performance and patient outcome:a critical appraisal of research. Ann Intern Med 1994; 120:135–142.PubMedCrossRefGoogle Scholar
  55. 55.
    Wyatt J, Spiegelhalter D. Field trials of medical decision-aids: potential problems and solutions. Proc Annu Symp Comput Appl Med Care 1991:3–7.Google Scholar
  56. 56.
    Miller PL. The evaluation of artificial intelligence systems in medicine. Comput Methods Programs Biomed 1986; 22:5–11.PubMedGoogle Scholar
  57. 57.
    McAdam WA, Brock BM, Armitage T et al. Twelve years experience of computer-aided diagnosis in a district general hospital. Ann R Coll Surg Engl 1990; 72:140–146.PubMedGoogle Scholar
  58. 58.
    de Dombal FT. Computer-aided decision support:in praise of level playing fields. Methods Inf Med 1994; 33:161–163.PubMedGoogle Scholar
  59. 59.
    Berner ES, Maisiak RS. Physician use of interactive functions in diagnostic decision support systems. Proc AMIA Fall Symp 1997; 842.Google Scholar
  60. 60.
    Miller RA. Evaluating evaluations of medical diagnostic systems. JAMIA 1996; 3:429–431.PubMedCrossRefGoogle Scholar
  61. 61.
    Welford CR. A comprehensive computerized patient record with automated linkage to QMR. Proc Annu Symp Comput Appl Med Care 1994: 814–818.Google Scholar

Copyright information

© Springer Science+Business Media New York 1999

Authors and Affiliations

  • Eta S. Berner

There are no affiliations available

Personalised recommendations