Abstract
As with any new agent introduced into medical practice, a medical expert system must be critically evaluated for its safety and efficacy in a clinical setting prior to its use in patient care. Assessing expert system performance in this way is similar to data collection in that it can be a difficult and time-consuming process. At least three aspects of system performance can be considered*: accuracy, usefulness, and transferability.2 In this section a framework for viewing expert system performance assessment is outlined. The next section illustrates the concepts involved by discussing the clinical evaluation of an expert system that assists with management of patients with transient ischemic attacks (TIAs). The final section identifies factors that were crucial in the efficient development and testing of the TIA expert system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Chandrasekaran B: On evaluating AI systems for medical diagnosis. In: Proceeding MEDCOMP ’82. Philadelphia: IEEE, 1982.
Reggia J, Tuhrim S: An overview of computer-assisted medical decision making. In Reggia J, Tuhrim S (eds): Computer-Assisted Medical Decision Making. New York: Springer Verlag, 1985, pp. 3–45.
McNeil B, Keeler E, Adelstein S: Primer on certain elements of medical decision making. N Engl J Med 293:211, 1975.
Salamon R, Bernadet M, Samson M, Derouesne C, Gremy F: Bayesian method applied to decision making in neurology—methodological considerations. Methods Inf Med 15:174, 1976.
Miller R, Pople H, Myers J: INTERNIST-I, an experimental computer-based diagnostic consultant for general internal medicine. N Engl J Med 307: 468, 1982.
Yu VL, Fagan LM, Wraith SM, Clancey WJ, Scott AC, Hanningan J, Blum RL, Buchanan BG, Cohen SN: Antimicrobial selection by a computer—a blinded evaluation by infectious disease experts. JAMA 42:1279, 1979.
Weiss S, Kulikowski C, Safir A: Glaucoma consultation by computer. Comput Biol Med 8:25, 1978.
Spitzer R, Cohen J, Fleiss J, Endicott J: Quantification of agreement in psychiatric diagnosis. Arch Gen Psychiatry 17:83, 1967.
Cohen J: Weighted kappa. Psychol Bull 70:213, 1968.
Shapiro A: The evaluation of clinical predictions. N Engl J Med 296:1509, 1977.
Zagoria R, Reggia J: Transferability of medical decision support systems based on Bayesian classification. Med Decis Making 3:501, 1983.
Dannenberg A, Shapiro A, Fries J: Enhancement of clinical predictive ability by computer consultation. Methods Inf Med 18:10, 1979.
Miller P: Critiquing-a different approach to expert computer advice in medicine. p. 17. In: Proceedings Eighth Annual Symposium on Computer Application in Medical Care. Washington DC: IEEE, 1984.
Langlotz C, Shortliffe E: Adopting a consultation system to critique user plans. Int J Man-Machine Stud 19:479, 1983.
Jelliffe R, Buell J, Kalaba R: Reduction of digitalis toxicity by computer-assisted glycoside dosage regimens. Ann Intern Med 77:891, 1972.
DeDombal F: Computer-aided diagnosis of acute abdominal pain. Br Med J 2:9, 1972.
Pryor T, Gardner R, Clayton P, Warner H: The HELP system. p. 109. In Blum B (ed): Information Systems for Patient Care,New York: Springer Verlag, 1984.
Zoltie N, Horrocks J, deDombal F: Computer assisted diagnosis of dyspepsia-report of transferability of a system, with emphasis on early diagnosis of gastric cancer. Methods Inf Med 16:89, 1977.
Calanchini P, Swanson P, Gotshall R, Haerer A, Poshanzer D, Price T: Cooperative study of hospital frequency and character of transient ischemic attacks: IV. The reliability of diagnosis: JAMA 238:2029, 1977.
Fisher R, Hanley D, Whisnant J, Barnett H: Transient Ischemic Attacks. Read before the American Society for Neurological Investigation, San Francisco, 1981.
Haerer A, Gotshall R, Conneally P, Dyken M, Poshanzer D, Price T, Swanson P, Calanchini P: Cooperative study of hospital frequency and character of transient ischemic attacks. III. Variations in treatment. JAMA 238:142, 1977.
Swanson P, Calanchini P, Dyken M, Gotshall R, Haerer A, Poskanzer D, Price T, Conneally P: Cooperative study of hospital frequency and character of transient: ischemic attacks. II. Performance of angiography among six centers. JAMA 237:2202, 1977.
Toole, J, Howard G: Management of asymptomatic carotid artery murmur and transient ischemic attacks. Arch Neurol 38:443, 1981.
Reggia J, Pula T, Price T, and Perricone B: Towards an intelligent textbook of neurology. p. 190. In: Proceedings of the Fourth Annual Symposium on Computer Applications in Medical Care. New York: IEEE, 1980.
Shortliffe E: Computer-Based Medical Consultations: MYCIN New York: Elsevier, 1976.
Reggia J, Tabb R, Price T, Banko M, Hebel R: Computer-aided assessment of transient ischemic attacks. Arch Neurol 41:1248, 1984.
Dyken M, Conneally M, Haerer A, Gotshall R, Calanchini P, Poskanzer D, Price T, Swanson P: Cooperative study of hospital frequency and character of transient ischemic attacks. I. Background, organization, and clinical survey. JAMA 237:882, 1977.
Kunitz S, Havekost C, Gross C: Pilot data bank networks for neurological disorders, p. 793. In: Proceedings of the Third Annual Symposium on Computer Application in Medical Care. Long Beach, CA: IEEE, 1979.
Davis R: Expert systems: Where are we? Where do we go from here? Artif Intell Magazine, 3:3, 1982.
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1988 Springer-Verlag New York Inc.
About this chapter
Cite this chapter
Reggia, J.A. (1988). Evaluation of Medical Expert Systems: Case Study in Performance Assessment. In: Miller, P.L. (eds) Selected Topics in Medical Artificial Intelligence. Computers and Medicine. Springer, New York, NY. https://doi.org/10.1007/978-1-4613-8777-0_17
Download citation
DOI: https://doi.org/10.1007/978-1-4613-8777-0_17
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4613-8779-4
Online ISBN: 978-1-4613-8777-0
eBook Packages: Springer Book Archive