Performance evaluation of medical expert systems
The major problem in the evaluation of expert systems is the selection of the appropriate statistical measures of performance consistent with the parameters of the system domain. The objective of this paper is to develop the statistical evaluation methodology needed to assess the performance of medical expert systems including MEDAS — the Medical Emergency Decision Assistance System. The measures of performance are selected so as to have an operational interpretation and also reflect the predictive diagnostic capacity of a medical expert system. Certain summary measures are used that represent the sensitivity, specificity, and system response of a medical expert system. Measures of agreement such as the kappa statistic and the measure of conditional agrement are used to measure the agreement between the medical expert system and the physician. Goodman and Kruskal's lambda and tau measures of predictive association are introduced to evaluate the predictive capacity of a medical expert system. This methodology has been partially implemented in the performance evaluation of MEDAS.
KeywordsExpert System Kappa Statistic Predictive Association Operational Interpretation Medical Expert System
Unable to display preview. Download preview PDF.
- M. Ben-Bassat, R.W. Carlson, U.K. Puri, M.D. Davenport, J.A. Shriver, M. Latif, R. Smith, L.R. Portigal, E.H. Lipnick, and M.H. Weil. Pattern-Based Interactive Diagnosis of Multiple Disorders: The MEDAS System. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-2, no. 2, March, 1980, pp. 148–160.Google Scholar
- J.A. Reggia. Evaluation of Medical Expert Systems: A Case Study in Performance Analysis. Proceedings of the Ninth Annual Symposium on Computer Applications in Medical Care, Baltimore, MD, 1985. pp. 287–291.Google Scholar
- J. Cohen. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20, 1960, pp. 37–46.Google Scholar
- Y. Bishop, S. Fienberg, and P. Holland. Discrete Multivariate Analysis: Theory and Practice. MIT Press, Cambridge, MA. 1984.Google Scholar
- R. Light. Analysis of Variance for Categorical Data, with Applications to Agreement and Association. Ph.D. Dissertation, Department of Statistics, Harvard University, 1969..Google Scholar
- D.C. Georgakis, R. Rosenthal, D.A. Trace, and M. Evens. Measures of Performance of the MEDAS System. Proceedings of the Fourth Annual Artificial Intelligence and Advanced Computer Technology Conference, Long Beach, CA, May, 1988, pp. 50–65.Google Scholar
- B.S. Everitt. The Analysis of Contingency Tables. Halsted Press, John Wiley & Sons, New York, NY, 1977.Google Scholar