Performance evaluation of medical expert systems

  • D. Christine Georgakis
  • Martha Evens
  • Frank Naeymi-Rad
  • David A. Trace
Track 2: Artificial Intelligence
Part of the Lecture Notes in Computer Science book series (LNCS, volume 507)


The major problem in the evaluation of expert systems is the selection of the appropriate statistical measures of performance consistent with the parameters of the system domain. The objective of this paper is to develop the statistical evaluation methodology needed to assess the performance of medical expert systems including MEDAS — the Medical Emergency Decision Assistance System. The measures of performance are selected so as to have an operational interpretation and also reflect the predictive diagnostic capacity of a medical expert system. Certain summary measures are used that represent the sensitivity, specificity, and system response of a medical expert system. Measures of agreement such as the kappa statistic and the measure of conditional agrement are used to measure the agreement between the medical expert system and the physician. Goodman and Kruskal's lambda and tau measures of predictive association are introduced to evaluate the predictive capacity of a medical expert system. This methodology has been partially implemented in the performance evaluation of MEDAS.


Expert System Kappa Statistic Predictive Association Operational Interpretation Medical Expert System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    M. Ben-Bassat, R.W. Carlson, U.K. Puri, M.D. Davenport, J.A. Shriver, M. Latif, R. Smith, L.R. Portigal, E.H. Lipnick, and M.H. Weil. Pattern-Based Interactive Diagnosis of Multiple Disorders: The MEDAS System. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-2, no. 2, March, 1980, pp. 148–160.Google Scholar
  2. [2]
    M. Ben-Bassat, D. Campell, A. MacNeil, and M.H. Weil. Evaluating Multimembership Classifiers: A Methodology and Application to the MEDAS Diagnostic System. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-5, 2, March, 1983, pp. 225–229.CrossRefGoogle Scholar
  3. [3]
    W.G. Cochran. Sampling Techniques. John Wiley & Sons, New York, 1977.zbMATHGoogle Scholar
  4. [4]
    J.A. Reggia. Evaluation of Medical Expert Systems: A Case Study in Performance Analysis. Proceedings of the Ninth Annual Symposium on Computer Applications in Medical Care, Baltimore, MD, 1985. pp. 287–291.Google Scholar
  5. [5]
    J. Cohen. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20, 1960, pp. 37–46.Google Scholar
  6. [6]
    Y. Bishop, S. Fienberg, and P. Holland. Discrete Multivariate Analysis: Theory and Practice. MIT Press, Cambridge, MA. 1984.Google Scholar
  7. [7]
    R. Light. Analysis of Variance for Categorical Data, with Applications to Agreement and Association. Ph.D. Dissertation, Department of Statistics, Harvard University, 1969..Google Scholar
  8. [8]
    D.C. Georgakis, R. Rosenthal, D.A. Trace, and M. Evens. Measures of Performance of the MEDAS System. Proceedings of the Fourth Annual Artificial Intelligence and Advanced Computer Technology Conference, Long Beach, CA, May, 1988, pp. 50–65.Google Scholar
  9. [9]
    L. Goodman and W. Kruskal. Measures of Association for Cross-Classifications. Springer-Verlag, New York, 1979.zbMATHGoogle Scholar
  10. [10]
    B.S. Everitt. The Analysis of Contingency Tables. Halsted Press, John Wiley & Sons, New York, NY, 1977.Google Scholar
  11. [11]
    L. Goodman and W. Kruskal. Measures of Association for Cross-Classifications, Part I. Journal of the American Statistical Association, 49, 1954, pp. 732–764.zbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1991

Authors and Affiliations

  • D. Christine Georgakis
    • 1
  • Martha Evens
    • 2
  • Frank Naeymi-Rad
    • 3
  • David A. Trace
    • 3
  1. 1.Northeastern Illinois UniversityChicago
  2. 2.Illinois Institute of TechnologyChicago
  3. 3.The Chicago Medical SchoolUniversity of Health SciencesNorth Chicago

Personalised recommendations