AIME 87 pp 15-24 | Cite as

The evaluation of clinical decision support systems: a discussion of the methodology used in the ACORN project

  • Jeremy Wyatt
Part of the Lecture Notes in Medical Informatics book series (LNMED, volume 33)


The evaluation of medical expert systems, particularly those intended for decision support in the clinical domain, has not received sufficient emphasis. The techniques used for the evaluation of 14 medical systems are reviewed and contrasted with those used for the evaluation of new drugs. A three stage procedure is outlined. This is currently being used for the evaluation of ACORN, a decision support system to aid the admission of patients to cardiac care units.


Expert System Decision Support System Clinical Decision Support System Role Definition Clinical Prediction Rule 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Adams ID, Chan M, Clifford PC etalii, 1986: ‘Computer aided diagnosis of acute abdominal pain: a multicentre study’; BMJ vol. 293, pp. 800–804.PubMedCrossRefGoogle Scholar
  2. 2.
    Aikins JS, Kunz JC, Shortliffe EH, 1983: ‘PUFF: an expert system for the interpretation of pulmonary function data’; Computers & Biomedical Res. vol. 16, pp. 199–208.CrossRefGoogle Scholar
  3. 3.
    Aikins JS, 1983: ‘Prototypical knowledge for expert systems’; Artificial Intelligence vol. 20, pp. 163–210.CrossRefGoogle Scholar
  4. 4.
    Alvey P, Greaves MF, 1986: ‘Observations on the development of a high performance system for leukaemia diagnosis’; in Proc. Expert Systems ’86 ( Brighton ), ed. Bramer M., pub. Cambridge University Press.Google Scholar
  5. 5.
    Armitage P, 1971: ‘Statistical methods in medical research’; pub. Blackwell Scientific, Oxford.Google Scholar
  6. 6.
    Catanzerite VA, Greenburg AG, Bremerman HJ, 1982: ‘Computer consultation in neurology: subjective and objective evaluations of the Neurologist system’; Comput. Biol. Med. vol. 12, pp. 342–355.Google Scholar
  7. 7.
    Diamond GA, Staniloff HM, Forester JS etalii, 1983: ‘Computer assisted diagnosis in the noninvasive evaluation of patients with suspected coronary artery disease’; J Am Coll Cardiol. vol. 1, pp. 444–455.PubMedCrossRefGoogle Scholar
  8. 8.
    Ellam SV & Maisey MN, 1986: ‘A knowledge based system to assist in medical image interpretation: design and evaluation methodology’; in Proc. Expert Systems ’86 ( Brighton ), ed. Bramer M.,pub. CUP.Google Scholar
  9. 9.
    Fieschi M, Joubert M, Fieschi D etalii, 1983: ‘A program for expert diagnosis and therapeutic decision’; Med Inform, vol. 8, pp. 127–135.CrossRefGoogle Scholar
  10. 10.
    Fox J, Barber D, Bardhan KD, 1980: ‘Alternatives to Bayes: a quantitative comparison with rule-based diagnostic inference’; Meth. Inform. Med. vol. 19, pp. 210–215.PubMedGoogle Scholar
  11. 11.
    Fox J, Myers CD, Greaves MF, Pegram S, 1985: ‘Knowledge acquisition for expert systems: experience in leukaemia diagnosis’; Meth. Inf. Med. vol. 24, pp. 65–72.Google Scholar
  12. 12.
    Fox J, Duncan TD, Frost D, Glowinski A, Hajnal S, O’Neill M, F, 1986: ‘Organising a large knowledge base: the Oxford System of Medicine’, paper presented at Expert Systems ’86 (Brighton).Google Scholar
  13. 13.
    Gaschnig J, Klahr P, Pople H, Shortliffe E, Terry A, 1983: ‘Evaluation of expert systems: issues and case studies’; in Hayes-Roth F, Waterman DA & Lenat D (eds), ‘Building expert systems’, Addison Wesley 1983.Google Scholar
  14. 14.
    Hudson DL, Cohen ME, Deedwania PC, 1984: ‘Emerge: a rule based expert system implemented on a microcomputer’; Microcomputer Applications vol. 3, pp. 79–83.Google Scholar
  15. 15.
    Mercer J, Talbot IC, 1985: ‘Clinical diagnosis: a post-mortem assessment of accuracy an the 1980s’; Postgrad. Med. J. vol. 61, pp. 713–716.PubMedCrossRefGoogle Scholar
  16. 16.
    Miller PL, 1983: ‘Medical plan-analysis: the ATTENDING system’; proc. IJCAI-83, pp. 239–241.Google Scholar
  17. 17.
    Politakis P, Weiss SM, 1982: ‘A system for empirical experimentation with expert knowledge’; Proc. 15th Hawaii Int. Conf. on Systems Science, 2, pp. 649–657.Google Scholar
  18. 18.
    Schwartz WB, Patil RS, Szolovits P, 1987: ‘Artificial intelligence in medicine: where do we stand ?’, NEJMed vol. 316, pp. 685–688.Google Scholar
  19. 19.
    Shortliffe EH, Davis R, Axline SG etalii, 1975: ‘Computer based consultations in clinical therapeutics… the MYCIN system’; Computers & Biomed. Res. vol. 8, pp. 303–320CrossRefGoogle Scholar
  20. 20.
    Shortliffe EH, Clancey WJ, 1984: ‘Anticipating the second decade’; in ‘Readings in medical AI’, Shortliffe & Clancey, pub. Addison Wesley 1984.Google Scholar
  21. 21.
    21.Spiegelhalter DJ 1983: ‘Evaluation of clinical decision aids, with an application to a system for dyspepsia’; Statistics in Med vol. 2, pp. 207–216.CrossRefGoogle Scholar
  22. 22.
    22.Szolovits P, Long WJ, 1982: ‘The development of clinical expertise in the computer’; chap. 4 in ‘Artificial inteligence in medicine’, ed. Szolovits P, pub. Westview, for AAAS, Washington.Google Scholar
  23. 23.
    Wasson JH, Sox HC, Neff RK & Goldman L: 1985: ‘Clinical prediction rules: applications and methodological standards’; NEJMed vol. 313, pp. 793–799.Google Scholar
  24. 24.
    Weiss SM, Kulikowski CA, Galen RS, 1981: ‘Developing microprocessor-based expert models for instrument interpretation’; in Proc. of seventh IJCAI, pp. 853–855, pub. Kaufmann Inc, Los Altos.Google Scholar
  25. 25.
    Weiss SM, Kulikowski CA, Amarel S, 1978: ‘A model-based method for computer-aided medical decision-making’; Artificial Intelligencevolil, pp. 145–172.Google Scholar
  26. 26.
    Wulff HR, 1976: ‘Rational diagnosis and treatment’; pub. Blackwell Scientific Publications, Oxford.Google Scholar
  27. 27.
    Wyatt JC, Emerson PA, Crichton NJ, 1986: ’Computer aided diagnosis of acute abdominal pain (letter); BMJ vol. 293, p. 1305.Google Scholar
  28. 28.
    Yu VL, Fagan LM, Wraith SM etalii, 1979: ‘Antimicrobial selection by computer: a blinded evaluation by infectious disease experts’; JAMA, vol. 242, pp. 1279–1282.PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1987

Authors and Affiliations

  • Jeremy Wyatt
    • 1
  1. 1.Dept. of Thoracic MedicineWestminster HospitalLondonUK

Personalised recommendations