Design, Conduct, and Analysis of Demonstration Studies

  • Charles P. Friedman
  • Jeremy C. Wyatt
Part of the Computers and Medicine book series (C+M)


Demonstration studies answer questions about an information resource, exploring such issues as the resource’s value to a certain professional group or its impact on the processes and outcomes of health care.1 Recall from Chapter 4 that measurement studies are required to test, refine and validate measurement processes before they can be used to answer questions about a resource or its impact. Chapters 5 and 6 explained these ideas and how to conduct measurement studies in more detail. In this chapter we assume that measurement methods are available and have been verified by appropriate measurement studies. To answer questions via a demonstration study, appropriate evaluation strategies and study designs must be formulated, the sample of subjects and tasks defined, any threats to validity identified and either eliminated or controlled for, and the results analyzed.2, 3 These issues are discussed in this chapter.


Care Provider Decision Support System Receiver Operating Characteristic Information Resource Hawthorne Effect 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Donabedian A: 1966. Evaluating the quality of medical care. Millbank Mem Q 1966;44:166–206.CrossRefGoogle Scholar
  2. 2.
    Wasson JH, Sox HC, Neff RK, Goldman L: Clinical prediction rules: applications and methodological standards. N Engl J Med 1985;313:793–799.PubMedCrossRefGoogle Scholar
  3. 3.
    Wyatt J, Spiegelhalter D: Evaluating medical expert systems: what to test and how? Med Inf(Lond) 1990;15:205–217.CrossRefGoogle Scholar
  4. 4.
    Schwartz D, Lellouch J: Explanatory and pragmatic attitudes in therapeutic trials. J Chron Dis 1967;20:637–648.PubMedCrossRefGoogle Scholar
  5. 5.
    Anderson C: Measuring what works in health care. Science 1994;263:1080–1081.PubMedCrossRefGoogle Scholar
  6. 6.
    Byar DP: Why data bases should not replace randomised controlled clinical trials. Biometrics 1980;36:337–342.PubMedCrossRefGoogle Scholar
  7. 7.
    Airman D: Practical Statistics for Medical Research. London: Chapman & Hall, 1991.Google Scholar
  8. 8.
    Cochran WG, Cox GM: Experimental Designs. New York: Wiley, 1957.Google Scholar
  9. 9.
    Winer BJ: Statistical Principles in Experimental Design. New York: McGraw Hill, 1991.Google Scholar
  10. 10.
    Buck C, Donner A: The design of controlled experiments in the evaluation of non-therapeutic interventions. J Chronic Dis 1982;35:531–538.PubMedCrossRefGoogle Scholar
  11. 11.
    Diwan VK, Eriksson B, Sterky G, Tomson G: Randomization by group in studying the effect of drug information in primary care. Int J Epidemiol 1992;21:124–130.PubMedCrossRefGoogle Scholar
  12. 12.
    Grimshaw JM, Russell IT: Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet 1993;342:1317–1322.PubMedCrossRefGoogle Scholar
  13. 13.
    Adams ID, Chan M, Clifford PC, et al: Computer aided diagnosis of acute abdominal pain: a multicentre study. BMJ 1986;293:800–804.PubMedCrossRefGoogle Scholar
  14. 14.
    Myers DH, Leahy A, Shoeb H, Ryder J: The patient’s view of life in a psychiatric hospital: a questionnaire study and associated methodological considerations. Brit J Psychiatry 1990;156:853–860.CrossRefGoogle Scholar
  15. 15.
    Van Way CW, Murphy JR, Dunn EL, Elerding SC: A feasibility study of computer-aided diagnosis in appendicitis. Surg Gynecol Obstet 1982;155:685–688.PubMedGoogle Scholar
  16. 16.
    Wyatt JC, Altman DG: Prognostic models: clinically useful, or quickly forgotten? BMJ 1995;311:1539–1541.CrossRefGoogle Scholar
  17. 17.
    Knaus W, Wagner D, Lynn J: Short term mortality predictions for critically ill hospitalised patients: science and ethics. Science 1991;254:389–394.PubMedCrossRefGoogle Scholar
  18. 18.
    Sacks H, Chalmers TC, Smith H: Randomised vs. historical controls for clinical trials. Am J Med 1982;72:233–240.PubMedCrossRefGoogle Scholar
  19. 19.
    Cornfield J: Randomisation by group: a formal analysis. Am J Epidemiol 1978;108:100–102.PubMedGoogle Scholar
  20. 20.
    Tierney WM, Overhage JM, McDonald CJ: A plea for controlled trials in medical informatics. J Am Med Inf Assoc 1994;1:353–355.CrossRefGoogle Scholar
  21. 21.
    Lyon HC Jr, Healy JC, Bell JR, et al: PlanAlyzer, an interactive computer-assisted program to teach clinical problem solving in diagnosing anemia and coronary artery disease. Acad Med 1992;67:821–828.PubMedCrossRefGoogle Scholar
  22. 22.
    McDonald CJ, Hui SL, Smith DM, et al: Reminders to physicians from an introspective computer medical record. A two-year randomized trial. Ann Intern Med 1984;100:130–138.PubMedGoogle Scholar
  23. 23.
    Freiman JA, Chalmers TC, Smith H, Kuebler RR: The importance of beta, the Type II error and sample size in the design and interpretation of the randomised controlled trial. N Engl J Med 1978;299:690–694.PubMedCrossRefGoogle Scholar
  24. 24.
    Cohen J: Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ: Lawrence Erlbaum, 1988.Google Scholar
  25. 25.
    Wyatt J: Lessons learned from the field trial of ACORN, an expert system to advise on chest pain. In: Barber B, Cao D, Qin D (eds) Proceedings of the Sixth World Conference on Medical Informatics, Singapore. Amsterdam: North Holland 1989:111–115.Google Scholar
  26. 26.
    Yu VL, Fagan LM, Wraith SM, et al: Antimicrobial selection by computer: a blinded evaluation by infectious disease experts. JAMA 1979;242:1279–1282.PubMedCrossRefGoogle Scholar
  27. 27.
    Schultz KF, Chalmers I, Hayes RJ, Altman DG: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408–412.CrossRefGoogle Scholar
  28. 28.
    Roethligsburger FJ, Dickson WJ: Management and the Worker. Cambridge, MA: Harvard University Press, 1939.Google Scholar
  29. 29.
    Parsons HM: What happened at Hawthorne? Science 1974;183:922–932.PubMedCrossRefGoogle Scholar
  30. 30.
    Pozen MW, d’Agostino RB, Selker HP Sytkowski PA, Hood WB: A predictive instrument to improve coronary care unit admission in acute ischaemic heart disease. N Eng J Med 1984;310:1273–1278.CrossRefGoogle Scholar
  31. 31.
    Murray GD, Murray LS, Barlow P, et al: Assessing the performance and clinical impact of a computerized prognostic system in severe head injury. Stat Med 1986;5:403–410.PubMedCrossRefGoogle Scholar
  32. 32.
    deBliek R, Friedman CP, Wildemuth BM. Martz JM, Twarog RG, File D: Information retieval from a database and the augmentation of personal knowledge. J Am Med Inf Assoc 1994;1:328–338.CrossRefGoogle Scholar
  33. 33.
    Cartmill RSV, Thornton JG: Effect of presentation of partogram information on obstetric decision-making. Lancet 1992;339:1520–1522.PubMedCrossRefGoogle Scholar
  34. 34.
    Suermondt HJ, Cooper GF: An evaluation of explanations of probabilistic inference. Comput Biomed Res 1993;26:242–254.PubMedCrossRefGoogle Scholar
  35. 35.
    Horrocks JC, Lambert DE, McAdam WAF, et al: Transfer of computer-aided diagnosis of dyspepsia from one geographical area to another. Gut 1976;17:640–644.PubMedCrossRefGoogle Scholar
  36. 36.
    Centor RM, Yarbrough B, Wood JP: Inability to predict relapse in acute asthma. New Eng J Med 1984;310:577–580.PubMedCrossRefGoogle Scholar
  37. 37.
    Hart A, Wyatt J: Evaluating black boxes as medical decision-aids: issues arising from a study of neural networks. Med Inf (Lond) 1990;15:229–236.CrossRefGoogle Scholar
  38. 38.
    Wellwood J, Spiegelhalter DJ, Johannessen S: How does computer-aided diagnosis improve the management of acute abdominal pain? Ann R Coll Surg Engl 1992;74:140–146.Google Scholar
  39. 39.
    Indurkhya N, Weiss SM: Models for measuring performance of medical expert systems. AI Med 1989;1:61–70.Google Scholar
  40. 40.
    Titterington DM, Murray GD, Murray LS, et al: Comparison of discriminant techniques applied to a complex data set of head injured patients (with discussion). JR Stat Soc A 1981;144:145–175.CrossRefGoogle Scholar
  41. 41.
    Wald N: Rational use of investigations in clinical practice. In: Hopkins A (ed) Appropriate Investigation and Treatment in Medical Practice. London: Royal College of Physicians, 1990:7–20.Google Scholar
  42. 42.
    Cohen J: Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull 1968;70:213–220.PubMedCrossRefGoogle Scholar
  43. 43.
    Fleiss JL: Measuring agreement between two judges on the presence or absence of a trait. Biometrics 1975;31:357–370.CrossRefGoogle Scholar
  44. 44.
    Hilden J, Habbema DF: Evaluation of clinical decision-aids: more to think about. Med Inf (Lond) 1990;15:275–284.CrossRefGoogle Scholar
  45. 45.
    Hanley JA, McNeil BJ: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143:29–36.PubMedGoogle Scholar
  46. 46.
    Swets JA: Measuring the accuracy of diagnostic systems. Science 1988;240: 1285–1293.PubMedCrossRefGoogle Scholar
  47. 47.
    O’Neil M, Glowinski A: Evaluating and validating very large knowledge-based systems. Med Inf 1990;15:237–252.CrossRefGoogle Scholar
  48. 48.
    Bobbio et al: Completeness of reporting trial results: effect on physicians’ willingness to prescribe. Lancet 1994;343:1209–1211.PubMedCrossRefGoogle Scholar
  49. 49.
    Friedman CP, Wildemuth BM, Gant SP, Muriuki M, File DD: A comparison of Boolean and Hypertext access to a basic science database. Presented to the Annual Conference on Research in Medical Education, November 1995 [abstract].Google Scholar
  50. 50.
    Habbema JDF, Hilden J, Bjerregaard B: The measurement of performance in probabilistic diagnosis; general recommendations. Meth Inf Med 1981;20:97–100.Google Scholar
  51. 51.
    Murphy AH, Winkler RL: Probability forecasting in meteorology. J Amer Statist Assoc 1984;79:489–500.Google Scholar

Copyright information

© Springer Science+Business Media New York 1997

Authors and Affiliations

  • Charles P. Friedman
    • 1
    • 2
  • Jeremy C. Wyatt
    • 3
  1. 1.University of North CarolinaPittsburghUSA
  2. 2.Center for Biomedical InformaticsUniversity of PittsburghPittsburghUSA
  3. 3.Imperial Cancer Research FundLondonUK

Personalised recommendations