Skip to main content
SpringerLink
Log in

Design, Conduct, and Analysis of Demonstration Studies

  • Chapter
Book cover Evaluation Methods in Medical Informatics

Part of the book series: Computers and Medicine ((C+M))

Abstract

Demonstration studies answer questions about an information resource, exploring such issues as the resource’s value to a certain professional group or its impact on the processes and outcomes of health care.1 Recall from Chapter 4 that measurement studies are required to test, refine and validate measurement processes before they can be used to answer questions about a resource or its impact. Chapters 5 and 6 explained these ideas and how to conduct measurement studies in more detail. In this chapter we assume that measurement methods are available and have been verified by appropriate measurement studies. To answer questions via a demonstration study, appropriate evaluation strategies and study designs must be formulated, the sample of subjects and tasks defined, any threats to validity identified and either eliminated or controlled for, and the results analyzed.2, 3 These issues are discussed in this chapter.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 74.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Donabedian A: 1966. Evaluating the quality of medical care. Millbank Mem Q 1966;44:166–206.

    Article  Google Scholar 

  2. Wasson JH, Sox HC, Neff RK, Goldman L: Clinical prediction rules: applications and methodological standards. N Engl J Med 1985;313:793–799.

    Article  PubMed  CAS  Google Scholar 

  3. Wyatt J, Spiegelhalter D: Evaluating medical expert systems: what to test and how? Med Inf(Lond) 1990;15:205–217.

    Article  CAS  Google Scholar 

  4. Schwartz D, Lellouch J: Explanatory and pragmatic attitudes in therapeutic trials. J Chron Dis 1967;20:637–648.

    Article  PubMed  CAS  Google Scholar 

  5. Anderson C: Measuring what works in health care. Science 1994;263:1080–1081.

    Article  PubMed  CAS  Google Scholar 

  6. Byar DP: Why data bases should not replace randomised controlled clinical trials. Biometrics 1980;36:337–342.

    Article  PubMed  CAS  Google Scholar 

  7. Airman D: Practical Statistics for Medical Research. London: Chapman & Hall, 1991.

    Google Scholar 

  8. Cochran WG, Cox GM: Experimental Designs. New York: Wiley, 1957.

    Google Scholar 

  9. Winer BJ: Statistical Principles in Experimental Design. New York: McGraw Hill, 1991.

    Google Scholar 

  10. Buck C, Donner A: The design of controlled experiments in the evaluation of non-therapeutic interventions. J Chronic Dis 1982;35:531–538.

    Article  PubMed  CAS  Google Scholar 

  11. Diwan VK, Eriksson B, Sterky G, Tomson G: Randomization by group in studying the effect of drug information in primary care. Int J Epidemiol 1992;21:124–130.

    Article  PubMed  CAS  Google Scholar 

  12. Grimshaw JM, Russell IT: Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet 1993;342:1317–1322.

    Article  PubMed  CAS  Google Scholar 

  13. Adams ID, Chan M, Clifford PC, et al: Computer aided diagnosis of acute abdominal pain: a multicentre study. BMJ 1986;293:800–804.

    Article  PubMed  CAS  Google Scholar 

  14. Myers DH, Leahy A, Shoeb H, Ryder J: The patient’s view of life in a psychiatric hospital: a questionnaire study and associated methodological considerations. Brit J Psychiatry 1990;156:853–860.

    Article  CAS  Google Scholar 

  15. Van Way CW, Murphy JR, Dunn EL, Elerding SC: A feasibility study of computer-aided diagnosis in appendicitis. Surg Gynecol Obstet 1982;155:685–688.

    PubMed  Google Scholar 

  16. Wyatt JC, Altman DG: Prognostic models: clinically useful, or quickly forgotten? BMJ 1995;311:1539–1541.

    Article  Google Scholar 

  17. Knaus W, Wagner D, Lynn J: Short term mortality predictions for critically ill hospitalised patients: science and ethics. Science 1991;254:389–394.

    Article  PubMed  CAS  Google Scholar 

  18. Sacks H, Chalmers TC, Smith H: Randomised vs. historical controls for clinical trials. Am J Med 1982;72:233–240.

    Article  PubMed  CAS  Google Scholar 

  19. Cornfield J: Randomisation by group: a formal analysis. Am J Epidemiol 1978;108:100–102.

    PubMed  CAS  Google Scholar 

  20. Tierney WM, Overhage JM, McDonald CJ: A plea for controlled trials in medical informatics. J Am Med Inf Assoc 1994;1:353–355.

    Article  CAS  Google Scholar 

  21. Lyon HC Jr, Healy JC, Bell JR, et al: PlanAlyzer, an interactive computer-assisted program to teach clinical problem solving in diagnosing anemia and coronary artery disease. Acad Med 1992;67:821–828.

    Article  PubMed  Google Scholar 

  22. McDonald CJ, Hui SL, Smith DM, et al: Reminders to physicians from an introspective computer medical record. A two-year randomized trial. Ann Intern Med 1984;100:130–138.

    PubMed  CAS  Google Scholar 

  23. Freiman JA, Chalmers TC, Smith H, Kuebler RR: The importance of beta, the Type II error and sample size in the design and interpretation of the randomised controlled trial. N Engl J Med 1978;299:690–694.

    Article  PubMed  CAS  Google Scholar 

  24. Cohen J: Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ: Lawrence Erlbaum, 1988.

    Google Scholar 

  25. Wyatt J: Lessons learned from the field trial of ACORN, an expert system to advise on chest pain. In: Barber B, Cao D, Qin D (eds) Proceedings of the Sixth World Conference on Medical Informatics, Singapore. Amsterdam: North Holland 1989:111–115.

    Google Scholar 

  26. Yu VL, Fagan LM, Wraith SM, et al: Antimicrobial selection by computer: a blinded evaluation by infectious disease experts. JAMA 1979;242:1279–1282.

    Article  PubMed  CAS  Google Scholar 

  27. Schultz KF, Chalmers I, Hayes RJ, Altman DG: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408–412.

    Article  Google Scholar 

  28. Roethligsburger FJ, Dickson WJ: Management and the Worker. Cambridge, MA: Harvard University Press, 1939.

    Google Scholar 

  29. Parsons HM: What happened at Hawthorne? Science 1974;183:922–932.

    Article  PubMed  CAS  Google Scholar 

  30. Pozen MW, d’Agostino RB, Selker HP Sytkowski PA, Hood WB: A predictive instrument to improve coronary care unit admission in acute ischaemic heart disease. N Eng J Med 1984;310:1273–1278.

    Article  CAS  Google Scholar 

  31. Murray GD, Murray LS, Barlow P, et al: Assessing the performance and clinical impact of a computerized prognostic system in severe head injury. Stat Med 1986;5:403–410.

    Article  PubMed  CAS  Google Scholar 

  32. deBliek R, Friedman CP, Wildemuth BM. Martz JM, Twarog RG, File D: Information retieval from a database and the augmentation of personal knowledge. J Am Med Inf Assoc 1994;1:328–338.

    Article  CAS  Google Scholar 

  33. Cartmill RSV, Thornton JG: Effect of presentation of partogram information on obstetric decision-making. Lancet 1992;339:1520–1522.

    Article  PubMed  CAS  Google Scholar 

  34. Suermondt HJ, Cooper GF: An evaluation of explanations of probabilistic inference. Comput Biomed Res 1993;26:242–254.

    Article  PubMed  CAS  Google Scholar 

  35. Horrocks JC, Lambert DE, McAdam WAF, et al: Transfer of computer-aided diagnosis of dyspepsia from one geographical area to another. Gut 1976;17:640–644.

    Article  PubMed  CAS  Google Scholar 

  36. Centor RM, Yarbrough B, Wood JP: Inability to predict relapse in acute asthma. New Eng J Med 1984;310:577–580.

    Article  PubMed  CAS  Google Scholar 

  37. Hart A, Wyatt J: Evaluating black boxes as medical decision-aids: issues arising from a study of neural networks. Med Inf (Lond) 1990;15:229–236.

    Article  CAS  Google Scholar 

  38. Wellwood J, Spiegelhalter DJ, Johannessen S: How does computer-aided diagnosis improve the management of acute abdominal pain? Ann R Coll Surg Engl 1992;74:140–146.

    Google Scholar 

  39. Indurkhya N, Weiss SM: Models for measuring performance of medical expert systems. AI Med 1989;1:61–70.

    Google Scholar 

  40. Titterington DM, Murray GD, Murray LS, et al: Comparison of discriminant techniques applied to a complex data set of head injured patients (with discussion). JR Stat Soc A 1981;144:145–175.

    Article  Google Scholar 

  41. Wald N: Rational use of investigations in clinical practice. In: Hopkins A (ed) Appropriate Investigation and Treatment in Medical Practice. London: Royal College of Physicians, 1990:7–20.

    Google Scholar 

  42. Cohen J: Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull 1968;70:213–220.

    Article  PubMed  CAS  Google Scholar 

  43. Fleiss JL: Measuring agreement between two judges on the presence or absence of a trait. Biometrics 1975;31:357–370.

    Article  Google Scholar 

  44. Hilden J, Habbema DF: Evaluation of clinical decision-aids: more to think about. Med Inf (Lond) 1990;15:275–284.

    Article  CAS  Google Scholar 

  45. Hanley JA, McNeil BJ: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143:29–36.

    PubMed  CAS  Google Scholar 

  46. Swets JA: Measuring the accuracy of diagnostic systems. Science 1988;240: 1285–1293.

    Article  PubMed  CAS  Google Scholar 

  47. O’Neil M, Glowinski A: Evaluating and validating very large knowledge-based systems. Med Inf 1990;15:237–252.

    Article  Google Scholar 

  48. Bobbio et al: Completeness of reporting trial results: effect on physicians’ willingness to prescribe. Lancet 1994;343:1209–1211.

    Article  PubMed  CAS  Google Scholar 

  49. Friedman CP, Wildemuth BM, Gant SP, Muriuki M, File DD: A comparison of Boolean and Hypertext access to a basic science database. Presented to the Annual Conference on Research in Medical Education, November 1995 [abstract].

    Google Scholar 

  50. Habbema JDF, Hilden J, Bjerregaard B: The measurement of performance in probabilistic diagnosis; general recommendations. Meth Inf Med 1981;20:97–100.

    CAS  Google Scholar 

  51. Murphy AH, Winkler RL: Probability forecasting in meteorology. J Amer Statist Assoc 1984;79:489–500.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of North Carolina, 8074, Forbes Tower, Pittsburgh, PA, 15213, USA

    Charles P. Friedman Ph.D. (Professor and Director, Formerly Assistant Dean for Medical Education and Informatics)

  2. Center for Biomedical Informatics, University of Pittsburgh, 8074, Forbes Tower, Pittsburgh, PA, 15213, USA

    Charles P. Friedman Ph.D. (Professor and Director, Formerly Assistant Dean for Medical Education and Informatics)

  3. Imperial Cancer Research Fund, P.O. Box 123, Lincoln’s Inn Fields, London, WC2A 3PX, UK

    Jeremy C. Wyatt M.B., B.S., D.M., M.R.C.P. (Consultant, Medical Informatics)

Authors
  1. Charles P. Friedman Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jeremy C. Wyatt M.B., B.S., D.M., M.R.C.P.
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 1997 Springer Science+Business Media New York

About this chapter

Cite this chapter

Friedman, C.P., Wyatt, J.C. (1997). Design, Conduct, and Analysis of Demonstration Studies. In: Evaluation Methods in Medical Informatics. Computers and Medicine. Springer, New York, NY. https://doi.org/10.1007/978-1-4757-2685-5_7

Download citation

  • DOI: https://doi.org/10.1007/978-1-4757-2685-5_7

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4757-2687-9

  • Online ISBN: 978-1-4757-2685-5

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation