Skip to main content
SpringerLink
Log in

Trust and Transparency in Machine Learning-Based Clinical Decision Support

  • Chapter
  • First Online:
Human and Machine Learning

Part of the book series: Human–Computer Interaction Series ((HCIS))

Abstract

Machine learning and other statistical pattern recognition techniques have the potential to improve diagnosis in medicine and reduce medical error. But technology can be both a solution to and a source of errors. Machine learning-based clinical decision support systems may cause new errors due to automation bias and automation complacency which arise from inappropriate trust in the technology. Transparency into a systems internal logic can improve trust in automation, but is hard to achieve in practice. This chapter discusses the clinical and technology related factors that influence clinician trust in automated systems, and can affect the need for transparency when developing machine learning-based clinical decision support systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Alberdi, E., Povyakalo, A., Strigini, L., Ayton, P.: Effects of incorrect computer-aided detection (CAD) output on human decision-making in mammography. Acad. Radiol. 11(8), 909–918 (2004)

    Article  Google Scholar 

  2. Bahner, J.E., Hüper, A.D., Manzey, D.: Misuse of automated decision aids: Complacency, automation bias and the impact of training experience. Int. J. Human Comput. Stud. 66(9), 688–699 (2008)

    Article  Google Scholar 

  3. Battles, J.B., Keyes, M.A.: Technology and patient safety: a two-edged sword (2002)

    Google Scholar 

  4. Berlin, L.: Radiologic errors, past, present and future. Diagnosis 1(1), 79–84 (2014)

    Article  Google Scholar 

  5. Berner, E.S., Graber, M.L.: Overconfidence as a cause of diagnostic error in medicine. Am. J. Med. 121(5) (2008)

    Article  Google Scholar 

  6. Berner, E.S., La Lande, T.J.: Overview of clinical decision support systems. In: Clinical Decision Support Systems, pp. 1–17. Springer, Cham (2016)

    Google Scholar 

  7. Berner, E.S., Maisiak, R.S., Heudebert, G., Young, K.: Clinician performance and prominence of diagnoses displayed by a clinical diagnostic decision support system (2003)

    Google Scholar 

  8. Campbell, S.G., Croskerry, P., Bond, W.F.: Profiles in patient safety: a perfect storm in the emergency department. Acad. Emerg. Med. 14(8), 743–749 (2007)

    Google Scholar 

  9. Carayon, P., Kianfar, S., Li, Y., Xie, A., Alyousef, B., Wooldridge, A.: A systematic review of mixed methods research on human factors and ergonomics in health care (2015)

    Article  Google Scholar 

  10. Carayon, P., Schoofs Hundt, A., Karsh, B.T., Gurses, A.P., Alvarado, C.J., Smith, M., Flatley Brennan, P.: Work system design for patient safety: the SEIPS model. Qual. Saf. Health Care 15(suppl–1), i50–i58 (2006)

    Article  Google Scholar 

  11. Castelvecchi, D.: Can we open the black box of AI? Nature 538(7623), 20–23 (2016)

    Article  Google Scholar 

  12. Coiera, E.: Technology, cognition and error. BMJ Qual. Saf. 24(7), 417–422 (2015)

    Article  Google Scholar 

  13. Cramer, H., Evers, V., Ramlal, S., Van Someren, M., Rutledge, L., Stash, N., Aroyo, L., Wielinga, B.: The effects of transparency on trust in and acceptance of a content-based art recommender. User Model. User-Adapt. Interact. 18(5), 455–496 (2008)

    Article  Google Scholar 

  14. Croskerry, P.: The feedback sanction. Acad. Emerg. Med. 7(11), 1232–8 (2000)

    Article  Google Scholar 

  15. Dreiseitl, S., Binder, M.: Do physicians value decision support? A look at the effect of decision support systems on physician opinion. Artif. Intell. Med. 33(1), 25–30 (2005)

    Article  Google Scholar 

  16. Dworkin, : Autonomy and informed consent. President’s commission for the study of ethical problems in medicine and biomedical and behavioral research making health care decisions. Fed. Regist. 3(226), 52,880–52,930 (1982)

    Google Scholar 

  17. Dzindolet, M.T., Peterson, S.A., Pomranky, R.A., Pierce, L.G., Beck, H.P.: The role of trust in automation reliance. Int. J. Human Comput. Stud. 58(6), 697–718 (2003)

    Article  Google Scholar 

  18. Friedman, C.P., Elstein, A.S., Wolf, F.M., Murphy, G.C., Franz, T.M., Heckerling, P.S., Fine, P.L., Miller, T.M., Abraham, V.: Enhancement of clinicians’ diagnostic reasoning by computer-based consultation: a multisite study of 2 systems. JAMA 282(19), 1851–1856 (1999)

    Article  Google Scholar 

  19. Friedman, C.P., Gatti, G.G., Franz, T.M., Murphy, G.C., Wolf, F.M., Heckerling, P.S., Fine, P.L., Miller, T.M., Elstein, A.S.: Do physicians know when their diagnoses are correct? Implications for decision support and error reduction. J. Gen. Intern. Med. 20(4), 334–339 (2005)

    Article  Google Scholar 

  20. Goddard, K., Roudsari, A., Wyatt, J.C.: Automation bias: Empirical results assessing influencing factors. Int. J. Med. Inf. (2014)

    Google Scholar 

  21. Goddard, K., Roudsari, A., Wyatt, J.C.: Automation bias: a systematic review of frequency, effect mediators, and mitigators. J. Am. Med. Inf. Assoc. 19(1), 121–127 (2012)

    Article  Google Scholar 

  22. Goodman, B., Flaxman, S.: EU regulations on algorithmic decision-making and a right to explanation. In: 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016) (Whi), pp. 26–30. xx, xx (2016)

    Google Scholar 

  23. Holst, H., Aström, K., Järund, A., Palmer, J., Heyden, A., Kahl, F., Tägil, K., Evander, E., Sparr, G., Edenbrandt, L.: Automated interpretation of ventilation-perfusion lung scintigrams for the diagnosis of pulmonary embolism using artificial neural networks. Eur. J. Nucl. Med. 27(4), 400–406 (2000)

    Article  Google Scholar 

  24. Kaufman, S., Rosset, S.: Leakage in data mining: formulation, detection, and avoidance. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 556–563 (2012)

    Google Scholar 

  25. Kohn, Linda T.; Corrigan, Janet M.; Donaldson, M.S.: [To err is human: building a safer health system], vol. 21 (2002)

    Google Scholar 

  26. Koppel, R., Kreda, D.: Health care information technology vendors’ hold harmless clause. JAMA 301(12), 1276–1278 (2009)

    Article  Google Scholar 

  27. Korunka, C., Weiss, A., Karetta, B.: Effects of new technologies with special regard for the implementation process per se. J. Organ. Behav. 14(4), 331–348 (1993)

    Article  Google Scholar 

  28. Ledley, R.S., Lusted, L.B.: Reasoning foundations of medical diagnosis; symbolic logic, probability, and value theory aid our understanding of how physicians reason. Science (New York, N.Y.) 130(3366), 9–21 (1959)

    Article  Google Scholar 

  29. Lee, C.S., Nagy, P.G., Weaver, S.J., Newman-Toker, D.E.: Cognitive and system factors contributing to diagnostic errors in radiology (2013)

    Article  Google Scholar 

  30. Leveson, N.G., Turner, C.S.: An investigation of the Therac-25 accidents. Computer 26(7), 18–41 (1993)

    Article  Google Scholar 

  31. Li, L., Cheng, W.Y., Glicksberg, B.S., Gottesman, O., Tamler, R., Chen, R., Bottinger, E.P., Dudley, J.T.: Identification of type 2 diabetes subgroups through topological analysis of patient similarity. Sci. Trans. Med. 7(311), 311ra174–311ra174 (2015)

    Article  Google Scholar 

  32. Lyell, D., Magrabi, F., Raban, M.Z., Pont, L., Baysari, M.T., Day, R.O., Coiera, E.: Automation bias in electronic prescribing. BMC Med. Inf. Decis. Mak. 17(1), 28 (2017)

    Article  Google Scholar 

  33. Makary, M.A., Daniel, M.: Medical error the third leading cause of death in the US. BMJ, i2139 (2016)

    Google Scholar 

  34. Marakas, G.: Decision Support Systems in the 21st Century. Prentice Hall (1999)

    Google Scholar 

  35. Mehdy, M.M., Ng, P.Y., Shair, E.F., Saleh, N.I.M., Gomes, C.: Artificial neural networks in image processing for early detection of breast cancer. Comput. Math. Methods Med. 2017, 1–15 (2017)

    Article  Google Scholar 

  36. Metzger, J., MacDonald, K.: Clinical decision support for the independent physician practice (2002)

    Google Scholar 

  37. Miller, R.A., Gardner, R.M.: Summary recommendations for responsible monitoring and regulation of clinical software systems. Ann. Intern. Med. 127(9), 842–845 (1997)

    Article  Google Scholar 

  38. Miller, R.A., Masarie, F.E.: The demise of the Greek Oracle model for medical diagnostic systems. Methods Inf. Med. 29(1), 1–2 (1990)

    Article  Google Scholar 

  39. Monteiro, C., Avelar, A.F.M., Pedreira, MdLG: Interruptions of nurses’ activities and patient safety: an integrative literature review. Revista Latino-Americana de Enfermagem 23(1), 169–179 (2015)

    Article  Google Scholar 

  40. Naguib, R.N.G., Sherbet, G.V.: Artificial Neural Networks in Cancer Diagnosis, Prognosis, and Patient Management (2001)

    Google Scholar 

  41. National Patient Safety Agency: Healthcare risk assessment made easy. National Patient Safety Agency, 3 (March) (2007)

    Google Scholar 

  42. Norman, D.a.: The Design of Everyday Things: Revised and Expanded Edition (1988)

    Google Scholar 

  43. Osheroff, J.A.: Improving Medication Use and Outcomes with Clinical Decision Support:: A Step by Step Guide. HIMSS (2009)

    Google Scholar 

  44. Persell, S., Friedberg, M., Meeker, D., Linder, J., Fox, C., Goldstein, N., Shah, P., Doctor, J., Knight, T.: Use of behavioral economics and social psychology to improve treatment of acute respiratory infections (BEARI): rationale and design of a cluster randomized controlled trial [1RC4AG039115-01] - study protocol and baseline practice and provider characteris. BMC Infect. Dis. 13, 290 (2013)

    Article  Google Scholar 

  45. Quionero-Candela, J., Sugiyama, M., Schwaighofer, A., Lawrence, N.: Dataset Shift in Machine Learning. MIT Press (2008)

    Google Scholar 

  46. Rogers, Y., Rogers, Y.: A brief introduction to distributed cognition. Cogn. Sci. 0000, 00000 (1997)

    Google Scholar 

  47. Rosset, S., Perlich, C., Świrszcz, G., Melville, P., Liu, Y.: Medical data mining: Insights from winning two competitions. Data Min. Knowl. Discov. 20(3), 439–468 (2010)

    Article  MathSciNet  Google Scholar 

  48. Samaranayake, N.R., Cheung, S.T., Chui, W.C., Cheung, B.M.: Technology-related medication errors in a tertiary hospital: A 5-year analysis of reported medication incidents. Int. J. Med. Inf. 81(12), 828–833 (2012)

    Article  Google Scholar 

  49. Shojania, K.G., Dixon-Woods, M.: Estimating deaths due to medical error: the ongoing controversy and why it matters: Table1. BMJ Quality and Safety pp. bmjqs–2016–006,144 (2016)

    Google Scholar 

  50. Tan, J., Sheps, S.: Health Decision Support Systems (1998)

    Google Scholar 

  51. Torsvik, T., Lillebo, B., Mikkelsen, G.: Presentation of clinical laboratory results: an experimental comparison of four visualization techniques. J. Am. Med. Inf. Assoc. 20(2), 325–331 (2013)

    Article  Google Scholar 

  52. Weingart, S.N., McL Wilson R, R.M., Gibberd, R.W., Harrison, B.: Epidemiology of medical error. West. J. Med. 172(6), 390–3 (2000)

    Article  Google Scholar 

  53. Westbrook, J.I.: Association of interruptions with an increased risk and severity of medication administration errors. Arch. Intern. Med. 170(8), 683 (2010)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University College London, School of Management, Level 38, 1 Canada Square, London, E14 5AA, UK

    Cosima Gretton

Authors
  1. Cosima Gretton
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cosima Gretton .

Editor information

Editors and Affiliations

  1. DATA61, CSIRO, Eveleigh, New South Wales, Australia

    Jianlong Zhou

  2. DATA61, CSIRO, Eveleigh, New South Wales, Australia

    Fang Chen

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gretton, C. (2018). Trust and Transparency in Machine Learning-Based Clinical Decision Support. In: Zhou, J., Chen, F. (eds) Human and Machine Learning. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-319-90403-0_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-90403-0_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-90402-3

  • Online ISBN: 978-3-319-90403-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation