Skip to main content
SpringerLink
Log in

Markov Decision Processes for Screening and Treatment of Chronic Diseases

  • Chapter
  • First Online:
Book cover Markov Decision Processes in Practice

Part of the book series: International Series in Operations Research & Management Science ((ISOR,volume 248))

Abstract

In recent years, Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs) have found important applications to medical decision making in the context of prevention, screening, and treatment of diseases. In this chapter, we provide a review of state-of-the-art models and methods that have been applied to chronic diseases. We provide a tutorial about how to formulate and solve these important problems emphasizing some of the challenges specific to chronic diseases such as diabetes, heart disease, and cancer. Then, we illustrate important considerations for model formulation and solution methods through two examples. The first example is an MDP model for optimal control of drug treatment decisions for managing the risk of heart disease and stroke in patients with type 2 diabetes. The second example is a POMDP model for optimal design of biomarker-based screening policies in the context of prostate cancer. We end the chapter with a discussion of the challenges of using MDPs and POMDPs for medical contexts and describe some important future directions for research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 229.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 299.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 299.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. World Health Organization, The top 10 causes of death (2013), Available at: who. int/mediacentre/factsheets/fs310/en/. 2014.

    Google Scholar 

  2. M.L. Brandeau, F. Sainfort, W.P. Pierskalla, Operations Research and Health Care (Kluwer Academic Publishers, Boston, 2004)

    Google Scholar 

  3. B.T. Denton, Handbook of Healthcare Operations Management: Methods and Applications (Springer, New York, 2013)

    Book  Google Scholar 

  4. G.S. Zaric, Operations Research and Health Care Policy (Springer, New York, 2013)

    Book  Google Scholar 

  5. O. Alagoz, L.M. Maillart, A.J. Schaefer, M.S. Roberts, The optimal timing of living donor liver transplantation. Manag. Sci. 50 (10), 1420–1430 (2004)

    Article  Google Scholar 

  6. O. Alagoz, C.L. Bryce, S.M. Shechter, A.J. Schaefer, C.-C.H. Chang, D.C. Angus, M.S. Roberts, Incorporating biological natural history in simulation models: empiric estimates of the progression of end-stage liver disease. Med. Decis. Mak. 25, 620–632 (2005)

    Article  Google Scholar 

  7. O. Alagoz, L.M. Maillart, A.J. Schaefer, M.S. Roberts, Which waiting lists should an end-stage liver disease patient join? Technical Report, University of Pittsburgh, 2006

    Google Scholar 

  8. O. Alagoz, L.M. Maillart, A.J. Schaefer, M.S. Roberts, Choosing among living-donor and cadaveric livers. Manag. Sci. 53 (11), 1702–1715 (2007)

    Article  Google Scholar 

  9. D.L. Segev, S.E. Gentry, D.S. Warren, B. Reeb, R.A. Montgomery, Kidney paired donation and optimizing the use of liver donor organs. J. Am. Med. Assoc. 295, 1655–1663 (2005)

    Google Scholar 

  10. S.A. Zenios, G.M. Chertow, L.M. Wein, Dynamic allocation of kidneys to candidates on the trasplant waiting list. Oper. Res. 48 (4), 549–569 (2000)

    Article  Google Scholar 

  11. X. Su, S. Zenios, Patient choice in kidney allocation: the role of the queuing discipline. Manuf. Serv. Oper. Manag. 6 (4), 280–301 (2005)

    Google Scholar 

  12. L.M. Maillart, J.S. Ivy, D. Kathleen, S. Ransom, Assessing dynamic breast cancer screening policies. Oper. Res. 56 (6), 1411–1427 (2008)

    Article  Google Scholar 

  13. J. Chhatwal, O. Alagoz, E.S. Burnside, Optimal breast biopsy decision-making based on mammographic features and demographic factors. Oper. Res. 58 (6), 1577–1591 (2010)

    Article  Google Scholar 

  14. E.K. Lee, T. Fox, I. Crocker, Integer programming applied to intensity-modulated radiation therapy treatment planning. Ann. Oper. Res. 119, 165–181 (2003)

    Article  Google Scholar 

  15. A. Holder, Designing radiotherapy plans with elastic constraints and interior point methods. Health Care Manag. Sci. 6, 5–16 (2003)

    Article  Google Scholar 

  16. F. Preciado-Walters, R. Rardin, M. Langer, V. Thai, A coupled column generation, mixed integer approach to optimal planning of intensity modulated radiation therapy for cancer. Math. Program. 101, 319–338 (2004)

    Article  Google Scholar 

  17. E.K. Lee, R.J. Gallagher, D. Silvern, C. Wu, M. Zaider, Treatment planning for brachytherapy: an integer programming model, two computational approaches, and experiments with permanent prostate implant planning. Phys. Med. Biol. 44, 145–165 (1999)

    Article  Google Scholar 

  18. S.M. Shechter, M.D. Bailey, A.J. Schaefer, M.S. Roberts, The optimal time to initiate HIV therapy under ordered health states. Oper. Res. 56 (1), 20–33 (2008)

    Article  Google Scholar 

  19. E.H. Kaplan, Probability models of needle exchange. Oper. Res. 43 (4), 558–569 (1995)

    Article  Google Scholar 

  20. G. Zaric, M.L. Brandeau, Optimal investment in a portfolio of HIV prevention programs. Med. Decis. Mak. 21, 391–408 (2001)

    Article  Google Scholar 

  21. R. Siegel, J. Ma, Z. Zou, A. Jemal, Cancer statistics, 2014. CA Cancer J. Clin. 64 (1), 9–29 (2014)

    Article  Google Scholar 

  22. T. Ayer, O. Alagoz, N.K. Stout, A POMDP approach to personalize mammography screening decisions. Oper. Res. 60 (5), 1019–1034 (2012)

    Article  Google Scholar 

  23. O. Alagoz, H. Hsu, A.J. Schaefer, M.S. Roberts, Markov decision processes: a tool for sequential decision making under uncertainty. Med. Decis. Mak. 30 (4), 474–483 (2010)

    Article  Google Scholar 

  24. A.J. Schaefer, M.D. Bailey, S.M. Shechter, M.S. Roberts, Modeling medical treatment using Markov decision processes, in Handbook of Operations Research/Management Science Applications in Health Care, ed. by M. Brandeau, F. Sainfort, W. Pierskalla (Kluwer Academic, Dordrecht, 2004), pp. 597–616

    Google Scholar 

  25. E. Regnier, S.M. Shechter, State-space size considerations for disease-progression models. Stat. Med. 32 (22), 3862–3880 (2013)

    Article  Google Scholar 

  26. G.E. Monohan, A survey of partially observable Markov decision processes: theory, models, and algorithms. Manag. Sci. 28 (1), 1–16 (1982)

    Article  Google Scholar 

  27. W.S. Lovejoy, A survey of algorithmic methods for partially observed Markov decision processes. Ann. Oper. Res. 28, 47–66 (1991)

    Article  Google Scholar 

  28. K.A. Anderson, P.M. Odel, P.W.F. Wilson, W.B. Kannel, Cardiovascular disease risk profiles. Am. Heart J. 121, 293–298 (1991)

    Article  Google Scholar 

  29. P.A. Wolf, R.B. D’Agostino, A.J. Belanger, W.B. Kannel, Probability of stroke: a risk profile from the Framingham study. Stroke 22 (3), 312–318 (1991)

    Article  Google Scholar 

  30. P.W.F Wilson, R.B. D?Agostino, D. Levy, A.M. Belanger, H. Silbershatz, W.B. Kannel, Prediction of coronary heart disease using risk factor categories. Circulation 97 (18), 1837–1847 (1998)

    Google Scholar 

  31. R.C. Turner, The uk prospective diabetes study - a review. Diabetes Care 21, C35–C38 (1998)

    Article  Google Scholar 

  32. T.M.E. Davis, H. Millns, I.M. Stratton, R.R. Holman, R.C. Turner, Risk factors for stroke in type 2 diabetes mellitus - United Kingdom Prospective Diabetes Study (UKPDS) 29. Arch. Intern. Med. 159 (10), 1097–1103 (1999)

    Article  Google Scholar 

  33. R.J. Stevens, V. Kothari, A.I. Adler, I.M. Stratton, R.R. Holman, The UKPDS risk engine: a model for the risk of coronary heart disease in type ii diabetes (UKPDS 56). Clin. Sci. 101 (6), 671–679 (2001)

    Article  Google Scholar 

  34. V. Kothari, R.J. Stevens, A.I. Adler, I.M. Stratton, S.E. Manley, H.A. Neil, R.R. Holman et al., UKPDS 60 risk of stroke in type 2 diabetes estimated by the UK Prospective Diabetes Study risk engine. Stroke 33 (7), 1776–1781 (2002)

    Article  Google Scholar 

  35. D.C. Goff, D.M. Lloyd-Jones, G. Bennett, C.J. O’Donnell, S. Coady, J. Robinson, 2013 ACC/AHA guideline on the assessment of cardiovascular risk. J. Am. Coll. Cardiol. 129, S49–S73 (2014)

    Google Scholar 

  36. S.L. Murphy, J. Xu, K.D. Kochanek, Deaths: final data for 2010. National Vital Statistics Reports: from the Centers for Disease Control and Prevention, National Center for Health Statistics. Natl. Vital Stat. Rep. 61 (4), 1–117 (2013)

    Google Scholar 

  37. D. Gold, M.R. Stevenson, D.G. Fryback, HALYS and QALYS and DALYS, oh my: similarities and differences in summary measures of population health. Annu. Rev. Public Health 23, 115–134 (2002)

    Article  Google Scholar 

  38. K.L. Rascati, The $64,000 question – what is a quality-adjusted life year worth? Clin. Ther. 28 (7), 1042–1043 (2006)

    Article  Google Scholar 

  39. D.P. Bertsekas, J.N. Tsitsiklis, Neuro-dynamic programming: an overview, in Proceedings of the 34th IEEE Conference on Decision and Control, 1995, vol. 1 (IEEE, New York, 1995), pp. 560–564

    Google Scholar 

  40. W.B. Powell, Approximate Dynamic Programming (Wiley, Hoboken, 2007)

    Book  Google Scholar 

  41. M.L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming. (Wiley, Hoboken, 1994)

    Google Scholar 

  42. M. Kurt, B.T. Denton, A.J. Schaefer, N.D. Shah, S.A. Smith, The structure of optimal statin initiation policies for patients with type 2 diabetes. IIE Trans. Healthc. Eng. 1, 49–65 (2011)

    Article  Google Scholar 

  43. R.D. Smallwood, E.J. Sondik, The optimal control of partially observable Markov processes over a finite horizon. Oper. Res. 21 (5), 1071–1088 (1973)

    Article  Google Scholar 

  44. A. Cassandra, M.L. Littman, N.L. Zhang, Incremental pruning: a simple, fast, exact method for partially observable Markov decision processes, in Proceedings Thirteenth Annual Conference on Uncertainty in Artificial Intelligence, San Francisco, CA (1997), pp. 54–61

    Google Scholar 

  45. A.R. Cassandra, L.P. Kaelbling, M.L. Littman. Acting optimally in partially observable stochastic domains. AAAI. Vol. 94 (1994). ftp://ftp.cs.brown.edu/pub/techreports/94/cs94-20.pdf

  46. J.E. Eckles, Optimum replacement of stochastically failing systems, Ph.D. Dissertation, Dept. Eng.-Econ. Syst., Stanford University, Stanford (1966)

    Google Scholar 

  47. W.S. Lovejoy, Computationally feasible bounds for partially observed Markov decision processes. Oper. Res. 39 (1), 162–175 (1991)

    Article  Google Scholar 

  48. Centers for Disease Control and Prevention, National Diabetes Statistics Report: Estimates of Diabetes and Its Burden in the United States, 2014. US Department of Health and Human Services, Atlanta, GA, 2014

    Google Scholar 

  49. V. Snow, M.D. Aronson, E.R. Hornbake, C. Mottur-Pilson, K.B. Weiss, Lipid control in the management of type 2 diabetes mellitus: a clinical practice guideline from the American College of Physicians. Ann. Intern. Med. 140 (8), 644–649 (2004)

    Article  Google Scholar 

  50. D.G. Manuel, K. Kwong, P. Tanuseputro, J. Lim, C.A. Mustard, G.M. Anderson, S. Ardal, D.A. Alter, A. Laupacis, Effectiveness and efficiency of different guidelines on statin treatment for preventing deaths from coronary heart disease: modelling study. Br. Med. J. 332 (7555), 1419–1422 (2006)

    Article  Google Scholar 

  51. P.N. Durrington, H. Prais, D. Bhatnagar, M. France, V. Crowley, J. Khan, J. Morgan, Indications for cholesterol-lowering medication: comparison of risk-assessment methods. Lancet, 353 (9149), 278–281 (1999)

    Article  Google Scholar 

  52. B.T. Denton, M. Kurt, N.D. Shah, S.C. Bryant, S.A. Smith, Optimizing the start time of statin therapy for patients with diabetes. Med. Decis. Mak. 29 (3), 351–367 (2009)

    Article  Google Scholar 

  53. J.E. Mason, D.A. England, B.T. Denton, S.A. Smith, M. Kurt, N.D. Shah, Optimizing statin treatment decisions for diabetes patients in the presence of uncertain future adherence. Med. Decis. Mak. 32 (1), 154–166 (2012)

    Article  Google Scholar 

  54. J.E. Mason, B.T. Denton, N.D. Shah, S.A. Smith, Optimizing the simultaneous management of blood pressure and cholesterol for type 2 diabetes patients. Eur. J. Oper. Res. 233 (3), 727–738 (2014)

    Article  Google Scholar 

  55. D.K. Miller, S.M. Homan, Determining transition probabilities confusion and suggestions. Med. Decis. Mak. 14 (1), 52–58 (1994)

    Article  Google Scholar 

  56. M.R. Gold, J.E. Siegel, L.B. Russell, M.C. Weinstein, Cost-Effectiveness in Health and Medicine (Oxford University Press, New York, 1996)

    Google Scholar 

  57. P.A. James, S. Oparil, B.L. Carter, W.C. Cushman, C. Dennison-Himmelfarb, J. Handler, D.T. Lackland, M.L. LeFevre, T.D. MacKenzie, O. Ogedegbe et al., 2014 evidence-based guideline for the management of high blood pressure in adults: report from the panel members appointed to the eighth joint national committee (jnc 8). JAMA, 311 (5), 507–520 (2014)

    Article  Google Scholar 

  58. N.J. Stone, J.G. Robinson, A.H. Lichtenstein, 2013 acc/aha guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the american college of cardiology/american heart association task force on practice guidelines. J. Am. Coll. Cardiol. 163, 2889–2934 (2014); correction. J. Am. Coll. Cardiol. 63 (25), 3024–3025 (2014)

    Google Scholar 

  59. O.H. Franco, E.W. Steyerberg, F.B. Hu, J. Mackenbach, W. Nusselder, Associations of diabetes mellitus with total life expectancy and life expectancy with and without cardiovascular disease. Arch. Intern. Med. 167 (11), 1145–1151 (2007)

    Article  Google Scholar 

  60. A.V. Chobanian, G.L. Bakris, H.R. Black, W.C. Cushman, L.A. Green, J.L. Izzo Jr., D.W. Jones, B.J. Materson, S. Oparil, J.T. Wright Jr. et al., The seventh report of the Joint National Committee on prevention, detection, evaluation, and treatment of high blood pressure: the JNC 7 report. JAMA 289 (19), 2560–2571 (2003)

    Article  Google Scholar 

  61. National Cholesterol Education Program NCEP Expert Panel, Third report of the National Cholesterol Education program (NCEP) expert panel on detection, evaluation, and treatment of high blood cholesterol in adults (Adult Treatment Panel III) final report. Circulation 106 (25), 3143 (2002)

    Google Scholar 

  62. J. Shah, N.D. Mason, M. Kurt, B.T. Denton, A. Schaefer, V. Montori, S. Smith, Comparative effectiveness of guidelines for the management of hyperlipidemia and hypertension for type 2 diabetes patients. Plos One 6 (1), e16170 (2011)

    Google Scholar 

  63. J. Zhang, H. Balasubramanian, B.T. Denton, N. Shah, B. Inman, Optimization of prostate cancer screening decisions: a comparison of patient and societal perspectives. Med. Decis. Mak. 32 (2), 337–349 (2011). doi:10.1177/0272989X11416513

    Article  Google Scholar 

  64. J. Zhang, B.T. Denton, H. Balasubramanian, N.D. Shah, B.A. Inman, Optimization of prostate biopsy referral decisions. Manuf. Serv. Oper. Manag. 14 (4), 529–547 (2012)

    Google Scholar 

  65. G.P. Haas, R.F. Delongchamps, V. Jones, V. Chandan, A.M. Seriod, A.J. Vickers, M. Jumbelic, G. Threatte, R. Korets, H. Lilja, G. De la Roza, Needle biopsies on autopsy prostates: sensitivity of cancer detection based on true prevalence. J. Natl. Cancer Inst. 99, 1484–1849 (2007)

    Article  Google Scholar 

  66. I.M. Thompson, D.P. Ankerst, C. Chi, P.J. Goodman, C.M. Tangen, M.S. Lucia, Z. Feng, H.L. Parnes, C.A. Coltman, Assessing prostate cancer risk: results from the prostate cancer prevention trial. J. Natl. Cancer Inst. 98 (8), 529–534 (2006)

    Article  Google Scholar 

  67. R. Gulati, L. Inoue, J. Katcher, W. Hazelton, R. Etzioni, Calibrating disease progression models using population data: a critical precursor to policy development in cancer control. Biostatistics 11 (4), 707–719 (2010)

    Article  Google Scholar 

  68. W.J. Catalona, P.T. Scardino, J.R. Beck, B.J. Miles, G.W. Chodak, R.A. Thisted, Conservative management of prostate cancer. N. Engl. J. Med. 330 (25), 1830–1832 (1994)

    Google Scholar 

  69. L. Bubendorf, A. Schöpfer, U. Wagner, G. Sauter, H. Moch, N. Willi, T.C. Gasser, M.J. Mihatsch, Metastatic patterns of prostate cancer: an autopsy study of 1,589 patients. Hum. Pathol. 31 (5), 578–583 (2000)

    Article  Google Scholar 

  70. M. Heron, Deaths: leading causes for 2004. Natl. Vital Stat. Rep. 56 (5), 1–96 (2007)

    Google Scholar 

  71. National Cancer Institute, Surveillance Epidemiology and End Results (SEER). SEER Stat Fact Sheets, Cancer: Prostate (2009). http://seer.cancer.gov/statfacts/html/. Accessed May 2015

  72. G.S. Kulkarni, S.M.H. Alibhai, A. Finelli, N.E. Fleshner, M.A.S. Jewett, S.R. Lopushinsky, A.M. Bayoumi, Cost-effectiveness analysis of immediate radical cystectomy versus intravesical bacillus Calmette-Guerin therapy for high-risk, high-grade (t1g3) bladder cancer. Cancer 115 (23), 5450–5459 (2009)

    Article  Google Scholar 

  73. K.E. Bremner, C.A.K.Y. Chong, G. Tomlinson, S.M.H Alibhai, M.D Krahn, A review and meta-analysis of prostate cancer utilities. Med. Decis. Making 27, 288–298 (2007)

    Google Scholar 

  74. G.E. Monohan, A survey of partially observable Markov decision processes: theory, mondels, and algorithms. Manag. Sci. 28 (1), 1–16 (1982)

    Article  Google Scholar 

  75. L.P. Kaelbling, M.L. Littman, A.R. Cassandra, Planning and acting in partially observable stochastic domains. Artif. Intell. 101 (1), 99–134 (1998)

    Article  Google Scholar 

  76. D. Underwood, Risk-based simulation optimization of PSA-based prostate cancer screening. Ph.D. Thesis, North Carolina State University, 2015

    Google Scholar 

  77. G.N. Iyengar, Robust dynamic programming. Math. Oper. Res. 30 (2), 257–280 (2005)

    Article  Google Scholar 

  78. A. Nilim, L.E. Ghaoui, Robust control of Markov decision processes with uncertain transition matrices. Oper. Res. 55 (5), 780–798 (2005)

    Article  Google Scholar 

Download references

Acknowledgements

This material is based in part on work supported by the National Science Foundation under grant numbers CMMI 1462060 (Brian T. Denton) and DGE 1256260 (Lauren N. Steimle). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Author information

Authors and Affiliations

  1. Department of Industrial and Operations Engineering, University of Michigan, 1205 Beal Ave, Ann Arbor, MI, 48109, USA

    Lauren N. Steimle & Brian T. Denton

Authors
  1. Lauren N. Steimle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brian T. Denton
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lauren N. Steimle .

Editor information

Editors and Affiliations

  1. Stochastic Operations Research, University of Twente, Enschede, The Netherlands

    Richard J. Boucherie

  2. Stochastic Operations Research, University of Twente, Enschede, The Netherlands

    Nico M. van Dijk

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Steimle, L.N., Denton, B.T. (2017). Markov Decision Processes for Screening and Treatment of Chronic Diseases. In: Boucherie, R., van Dijk, N. (eds) Markov Decision Processes in Practice. International Series in Operations Research & Management Science, vol 248. Springer, Cham. https://doi.org/10.1007/978-3-319-47766-4_6

Download citation

Publish with us

Policies and ethics

Search

Navigation