Markov Decision Processes for Screening and Treatment of Chronic Diseases

  • Lauren N. SteimleEmail author
  • Brian T. Denton
Part of the International Series in Operations Research & Management Science book series (ISOR, volume 248)


In recent years, Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs) have found important applications to medical decision making in the context of prevention, screening, and treatment of diseases. In this chapter, we provide a review of state-of-the-art models and methods that have been applied to chronic diseases. We provide a tutorial about how to formulate and solve these important problems emphasizing some of the challenges specific to chronic diseases such as diabetes, heart disease, and cancer. Then, we illustrate important considerations for model formulation and solution methods through two examples. The first example is an MDP model for optimal control of drug treatment decisions for managing the risk of heart disease and stroke in patients with type 2 diabetes. The second example is a POMDP model for optimal design of biomarker-based screening policies in the context of prostate cancer. We end the chapter with a discussion of the challenges of using MDPs and POMDPs for medical contexts and describe some important future directions for research.


Prostate Cancer Human Immunodeficiency Virus Prostate Specific Antigen Optimal Policy Action Space 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This material is based in part on work supported by the National Science Foundation under grant numbers CMMI 1462060 (Brian T. Denton) and DGE 1256260 (Lauren N. Steimle). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.


  1. 1.
    World Health Organization, The top 10 causes of death (2013), Available at: who. int/mediacentre/factsheets/fs310/en/. 2014.Google Scholar
  2. 2.
    M.L. Brandeau, F. Sainfort, W.P. Pierskalla, Operations Research and Health Care (Kluwer Academic Publishers, Boston, 2004)Google Scholar
  3. 3.
    B.T. Denton, Handbook of Healthcare Operations Management: Methods and Applications (Springer, New York, 2013)CrossRefGoogle Scholar
  4. 4.
    G.S. Zaric, Operations Research and Health Care Policy (Springer, New York, 2013)CrossRefGoogle Scholar
  5. 5.
    O. Alagoz, L.M. Maillart, A.J. Schaefer, M.S. Roberts, The optimal timing of living donor liver transplantation. Manag. Sci. 50 (10), 1420–1430 (2004)CrossRefGoogle Scholar
  6. 6.
    O. Alagoz, C.L. Bryce, S.M. Shechter, A.J. Schaefer, C.-C.H. Chang, D.C. Angus, M.S. Roberts, Incorporating biological natural history in simulation models: empiric estimates of the progression of end-stage liver disease. Med. Decis. Mak. 25, 620–632 (2005)CrossRefGoogle Scholar
  7. 7.
    O. Alagoz, L.M. Maillart, A.J. Schaefer, M.S. Roberts, Which waiting lists should an end-stage liver disease patient join? Technical Report, University of Pittsburgh, 2006Google Scholar
  8. 8.
    O. Alagoz, L.M. Maillart, A.J. Schaefer, M.S. Roberts, Choosing among living-donor and cadaveric livers. Manag. Sci. 53 (11), 1702–1715 (2007)CrossRefGoogle Scholar
  9. 9.
    D.L. Segev, S.E. Gentry, D.S. Warren, B. Reeb, R.A. Montgomery, Kidney paired donation and optimizing the use of liver donor organs. J. Am. Med. Assoc. 295, 1655–1663 (2005)Google Scholar
  10. 10.
    S.A. Zenios, G.M. Chertow, L.M. Wein, Dynamic allocation of kidneys to candidates on the trasplant waiting list. Oper. Res. 48 (4), 549–569 (2000)CrossRefGoogle Scholar
  11. 11.
    X. Su, S. Zenios, Patient choice in kidney allocation: the role of the queuing discipline. Manuf. Serv. Oper. Manag. 6 (4), 280–301 (2005)Google Scholar
  12. 12.
    L.M. Maillart, J.S. Ivy, D. Kathleen, S. Ransom, Assessing dynamic breast cancer screening policies. Oper. Res. 56 (6), 1411–1427 (2008)CrossRefGoogle Scholar
  13. 13.
    J. Chhatwal, O. Alagoz, E.S. Burnside, Optimal breast biopsy decision-making based on mammographic features and demographic factors. Oper. Res. 58 (6), 1577–1591 (2010)CrossRefGoogle Scholar
  14. 14.
    E.K. Lee, T. Fox, I. Crocker, Integer programming applied to intensity-modulated radiation therapy treatment planning. Ann. Oper. Res. 119, 165–181 (2003)CrossRefGoogle Scholar
  15. 15.
    A. Holder, Designing radiotherapy plans with elastic constraints and interior point methods. Health Care Manag. Sci. 6, 5–16 (2003)CrossRefGoogle Scholar
  16. 16.
    F. Preciado-Walters, R. Rardin, M. Langer, V. Thai, A coupled column generation, mixed integer approach to optimal planning of intensity modulated radiation therapy for cancer. Math. Program. 101, 319–338 (2004)CrossRefGoogle Scholar
  17. 17.
    E.K. Lee, R.J. Gallagher, D. Silvern, C. Wu, M. Zaider, Treatment planning for brachytherapy: an integer programming model, two computational approaches, and experiments with permanent prostate implant planning. Phys. Med. Biol. 44, 145–165 (1999)CrossRefGoogle Scholar
  18. 18.
    S.M. Shechter, M.D. Bailey, A.J. Schaefer, M.S. Roberts, The optimal time to initiate HIV therapy under ordered health states. Oper. Res. 56 (1), 20–33 (2008)CrossRefGoogle Scholar
  19. 19.
    E.H. Kaplan, Probability models of needle exchange. Oper. Res. 43 (4), 558–569 (1995)CrossRefGoogle Scholar
  20. 20.
    G. Zaric, M.L. Brandeau, Optimal investment in a portfolio of HIV prevention programs. Med. Decis. Mak. 21, 391–408 (2001)CrossRefGoogle Scholar
  21. 21.
    R. Siegel, J. Ma, Z. Zou, A. Jemal, Cancer statistics, 2014. CA Cancer J. Clin. 64 (1), 9–29 (2014)CrossRefGoogle Scholar
  22. 22.
    T. Ayer, O. Alagoz, N.K. Stout, A POMDP approach to personalize mammography screening decisions. Oper. Res. 60 (5), 1019–1034 (2012)CrossRefGoogle Scholar
  23. 23.
    O. Alagoz, H. Hsu, A.J. Schaefer, M.S. Roberts, Markov decision processes: a tool for sequential decision making under uncertainty. Med. Decis. Mak. 30 (4), 474–483 (2010)CrossRefGoogle Scholar
  24. 24.
    A.J. Schaefer, M.D. Bailey, S.M. Shechter, M.S. Roberts, Modeling medical treatment using Markov decision processes, in Handbook of Operations Research/Management Science Applications in Health Care, ed. by M. Brandeau, F. Sainfort, W. Pierskalla (Kluwer Academic, Dordrecht, 2004), pp. 597–616Google Scholar
  25. 25.
    E. Regnier, S.M. Shechter, State-space size considerations for disease-progression models. Stat. Med. 32 (22), 3862–3880 (2013)CrossRefGoogle Scholar
  26. 26.
    G.E. Monohan, A survey of partially observable Markov decision processes: theory, models, and algorithms. Manag. Sci. 28 (1), 1–16 (1982)CrossRefGoogle Scholar
  27. 27.
    W.S. Lovejoy, A survey of algorithmic methods for partially observed Markov decision processes. Ann. Oper. Res. 28, 47–66 (1991)CrossRefGoogle Scholar
  28. 28.
    K.A. Anderson, P.M. Odel, P.W.F. Wilson, W.B. Kannel, Cardiovascular disease risk profiles. Am. Heart J. 121, 293–298 (1991)CrossRefGoogle Scholar
  29. 29.
    P.A. Wolf, R.B. D’Agostino, A.J. Belanger, W.B. Kannel, Probability of stroke: a risk profile from the Framingham study. Stroke 22 (3), 312–318 (1991)CrossRefGoogle Scholar
  30. 30.
    P.W.F Wilson, R.B. D?Agostino, D. Levy, A.M. Belanger, H. Silbershatz, W.B. Kannel, Prediction of coronary heart disease using risk factor categories. Circulation 97 (18), 1837–1847 (1998)Google Scholar
  31. 31.
    R.C. Turner, The uk prospective diabetes study - a review. Diabetes Care 21, C35–C38 (1998)CrossRefGoogle Scholar
  32. 32.
    T.M.E. Davis, H. Millns, I.M. Stratton, R.R. Holman, R.C. Turner, Risk factors for stroke in type 2 diabetes mellitus - United Kingdom Prospective Diabetes Study (UKPDS) 29. Arch. Intern. Med. 159 (10), 1097–1103 (1999)CrossRefGoogle Scholar
  33. 33.
    R.J. Stevens, V. Kothari, A.I. Adler, I.M. Stratton, R.R. Holman, The UKPDS risk engine: a model for the risk of coronary heart disease in type ii diabetes (UKPDS 56). Clin. Sci. 101 (6), 671–679 (2001)CrossRefGoogle Scholar
  34. 34.
    V. Kothari, R.J. Stevens, A.I. Adler, I.M. Stratton, S.E. Manley, H.A. Neil, R.R. Holman et al., UKPDS 60 risk of stroke in type 2 diabetes estimated by the UK Prospective Diabetes Study risk engine. Stroke 33 (7), 1776–1781 (2002)CrossRefGoogle Scholar
  35. 35.
    D.C. Goff, D.M. Lloyd-Jones, G. Bennett, C.J. O’Donnell, S. Coady, J. Robinson, 2013 ACC/AHA guideline on the assessment of cardiovascular risk. J. Am. Coll. Cardiol. 129, S49–S73 (2014)Google Scholar
  36. 36.
    S.L. Murphy, J. Xu, K.D. Kochanek, Deaths: final data for 2010. National Vital Statistics Reports: from the Centers for Disease Control and Prevention, National Center for Health Statistics. Natl. Vital Stat. Rep. 61 (4), 1–117 (2013)Google Scholar
  37. 37.
    D. Gold, M.R. Stevenson, D.G. Fryback, HALYS and QALYS and DALYS, oh my: similarities and differences in summary measures of population health. Annu. Rev. Public Health 23, 115–134 (2002)CrossRefGoogle Scholar
  38. 38.
    K.L. Rascati, The $64,000 question – what is a quality-adjusted life year worth? Clin. Ther. 28 (7), 1042–1043 (2006)CrossRefGoogle Scholar
  39. 39.
    D.P. Bertsekas, J.N. Tsitsiklis, Neuro-dynamic programming: an overview, in Proceedings of the 34th IEEE Conference on Decision and Control, 1995, vol. 1 (IEEE, New York, 1995), pp. 560–564Google Scholar
  40. 40.
    W.B. Powell, Approximate Dynamic Programming (Wiley, Hoboken, 2007)CrossRefGoogle Scholar
  41. 41.
    M.L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming. (Wiley, Hoboken, 1994)Google Scholar
  42. 42.
    M. Kurt, B.T. Denton, A.J. Schaefer, N.D. Shah, S.A. Smith, The structure of optimal statin initiation policies for patients with type 2 diabetes. IIE Trans. Healthc. Eng. 1, 49–65 (2011)CrossRefGoogle Scholar
  43. 43.
    R.D. Smallwood, E.J. Sondik, The optimal control of partially observable Markov processes over a finite horizon. Oper. Res. 21 (5), 1071–1088 (1973)CrossRefGoogle Scholar
  44. 44.
    A. Cassandra, M.L. Littman, N.L. Zhang, Incremental pruning: a simple, fast, exact method for partially observable Markov decision processes, in Proceedings Thirteenth Annual Conference on Uncertainty in Artificial Intelligence, San Francisco, CA (1997), pp. 54–61Google Scholar
  45. 45.
    A.R. Cassandra, L.P. Kaelbling, M.L. Littman. Acting optimally in partially observable stochastic domains. AAAI. Vol. 94 (1994).
  46. 46.
    J.E. Eckles, Optimum replacement of stochastically failing systems, Ph.D. Dissertation, Dept. Eng.-Econ. Syst., Stanford University, Stanford (1966)Google Scholar
  47. 47.
    W.S. Lovejoy, Computationally feasible bounds for partially observed Markov decision processes. Oper. Res. 39 (1), 162–175 (1991)CrossRefGoogle Scholar
  48. 48.
    Centers for Disease Control and Prevention, National Diabetes Statistics Report: Estimates of Diabetes and Its Burden in the United States, 2014. US Department of Health and Human Services, Atlanta, GA, 2014Google Scholar
  49. 49.
    V. Snow, M.D. Aronson, E.R. Hornbake, C. Mottur-Pilson, K.B. Weiss, Lipid control in the management of type 2 diabetes mellitus: a clinical practice guideline from the American College of Physicians. Ann. Intern. Med. 140 (8), 644–649 (2004)CrossRefGoogle Scholar
  50. 50.
    D.G. Manuel, K. Kwong, P. Tanuseputro, J. Lim, C.A. Mustard, G.M. Anderson, S. Ardal, D.A. Alter, A. Laupacis, Effectiveness and efficiency of different guidelines on statin treatment for preventing deaths from coronary heart disease: modelling study. Br. Med. J. 332 (7555), 1419–1422 (2006)CrossRefGoogle Scholar
  51. 51.
    P.N. Durrington, H. Prais, D. Bhatnagar, M. France, V. Crowley, J. Khan, J. Morgan, Indications for cholesterol-lowering medication: comparison of risk-assessment methods. Lancet, 353 (9149), 278–281 (1999)CrossRefGoogle Scholar
  52. 52.
    B.T. Denton, M. Kurt, N.D. Shah, S.C. Bryant, S.A. Smith, Optimizing the start time of statin therapy for patients with diabetes. Med. Decis. Mak. 29 (3), 351–367 (2009)CrossRefGoogle Scholar
  53. 53.
    J.E. Mason, D.A. England, B.T. Denton, S.A. Smith, M. Kurt, N.D. Shah, Optimizing statin treatment decisions for diabetes patients in the presence of uncertain future adherence. Med. Decis. Mak. 32 (1), 154–166 (2012)CrossRefGoogle Scholar
  54. 54.
    J.E. Mason, B.T. Denton, N.D. Shah, S.A. Smith, Optimizing the simultaneous management of blood pressure and cholesterol for type 2 diabetes patients. Eur. J. Oper. Res. 233 (3), 727–738 (2014)CrossRefGoogle Scholar
  55. 55.
    D.K. Miller, S.M. Homan, Determining transition probabilities confusion and suggestions. Med. Decis. Mak. 14 (1), 52–58 (1994)CrossRefGoogle Scholar
  56. 56.
    M.R. Gold, J.E. Siegel, L.B. Russell, M.C. Weinstein, Cost-Effectiveness in Health and Medicine (Oxford University Press, New York, 1996)Google Scholar
  57. 57.
    P.A. James, S. Oparil, B.L. Carter, W.C. Cushman, C. Dennison-Himmelfarb, J. Handler, D.T. Lackland, M.L. LeFevre, T.D. MacKenzie, O. Ogedegbe et al., 2014 evidence-based guideline for the management of high blood pressure in adults: report from the panel members appointed to the eighth joint national committee (jnc 8). JAMA, 311 (5), 507–520 (2014)CrossRefGoogle Scholar
  58. 58.
    N.J. Stone, J.G. Robinson, A.H. Lichtenstein, 2013 acc/aha guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the american college of cardiology/american heart association task force on practice guidelines. J. Am. Coll. Cardiol. 163, 2889–2934 (2014); correction. J. Am. Coll. Cardiol. 63 (25), 3024–3025 (2014)Google Scholar
  59. 59.
    O.H. Franco, E.W. Steyerberg, F.B. Hu, J. Mackenbach, W. Nusselder, Associations of diabetes mellitus with total life expectancy and life expectancy with and without cardiovascular disease. Arch. Intern. Med. 167 (11), 1145–1151 (2007)CrossRefGoogle Scholar
  60. 60.
    A.V. Chobanian, G.L. Bakris, H.R. Black, W.C. Cushman, L.A. Green, J.L. Izzo Jr., D.W. Jones, B.J. Materson, S. Oparil, J.T. Wright Jr. et al., The seventh report of the Joint National Committee on prevention, detection, evaluation, and treatment of high blood pressure: the JNC 7 report. JAMA 289 (19), 2560–2571 (2003)CrossRefGoogle Scholar
  61. 61.
    National Cholesterol Education Program NCEP Expert Panel, Third report of the National Cholesterol Education program (NCEP) expert panel on detection, evaluation, and treatment of high blood cholesterol in adults (Adult Treatment Panel III) final report. Circulation 106 (25), 3143 (2002)Google Scholar
  62. 62.
    J. Shah, N.D. Mason, M. Kurt, B.T. Denton, A. Schaefer, V. Montori, S. Smith, Comparative effectiveness of guidelines for the management of hyperlipidemia and hypertension for type 2 diabetes patients. Plos One 6 (1), e16170 (2011)Google Scholar
  63. 63.
    J. Zhang, H. Balasubramanian, B.T. Denton, N. Shah, B. Inman, Optimization of prostate cancer screening decisions: a comparison of patient and societal perspectives. Med. Decis. Mak. 32 (2), 337–349 (2011). doi:10.1177/0272989X11416513CrossRefGoogle Scholar
  64. 64.
    J. Zhang, B.T. Denton, H. Balasubramanian, N.D. Shah, B.A. Inman, Optimization of prostate biopsy referral decisions. Manuf. Serv. Oper. Manag. 14 (4), 529–547 (2012)Google Scholar
  65. 65.
    G.P. Haas, R.F. Delongchamps, V. Jones, V. Chandan, A.M. Seriod, A.J. Vickers, M. Jumbelic, G. Threatte, R. Korets, H. Lilja, G. De la Roza, Needle biopsies on autopsy prostates: sensitivity of cancer detection based on true prevalence. J. Natl. Cancer Inst. 99, 1484–1849 (2007)CrossRefGoogle Scholar
  66. 66.
    I.M. Thompson, D.P. Ankerst, C. Chi, P.J. Goodman, C.M. Tangen, M.S. Lucia, Z. Feng, H.L. Parnes, C.A. Coltman, Assessing prostate cancer risk: results from the prostate cancer prevention trial. J. Natl. Cancer Inst. 98 (8), 529–534 (2006)CrossRefGoogle Scholar
  67. 67.
    R. Gulati, L. Inoue, J. Katcher, W. Hazelton, R. Etzioni, Calibrating disease progression models using population data: a critical precursor to policy development in cancer control. Biostatistics 11 (4), 707–719 (2010)CrossRefGoogle Scholar
  68. 68.
    W.J. Catalona, P.T. Scardino, J.R. Beck, B.J. Miles, G.W. Chodak, R.A. Thisted, Conservative management of prostate cancer. N. Engl. J. Med. 330 (25), 1830–1832 (1994)Google Scholar
  69. 69.
    L. Bubendorf, A. Schöpfer, U. Wagner, G. Sauter, H. Moch, N. Willi, T.C. Gasser, M.J. Mihatsch, Metastatic patterns of prostate cancer: an autopsy study of 1,589 patients. Hum. Pathol. 31 (5), 578–583 (2000)CrossRefGoogle Scholar
  70. 70.
    M. Heron, Deaths: leading causes for 2004. Natl. Vital Stat. Rep. 56 (5), 1–96 (2007)Google Scholar
  71. 71.
    National Cancer Institute, Surveillance Epidemiology and End Results (SEER). SEER Stat Fact Sheets, Cancer: Prostate (2009). Accessed May 2015
  72. 72.
    G.S. Kulkarni, S.M.H. Alibhai, A. Finelli, N.E. Fleshner, M.A.S. Jewett, S.R. Lopushinsky, A.M. Bayoumi, Cost-effectiveness analysis of immediate radical cystectomy versus intravesical bacillus Calmette-Guerin therapy for high-risk, high-grade (t1g3) bladder cancer. Cancer 115 (23), 5450–5459 (2009)CrossRefGoogle Scholar
  73. 73.
    K.E. Bremner, C.A.K.Y. Chong, G. Tomlinson, S.M.H Alibhai, M.D Krahn, A review and meta-analysis of prostate cancer utilities. Med. Decis. Making 27, 288–298 (2007)Google Scholar
  74. 74.
    G.E. Monohan, A survey of partially observable Markov decision processes: theory, mondels, and algorithms. Manag. Sci. 28 (1), 1–16 (1982)CrossRefGoogle Scholar
  75. 75.
    L.P. Kaelbling, M.L. Littman, A.R. Cassandra, Planning and acting in partially observable stochastic domains. Artif. Intell. 101 (1), 99–134 (1998)CrossRefGoogle Scholar
  76. 76.
    D. Underwood, Risk-based simulation optimization of PSA-based prostate cancer screening. Ph.D. Thesis, North Carolina State University, 2015Google Scholar
  77. 77.
    G.N. Iyengar, Robust dynamic programming. Math. Oper. Res. 30 (2), 257–280 (2005)CrossRefGoogle Scholar
  78. 78.
    A. Nilim, L.E. Ghaoui, Robust control of Markov decision processes with uncertain transition matrices. Oper. Res. 55 (5), 780–798 (2005)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Department of Industrial and Operations EngineeringUniversity of MichiganAnn ArborUSA

Personalised recommendations