Analysis via Markov decision process to evaluate glycemic control strategies of a large retrospective cohort with type 2 diabetes: the ameliorate study



Our aim was to explore optimal treatment decisions for HbA1c control for type 2 diabetes mellitus patients and assess the impact on potential improvements in quality of life compared with current guidelines.


We analyzed a large dataset of HbA1c levels, diabetes-related key risk factors and medication dispensed to 70,069 patients with type 2 diabetes from polyclinics and a large public hospital in Singapore during January 1, 2008, to December 31, 2015. A Markov decision process (MDP) model was developed to determine the optimal treatment policy concerning medication management for glycemic control over a long-term treatment period. We assessed the model performance by comparing quality-adjusted life years (QALYs) gained by the model with those derived by a conventional Markov model informed by current clinical guidelines.


Numerical results showed that optimal treatment strategies derived by the MDP model could increase the total expected QALYs by as much as 0.27 years for patients at higher risk such as old age, high HbA1c levels and smokers. In particular, the improvements in QALYs gained for patients with HbA1c levels of 9% (75 mmol/mol) and above were higher than those with lower HbA1c levels. However, the potential improvements appeared to be marginal for patients at lower risk compared with current guidelines.


Use of data-driven prescriptive analytics would help clinicians make evidence-based treatment decisions for HbA1c control for patients with type 2 diabetes, in particular for those at high risk.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2

Availability of data and materials

The corresponding author has all the data and materials.


  1. 1.

    Chan JC, Malik V, Jia W, Kadowaki T, Yajnik CS, Yoon KH et al (2009) Diabetes in Asia: epidemiology, risk factors, and pathophysiology. JAMA 301:2129–2140

    CAS  Article  Google Scholar 

  2. 2.

    Ramachandran A, Snehalatha C, Shetty AS, Nanditha A (2012) Trends in prevalence of diabetes in Asian countries. World J Diabetes 3:110–117

    Article  Google Scholar 

  3. 3.

    International Diabetes Federation (2019) IDF diabetes atlas, 9th edn. International Diabetes Federation, Brussels, Belgium. Accessed 6 Jan 2020

  4. 4.

    Phan TP, Alkema L, Tai ES, Tan CHX, Yang Q, Lim W-Y et al (2014) Forecasting the burden of type 2 diabetes in Singapore using a demographic epidemiological model of Singapore. BMJ Open Diabetes Res Care 2:e000012

    Article  Google Scholar 

  5. 5.

    Ministry of Health, Singapore. News Highlights on 7th May, 2019. Accessed 6 Jan 2020

  6. 6.

    Gold M, Siegel J, Russell L, Weinstein M (1996) Cost-effectiveness in health and medicine. Oxford University Press, New York

    Google Scholar 

  7. 7.

    Gold M, Stevenson D, Fryback D (2002) HALYs and QALYs and DALYs, OH MY: similarities and differences in summary measures of population health. Annu Rev Publ Health 23:115–134

    Article  Google Scholar 

  8. 8.

    Weinstein MC, Torrance G, McGuire A (2009) QALYs: the basics. Value Health 12:S5–S9

    Article  Google Scholar 

  9. 9.

    Ministry of Health (2018) Clinical practice guidelines: diabetes mellitus (summary booklet). Ministry of Health, Singapore.; 2014. Accessed 15 Mar 2018

  10. 10.

    Roberts MS (1992) Markov process-based Monte Carlo simulation: a tool for modeling complex disease and its application to the timing of liver transplantation. In: Proceedings of the 24th conference on winter simulation 1992, Arlington, VA, pp 1034–1040

  11. 11.

    Puterman ML (2008) Markov decision processes: discrete stochastic dynamic programming. Wiley, Hoboken

    Google Scholar 

  12. 12.

    Alagoz O, Hsu H, Schaefer AJ, Roberts MS (2010) Markov decision processes: a tool for sequential decision making under uncertainty. Med Decis Mak 30:474–483

    Article  Google Scholar 

  13. 13.

    Alagoz O, Maillart LM, Schaefer AJ, Roberts MS (2007) Choosing among living-donor and cadaveric livers. Manag Sci 53:1702–1715

    Article  Google Scholar 

  14. 14.

    Schechter SM, Bailey MD, Schaefer AJ, Roberts MS (2008) The optimal time to initiate HIV therapy under ordered health states. Oper Res 56:20–33

    Article  Google Scholar 

  15. 15.

    Mason JE, Denton BT, Shah ND, Smith SA (2014) Optimizing the simultaneous management of blood pressure and cholesterol for type 2 diabetes patients. Eur J Oper Res 233:727–738

    Article  Google Scholar 

  16. 16.

    Kurt M, Denton BT, Schaefer AJ, Shah ND, Smith SA (2011) The structure of optimal statin initiation policies for patients with type 2 diabetes. IIE Trans Healthc Syst Eng 1:49–65

    Article  Google Scholar 

  17. 17.

    Denton B, Kurt M, Shah N, Bryant S, Smith S (2009) Optimizing the start time of statin therapy for patients with diabetes. Med Decis Mak 29:351–367

    Article  Google Scholar 

  18. 18.

    American Diabetes Association (2013) Standards of medical care in diabetes. Diabetes Care 36:S11–S66

    Article  Google Scholar 

  19. 19.

    Zhang Y, McCoy RG, Mason JE, Smith SA, Shah ND, Denton BT (2014) Second-line agents for glycemic control for type 2 diabetes: are newer agents better? Diabetes Care 37:1338–1345

    CAS  Article  Google Scholar 

  20. 20.

    Arias E (2011) United States Life Tables, 2007. National Vital Statistics Reports, vol 59, no 9

  21. 21.

    Clarke PM, Gray AM, Briggs A, Farmer AJ, Fenn P, Stevens RJ et al (2004) A model to estimate the lifetime health outcomes of patients with type 2 diabetes: the United Kingdom Prospective Diabetes Study (UKPDS) Outcomes Model (UKPDS no. 68). Diabetologia 47:1747–1759

    CAS  Article  Google Scholar 

  22. 22.

    Expert panel on detection, evaluation, and treatment of high blood cholesterol in adults (2001) Executive summary of the third report of the national cholesterol education program (NCEP) expert panel on detection evaluation, and treatment of high blood cholesterol in adults (Adult Treatment Panel III). JAMA 285:2486–2497

    Article  Google Scholar 

  23. 23.

    Hsu WC, Araneta MRG, Kanaya AM, Chiang JL, Fujimoto W (2015) BMI cut points to identify at-risk Asian Americans for type 2 diabetes screening. Diabetes Care 38:150–158

    Article  Google Scholar 

  24. 24.

    Fitch K, Iwasaki K, Pyenson B (2012) The cost and quality gap in diabetes care: an actuarial analysis. Milliman Client Report, commissioned by At&T Services, Inc

Download references


F. Meng acknowledges funding support by the National Medical Research Council, Singapore, Grant Number NMRC/HSRNIG/0008/2015. The authors thank Professor Melvyn Sim, the Department of Analytics & Operations of National University of Singapore, for providing his modeling expertise in decision science under uncertain environments. We thank the anonymous reviewers for their constructive suggestions and comments which helped to improve the presentation of the paper.


This work was supported by the National Medical Research Council, Singapore, Grant Number NMRC/HSRNIG/0008/2015.

Author information




FM contributed to the definition of the problem, model development, data analysis, writing, reviewing and editing the paper. YS contributed to the definition of the problem, model development, reviewing and editing the paper. BHH contributed to the definition of the problem, constructive scientific and clinical advice, reviewing and editing the paper. MKSL contributed to the definition of the problem, model development, constructive scientific and clinical advice, reviewing and editing the paper. FM performed numerical experiments and is the principal investigator of this work. All authors take responsibility for the integrity of the data and the accuracy of the data analysis.

Corresponding author

Correspondence to Fanwen Meng.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Consent for publication

All the authors have approved the final manuscript and consented for publication.

Ethics standard statement

The data of this study were acquired in a retrospective study entitled “Evaluation of treatment strategies in prevention of stroke and coronary heart disease among type 2 diabetic patients using Markov decision process” and ethically approved by the Domain Specific Review Board (DSRB reference number—2015/00698) of National Healthcare Group, Singapore, which was determined by the DSRB to be an exempt category.

Informed consent

Informed consent from patients was not taken because this is a retrospective study and the data were de-identified.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Managed By Antonio Secchi.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 25 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Meng, F., Sun, Y., Heng, B.H. et al. Analysis via Markov decision process to evaluate glycemic control strategies of a large retrospective cohort with type 2 diabetes: the ameliorate study. Acta Diabetol 57, 827–834 (2020).

Download citation


  • Type 2 diabetes
  • Glycemic control
  • Treatment strategy
  • Markov decision process