Optimizing diabetes screening frequencies for at-risk groups

Abstract

There is strong evidence that diabetes is underdiagnosed in the US: the Centers for Disease Control and Prevention (CDC) estimates that approximately 25% of diabetic patients are unaware of their condition. To encourage timely diagnosis of at-risk patients, we develop screening guidelines stratified by body mass index (BMI), age, and prior test history by using a Partially Observed Markov Decision Process (POMDP) framework to provide more personalized screening frequency recommendations. We identify structural results that prove the existence of threshold solutions in our problem and allow us to determine the relative timing and frequency of screening given different risk profiles. We then use nationally representative empirical data to identify a policy that provides the optimal action (screen or wait) every six months from age 45 to 90. We find that the current screening guidelines are suboptimal, and the recommended diabetes screening policy should be stratified by age and by finer BMI thresholds than in the status quo. We identify age ranges and BMI categories for which relatively less or more screening is needed compared to the existing guidelines to help physicians target patients most at risk. Compared to the status quo, we estimate that an optimal screening policy would generate higher net monetary benefits by $3,200-$3,570 and save $120-$1,290 in health expenditures per individual in the US above age 45.

Appendices

Appendix:

Table 7 Model notation

A Proofs

A.1 Observations on TP2 matrices

We first present properties to be used in Lemmas 1 and 2. We define the determinant of a two by two submatrix of a matrix A to be:

$$ \begin{array}{@{}rcl@{}} \left\|\begin{array}{ll} a_{i_{1}j_{1}} & a_{i_{1}j_{2}} \\ a_{i_{2}j_{1}} & a_{i_{2}j_{2}} \end{array}\right\| = a_{i_{1}j_{1}}a_{i_{2}j_{2}} - a_{i_{1}j_{2}}a_{i_{2}j_{1}}, \forall i_{2} \geq i_{1}, j_{2} \geq j_{1} \end{array} $$

Property 1 For two nonnegative vectors with the first vector ordered from largest to smallest and the second one ordered from smallest to largest, the determinant of any two by two submatrix is nonnegative.

Proof

For any two column vectors where $a_{i_{1}j_{1}} \geq a_{i_{2}j_{1}} \geq 0$ and $a_{i_{2}j_{2}} \geq a_{i_{1}j_{2}} \geq 0$, then $a_{i_{1}j_{1}}a_{i_{2}j_{2}} - a_{i_{1}j_{2}}a_{i_{2}j_{1}} \geq 0$. Similarly, for any two row vectors where $a_{i_{1}j_{1}} \geq a_{i_{1}j_{2}} \geq 0$ and $a_{i_{2}j_{2}} \geq a_{i_{2}j_{1}} \geq 0$, we have $a_{i_{1}j_{1}}a_{i_{2}j_{2}} - a_{i_{1}j_{2}}a_{i_{2}j_{1}}$ ≥ 0. □

Property 2 For any nonnegative two by two matrix, if $a_{i_{1}j_{2}}$ or $a_{i_{2}j_{1}}$ = 0, then the determinant is nonnegative.

Proof

We have have $a_{i_{1}j_{1}}, a_{i_{1}j_{2}}, a_{i_{2}j_{1}}, a_{i_{2}j_{2}} \geq 0$ and $a_{i_{1}j_{2}}$ or $a_{i_{2}j_{1}}$ = 0. Thus, $a_{i_{1}j_{1}}a_{i_{2}j_{2}} - a_{i_{1}j_{2}}a_{i_{2}j_{1}} = a_{i_{1}j_{1}}a_{i_{2}j_{2}} \geq 0$. □

Property 3 For any nonnegative two by two matrix, if $a_{i_{1}j_{1}} \geq a_{i_{2}j_{1}} \geq 0$ and $a_{i_{2}j_{2}} \geq a_{i_{1}j_{2}} \geq 0$ or $a_{i_{1}j_{1}} \geq a_{i_{1}j_{2}} \geq 0$ and $a_{i_{2}j_{2}} \geq a_{i_{2}j_{1}} \geq 0$, then the determinant is nonnegative.

Proof

In the first case, we have have $a_{i_{1}j_{1}}, a_{i_{1}j_{2}},a_{i_{2}j_{1}},$ $a_{i_{2}j_{2}} \geq 0$ and $a_{i_{1}j_{1}} \geq a_{i_{2}j_{1}} \geq 0$ and $a_{i_{2}j_{2}} \geq a_{i_{1}j_{2}} \geq 0$. Thus, $a_{i_{1}j_{1}}a_{i_{2}j_{2}} - a_{i_{1}j_{2}}a_{i_{2}j_{1}} \geq 0$. In the second case, we have $a_{i_{1}j_{1}}, a_{i_{1}j_{2}}, a_{i_{2}j_{1}}, a_{i_{2}j_{2}} \geq 0$ and $a_{i_{1}j_{1}} \geq a_{i_{1}j_{2}} \geq 0$ and $a_{i_{2}j_{2}} \geq a_{i_{2}j_{1}} \geq 0$. Thus, $a_{i_{1}j_{1}}a_{i_{2}j_{2}} - a_{i_{1}j_{2}}a_{i_{2}j_{1}} \geq 0$. □

Lemma 1

If the following conditions hold, then in a problem with three health states (orderable from the healthiest to sickest) and observable death (the fourth health state), the observation matrix, O^a, is TP2 given action a.

• 1. Observation results have a greater than 50% chance to coincide with the true health state.

• 2. O^a has the IFR property.

• 3. $q^{{a}}_{12} \geq q^{{a}}_{13}$ and $q^{{a}}_{32} \geq q^{{a}}_{31}$

• 4. $q^{{a}}_{12}q^{{a}}_{23} \geq q^{{a}}_{13}q^{{a}}_{22}$ and $q^{{a}}_{21}q^{{a}}_{32} \geq q^{{a}}_{22}q^{{a}}_{31}$.

A.2 Proof of Lemma 1

The death state is observable, so our observation matrix, O^a, is fully captured by a three by three matrix. For a matrix to be TP2, all determinants of second-order submatrices need to be greater than or equal to 0. With three combinations of columns and rows, we have nine submatrices.

• Conditions 1-3 ensure that $q^{{a}}_{11}\geq q^{{a}}_{21} \geq q^{{a}}_{31}$ and $q^{{a}}_{33}\geq q^{{a}}_{23} \geq q^{{a}}_{13}$. Property 1 then tells us submatrices formed from columns 1 and 3 are non-negative.

• Conditions 1-3 also ensure that $q^{{a}}_{11}\geq q^{{a}}_{12}$, $q^{{a}}_{11}\geq q^{{a}}_{21}$, $q^{{a}}_{22} \geq q^{{a}}_{12}$, and $q^{{a}}_{22} \geq q^{a}_{21}$. Property 3 then ensures that $q^{a}_{11}q^{a}_{22} - q^{a}_{12}q^{a}_{21} \geq 0$ and $q^{a}_{22}q^{a}_{33} - q^{a}_{23}q^{a}_{32} \geq 0$ must be satisfied.

• Condition 4 ensures all remaining submatrices are non-negative: $q^{{a}}_{12}q^{{a}}_{23} - q^{{a}}_{13}q^{{a}}_{22} \!\geq \! 0$ and $q^{{a}}_{21}q^{{a}}_{32} - q^{a}_{22}q^{a}_{31} \!\geq$ 0.

Therefore determinants of all second-order submatrices are nonnegative.

The TP2 property enables us to order the health states as we observe test results over time. These conditions specify what it means to be a “good” test, where test results are more likely to be “close” to the true state than otherwise, in a context with multiple health states. This means that in a case with a good diagnosis test, one only needs to verify condition four to establish that the observations are TP2. We next turn to establishing similar conditions for the transition matrix, P^t,BMI.

Lemma 2

If the following conditions hold, then in a problem with three health states (orderable from the healthiest to sickest) and death (the fourth health state), the transition matrix, P^t,BMI, is TP2 given time t and BMI category BMI.

• 1. Once you are in the sickest (non-dead) state, one cannot recover to healthier states.

• 2. Health states are more than 50% likely to stay the same between transitions.

• 3. P^t,BMI has the IFR property.

• 4. $p^{t,BMI}_{23} \geq p^{t,BMI}_{13}$.

• 5. $p^{t,BMI}_{12} \geq p^{t,BMI}_{13} \geq p^{t,BMI}_{14}$ and $p^{t,BMI}_{23} \geq p^{t,BMI}_{24}$

• 6. $p^{t,BMI}_{i_{1}j_{1}}p^{t,BMI}_{i_{2}j_{2}} \geq p^{t,BMI}_{i_{1}j_{2}}p^{t,BMI}_{i_{2}j_{1}}$ where i₁,i₂ ∈{1, 2, 3} and j₁,j₂ ∈{2, 3, 4} ∀i₂ ≥ i₁,j₂ ≥ j₁

A.3 Proof of Lemma 2

For a matrix to be TP2, all determinants of second-order submatrices need to be greater than or equal to 0.

• The death state is an absorbing state, so the last row of P^t,BMI is comprised of 0s and 1. Using Property 2, the determinants of all submatrices that include this row are greater or equal to zero.

• Conditions 1-5 ensure that $p^{t,BMI}_{11}\geq p^{t,BMI}_{21} \geq p^{t,BMI}_{31}$ and $p^{t,BMI}_{32}\geq p^{t,BMI}_{22} \geq p^{t,BMI}_{12}$ and $p^{t,BMI}_{33}\geq p^{t,BMI}_{23} \geq p^{t,BMI}_{13}$ and $p^{t,BMI}_{34}\geq p^{t,BMI}_{24} \geq p^{t,BMI}_{14}$. Property 1 then tells us submatrices formed from columns 1 and any of the remaining columns are non-negative.

• Condition 1 also ensures that $p^{t,BMI}_{31} = p^{t,BMI}_{32} = 0$. Then $p^{t,BMI}_{12}p^{t,BMI}_{33} - p^{t,BMI}_{13}p^{t,BMI}_{32} \geq 0$ and $p^{t,BMI}_{22}p^{t,BMI}_{33} - p^{t,BMI}_{23}p^{t,BMI}_{32} \geq 0$ and $p^{t,BMI}_{12}p^{t,BMI}_{34} - p^{t,BMI}_{14}p^{t,BMI}_{32} \geq 0$ and $p^{t,BMI}_{22}p^{t,BMI}_{34} - p^{t,BMI}_{24}p^{t,BMI}_{32} \geq 0$ must be satisfied according to Property 2.

• Condition 6 ensures all remaining submatrices are non-negative: $p^{t,BMI}_{12}p^{t,BMI}_{23} - p^{t,BMI}_{13}p^{t,BMI}_{22} \geq 0$ and $p^{t,BMI}_{12}p^{t,BMI}_{24} - p^{t,BMI}_{14}p^{t,BMI}_{22} \geq 0$ and $p^{t,BMI}_{13}$ $p^{t,BMI}_{24} - p^{t,BMI}_{14}p^{t,BMI}_{23} \geq 0$ and $p^{t,BMI}_{13}p^{t,BMI}_{34} - p^{t,BMI}_{14}p^{t,BMI}_{33} \geq 0$ and $p^{t,BMI}_{23}$ $p^{t,BMI}_{34} -p^{t,BMI}_{24}$ $p^{t,BMI}_{33} \geq 0$.

Therefore determinants of all second-order submatrices are nonnegative.

We will use this result to show when the optimal policy has a threshold solution. There are more conditions for the transition matrix to be TP2 than the observation matrix because we cannot exclude the death state (but we can in the observation analysis because it is fully observable). Evaluating TP2 directly for a four by four matrix requires thirty-six submatrices; here, we have reduced the conditions so only five submatrices need to be checked. These lemmas allow us to examine when a monotonic value function exists.

A.4 Proof of proposition 1

This problem is a special case studied by Krishnamurthy (2016), where our linear reward function satisfies the assumption that reward function is first-order stochastically decreasing with respect to belief π^t for each action a^t [51]. To see this, define their Hidden Markov model (HMM) filter T(π^t, O^a,a) and the normalization measure σ(π^t,O^a,a) as:

$$ \begin{aligned} T(\pi^{t},O^{a},a)=\pi^{t+1} \\ \textrm{where } \pi^{t+1}(j)=\frac{{\sum}_{i \in \mathcal{S}} \pi^{t}(i) p^{t,BMI}_{ij} q^{{a}}_{jk}}{{\sum}_{i \in \mathcal{S}} {\sum}_{l \in \mathcal{S}}\pi^{t}(i) p^{t,BMI}_{il} q^{{a}}_{lk}}\ \end{aligned} $$

(A1)

$$ \sigma(\pi^{t},O^{a},a)=\sum\limits_{i \in \mathcal{S}} \sum\limits_{l \in \mathcal{S}}\pi^{t}(i) p^{t,BMI}_{il} q^{{a}}_{lk}\ $$

(A2)

Therefore, the proposition is true by Theorem 11.2.1 in Krishnamurthy (2016) [51].

A.5 Proof of Theorem 1

A threshold π^∗,t can be defined as the π^∗,t such that v^t(π^∗,t,a^t = a₁) = v^t(π^∗,t,a^t = a₂). We consider two cases: (1) there is no such threshold, or (2) there are one or more thresholds. For the case that there does not exist a threshold π^∗,t, we can define the threshold to be [1 0 0 0] or [0 0 0 1] since one action dominates the other over the entire belief simplex (always screen or wait for any belief at time t). We therefore restrict our analysis to the case where there exists at least one π^∗,t.

We use a proof by contradiction to show that there cannot be two or more thresholds given the conditions in Theorem 1. We start with an observation on the single threshold case.

Let us start with the single threshold ($\pi ^{\ast ,t}_{1}$) case. Recall that we have v^t(π^t,a₁) − v^t(π^t,a₂) is MLR decreasing in π^t, by assumption. If the current belief $\pi ^{t} \leq _{r} \pi ^{\ast ,t}_{1}$, we then must have:

$$v^{t}(\pi^{t},a_{1}) - v^{t}(\pi^{t},a_{2}) \geq v^{t}(\pi^{\ast,t}_{1},a_{1}) - v^{t}(\pi^{\ast,t}_{1}, a_{2}) = 0$$

Where the last equality is by the definition of threshold. Thus, we have v^t(π^t,a₁) ≥ v^t(π^t,a₂). Therefore, if $\pi ^{t} \leq _{r} \pi ^{\ast ,t}_{1}$, then the optimal action is a₁.

Similarly, if the current belief $\pi ^{t} \geq _{r} \pi ^{\ast ,t}_{1}$, then we have:

$$v^{t}(\pi^{t}, a_{1}) - v^{t}(\pi^{t}, a_{2}) \leq v^{t}(\pi^{\ast,t}_{1},a_{1}) - v^{t}(\pi^{\ast,t}_{1},a_{2}) = 0$$

Thus, we have v^t(π^t,a₁) ≤ v^t(π^t,a₂), so then the optimal action must be a₂.

Therefore, if there is a single threshold, the optimal action must differ on either side of it. To show that two or more thresholds cannot exist simultaneously, we use this rationale in a proof by contradiction.

Suppose there are two thresholds: $\pi ^{\ast ,t}_{1} \leq _{r} \pi ^{\ast ,t}_{2}$. From the observation above, we can see that if $\pi ^{t} \leq _{r} \pi ^{\ast ,t}_{1}$, then the optimal action is a₁, and if $\pi ^{\ast ,t}_{2} \leq _{r} \pi ^{t}$, then the optimal action is a₂.

Let $\pi ^{\ast ,t}_{1} \leq _{r} \pi ^{t} \leq _{r} \pi ^{\ast ,t}_{2}$. By repeating the previous procedure we can see that if $\pi ^{\ast ,t}_{1} \leq _{r} \pi ^{t}$, then the optimal action is a₂, and if $\pi ^{t} \leq _{r} \pi ^{\ast ,t}_{2}$, then the optimal action is a₁. This is a contradiction since the MLR decreasing assumption is violated and you can not take two actions at the same time. This means that if the optimal action is a₁ for $\pi ^{\ast ,t}_{1} \leq _{r} \pi ^{t} \leq _{r} \pi ^{\ast ,t}_{2}$, then $\pi ^{\ast ,t}_{2}$ is the single threshold. Otherwise, if the optimal action is a₂, then $\pi ^{\ast ,t}_{1}$ is the single threshold.

Following the same rationale, we see that there cannot be n thresholds for n > 1: $\pi ^{\ast ,t}_{1} \leq _{r} \pi ^{\ast ,t}_{2} \leq _{r} ... \leq _{r} \pi ^{\ast ,t}_{n}$.

A.6 Proof of Theorem 2

We have monotonic value functions and unique threshold policies for both individuals, so using Theorem 1, we can rewrite our threshold solution: if ${v^{t}_{i}}({\pi ^{t}_{i}},a^{t} = a_{1}) \geq {v^{t}_{i}}(\pi ^{\ast ,t}_{i},a^{t} = a_{1})$ then a₁, else a₂. Note that by definition we have ${v^{t}_{i}}(\pi ^{\ast ,t}_{i},a^{t} = a_{1}) = {v^{t}_{i}}(\pi ^{\ast ,t}_{i},a^{t} = a_{2})$. Thus, ${v^{t}_{1}}({\pi ^{t}_{1}}, a^{t} = a_{1}) - {v^{t}_{1}}(\pi ^{\ast ,t}_{1}, a^{t} = a_{2}) \geq {v^{t}_{2}}({\pi ^{t}_{2}}, a^{t} = a_{1}) - {v^{t}_{2}}(\pi ^{\ast ,t}_{2}, a^{t} = a_{2})$ ensures that when the optimal action for individual 1 is a₂ (${v^{t}_{1}}({\pi ^{t}_{1}},a^{t} = a_{1}) \leq {v^{t}_{1}}(\pi ^{\ast ,t}_{1},a^{t} = a_{2})$), individual 2’s optimal action must also be a₂ (${v^{t}_{2}}({\pi ^{t}_{2}},a^{t} = a_{1}) \leq {v^{t}_{2}}(\pi ^{\ast ,t}_{2},a^{t} = a_{2})$) but not vice versa. Similarly, when individual 2’s optimal action is a₁ (${v^{t}_{2}}({\pi ^{t}_{2}},a^{t} = a_{1}) \geq {v^{t}_{2}}(\pi ^{\ast ,t}_{2},a^{t} = a_{2})$), individual 1’s optimal must also be a₁ (${v^{t}_{1}}({\pi ^{t}_{1}},a^{t} = a_{1}) \geq {v^{t}_{1}}(\pi ^{\ast ,t}_{1},a^{t} = a_{2})$) but not vice versa. We also know $\pi ^{t=0}_{1} \leq _{r} \pi ^{t=0}_{2}$ and ${\pi ^{t}_{1}} \leq _{r} {\pi ^{t}_{2}}$ then ${\pi ^{t}_{1}} P^{t,BMI_{1}}_{1} \leq _{r} {\pi ^{t}_{2}} P^{t,BMI_{2}}_{2}$, which ensures that the belief ordering over time will be preserved given BMI₁ ≤ BMI₂. Therefore, in expectation, individual 2 will take action a₂ earlier in time and more frequently across decision epochs.

A.7 Proof of Corollary 1

$\pi ^{\ast ,t}_{2} \leq _{r} \pi ^{\ast ,t}_{1}$, and ${v^{t}_{1}}(\pi ^{\ast ,t}_{1}) \leq {v^{t}_{2}}(\pi ^{\ast ,t}_{2})$ for all t ensure that ${v^{t}_{2}}(\pi ^{\ast ,t}_{2}, a^{t} = a_{2}) \geq {v^{t}_{1}}(\pi ^{\ast ,t}_{1}, a^{t} = a_{2})$ for all t. We also have ${v^{t}_{1}}({\pi ^{t}_{1}}, a^{t} = a_{1}) \geq {v^{t}_{2}}({\pi ^{t}_{2}}, a^{t} = a_{1})$ for all t by Theorem 1. Therefore, ${v^{t}_{1}}({\pi ^{t}_{1}}, a^{t} = a_{1}) - {v^{t}_{1}}(\pi ^{\ast ,t}_{1}, a^{t} = a_{2}) \geq {v^{t}_{2}}({\pi ^{t}_{2}}, a^{t} = a_{1}) - {v^{t}_{2}}(\pi ^{\ast ,t}_{2}, a^{t} = a_{2})$, which is condition 2 of Theorem 2. The results of Theorem 2 therefore apply.

B Estimating transition probabilities using multinomial logistic regressions

Since HRS Biomarker waves are every four years, the merged HRS and HRS Biomarker data from 2006-2014 allows us to estimate the 4-year transition probability matrix by fitting multinomial logistic regressions. We use the current health state, age, race, gender, and BMI to estimate the likelihood an individual is in each health state (healthy, prediabetic, diabetic, or dead) every four years. We define categories for BMI = {1, 2, 3} where 1 is for BMI < 25, 2 for 25 ≤ BMI < 30, and 3 for BMI ≥ 30 following the CDC guidelines and the definition in HRS. Each BMI category therefore uses a separate set of transition matrices. To estimate the transition probabilities (staying diabetic in 6 months or being dead within 6 months) given a patient is on treatment, we use the relative risk (treated to untreated) of diabetic mortality found in the medical literature to estimate mortality for undiagnosed diabetic patients [60], assuming all patients with diagnosed diabetes in the HRS are on treatment. After modifying the death probabilities, we renormalize the outcomes to ensure the result produces a stochastic matrix. We then use the eigendecomposition numerical approximation methods proposed by Chhatwal et al. (2016) to estimate half-year transition matrices from the 4-year data by raising them to the 1/8 power [64]. This allows us to generate different transition probability matrices for each age, each BMI category, and each diabetes treatment status (treated or untreated). See Tables 8 and 9 for examples. The transition probabilities for untreated individuals are plotted in Appendix Figs. 7, 8 and 9.

Table 8 6-month transition matrix for 65-year-old white male with BMI ≥ 30 and without treatment

Table 9 6-month transition matrix for 65-year-old white male with BMI ≥ 30 and on treatment

Fig. 7

The diagonal elements of P^t,BMI are greater than 0.5 – this property is used in Lemma 2 to show that P^t,BMI is TP2

Fig. 8

P(4∣3) ≥ P(4∣2) ≥ P(4∣1) for all age groups t given BMI

C Numerical diabetes example: the IFR property

The transition probabilities change with age. To verify that the transition matrices used for our diabetes numerical example have the IFR property, we confirm that, for the transition matrix associated with each age:

• The diagonal elements from the transition matrices are greater than 0.5 (see Fig. 7)

• $p^{t,BMI}_{34} \geq p^{t,BMI}_{24} \geq p^{t,BMI}_{14}$ or P(4∣3) ≥ P(4∣2) ≥ P(4∣1) (see Fig. 8)

• $p^{t,BMI}_{23} \geq p^{t,BMI}_{13}$ or P(3∣2) ≥ P(3∣1) (see Fig. 9)

This is sufficient to guarantee that each transition matrix has IFR property. The following graphs show the transition probabilities for white males with BMI ≥ 30.

Fig. 9

P(3∣2) ≥ P(3∣1) for all age groups t given BMI

D Numerical verification of a threshold solution

We now evaluate our problem for the existence of a threshold policy. Our theoretical results show that we will have a threshold policy for any problem with two actions, a monotonic value function, and that the difference in v^t is MLR decreasing in π^t with two actions. There are only two actions in our problem, so we turn first to establishing that the value function is monotonic.

We use Proposition 1, which states that if r^t(π^t,a^t) is linear and decreasing in π^t for each action a^t, and the product of P^t,BMI and O^a is TP2, then the value function is monotonic. In this problem, r^t(π^t,a^t) is the NMB associated with the patient’s health state, which is a linear function of QALYs and costs: λ QALY(π^t,a^t) - Cost(π^t,a^t). Note that the linear combination of linear functions is still linear (see notation section for definition of NMB given π^t). Our QALY values are decreasing in health states and our costs are increasing in health states, so we know that r^t(π^t,a^t) is linear and decreasing.

The second condition in Proposition 1 requires the product of the transition matrix and the observation matrix to be TP2 for a fixed action a^t. Our observation matrix, O^a, is fully captured by a three by three matrix since the death state is observable. To show the product is TP2, it suffices to show the transition matrix for the partially observable states to be TP2. Then, we directly verify these matrices are TP2 numerically using the conditions in Lemma 1 and Lemma 2. Our problem therefore has a monotonic value function.

To show that the policy has a unique threshold solution, we additionally need the difference in v^t to be MLR decreasing in π^t with two actions for all t. We verify this result numerically by sampling from the belief simplex and calculating the value function for both actions at each time period given that sampled belief (similar to plotting Fig. 4 for each decision epoch).

We then repeat the above process for all three BMI groups. This establishes that our problem for BMI-stratified diabetes screening in the US has a threshold solution, ensuring that our optimal policy is more easily interpretable and communicable to policy-makers. Additionally, a grid approximation method is appropriate for this problem [23].