A systematic approach for examining the impact of calibration uncertainty in disease modeling

Abstract

In model-based analysis for comparative evaluation of strategies for disease treatment and management, the model of the disease is arguably the most critical element. A fundamental challenge in identifying model parameters arises from the limitations of available data, which challenges the ability to uniquely link model parameters to calibration targets. Consequently, the calibration of disease models leads to the discovery of multiple models that are similarly consistent with available data. This phenomenon is known as calibration uncertainty and its effect is transferred to the results of the analysis. Insufficient examination of the breadth of potential model parameters can create a false sense of confidence in the model recommendation, and ultimately cast doubt on the value of the analysis. This paper introduces a systematic approach to the examination of calibration uncertainty and its impact. We begin with a model of the calibration process as a constrained optimization problem and introduce the notion of plausible models which define the uncertainty region for model parameters. We illustrate the approach using a fictitious disease, and explore various methods for interpreting the outputs obtained.

Appendix: The calibration problem

With the assumption that $\pi _{\text {H}}(0) = 1$ (i.e., $\pi (0) = [1\ 0\ 0\ \ldots 0]$), we obtain the calibration targets $\hat{\tau } = \{ \hat{\pi }_j(a): j \in \{1D, 2D, 3D, DD, DO\}, a \in \mathcal{{A}}\}\, \cup \, \{ \hat{\alpha }_j(a): j \in \{1D, 2D, 3D\}, a \in \mathcal{{A}}\}$ where $\{ \hat{\pi }_j(a)\}$ and $\{\hat{\alpha }_j(a)\}$ are calculated via (1) and (2) using $\mathbb {P}^{\text {true}}$ and $\pi (0)$, where

Note that from (1) and (2), the modelled outcomes can be represented as:

• $\pi _j(a) = [\pi (0) \mathbb {P}^a]_j, j \in \mathcal{{S}}$

• $\alpha _j(a) = \sum \limits _{i \in \{H,1U,2U,3U\}} [\pi (0) \mathbb {P}^{a-1}]_i P_{i,j} , j \in \{1D, 2D, 3D\}$

where $[\cdot ]_j$ represents the jth element.

By representing $\mathbb {P}$ as the decision variable, the Calibration Problem for our fictitious disease may be stated as:

$$\begin{aligned}&\text {optimize} \sum \limits _{j \in \{1D, 2D, 3D, DD, DO\}} \sum \limits _{a \in \mathcal{{A}}} \left( [\pi (0) \mathbb {P}^a]]_j - \hat{\pi }_j(a) \right) ^2\\&\quad + \sum \limits _{j \in \{1D, 2D, 3D\}} \sum \limits _{a \in \mathcal{{A}}} \left( \sum \limits _{i \in \{H,1U,2U,3U\}} [\pi (0) \mathbb {P}^{a-1}]_i P_{i,j} - \hat{\alpha }_j(a) \right) ^2\\&\text {subject to } \end{aligned}$$

$$\begin{aligned}&\sum \limits _{j \in \mathcal{{S}}} P_{i,j} = 1 \quad \forall i \in \mathcal{{S}}\end{aligned}$$

(10)

$$\begin{aligned}&0 \le P_{i,j} \le 1 \quad \forall i,j \in \mathcal{{S}}\end{aligned}$$

(11)

$$\begin{aligned}&P_{iU,j} = P_{iD,j} \quad \forall i \in \{1, 2, 3 \}, j \in \{ DO, DD\} \end{aligned}$$

(12)

$$\begin{aligned}&P_{H,H} \ge P_{H,1U} \ge P_{H,2U} \ge P_{H,3U} \end{aligned}$$

(13)

$$\begin{aligned}&P_{H,H} \ge P_{H,1D} \ge P_{H,2D} \ge P_{H,3D} \end{aligned}$$

(14)

$$\begin{aligned}&P_{1U,1U} \ge P_{1U,2U} \ge P_{1U,3U} \end{aligned}$$

(15)

$$\begin{aligned}&P_{1U,1D} \ge P_{1U,2D} \ge P_{1U,3D} \end{aligned}$$

(16)

$$\begin{aligned}&P_{2U,2U} \ge P_{2U,3U} \end{aligned}$$

(17)

$$\begin{aligned}&P_{2U,2D} \ge P_{2U,3D} \end{aligned}$$

(18)

$$\begin{aligned}&P_{1D,1D} \ge P_{1D,2D} \ge P_{1D,3D} \end{aligned}$$

(19)

$$\begin{aligned}&P_{2D,2D} \ge P_{2D,3D} \end{aligned}$$

(20)

$$\begin{aligned}&P_{i,jU} \ge P_{i,jD} \quad \forall i \in \{H,1U,2U,3U\}, j \in \{1, 2, 3\} \end{aligned}$$

(21)

$$\begin{aligned}&P_{H,1U} + P_{H,1D} \ge P_{H,2U} + P_{H,2D} \ge P_{H,3U} + P_{H,3D} \end{aligned}$$

(22)

$$\begin{aligned}&P_{1U,1U} + P_{1U,1D} \ge P_{1U,2U} + P_{1U,2D} \ge P_{1U,3U} + P_{1U,3D} \end{aligned}$$

(23)

$$\begin{aligned}&P_{2U,2U} + P_{2U,2D} \ge P_{2U,3U} + P_{2U,3D} \end{aligned}$$

(24)

$$\begin{aligned}&P_{i,H} = 0 \quad \forall i \notin \{H\} \end{aligned}$$

(25)

$$\begin{aligned}&P_{i,1U} = 0 \quad \forall i \notin \{H, 1U\} \end{aligned}$$

(26)

$$\begin{aligned}&P_{i,2U} = 0 \quad \forall i \notin \{H, 1U, 2U\} \end{aligned}$$

(27)

$$\begin{aligned}&P_{i,3U} = 0 \quad \forall i \notin \{H,1U,2U,3U\} \end{aligned}$$

(28)

$$\begin{aligned}&P_{i,1D} = 0 \quad \forall i \notin \{H, 1U, 1D\} \end{aligned}$$

(29)

$$\begin{aligned}&P_{i,2D} = 0 \quad \forall i \notin \{H, 1U, 1D, 2U, 2D\} \end{aligned}$$

(30)

$$\begin{aligned}&P_{DO,DO} = 1 \end{aligned}$$

(31)

$$\begin{aligned}&P_{DD,DD} = 1 \end{aligned}$$

(32)

$$\begin{aligned}&P_{i,j} \ge 1 \times 10^{-7} \quad \forall i \notin \{DO, DD\}, j \in \{DO, DD\} \end{aligned}$$

(33)

$$\begin{aligned}&0.75 \le P_{1D,1D} \le 0.85 \end{aligned}$$

(34)

$$\begin{aligned}&0.1 \le P_{1D,2D} \le 0.15 \end{aligned}$$

(35)

$$\begin{aligned}&0.06 \le P_{1D,3D} \le 0.09 \end{aligned}$$

(36)

$$\begin{aligned}&0.0015 \le P_{1D,DO} \le 0.003 \end{aligned}$$

(37)

$$\begin{aligned}&0.0002 \le P_{1D,DD} \le 0.0025 \end{aligned}$$

(38)

$$\begin{aligned}&0.68 \le P_{2D,2D} \le 0.72 \end{aligned}$$

(39)

$$\begin{aligned}&0.28 \le P_{2D,3D} \le 0.31 \end{aligned}$$

(40)

$$\begin{aligned}&0.0000001 \le P_{2D,DO} \le 0.0015 \end{aligned}$$

(41)

$$\begin{aligned}&0.0000001 \le P_{2D,DD} \le 0.003 \end{aligned}$$

(42)

$$\begin{aligned}&0.95 \le P_{3D,3D} \le 1 \end{aligned}$$

(43)

$$\begin{aligned}&0.0005 \le P_{3D,DO} \le 0.0015 \end{aligned}$$

(44)

$$\begin{aligned}&0.0018 \le P_{3D,DD} \le 0.0024 \end{aligned}$$

(45)

Equation (12) states that mortality is independent of awareness of the disease. Equations (13)–(24) indicate that the disease is more likely to remain in the current state than progress to a worse health state or to become detected. Equation (33) indicates that there is a non-zero probability of dying from any state. Equations (34)–(45) represent the biologically plausible ranges for observable transition probabilities. These constraints represent the expert judgment that would likely result from observations of the diseased patients.