Evaluating Incremental Values from New Predictors with Net Reclassification Improvement in Survival Analysis

Abstract

Developing individualized prediction rules for disease risk and prognosis has played a key role in modern medicine. When new genomic or biological markers become available to assist in risk prediction, it is essential to assess the improvement in clinical usefulness of the new markers over existing routine variables. Net Reclassification Improvement (NRI) has been proposed to assess improvement in risk reclassification in the context of comparing two risk models and the concept has been quickly adopted in medical journals (Pencina et al., Stat Med 27:157–172, 2008). We propose both nonparametric and semiparametric procedures for calculating NRI as a function of a future prediction time t with a censored failure time outcome. The proposed methods accommodate covariate-dependent censoring, therefore providing more robust and sometimes more efficient procedures compared with the existing nonparametric-based estimators (Pencina et al., Stat Med 30: 11–21, 2011; Uno et al., Stat Med 32:2430–42, 2013). Simulation results indicate that the proposed procedures perform well in finite samples. We illustrate these procedures by evaluating a new risk model for predicting the onset of cardiovascular disease.

Appendix

Throughout, we assume that the joint density of (T, C, Y) is twice continuously differentiable, Y are bounded, and 1 > P(T > t) > 0, 1 > P(C > t) > 0. The kernel function K is a symmetric probability density function with compact support and bounded second derivative. The bandwidth h → 0 such that nh ⁴ → 0. In addition, the estimator $\hat{\boldsymbol{\theta }}_{k}$ converges to $\boldsymbol{\theta }_{0k}$ for k = 1, 2 as n → ∞ [13], where $\boldsymbol{\beta }_{k0}$ is the unique maximizer of the expected value of the corresponding partial likelihood and Λ _k0 is the baseline cumulative hazard for k = 1, 2. We denote the parameter space for $\boldsymbol{\theta }_{k}$ by Ω _k and assume that Ω _k is a compact set containing $\boldsymbol{\theta }_{0k}$. Furthermore, we assume that $\boldsymbol{\beta }_{2}\neq 0$ and note that $Q(\boldsymbol{\theta }_{2}) = 1 -\exp \{\varLambda _{02}(t){e}^{\boldsymbol{\beta }_{2}^{\mathsf{T}}\mathbf{Y}_{ (2)}}\}$ and $P(\boldsymbol{\theta }_{1}) = 1 -\exp \{\varLambda _{01}(t){e}^{\boldsymbol{\beta }_{1}^{\mathsf{T}}\mathbf{Y}_{ (1)}}\}$ are the respective limits of $Q(\hat{\boldsymbol{\theta }}_{2})$ and $P(\hat{\boldsymbol{\theta }}_{1})$, for any given Y ₍₂₎ and Y ₍₁₎. The in-probability convergence of $Q(\hat{\boldsymbol{\theta }}_{2}) \rightarrow Q(\boldsymbol{\theta }_{02})$ and $P(\hat{\boldsymbol{\theta }}_{1})$ and $P(\boldsymbol{\theta }_{01})$ are uniform in Y ₍₂₎ and Y ₍₁₎ due to the convergence of $\hat{\boldsymbol{\theta }} \rightarrow \boldsymbol{\theta }_{0} = {(\boldsymbol{\theta }_{01}^{\mathsf{T}},\boldsymbol{\theta }_{02}^{\mathsf{T}})}^{\mathsf{T}}$.

Asymptotic Properties of $\widetilde{\mathrm{NRI}}(\hat{\boldsymbol{\theta }},t)$

From the same arguments as given in [3] and [7], it follows that we have the uniform consistency of $\tilde{H}_{q}^{(\iota )}(t)$ to $H_{q}^{(\iota )}(t) = P\{C \geq t\mid Q(\boldsymbol{\theta }_{2}) = q,\varDelta (\boldsymbol{\theta }) \in \mho _{\iota }\}$, where ℧ ₁ = 1 and ℧ _• = { 0, 1}, for ι = 1 and •. It follows, using the uniform law of large numbers [20], that

$$\displaystyle{\sup _{\boldsymbol{\theta }}\vert \widetilde{\mbox{ NRI}}(\boldsymbol{\theta },t) -\mbox{ NRI}(\boldsymbol{\theta },t)\vert \rightarrow 0.}$$

This along with the convergence of $\hat{\boldsymbol{\theta }}$ to $\boldsymbol{\theta }_{0}$ implies that $\widetilde{\mbox{ NRI}}(\hat{\boldsymbol{\theta }},t)$ is uniformly consistent for $\mbox{ NRI}(\boldsymbol{\theta }_{0},t)$.

Throughout, we will use the fact that $E\{\varDelta _{i}(\boldsymbol{\theta })I(X_{i} \leq t)\delta _{i}H_{Q_{i}(\boldsymbol{\theta }_{2})}^{(1)}{(X_{i})}^{-1}\mid Q_{i}(\boldsymbol{\theta }_{2}) = q\} = P(\varDelta _{i}(\boldsymbol{\theta }) = 1,T_{i} \leq t\mid Q_{i}(\boldsymbol{\theta }_{2}) = q)$ if either C ⊥ T, Y ₍₂₎ (model may be misspecified) or $Q(\boldsymbol{\theta }_{2}) = \mbox{ Pr}(T \leq t\vert Y _{(2)})$ i.e. the Cox model is correctly specified though censoring may be such that C ⊥ T∣Y ₍₂₎ (double robustness). We first write the i.i.d representation of $\sqrt{n}[\widetilde{\mbox{ NRI}}(\boldsymbol{\theta },t) -\mbox{ NRI}(\boldsymbol{\theta },t)]$ for any $\boldsymbol{\theta }$. Note that $\sqrt{n}\{\widetilde{\mbox{ NRI}}(\boldsymbol{\theta },t)-\mbox{ NRI}(\boldsymbol{\theta },t)\} = 2\sqrt{n}\{\widetilde{\mbox{ Pr}}(\varDelta (\boldsymbol{\theta }) = 1\vert T \leq t)-\mbox{ Pr}(\varDelta (\boldsymbol{\theta }) = 1\vert T \leq t)\}-2\sqrt{n}\{\widetilde{\mbox{ Pr}}(\varDelta (\boldsymbol{\theta }) = 1\vert T > t)-\mbox{ Pr}(\varDelta (\boldsymbol{\theta }) = 1\vert T > t)\}$. We first examine the initial component,

$$\displaystyle\begin{array}{rcl} \widetilde{\mbox{ Pr}}(\varDelta (\hat{\boldsymbol{\theta }}) = 1\vert T \leq t)& =& \frac{\sum _{i}\varDelta (\hat{\boldsymbol{\theta }})I(X_{i} \leq t)\delta _{i}/\tilde{H}_{Q_{i}(\hat{\boldsymbol{\theta }}_{2})}^{(1)}(X_{i})} {\sum _{i}I(X_{i} \leq t)\delta _{i}/\tilde{H}_{Q_{i}(\hat{\boldsymbol{\theta }}_{2})}^{(\bullet )}(X_{i})} \equiv \frac{\hat{N}(t,\hat{\boldsymbol{\theta }},\tilde{H})} {\hat{D}(t,\hat{\boldsymbol{\theta }},\tilde{H})} {}\\ \end{array}$$

where $\hat{N}(t,\boldsymbol{\theta },H) = {n}^{-1}\sum _{i}\varDelta _{i}(\boldsymbol{\theta })I(X_{i} \leq t)\delta _{i}/H_{Q_{i}(\boldsymbol{\theta }_{2})}^{(1)}(X_{i})$ and $\hat{D}(t,\boldsymbol{\theta },H) = {n}^{-1}\sum _{i}I(X_{i} \leq t)\delta _{i}/H_{Q_{i}(\boldsymbol{\theta }_{2})}^{(\bullet )}(X_{i})$. Let $N(t,\boldsymbol{\theta }) = \mbox{ Pr}(\varDelta (\boldsymbol{\theta }) = 1,T \leq t)$ and D(t) = Pr(T ≤ t). Then by the uniform consistency of the IPW weights, we have

$$\displaystyle{\sqrt{ n}\{\widetilde{\mbox{ Pr}}(\varDelta (\boldsymbol{\theta }) = 1\vert T \leq t)-\mbox{ Pr}(\varDelta (\boldsymbol{\theta }) = 1\vert T \leq t)\} \approx \sqrt{n}\{\hat{N}(t,\boldsymbol{\theta },\tilde{H})D(t)-N(t,\boldsymbol{\theta })\hat{D}(t,\boldsymbol{\theta },\tilde{H})\}/D{(t)}^{2}.}$$

Examining the numerator, $\sqrt{n}\{\hat{N}(t,\boldsymbol{\theta },\tilde{H})D(t) - N(t,\boldsymbol{\theta })\hat{D}(t,\boldsymbol{\theta },\tilde{H})\} = \sqrt{n}\{(1) + (2) - (3)\}$ where $(1) =\hat{ N}(t,\boldsymbol{\theta },H)D(t) -\hat{ D}(t,\boldsymbol{\theta },H)N(t,\boldsymbol{\theta }),\quad (2) =\hat{ N}(t,\boldsymbol{\theta },\tilde{H})D(t) -\hat{ N}(t,\boldsymbol{\theta },H)D(t),$ and $(3) = [N(t,\boldsymbol{\theta })\hat{D}(t,\boldsymbol{\theta },\tilde{H}) -\hat{ D}(t,\boldsymbol{\theta },H)N(t,\boldsymbol{\theta })].$ Note that

$$\displaystyle\begin{array}{rcl} (1)& =& \sqrt{n}(\hat{N}(t,\boldsymbol{\theta },H)D(t) -\hat{ D}(t,\boldsymbol{\theta },H)N(t,\boldsymbol{\theta })) = {n}^{-\frac{1} {2} }\sum U_{1i}(t),\quad \mbox{ where} {}\\ U_{1i}(t)& =& \frac{I(X_{i} \leq t)\delta _{i}} {H_{Q_{i}(\boldsymbol{\theta }_{2})}^{(1)}(X_{i})}\varDelta _{i}(\boldsymbol{\theta })D(t) - \frac{I(X_{i} \leq t)\delta _{i}} {H_{Q_{i}(\boldsymbol{\theta }_{2})}^{(\bullet )}(X_{i})}N(t,\boldsymbol{\theta }) {}\\ \end{array}$$

Using a Taylor series expansion, Lemma A.3 of [2] and the asymptotic expansion for $\hat{\varLambda }_{q}(t)$ given in [8],

$$\displaystyle\begin{array}{rcl} (2)& =& D(t)\sqrt{n}\{\hat{N}(t,\boldsymbol{\theta },\tilde{H}) -\hat{ N}(t,\boldsymbol{\theta },H)\} {}\\ & =& D(t){n}^{-1/2}\sum _{ i} \frac{\varDelta _{i}(\boldsymbol{\theta })I(X_{i} \leq t)\delta _{i}} {H_{Q_{i}(\boldsymbol{\theta }_{2})}^{(1)}(X_{i})}\left [\frac{H_{Q_{i}(\boldsymbol{\theta }_{2})}^{(1)}(X_{i})} {\tilde{H}_{Q_{i}(\boldsymbol{\theta }_{2})}^{(1)}(X_{i})} - 1\right ] {}\\ & =& D(t){n}^{-1/2}\int \int _{ 0}^{t}\left [\frac{H_{q}^{(1)}(s)} {\tilde{H}_{q}^{(1)}(s)} - 1\right ]d\sum _{i}\frac{\varDelta _{i}(\boldsymbol{\theta })\delta _{i}I(X_{i} \leq s,Q_{i}(\boldsymbol{\theta }_{2}) \leq q)} {H_{Q_{i}(\boldsymbol{\theta }_{2})}^{(1)}(X_{i})} {}\\ & \approx & D(t)\int \int _{0}^{t}\sqrt{n}\left [\hat{\varLambda }_{ q}^{(1)}(s) -\varLambda _{ q}^{(1)}(s)\right ]d\left \{\frac{1} {n}\sum _{i}\frac{\varDelta _{i}(\boldsymbol{\theta })\delta _{i}I(X_{i} \leq s,Q_{i}(\boldsymbol{\theta }_{2}) \leq q)} {H_{Q_{i}(\boldsymbol{\theta }_{2}),}^{(1)}(X_{i})} \right \} {}\\ & \approx & D(t)\int \int _{0}^{t}\left [{n}^{-\frac{1} {2} }\sum K_{h}\left \{q-Q_{i}(\boldsymbol{\theta }_{2})\right \}M_{Cq}^{(1)}(s,X_{i},\delta _{i})\right ]dP(\varDelta (\boldsymbol{\theta })=1,T \leq t,Q(\boldsymbol{\theta }_{2}) \leq q) {}\\ \end{array}$$

where

$$\displaystyle{M_{Cq}^{(1)}(t,X_{ i},\delta _{i}) =\int _{ 0}^{t}\frac{dN_{Ci}(s) - I(X_{i} \geq s)d\varLambda _{q}^{(1)}(s)} {\pi _{s}^{(1)}(q)}.}$$

Now by a change of variable, $\psi = \frac{q-Q_{i}(\boldsymbol{\theta }_{2})} {h}$ and $f(t,q) \equiv {\partial }^{2}P(\varDelta (\boldsymbol{\theta }) = 1,$ $T \leq t,Q(\boldsymbol{\theta }_{2}) \leq q)/\partial t\partial q$,

$$\displaystyle\begin{array}{rcl} (2)& \approx & D(t)\int \int _{0}^{t}\sqrt{n}\left [ \frac{1} {n}\sum K\left (\psi \right )M_{C(\psi h+Q_{i}(\boldsymbol{\theta }_{2}))}(s,X_{i},\delta _{i})\right ]f(t,\psi h + Q_{i})dsd\psi {}\\ & =& D(t){n}^{-1/2}\sum \int \int _{ 0}^{t}K\left (\psi \right )a\{s,h\psi + Q_{ i}(\boldsymbol{\theta }_{2}),X_{i}\}dsd\psi = {n}^{-\frac{1} {2} }\sum U_{2i}(t), {}\\ \end{array}$$

where $U_{2i}(t) = D(t)\int _{0}^{t}a(s,{q}^{{\ast}},X_{i})ds$ and $a(t,q,X_{i}) = M_{C{q}^{{\ast}}}(t,X_{i},\delta _{i})f(t,{q}^{{\ast}})$. Similar arguments can be used to obtain an asymptotic expansion for (3) as $(3) \approx {n}^{-\frac{1} {2} }\sum U_{3i}(t)$ and therefore, the numerator, $\sqrt{n}\left [\hat{N}(t,\boldsymbol{\theta },\tilde{H})D(t) - N(t,\boldsymbol{\theta })\hat{D}(t,\boldsymbol{\theta },\tilde{H})\right ] \approx {n}^{-\frac{1} {2} }\sum \{U_{1i}(t) + U_{2i}(t) + U_{3i}(t)\}.$ The same arguments as given above can be used to obtain an asymptotic expansion for $\sqrt{n}\{\widetilde{\mbox{ Pr}}(\varDelta (\boldsymbol{\theta }) = 1\vert T > t) -\mbox{ Pr}(\varDelta (\boldsymbol{\theta }) = 1\vert T > t)\}$ as ${n}^{-\frac{1} {2} }\sum _{i=1}^{n}D(t)_{-}^{-2}\{U_{-1i}(t) + U_{-2i}(t) + U_{-3i}(t)\}$ where D(t)₋, $U_{-1i}(t),U_{-2i}(t),$ and U _−3i (t) are defined similarly to D(t), $U_{1i}(t),U_{2i}(t),$ and U _3i (t) with T ≤ t replaced with T > t. Therefore, $\sqrt{n}\{\widetilde{\mbox{ NRI}}(\boldsymbol{\theta },t)-\mbox{ NRI}(\boldsymbol{\theta },t)\} \approx {n}^{-\frac{1} {2} }\sum _{i=1}^{n}2[D{(t)}^{-2}\{U_{1i}(t)+U_{2i}(t)+U_{3i}(t)\}-D(t)_{-}^{-2}\{U_{-1i}(t)+U_{-2i}(t)+U_{-3i}(t)\}] = {n}^{-\frac{1} {2} }\sum _{i=1}^{n}\eta _{i}(t)$.

Note that regardless of correct model specification, $\sqrt{n}(\hat{\boldsymbol{\theta }} -\boldsymbol{\theta }_{ 0}) = {n}^{-1/2}\sum \psi _{ i} + o_{p}(1)$ where ψ _i are i.i.d mean zero random variables by Lin and Wei [16] and Uno et al. [24]. Using a Taylor series approximation and the i.i.d representation of $\sqrt{n}[\widetilde{\mbox{ NRI}}(\boldsymbol{\theta },t) -\mbox{ NRI}(\boldsymbol{\theta },t)]$ for any $\boldsymbol{\theta }$, we can write $\widetilde{\mathcal{W}}(t) = \sqrt{n}[\widetilde{\mbox{ NRI}}(\hat{\boldsymbol{\theta }},t) -\mbox{ NRI}(\boldsymbol{\theta }_{0},t)]$ as a sum of i.i.d terms, ${n}^{-1/2}\sum _{i=1}^{n}\epsilon _{i}(t)$ defined below.

$$\displaystyle\begin{array}{rcl} & & \sqrt{n}[\widetilde{\mbox{ NRI}}(\hat{\boldsymbol{\theta }},t) -\mbox{ NRI}(\boldsymbol{\theta }_{0},t)] {}\\ & & \quad = \sqrt{n}[\widetilde{\mbox{ NRI}}(\hat{\boldsymbol{\theta }},t) -\mbox{ NRI}(\hat{\boldsymbol{\theta }},t) + \mbox{ NRI}(\hat{\boldsymbol{\theta }},t) -\mbox{ NRI}(\boldsymbol{\theta }_{0},t)] {}\\ & & \quad \approx \sqrt{n}[\widetilde{\mbox{ NRI}}(\hat{\boldsymbol{\theta }},t) -\mbox{ NRI}(\hat{\boldsymbol{\theta }},t) + \frac{\partial \mbox{ NRI}(t)} {\partial \boldsymbol{\theta }} \vert _{\boldsymbol{\theta }_{0}}(\hat{\boldsymbol{\theta }} -\boldsymbol{\theta }_{0}) {}\\ & & \quad = \sqrt{n}[\widetilde{\mbox{ NRI}}(\hat{\boldsymbol{\theta }},t) -\mbox{ NRI}(\hat{\boldsymbol{\theta }},t)] + \sqrt{n}(\hat{\boldsymbol{\theta }} -\boldsymbol{\theta }_{0})\frac{\partial \mbox{ NRI}(t)} {\partial \boldsymbol{\theta }} \vert _{\boldsymbol{\theta }_{0}} {}\\ & & \quad \approx \sqrt{n}[\widetilde{\mbox{ NRI}}(\hat{\boldsymbol{\theta }},t)-\mbox{ NRI}(\hat{\boldsymbol{\theta }},t)]+{n}^{-1/2}\sum \psi _{ i}\frac{\partial \mbox{ NRI}(t)} {\partial \boldsymbol{\theta }} \vert _{\boldsymbol{\theta }_{0}} {}\\ & & \quad \approx {n}^{-1/2}\sum _{ i=1}^{n}\eta _{ i}(t) + {n}^{-1/2}\sum \psi _{ i}\frac{\partial \mbox{ NRI}(t)} {\partial \boldsymbol{\theta }} \vert _{\boldsymbol{\theta }_{0}} {}\\ & & \quad = {n}^{-1/2}\sum _{ i=1}^{n}\epsilon _{ i}(t) {}\\ \end{array}$$

where $\epsilon _{i}(u,v,t) =\eta _{i}(u,v,t) +\psi _{i}\frac{\partial \mbox{ NRI}(t)} {\partial \boldsymbol{\theta }} \vert _{\boldsymbol{\theta }_{0}}$. By a functional central limit theorem of [20], the process $\widetilde{\mathcal{W}}(t)$ converges weakly to a mean zero Gaussian process in t.

Asymptotic Properties of $\widehat{\mathrm{NRI}}(\hat{\boldsymbol{\theta }},t)$

Recall that we assume the Cox model is correctly specified and thus, $Q(\boldsymbol{\theta }_{2})$ $= Q(\boldsymbol{\theta }_{2},t,\mathbf{Y}_{(2)}) = \mbox{ Pr}(T \leq t\vert Y _{(2)}) = 1 -\exp \{\varLambda _{02}(t){e}^{\boldsymbol{\beta }_{2}^{\mathsf{T}}Y _{ (2)}}\}$ and $S_{Q_{i}(\boldsymbol{\theta }_{2})}(t) = \mbox{ Pr}(T > t\vert Y _{(2)}) =\exp \{\varLambda _{02}(t){e}^{\boldsymbol{\beta }_{2}Y _{(2)}}\}$. To derive asymptotic properties of $\widehat{\mbox{ NRI}}(\hat{\boldsymbol{\theta }},t)$ we assume the same regularity conditions as in [1]. The uniform consistency of $Q(\hat{\boldsymbol{\theta }}_{2},t,\mathbf{Y}_{(2)})$ for $Q(\boldsymbol{\theta }_{2},t,\mathbf{Y}_{(2)})$ in t and Y ₍₂₎ follows directly from the uniform consistency of $\hat{\varLambda }_{02}(t)$ and $\hat{\boldsymbol{\beta }}_{2}$. It follows from the uniform law of large numbers [20] that $\widehat{\mbox{ NRI}}(\hat{\boldsymbol{\theta }},t)$ is uniformly consistent for $\mbox{ NRI}(\boldsymbol{\theta }_{0},t)$. Andersen and Gill [1] show that $\sqrt{n}(\hat{\beta }_{2} -\beta _{02})$ is a normal random variable and $\sqrt{n}(\hat{\varLambda }_{02}(t) -\varLambda _{02}(t))$ converges to a Gaussian process. By the functional delta method it can be shown that $\sqrt{n}\{Q(\hat{\boldsymbol{\theta }}_{2},t,\mathbf{Y}_{(2)}) - Q(\boldsymbol{\theta }_{2},t,\mathbf{Y}_{(2)})\}$ converges to a zero mean Gaussian process in t and Y ₍₂₎. Similar to the derivation for $\widetilde{\mbox{ NRI}}(\hat{\boldsymbol{\theta }},t)$, it can be shown that the process $\widetilde{\mathcal{N}}(t) = \sqrt{n}[\widehat{\mbox{ NRI}}(\hat{\boldsymbol{\theta }},t) -\mbox{ NRI}(\boldsymbol{\theta }_{0},t)]$ is asymptotically equivalent to ${n}^{-1/2}\sum _{i=1}^{n}\zeta _{i}(u,v,t).$ In particular, for a fixed $\boldsymbol{\theta }$, $\sqrt{n}\{\widehat{\mbox{ NRI}}(\boldsymbol{\theta },t) -\mbox{ NRI}(\boldsymbol{\theta },t)\} \approx {n}^{-1/2 }\sum _{i=1}^{n}\eta _{i}^{{\ast}}(t)$ where $\eta _{i}^{{\ast}}(t) = 2[D{(t)}^{-2}\{\varDelta _{i}(\boldsymbol{\theta })Q_{i}(\boldsymbol{\theta }_{2})-\mbox{ Pr}(\varDelta _{i}(\boldsymbol{\theta }) = 1\vert T_{i} \leq t)Q_{i}(\boldsymbol{\theta }_{2})\}-D(t)_{-}^{-2}\{\varDelta _{i}(\boldsymbol{\theta })[1-Q_{i}(\boldsymbol{\theta }_{2})]-\mbox{ Pr}(\varDelta _{i}(\boldsymbol{\theta }) = 1\vert T_{i} > t)[1-Q_{i}(\boldsymbol{\theta }_{2})]\}]$. Thus, $\widetilde{\mathcal{N}}(t) \approx {n}^{-1/2}\sum _{i=1}^{n}\zeta _{i}(t)$ where $\zeta _{i}(u,v,t) =\eta _{ i}^{{\ast}}(t) +\psi _{i}\frac{\partial \mbox{ NRI}(t)} {\partial \boldsymbol{\theta }} \vert _{\boldsymbol{\theta }_{0}}$. Once again, using a functional central limit theorem, this implies that $\widetilde{\mathcal{N}}(t)$ converges to a Gaussian process with mean zero.