Robust variable selection for finite mixture regression models

Abstract

Finite mixture regression (FMR) models are frequently used in statistical modeling, often with many covariates with low significance. Variable selection techniques can be employed to identify the covariates with little influence on the response. The problem of variable selection in FMR models is studied here. Penalized likelihood-based approaches are sensitive to data contamination, and their efficiency may be significantly reduced when the model is slightly misspecified. We propose a new robust variable selection procedure for FMR models. The proposed method is based on minimum-distance techniques, which seem to have some automatic robustness to model misspecification. We show that the proposed estimator has the variable selection consistency and oracle property. The finite-sample breakdown point of the estimator is established to demonstrate its robustness. We examine small-sample and robustness properties of the estimator using a Monte Carlo study. We also analyze a real data set.

Appendix

In this “Appendix”, we list the conditions used in the theorems and outline the proofs of main results. For convenience of notation, we write $f_{\varvec{\theta }}(y)\$for $f_{\varvec{\theta },\eta }(y)$ defined by (4). The following technical conditions are imposed:

(C1) $E\Vert \mathbf {X}\Vert ^{2}<+\infty$ and $\max _{1\le i\le n}\Vert X_{i}\Vert =O_{p}(1)$. $\int \sup _{t\in \Theta }|\frac{\partial f_{t} (y)}{\partial t}|{\text{ d }}y<+\infty$, where $|b|=\max _{1\le i\le k}|b_{i}|$ for a vector $b=(b_{1},\ldots ,b_{k})^{T}$. $\ddot{S}_{\varvec{\theta }}^{(l,m)}(y)\in L_{2}$ for $1\le l,m\le J(p+2)-1$ and $H(\varvec{\theta }_{0})=-\int \ddot{S}_{\varvec{\theta }_{0}}(y)f_{\varvec{\theta }_{0}}^{1/2}(y){\text{ d }}y$ is a positive definite matrix, where $\ddot{S}_{\varvec{\theta }}^{(l,m)}(y)$ denotes the $(l,m)^{th}$ component of $\ddot{S}_{\varvec{\theta }}(y)$, $\ddot{S} _{_{\varvec{\theta }}}(y)=\frac{\partial ^{2}S_{\varvec{\theta }}(y)}{\partial \varvec{\theta }\partial \varvec{\theta }^{T}}$, and $S_{\varvec{\theta }} (y)=f_{\varvec{\theta }}^{1/2}(y)$.

(C2) The second and third continuous partial derivatives of g(y, z, u) exist w.r.t. y and $z,\,u,$ respectively. For a given $\tilde{L}\,>0$ and some $\epsilon$-neighborhood of $\theta ,B(\theta ,\epsilon ),$ define $\tilde{g}(y)=\inf _{\Vert x\Vert \le \tilde{L},t\in B(\theta ,\epsilon )}\min _{1\le j\le k}g(y,\mathbf {x}^{T}t_{j},u_{j}).$ Suppose that $1/\tilde{g}(y)$ is bounded on any compact subset of R and that, as $L\rightarrow \infty$,

$$\begin{aligned} \begin{array}{ll} \int _{|y|>L}\int _{\Vert x\Vert \le \tilde{L}}|x_{r}|\breve{g}_{z} (y,\mathbf {x}){\text{ d }}\eta (\mathbf {x}){\text{ d }}y\rightarrow 0, &{} \int _{|y|>L}\int _{\Vert x\Vert \le \tilde{L}}\breve{g}_{u}(y,\mathbf {x}){\text{ d }}\eta (\mathbf {x}){\text{ d }}y\rightarrow 0,\\ \int _{|y|>L}\int _{\Vert x\Vert \le \tilde{L}}g^{*}(y,\mathbf {x} ){\text{ d }}\eta (\mathbf {x}){\text{ d }}y\rightarrow 0, &{} \int _{|y|>L}\frac{\int _{\Vert x\Vert \le \tilde{L}}x_{r}^{2}(\dot{g}_{z}^{*}(y,\mathbf {x}))^{2}{\text{ d }}\eta (\mathbf {x})}{\tilde{g}(y)}{\text{ d }}y\rightarrow 0,\\ \int _{|y|>L}\frac{\int _{\Vert x\Vert \le \tilde{L}}(\dot{g}_{u}^{*}(y,\mathbf {x}))^{2}{\text{ d }}\eta (\mathbf {x})}{\tilde{g}(y)}{\text{ d }}y\rightarrow 0, &{} \int _{|y|>L}\frac{\int _{\Vert x\Vert \le \tilde{L}}x_{r}^{4}(\dot{g}_{z}^{*}(y,\mathbf {x}))^{4}{\text{ d }}\eta (\mathbf {x})}{\tilde{g}^{3}(y)}{\text{ d }}y\rightarrow 0,\\ \int _{|y|>L}\frac{\int _{\Vert x\Vert \le \tilde{L}}(\dot{g}_{u}^{*}(y,\mathbf {x}))^{4}{\text{ d }}\eta (\mathbf {x})}{\tilde{g}^{3}(y)}{\text{ d }}y\rightarrow 0, &{} \int _{|y|>L}\frac{\int _{\Vert x\Vert \le \tilde{L}}x_{r}^{2}x_{q}^{2}(\ddot{g}_{zz}^{*}(y,\mathbf {x}))^{2}{\text{ d }}\eta (\mathbf {x})}{\tilde{g}(y)} {\text{ d }}y\rightarrow 0,\\ \int _{|y|>L}\frac{\int _{\Vert x\Vert \le \tilde{L}}(\ddot{g}_{uu}^{*}(y,\mathbf {x}))^{2}{\text{ d }}\eta (\mathbf {x})}{\tilde{g}(y)}{\text{ d }}y\rightarrow 0, &{} \int _{|y|>L}\frac{\int _{\Vert x\Vert \le \tilde{L}}x_{r}^{2}(\ddot{g} _{zu}^{*}(y,\mathbf {x}))^{2}{\text{ d }}\eta (\mathbf {x})}{\tilde{g}(y)}{\text{ d }}y\rightarrow 0 \end{array} \end{aligned}$$

for $r,q=0,\ldots ,p$, where $x_{0}=1$, $\breve{g}_{z}(y,\mathbf {x})=\sup _{t\in \Theta }\max _{1\le j\le k}|\dot{g}_{z}(y,\mathbf {x}^{T}t_{j},u_{j})|$, $\breve{g}_{u}(y,\mathbf {x})=\sup _{t\in \Theta }\max _{1\le j\le k}|\dot{g} _{u}(y,\mathbf {x}^{T}t_{j},u_{j})|$,

$$\begin{aligned} g^{*}(y,\mathbf {x})=\sup _{t\in B(\theta ,\epsilon )}\max _{1\le j\le k}g(y,x^{T}t_{j},u_{j}),\ \ \dot{g}_{z}^{*}(y,x)=\sup _{t\in B(\theta ,\epsilon )}\max _{1\le j\le k}|\dot{g}_{z}(y,x^{T}t_{j},u_{j})|, \end{aligned}$$

$\ddot{g}_{zz}^{*}(y,\mathbf {x})=\sup _{t\in B(\theta ,\epsilon )}\max _{1\le j\le k}|\ddot{g}_{zz}(y,\mathbf {x}^{T}t_{j},u_{j})|$, $\dot{g}_{u}^{*}(y,\mathbf {x})$, $\ddot{g}_{zu}^{*}(y,\mathbf {x})$, and $\ddot{g} _{uu}^{*}(y,\mathbf {x})$ are defined in a similar fashion, $\dot{g} _{z}(y,z,u)=\frac{\partial g(y,z,u)}{\partial z},\dot{g}_{u}(y,z,u)=\frac{\partial g(y,z,u)}{\partial u}$, $\ddot{g}_{zz}(y,z,u)=\frac{\partial ^{2}g(y,z,u)}{\partial z^{2}},\$and $\ddot{g}_{zu}(y,z,u)=\frac{\partial ^{2}g(y,z,u)}{\partial z\partial u}$.

(C3) The kernel function $K(\cdot )$ is a bounded symmetric density with compact support $[-M,M]$.

(C4) $\sup _{y\in \mathbb {R}}\sup _{|v|\le M}\frac{f_{\varvec{\theta }_{0}}(y+v)}{f_{\varvec{\theta }_{0} }^{1/2}(y)}=O(1)$, $\sup _{y\in \mathbb {R}}\sup _{|v|\le M}\frac{[f_{\varvec{\theta }_{0}}^{\prime \prime }(y+v)]^{2}}{f_{\varvec{\theta }_{0}}^{7/4}(y)}=O(1)$, and as $L\rightarrow \infty \,\int _{|y|>L}\frac{\dot{S}_{\varvec{\theta }_{0} q}^{2}(y)}{f_{\varvec{\theta }_{0} }^{1/2}(y)}{\text{ d }}y\rightarrow 0$ for $q=1,\ldots ,J(p+2)-1$, where $f_{\varvec{\theta }} ^{\prime \prime }(y)=\frac{\partial ^{2}f_{\varvec{\theta }}(y)}{\partial y^{2}}$ and $\dot{S}_{\varvec{\theta } q}(y)$ is the $q^{th}$ entry of the vector $\dot{S}_{\varvec{\theta }}(y)$.

(C5) The bandwidth $h_{n}=b_{0}n^{-\gamma }$ for some $\gamma \in (1/8,1/2)$ and constant $b_{0}>0$. $E|Y|^{s}<+\infty$ for $s>6/(1-2\gamma )$. There exists some $l,\,1/s<l<(1-2\gamma )/6,$ satisfying $\sup _{|y|\le n^{l}}\sup _{|v|\le M}\frac{f_{\varvec{\theta }_{0}}(y+v)}{f_{\varvec{\theta }_{0} }(y)}=O(1)$, and as $n\rightarrow \infty$

$$\begin{aligned} (n^{1/2}h_{n})^{-1}\int _{|y|\le n^{l}}\frac{|\dot{g}_{jq}(y)|}{g_{j} (y)}{\text{ d }}y\rightarrow 0,\ \ \ (n^{1/2}h_{n})^{-1}\int _{|y|\le n^{l}}\frac{|\dot{g}_{j\gamma }(y)|}{g_{j}(y)}{\text{ d }}y\rightarrow 0, \end{aligned}$$

where $g_{j}(y)=g_{j}(y,\beta _{j},\gamma _{j})=\int g(y,x^{T}\beta _{j} ,\gamma _{j}){\text{ d }}\eta (x)$; $\dot{g}_{jq}(y)=\frac{\partial g_{j}(y)}{\partial \beta _{jq}}$ for $j=1,\ldots ,J$, $q=1,\ldots ,p$; and $\dot{g}_{j\gamma }(y)=\frac{\partial g_{j}(y)}{\partial \gamma _{j}}$.

(C6) $\int \frac{\int g^{4}(y,\mathbf {x}^{T}\varvec{\beta }_{j},\gamma _{j}){\text{ d }}\eta (x)}{g_{j}^{3}(y)}{\text{ d }}y<+\infty$; $\int \frac{\int x_{r}^{2}\dot{g} _{z}^{2}(y,\mathbf {x}^{T}\varvec{\beta }_{j},\gamma _{j}){\text{ d }}\eta (x)}{g_{j} (y)}{\text{ d }}y<+\infty$ for $j=1,\ldots ,J$, $r=1,\ldots ,p$; and $\int \frac{\int \dot{g}_{u}^{2}(y,\mathbf {x}^{T}\varvec{\beta }_{j},\gamma _{j}){\text{ d }}\eta (x)}{g_{j} (y)}{\text{ d }}y<+\infty .$

Conditions (C1) and (C2) guarantee that (2.5) and (2.6) of Beran (1977) hold about an expansion of the first and second partial derivatives in some neighborhood of $\varvec{\theta }_{0}$. Condition (C3) is also a typical assumption on kernels, including the family of symmetric beta kernel functions (Chen 1999). Conditions (C4)–(C6) are used to derive the asymptotic normality of the MHD estimators. When X is bounded and $g(y,x^{T}\beta _{j},\gamma _{j})=\exp \{-(y-x^{T}\beta _{j})^{2}/(2\gamma _{j}^{2})\}/(\sqrt{2\pi }\gamma _{j})$, or X is a normal random variable and $g(y,x,\beta _{j1},\beta _{j2},\gamma _{j})=\exp \{-[y-(\beta _{j1}+\beta _{j2}x)]^{2} /(2\gamma _{j}^{2})\}/(\sqrt{2\pi }\gamma _{j})$ for $j=1,\ldots ,J$, the above conditions are satisfied, see Remarks 3.4 and 3.4 of Tang and Karunamuni (2013) for details.

Lemma 1

Under the assumptions of Theorem 3, there exists a local minimizer $\hat{\varvec{\theta }}$ of (9) such that $\left\| \hat{\varvec{\theta }} -\varvec{\theta }_{0}\right\| =O_{p}(n^{-1/2})$.

Proof

Let

$$\begin{aligned} P_{n}(\varvec{\theta })=\sum _{j=1}^{J}\alpha _{j}^{1/2}\sum _{k=1}^{p} p_{\lambda _{nj}}^{\prime }(|\beta _{jk}^{(0)}|)|\beta _{jk}|,\ \ P_{n1} (\varvec{\theta })=\sum _{j=1}^{J}\alpha _{j}^{1/2}\sum _{k=1}^{d_{j}}p_{\lambda _{nj}}^{\prime }(|\beta _{jk}^{(0)}|)|\beta _{jk}| \end{aligned}$$

and $D_{n}(\varvec{\theta })=\int S_{\varvec{\theta },n}(y)\hat{f}_{n}^{1/2} (y){\text{ d }}y-P_{n}(\varvec{\theta })$. It suffices to prove that for any given $\varepsilon >0$, there exists a constant C such that

$$\begin{aligned} P\left\{ \sup _{\Vert v\Vert =C}D_{n}(\varvec{\theta }_{0}+n^{-1/2}v)<D_{n} (\varvec{\theta }_{0})\right\} \ge 1-\varepsilon . \end{aligned}$$

(20)

Note that

$$\begin{aligned} D_{n}(\varvec{\theta }_{0}+n^{-1/2}v)-D_{n}(\varvec{\theta }_{0})\le & {} \int [S_{\varvec{\theta }_{0}+n^{-1/2}v,n}(y)-S_{\varvec{\theta }_{0},n}(y)]\hat{f} _{n}^{1/2}(y){\text{ d }}y\nonumber \\&-[P_{n1}(\varvec{\theta }_{01}+n^{-1/2}v_{1})-P_{n1}(\varvec{\theta }_{01})]. \end{aligned}$$

(21)

By a Taylor expansion,

$$\begin{aligned}&\int [S_{\varvec{\theta }_{0}+n^{-1/2}v,n}(y)-S_{\varvec{\theta }_{0},n}(y)]\hat{f} _{n}^{1/2}(y){\text{ d }}y\\&\quad =n^{-1/2}v\int \dot{S}_{\varvec{\theta }_{0},n}(y)\hat{f}_{n}^{1/2}(y){\text{ d }}y+\frac{1}{2n}v^{T}\int \ddot{S}_{\varvec{\theta }^{*},n}(y)\hat{f}_{n}^{1/2}(y){\text{ d }}yv, \end{aligned}$$

where $\varvec{\theta }^{*}$ is between $\varvec{\theta }_{0}$ and $\varvec{\theta }_{0}+n^{-1/2}v$. As in the proof of Lemma 3.1 of Tang and Karunamuni (2013), we have

$$\begin{aligned} \int [\ddot{S}_{\varvec{\theta }^{*},n}(y)-\ddot{S}_{\varvec{\theta }_{0},n} (y)]\hat{f}_{n}^{1/2}(y){\text{ d }}y\le \left( \int [\ddot{S}_{\varvec{\theta }^{*},n} (y)-\ddot{S}_{\varvec{\theta }_{0},n}(y)]^{2}{\text{ d }}y\right) ^{1/2}\left( \int \hat{f}_{n} (y){\text{ d }}y\right) ^{1/2}=o_{p}(1). \end{aligned}$$

Similar to the proof of Theorem 3.2 of Tang and Karunamuni (2013), we obtain

$$\begin{aligned} \int \ddot{S}_{\varvec{\theta }_{0},n}(y)\hat{f}_{n}^{1/2}(y){\text{ d }}y=-H(\varvec{\theta }_{0} )+o_{p}(1), \end{aligned}$$

where $H(\varvec{\theta }_{0})=-\int \ddot{S}_{\varvec{\theta }_{0}} (y)f_{\varvec{\theta }_{0}}^{1/2}(y){\text{ d }}y$. Hence

$$\begin{aligned}&\int [S_{\varvec{\theta }_{0}+n^{-1/2}v,n}(y)-S_{\varvec{\theta }_{0},n}(y)]\hat{f} _{n}^{1/2}(y){\text{ d }}y\nonumber \\&\quad =n^{-1/2}v\int \dot{S}_{\varvec{\theta }_{0},n}(y)\hat{f}_{n}^{1/2}(y){\text{ d }}y-\frac{1}{2n}v^{T}H(\varvec{\theta }_{0})v[1+o_{p}(1)]. \end{aligned}$$

(22)

By (A.26) of Tang and Karunamuni (2013), it follows that $\int \dot{S}_{\varvec{\theta }_{0},n}(y)\hat{f}_{n}^{1/2}(y){\text{ d }}y=O_{p}(n^{-1/2})$. Since $\varvec{\theta }^{(0)}\rightarrow _{P}\varvec{\theta }_{0}$, we have $P\{P_{n1} (\varvec{\theta }_{01}+n^{-1/2}v_{1})-P_{n1}(\varvec{\theta }_{01})=0\}\rightarrow 1$ as $n\rightarrow \infty$. Hence, for sufficiently large C, (20) follows from (21) and (22) and the fact that $H(\varvec{\theta }_{0})$ is positive definite. The proof of Lemma 1 is complete. $\square$

Lemma 2

Under the assumptions of Theorem 3, for any $\varvec{\theta }=(\varvec{\theta }_{1}^{T},\varvec{\theta }_{2}^{T})^{T}$ such that $\Vert \varvec{\theta }-\varvec{\theta }_{0}\Vert =O(n^{-1/2})$ and $\varvec{\theta }_{2} \ne \varvec{0}$, with probability tending to 1, we have

$$\begin{aligned} D_{n}((\varvec{\theta }_{1},\varvec{\theta }_{2}))<D_{n}((\varvec{\theta }_{1},\varvec{0})). \end{aligned}$$

Proof

By a Taylor expansion, we obtain

$$\begin{aligned} S_{(\varvec{\theta }_{1},\varvec{\theta }_{2}),n}(y)=S_{(\varvec{\theta }_{1} ,\varvec{0}),n}(y)+\varvec{\theta }_{2}^{T}\frac{\partial S_{\varvec{\theta },n} (y)}{\partial \varvec{\theta }_{2}}\left| _{\varvec{\theta }=(\varvec{\theta }_{1} ,\varvec{0})}+\frac{1}{2}\varvec{\theta }_{2}^{T}\frac{\partial ^{2}S_{\varvec{\theta },n} (y)}{\partial \varvec{\theta }_{2}\partial \varvec{\theta }_{2}^{T}}\right| _{\varvec{\theta }=(\varvec{\theta }_{1},\varvec{\theta }_{2}^{*})}\varvec{\theta }_{2}, \end{aligned}$$

where $\varvec{\theta }_{2}^{*}$ is between $\varvec{0}$ and $\varvec{\theta }_{2}$. As in the proof of (22), it follows that

$$\begin{aligned} \int \frac{\partial ^{2}S_{\varvec{\theta },n}(y)}{\partial \varvec{\theta }_{2} \partial \varvec{\theta }_{2}^{T}}\left| _{\varvec{\theta }=(\varvec{\theta }_{1} ,\varvec{\theta }_{2}^{*})}\hat{f}_{n}^{1/2}(y){\text{ d }}y=\int \frac{\partial ^{2}S_{\varvec{\theta }}(y)}{\partial \varvec{\theta }_{2}\partial \varvec{\theta }_{2}^{T} }\right| _{\varvec{\theta }=\varvec{\theta }_{0}}f_{\varvec{\theta }_{0}} ^{1/2}(y){\text{ d }}y[1+o_{p}(1)]=O_{p}(1). \end{aligned}$$

By (A.26) of Tang and Karunamuni (2013), we have

$$\begin{aligned} \int \frac{\partial S_{\varvec{\theta },n}(y)}{\partial \varvec{\theta }_{2}}\left| _{\varvec{\theta }=(\varvec{\theta }_{1},\varvec{0})}\hat{f}_{n}^{1/2}(y){\text{ d }}y=O_{p} (n^{-1/2})\right. . \end{aligned}$$

Using the fact that $\Vert \varvec{\theta }_{2}\Vert =O(n^{-1/2})$ and $n^{1/2}\lambda _{nj}\rightarrow +\infty$, we deduce that with probability tending to 1, it holds that

$$\begin{aligned}&D_{n}((\varvec{\theta }_{1},\varvec{\theta }_{2}))-D_{n}((\varvec{\theta }_{1},\varvec{0}))\\&\quad =\int [S_{(\varvec{\theta }_{1},\varvec{\theta }_{2}),n}(y)-S_{(\varvec{\theta }_{1} ,\varvec{0}),n}(y)]\hat{f}_{n}^{1/2}(y){\text{ d }}y-\sum _{j=1}^{J}\alpha _{j}^{1/2} \sum _{k=d_{j}}^{p}p_{\lambda _{nj}}^{\prime }(|\beta _{jk}^{(0)}|)|\beta _{jk}|\\&\quad =O_{p}(n^{-1/2})\sum _{j=1}^{J}\sum _{k=d_{j}}^{p}|\beta _{jk}|-\sum _{j=1} ^{J}\alpha _{j}^{1/2}\sum _{k=d_{j}}^{p}p_{\lambda _{nj}}^{\prime }(|\beta _{jk}^{(0)}|)|\beta _{jk}|\\&\quad =\sum _{j=1}^{J}\lambda _{nj}\sum _{k=d_{j}}^{p}\left[ O_{p}((n^{1/2}\lambda _{nj} )^{-1})-\alpha _{j}^{1/2}\lambda _{nj}^{-1}p_{\lambda _{nj}}^{\prime }(|\beta _{jk}^{(0)}|)\right] |\beta _{jk}|<0. \end{aligned}$$

This completes the proof of Lemma 2. $\square$

Proof of Theorem 3

By Lemmas 1 and 2, there exists a $\sqrt{n}$-consistent local maximizer $\check{\varvec{\theta }}=(\check{\varvec{\theta }} _{1},\varvec{0}^{T})^{T}$ of (9). By a Taylor expansion, with probability tending 1, we have

$$\begin{aligned}&D_{n}((\hat{\varvec{\theta }}_{1},\hat{\varvec{\theta }}_{2}))\\&\quad =D_{n}((\check{\varvec{\theta }}_{1},\varvec{0}))+(\hat{\varvec{\theta }}-\check{\varvec{\theta }})^{T}\int \frac{\partial S_{\varvec{\theta },n}(y)}{\partial \varvec{\theta }}\left| _{\varvec{\theta }=(\check{\varvec{\theta }}_{1},0)}\hat{f}_{n}^{1/2}(y){\text{ d }}y\right. \\&\qquad +\,\frac{1}{2}(\hat{\varvec{\theta }}-\check{\varvec{\theta }})^{T}\int \frac{\partial ^{2}S_{\varvec{\theta },n}(y)}{\partial \varvec{\theta }\partial \varvec{\theta }^{T}}\left| _{\varvec{\theta }=\varvec{\theta }^{*}}\hat{f} _{n}^{1/2}(y){\text{ d }}y(\hat{\varvec{\theta }}-\check{\varvec{\theta }})-\sum _{j=1}^{J} \alpha _{j}^{1/2}\sum _{k=d_{k}+1}^{p}p_{\lambda _{nj}}^{\prime }\left( |\beta _{jk}^{(0)}|\right) |\hat{\beta }_{jk}|,\right. \end{aligned}$$

where $\varvec{\theta }^{*}$ is between $\hat{\varvec{\theta }}$ and $\check{\varvec{\theta }}$. By Theorem 2, it follows that $\hat{\varvec{\theta }} \rightarrow _{P}\varvec{\theta }_{0}$. Using an argument similar to the one used in the proof of (22), we obtain $\int \frac{\partial ^{2}S_{\varvec{\theta },n} (y)}{\partial \varvec{\theta }\partial \varvec{\theta }^{T}} _{\varvec{\theta }=\varvec{\theta }^{*}}\hat{f}_{n}^{1/2}(y){\text{ d }}y=-H(\varvec{\theta }_{0} )[1+o_{p}(1)]$. Noting that with probability tending to 1, $\int \frac{\partial S_{\varvec{\theta },n}(y)}{\partial \varvec{\theta }_{1}}_{\varvec{\theta }=(\check{\varvec{\theta }}_{1},0)}\hat{f}_{n}^{1/2}(y){\text{ d }}y=0$, we have

$$\begin{aligned}&D_{n}((\hat{\varvec{\theta }}_{1},\hat{\varvec{\theta }}_{2}))\nonumber \\&\quad =D_{n}((\check{\varvec{\theta }}_{1},\varvec{0}))+\hat{\varvec{\theta }}_{2}^{T}\int \frac{\partial S_{\varvec{\theta },n}(y)}{\partial \varvec{\theta }_{2}}\left| _{\varvec{\theta }=(\check{\varvec{\theta }}_{1},0)}\hat{f}_{n}^{1/2}(y){\text{ d }}y\right. \nonumber \\&\qquad -\,\frac{1}{2}(\hat{\varvec{\theta }}-\check{\varvec{\theta }})^{T}H(\varvec{\theta }_{0} )(\hat{\varvec{\theta }}-\check{\varvec{\theta }})[1+o_{p}(1)]-\sum _{j=1}^{J} \alpha _{j}^{1/2}\sum _{k=d_{k}+1}^{p}p_{\lambda _{nj}}^{\prime }(|\beta _{jk}^{(0)}|)|\hat{\beta }_{jk}|,\nonumber \\ \end{aligned}$$

(23)

Using a Taylor expansion, we have

$$\begin{aligned}&\int \frac{\partial S_{\varvec{\theta },n}(y)}{\partial \varvec{\theta }_{2}}\left| _{\varvec{\theta }=(\check{\varvec{\theta }}_{1},0)}\hat{f}_{n}^{1/2}(y){\text{ d }}y=\int \frac{\partial S_{\varvec{\theta },n}(y)}{\partial \varvec{\theta }_{2}}\right| _{\varvec{\theta }=\varvec{\theta }_{0}}\nonumber \\&\quad \hat{f}_{n}^{1/2}(y){\text{ d }}y+H_{21}(\varvec{\theta }_{0} )\check{\varvec{\theta }}_{1}[1+o_{p}(1)], \end{aligned}$$

where $H_{21}(\varvec{\theta }_{0})=\int \frac{\partial ^{2}S_{\varvec{\theta }} (y)}{\partial \varvec{\theta }_{2}\partial \varvec{\theta }_{1}^{T}} _{\varvec{\theta }=\varvec{\theta }_{0}}f_{\varvec{\theta }_{0}}^{1/2}(y){\text{ d }}y$. By (A.26) of Tang and Karunamuni (2013), we have $\int \frac{\partial S_{\varvec{\theta },n} (y)}{\partial \varvec{\theta }_{2}}\left| _{\varvec{\theta }=\varvec{\theta }_{0}} \hat{f}_{n}^{1/2}(y){\text{ d }}y=O_{p}(n^{-1/2})\right.$. Then $\check{\varvec{\theta }} _{1}=O_{p}(n^{-1/2})$ implies that $\int \frac{\partial S_{\varvec{\theta },n} (y)}{\partial \varvec{\theta }_{2}}_{\varvec{\theta }=(\check{\varvec{\theta }}_{1},0)} \hat{f}_{n}^{1/2}(y){\text{ d }}y=O_{p}(n^{-1/2})$. If $\hat{\varvec{\theta }}\ne \check{\varvec{\theta }}$, then by (23), we have $D_{n}((\hat{\varvec{\theta }} _{1},\hat{\varvec{\theta }}_{2}))<D_{n}((\check{\varvec{\theta }}_{1},\varvec{0}))$. This is a contradiction to the fact that $\hat{\varvec{\theta }}$ is a maximizer of (10). So $\hat{\varvec{\theta }}_{2}=0$ and $\hat{\varvec{\theta }}_{1}=\check{\varvec{\theta }}_{1}$.

We now prove the asymptotic normality part. Consider $D_{n}((\varvec{\theta }_{1} ,\varvec{0}))$ as a function of $\varvec{\theta }_{1}$. Noting that with probability tending 1, $\hat{\varvec{\theta }}_{1}$ is the $\sqrt{n}$-consistent maximizer of $D_{n}((\varvec{\theta }_{1},\varvec{0}))$ and satisfies

$$\begin{aligned} \frac{\partial D_{n}((\varvec{\theta }_{1},\varvec{0}))}{\partial \varvec{\theta }_{1} }\left| _{\varvec{\theta }_{1}=\hat{\varvec{\theta }}_{1}}=\int \frac{\partial S_{\varvec{\theta },n}(y)}{\partial \varvec{\theta }_{1}}\right| _{\varvec{\theta }=(\hat{\varvec{\theta }}_{1},\varvec{0})}\hat{f}_{n}^{1/2}(y){\text{ d }}y=0. \end{aligned}$$

By an argument similar to the one used in the proof of (22), we obtain

$$\begin{aligned} \int \frac{\partial S_{\varvec{\theta },n}(y)}{\partial \varvec{\theta }_{1}}\left| _{\varvec{\theta }=(\hat{\varvec{\theta }}_{1},\varvec{0})}\hat{f}_{n}^{1/2} (y){\text{ d }}y=\int \frac{\partial S_{\varvec{\theta },n}(y)}{\partial \varvec{\theta }_{1} }\right| _{\varvec{\theta }={\varvec{\theta }}_{0}}\hat{f}_{n}^{1/2} (y){\text{ d }}y-H_{1}(\varvec{\theta }_{01})(\hat{\varvec{\theta }}_{1}-\varvec{\theta }_{01} )[1+o_{p}(1)]. \end{aligned}$$

Hence

$$\begin{aligned} H_{1}(\varvec{\theta }_{01})(\hat{\varvec{\theta }}_{1}-\varvec{\theta }_{01} )[1+o_{p}(1)]=\int \frac{\partial S_{\varvec{\theta },n}(y)}{\partial \varvec{\theta }_{1}}\left| _{\varvec{\theta }={\varvec{\theta }}_{0}}\hat{f} _{n}^{1/2}(y){\text{ d }}y\right. . \end{aligned}$$

(24)

Using an argument similar to the one used in the proof of Theorem 3.3 of Tang and Karunamuni (2013), we obtain

$$\begin{aligned} n^{1/2}\left( \int \frac{\partial S_{\varvec{\theta },n}(y)}{\partial \varvec{\theta }_{1}}\left| _{\varvec{\theta }={\varvec{\theta }}_{0}}\hat{f} _{n}^{1/2}(y){\text{ d }}y-A_{n1}(\varvec{\theta }_{01})\right. \right) \rightarrow _{d}N\left( \varvec{0},\Sigma _{1}(\varvec{\theta }_{01})\right) . \end{aligned}$$

Now (11) follows from the preceding expression and (24). This completes the proof of Theorem 3. $\square$

Proof of Theorem 5

Note that

$$\begin{aligned}&2\left\| f_{n+n_{1}}^{1/2}(\hat{\varvec{\theta }}(h_{n+n_{1}}))-\hat{f}_{n+n_{1} }^{1/2}(h_{n+n_{1}})\right\| ^{2}\nonumber \\&\quad \ge 2\left\| f_{n+n_{1}}^{1/2}(\hat{\varvec{\theta }}(h_{n+n_{1}}))-f_{n}^{1/2} (\hat{\varvec{\theta }}(h_{n}))\right\| ^{2}-2\left\| f_{n}^{1/2}(\hat{\varvec{\theta }} (h_{n}))-\hat{f}_{n+n_{1}}^{1/2}(h_{n+n_{1}})\right\| ^{2}.\nonumber \\ \end{aligned}$$

(25)

Since

$$\begin{aligned}&\left\| f_{n+n_{1}}^{1/2}(\hat{\varvec{\theta }}(h_{n+n_{1}}))-\hat{f}_{n+n_{1} }^{1/2}(h_{n+n_{1}})\right\| ^{2}\\&\qquad +\,2\sum _{j=1}^{J}\hat{\alpha }_{j,n+n_{1}}^{1/2}\sum _{k=1}^{p}p_{\lambda _{(n+n_{1})j}}^{\prime }(|\beta _{jk,n+n_{1}}^{(0)}|)|\hat{\beta }_{jk,n+n_{1}}|\\&\quad \le \left\| f_{n+n_{1}}^{1/2}(\hat{\varvec{\theta }}(h_{n}))-\hat{f}_{n+n_{1}} ^{1/2}(h_{n+n_{1}})\right\| ^{2}+2\sum _{j=1}^{J}\hat{\alpha }_{j,n}^{1/2}\sum _{k=1}^{p}p_{\lambda _{(n+n_{1})j}}^{\prime }(|\beta _{jk,n+n_{1}}^{(0)} |)|\hat{\beta }_{jk,n}|, \end{aligned}$$

we have

$$\begin{aligned}&\left\| f_{n+n_{1}}^{1/2}(\hat{\varvec{\theta }}(h_{n+n_{1}}))-\hat{f}_{n+n_{1} }^{1/2}(h_{n+n_{1}})\right\| ^{2}\nonumber \\&\quad \le \left\| f^{1/2}(\hat{\varvec{\theta }}(h_{n}))-\hat{f}_{n+n_{1}}^{1/2} (h_{n+n_{1}})\right\| ^{2}+\left\| f_{n+n_{1}}^{1/2}(\hat{\varvec{\theta }} (h_{n}))-f^{1/2}(\hat{\varvec{\theta }}(h_{n}))\right\| ^{2}+2P_{n,n_{1}}^{*}.\nonumber \\ \end{aligned}$$

(26)

Clearly

$$\begin{aligned}&\left\| f_{n}^{1/2}(\hat{\varvec{\theta }}(h_{n}))-\hat{f}_{n+n_{1}}^{1/2} (h_{n+n_{1}})\right\| ^{2}\nonumber \\&\quad \le \left\| f^{1/2}(\hat{\varvec{\theta }}(h_{n}))-\hat{f}_{n+n_{1}}^{1/2} (h_{n+n_{1}})\right\| ^{2}+\left\| f_{n}^{1/2}(\hat{\varvec{\theta }}(h_{n} ))-f^{1/2}(\hat{\varvec{\theta }}(h_{n}))\right\| ^{2} \end{aligned}$$

(27)

and

$$\begin{aligned}&\left\| f_{n+n_{1}}^{1/2}(\hat{\varvec{\theta }}(h_{n+n_{1}}))-f_{n}^{1/2} (\hat{\varvec{\theta }}(h_{n}))\right\| ^{2}\nonumber \\&\ge \left\| f^{1/2}(\hat{\varvec{\theta }}(h_{n+n_{1}}))-f^{1/2}(\hat{\varvec{\theta }} (h_{n}))\right\| ^{2}-\left\| f_{n+n_{1}}^{1/2}(\hat{\varvec{\theta }}(h_{n+n_{1}}))\right. \nonumber \\&\left. -f^{1/2}(\hat{\varvec{\theta }}(h_{n+n_{1}}))\right\| ^{2}-\left\| f_{n}^{1/2} (\hat{\varvec{\theta }}(h_{n}))-f^{1/2}(\hat{\varvec{\theta }}(h_{n}))\right\| ^{2}. \end{aligned}$$

(28)

Combining (25)–(28), we conclude that

$$\begin{aligned} 4\left\| f^{1/2}(\hat{\varvec{\theta }}(h_{n}))-\hat{f}_{n+n_{1}}^{1/2}(h_{n+n_{1} })\right\| ^{2}\ge \left\| f^{1/2}(\hat{\varvec{\theta }}(h_{n+n_{1}}))-f^{1/2} (\hat{\varvec{\theta }}(h_{n}))\right\| ^{2}-\varepsilon _{n,n_{1}} \end{aligned}$$

If $\hat{\varvec{\theta }}(h_{n})$ breaks down, then $\sup _{\#\varvec{V}_{n_{1} }=n_{1}}d(\hat{\varvec{\theta }}(h_{n}),\hat{\varvec{\theta }}(h_{n+n_{1}}))=\infty$. So by the definition of $\delta ^{*}$, $\left\| f^{1/2}(\hat{\varvec{\theta }} (h_{n+n_{1}}))-f^{1/2}(\hat{\varvec{\theta }}(h_{n}))\right\| ^{2}\ge \delta ^{*}$. Hence, $\left\| f^{1/2}(\hat{\varvec{\theta }}(h_{n}))-\right. \left. \hat{f}_{n+n_{1}} ^{1/2}(h_{n+n_{1}})\right\| ^{2}\ge (\delta ^{*}-\varepsilon _{n,n_{1}})/4.$ The rest of the proof is similar to that of Tamura and Boos (1986) and is therefore omitted to save space. This completes the proof of Theorem 5. $\square$