Bayesian Grouped Horseshoe Regression with Application to Additive Models

Abstract

The Bayesian horseshoe estimator is known for its robustness when handling noisy and sparse big data problems. This paper presents two extensions of the regular Bayesian horseshoe: (i) the grouped Bayesian horseshoe and (ii) the hierarchical Bayesian grouped horseshoe. The advantages of the proposed methods are their flexibility in handling grouped variables through extra shrinkage parameters at the group and within-group levels. We apply the proposed methods to the important class of additive models where group structures naturally exist, and we demonstrate that the grouped hierarchical Bayesian horseshoe has promising performance on both simulated and real data.

Appendix: Full Conditional Distributions

The hierarchical specification of the complete model of the HBGHS is given in (7). By using the decomposition [10], the hierarchical representation becomes:

$$\begin{aligned} \begin{aligned} \mathbf {y}|\mathbf {X},\varvec{\beta },\sigma ^2&\sim \mathcal {N}(\mathbf {X}\varvec{\beta },\sigma ^2\mathbbm {1}_n) \\ \varvec{\beta }|\sigma ^2,\tau ^2,\lambda _1,\cdots ,\lambda _G,\delta _1,\cdots ,\delta _p&\sim \mathcal {N}(\varvec{0},\sigma ^2\tau ^2\mathbf {D}_{\varvec{\lambda }}\mathbf {D}_{\varvec{\delta }}) \\ \mathbf {D}_{\varvec{\lambda }}=\text {diag}(\lambda _{1}^2\mathbf {I}_{s_1},\cdots ,\lambda _{G}^2\mathbf {I}_{s_G})&,\quad \mathbf {D}_{\varvec{\delta }}=\text {diag}(\delta _1^2,\cdots ,\delta _p^2) \\ \lambda _g^2|t_g \sim \mathcal {IG}\left( \frac{1}{2},\frac{1}{t_g}\right)&,\ t_g \sim \mathcal {IG} \left( \frac{1}{2},1\right) ,\ g =1,\cdots ,G\\ \delta _j^2|c_j \sim \mathcal {IG}\left( \frac{1}{2},\frac{1}{c_j} \right)&, \ c_j \sim \mathcal {IG}\left( \frac{1}{2},1\right) , \ j=1,\cdots ,p\\ \tau ^2|v \sim \mathcal {IG}\left( \frac{1}{2},\frac{1}{v}\right)&,\ v \sim \mathcal {IG}\left( \frac{1}{2},1\right) \\ \sigma ^2&\sim \frac{1}{\sigma ^2}d\sigma ^2. \end{aligned} \end{aligned}$$

The full conditional distributions of $\varvec{\beta }$, $\sigma ^2$, $\lambda _1^2,\cdots ,\lambda _G^2$, $\delta _1^2,\cdots ,\delta _p^2$, $\tau$ are:

$$\begin{aligned} \begin{aligned}&\varvec{\beta }|\sigma ^2,\tau ^2,\lambda _1^2,\cdots ,\lambda _G^2,\delta _1^2,\cdots ,\delta _p^2 \sim \mathcal {N} \left( \mathbf {A}^{-1}\mathbf {X}^T\mathbf {y},\sigma ^2\mathbf {A}^{-1}\right) , \quad \mathbf {A}= \mathbf {X}^T \mathbf {X}+(\tau ^2\mathbf {D}_{\varvec{\lambda }}\mathbf {D}_{\varvec{\delta }})^{-1}\\&\sigma ^2|\varvec{\beta },\tau ^2,\lambda _1^2,\cdots ,\lambda _G^2,\delta _1^2,\cdots ,\delta _p^2 \sim \mathcal {IG}\left( \frac{n-1+p}{2},\frac{(\mathbf {y}-\mathbf {X}\varvec{\beta })^T(\mathbf {y}-\mathbf {X}\varvec{\beta })+\varvec{\beta }^T(\tau ^2\mathbf {D}_{\varvec{\lambda }}\mathbf {D}_{\varvec{\delta }})^{-1}\varvec{\beta }}{2}\right) \\&\lambda _g^2|\varvec{\beta },\sigma ^2,\tau ^2,t_g,\delta _1^2,\cdots ,\delta _p^2 \sim \mathcal {IG}\left( \frac{s_g+1}{2},\frac{\varvec{\beta }_g^T(\mathbf {D}_{\varvec{\delta }_g})^{-1}\varvec{\beta }_g}{2\sigma ^2\tau ^2}+\frac{1}{t_g}\right) ,\ t_g|\lambda _g^2 \sim \mathcal {IG}\left( 1,\frac{1}{\lambda _g^2}+1\right) \\&\delta _j^2 |\varvec{\beta },\sigma ^2,\tau ^2,\lambda _1^2,\cdots ,\lambda _G^2,c_j \sim \mathcal {IG}\left( 1,\frac{\beta _j^2}{2\sigma ^2\tau ^2\lambda _{gj}^2}+\frac{1}{c_j} \right) ,\ c_j|\delta _j^2 \sim \mathcal {IG} \left( 1,\frac{1}{\delta _j^2}+1\right) \\&\tau ^2|\varvec{\beta },\sigma ^2,\tau ^2,\lambda _1^2,\cdots ,\lambda _G^2,\delta _1^2,\cdots ,\delta _p^2,v \sim \mathcal {IG}\left( \frac{p+1}{2},\frac{\varvec{\beta }^T(\mathbf {D}_{\varvec{\lambda }}\mathbf {D}_{\varvec{\delta }})^{-1}\varvec{\beta }}{2\sigma ^2}+\frac{1}{v}\right) \\&v|\tau ^2 \sim \mathcal {IG}\left( 1,\frac{1}{\tau ^2}+1\right) . \end{aligned} \end{aligned}$$