Bayesian Variable Selection

  • Matthew SuttonEmail author
Part of the Lecture Notes in Mathematics book series (LNM, volume 2259)


In this chapter we survey Bayesian approaches for variable selection and model choice in regression models. We explore the methodological developments and computational approaches for these methods. In conclusion we note the available software for their implementation.



The author would like to acknowledge the Australian Research Council Centre of Excellence in Mathematical and Statistical Frontiers for funding.


  1. 1.
    T.J. Mitchell, J.J. Beauchamp, Bayesian variable selection in linear regression. J. Am. Stat. Assoc. 83(404), 1023–1032 (1988)MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    M. Barbieri, J.O. Berger, E.I. George, V. Rockova, The median probability model and correlated variables. arXiv:1807.08336 (2020)Google Scholar
  3. 3.
    M.M. Barbieri, J.O. Berger, Optimal predictive model selection. Ann. Stat. 32(3), 870–897 (2004)MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    F. Liang, Q. Song, K. Yu, Bayesian subset modeling for high-dimensional generalized linear models. J. Am. Stat. Assoc. 108(502), 589–606 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  5. 5.
    W. Jiang, Bayesian variable selection for high dimensional generalized linear models: convergence rates of the fitted densities. Ann. Stat. 35(4), 1487–1511 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    E.I. George, R.E. McCulloch, Variable selection via Gibbs sampling. J. Am. Stat. Assoc. 88(423), 881–889 (1993)CrossRefGoogle Scholar
  7. 7.
    M. Smith, R. Kohn, A Bayesian approach to nonparametric bivariate regression. J. Am. Stat. Assoc. 92(440), 1522–1535 (1997)MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    I. Castillo, J. Schmidt-Hieber, A. van der Vaart, Bayesian linear regression with sparse priors. Ann. Stat. 43(5), 1986–2018 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  9. 9.
    J.G. Scott, J.O. Berger, Bayes and empirical-bayes multiplicity adjustment in the variable-selection problem. Ann. Stat. 38(5), 2587–2619 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    F. Li, N.R. Zhang, Bayesian variable selection in structured high-dimensional covariate spaces with applications in genomics. J. Am. Stat. Assoc. 105(491), 1202–1214 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    M.E.J. Newman, G.T. Barkema, Monte Carlo Methods in Statistical Physics (Clarendon Press, Oxford, 1999)zbMATHGoogle Scholar
  12. 12.
    E.E. Leamer, Specification Searches: Ad hoc Inference with Nonexperimental Data, vol. 53 (Wiley, Hoboken, 1978)zbMATHGoogle Scholar
  13. 13.
    I. Castillo, R. Mismer, Empirical bayes analysis of spike and slab posterior distributions. Electron. J. Stat. 12, 3953–4001 (2018)MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    R.B. O’Hara, M.J. Sillanpää, A review of Bayesian variable selection methods: what, how and which. Bayesian Anal. 4(1), 85–117 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  15. 15.
    L. Kuo, B. Mallick, Variable selection for regression models. Sankhyā Indian J. Stat. Ser. B (1960–2002) 60(1), 65–81 (1998)Google Scholar
  16. 16.
    H. Chipman, E.I. George, R.E. McCulloch, The Practical Implementation of Bayesian Model Selection. Lecture Notes–Monograph Series, vol. 38 (Institute of Mathematical Statistics, Beachwood, 2001), pp. 65–116. Google Scholar
  17. 17.
    P. Dellaportas, J.J. Forster, I. Ntzoufras, Bayesian variable selection using the Gibbs sampler. BIOSTATISTICS-BASEL- 5, 273–286 (2000)MathSciNetzbMATHGoogle Scholar
  18. 18.
    B.P. Carlin, S. Chib, Bayesian model choice via Markov chain Monte Carlo methods. J. R. Stat. Soc. Ser. B Stat. Methodol. 57(3), 473–484 (1995)zbMATHGoogle Scholar
  19. 19.
    N.N. Narisetty, X. He, Bayesian variable selection with shrinking and diffusing priors. Ann. Stat. 42(2), 789–817 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  20. 20.
    H. Ishwaran, J.S. Rao, Detecting differentially expressed genes in microarrays using Bayesian model selection. J. Am. Stat. Assoc. 98(462), 438–455 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  21. 21.
    L. Fahrmeir, T. Kneib, S. Konrath, Bayesian regularisation in structured additive regression: a unifying perspective on shrinkage, smoothing and predictor selection. Stat. Comput. 20(2), 203–219 (2010)MathSciNetCrossRefGoogle Scholar
  22. 22.
    V. Ročková, E.I. George, The spike-and-slab lasso. J. Am. Stat. Assoc. 113(521), 431–444 (2018)MathSciNetzbMATHCrossRefGoogle Scholar
  23. 23.
    V.E. Johnson, D. Rossell, Bayesian model selection in high-dimensional settings. J. Am. Stat. Assoc. 107(498), 649–660 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  24. 24.
    D. Rossell, D. Telesca, Non-local priors for high-dimensional estimation. J. Am. Stat. Assoc. 112(517), 254–265 (2017)CrossRefGoogle Scholar
  25. 25.
    A. Nikooienejad, W. Wang, V.E. Johnson, Bayesian variable selection for binary outcomes in high-dimensional genomic studies using non-local priors. Bioinformatics 32(9), 1338–1345 (2016)CrossRefGoogle Scholar
  26. 26.
    R. Tibshirani, Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 58(1), 267–288 (1996)MathSciNetzbMATHGoogle Scholar
  27. 27.
    J. Fan, J. Lv, A selective overview of variable selection in high dimensional feature space. Stat. Sin. 20(1), 101–148 (2010)MathSciNetzbMATHGoogle Scholar
  28. 28.
    N.G. Polson, J.G. Scott, Local shrinkage rules, lévy processes and regularized regression. J. R. Stat. Soc. Ser. B Stat. Methodol. 74(2), 287–311 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  29. 29.
    J.E. Griffin, P.J. Brown, Inference with normal-gamma prior distributions in regression problems. Bayesian Anal. 5(1), 171–188 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  30. 30.
    C.M. Carvalho, N.G. Polson, J.G. Scott, The horseshoe estimator for sparse signals. Biometrika 97(2), 465–480 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  31. 31.
    A. Armagan, D.B. Dunson, J. Lee, Generalized double pareto shrinkage. Stat. Sin. 23(1), 119–143 (2013)MathSciNetzbMATHGoogle Scholar
  32. 32.
    A. Bhattacharya, D. Pati, N.S. Pillai, D.B. Dunson, Dirichlet–laplace priors for optimal shrinkage. J. Am. Stat. Assoc. 110(512), 1479–1490 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  33. 33.
    A. Bhadra, J. Datta, N.G. Polson, B. Willard, The horseshoe+ estimator of ultra-sparse signals. Bayesian Anal. 12(4), 1105–1131 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
  34. 34.
    P. Ghosh, X. Tang, M. Ghosh, A. Chakrabarti, Asymptotic properties of bayes risk of a general class of shrinkage priors in multiple hypothesis testing under sparsity. Bayesian Anal. 11(3), 753–796 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  35. 35.
    R. Bai, M. Ghosh, High-dimensional multivariate posterior consistency under global–local shrinkage priors. J. Multivar. Anal. 167, 157–170 (2018)MathSciNetzbMATHCrossRefGoogle Scholar
  36. 36.
    S. van der Pas, B. Szabó, A. van der Vaart, Uncertainty quantification for the horseshoe (with discussion). Bayesian Anal. 12(4), 1221–1274 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
  37. 37.
    P.R. Hahn, C.M. Carvalho, Decoupling shrinkage and selection in Bayesian linear models: a posterior summary perspective. J. Am. Stat. Assoc. 110(509), 435–448 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  38. 38.
    D. Madigan, J. York, D. Allard, Bayesian graphical models for discrete data. Int. Stat. Rev./Rev. Int. de Stat. 63(2), 215–232 (1995)zbMATHGoogle Scholar
  39. 39.
    C. Hans, A. Dobra, M. West, Shotgun stochastic search for “large p” regression. J. Am. Stat. Assoc. 102(478), 507–516 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  40. 40.
    F. Liang, W.H. Wong, Evolutionary monte carlo: applications to C p model sampling and change point problem. Stat. Sin. 10(2), 317–342 (2000)zbMATHGoogle Scholar
  41. 41.
    L. Bottolo, S. Richardson, Evolutionary stochastic search for Bayesian model exploration. Bayesian Anal. 5(3), 583–618 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  42. 42.
    G. Zanella, G. Roberts, Scalable importance tempering and Bayesian variable selection. J. R. Statist. Soc. B 81, 489–517 (2019)MathSciNetzbMATHCrossRefGoogle Scholar
  43. 43.
    R Core Team, R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, Vienna, 2013)Google Scholar
  44. 44.
    I. Ntzoufras, Gibbs variable selection usingbugs. J. Stat. Softw. 7(7), 1–19 (2002)CrossRefGoogle Scholar
  45. 45.
    D.J. Lunn, A. Thomas, N. Best, D. Spiegelhalter, Winbugs-a Bayesian modelling framework: concepts, structure, and extensibility. Stat. Comput. 10(4), 325–337 (2000)CrossRefGoogle Scholar
  46. 46.
    M. Plummer, et al., JAGS: A program for analysis of Bayesian graphical models using gibbs sampling, in Proceedings of the 3rd International Workshop on Distributed Statistical Computing, vol. 124 (2003)Google Scholar
  47. 47.
    B. Carpenter, A. Gelman, M.D. Hoffman, D. Lee, B. Goodrich, M. Betancourt, M. Brubaker, J. Guo, P. Li, A. Riddell, Stan: A probabilistic programming language. J. Stat. Softw. 76(1), 1–32 (2017)CrossRefGoogle Scholar
  48. 48.
    J. Piironen, A. Vehtari, Projection predictive model selection for gaussian processes, in 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP) (2016), pp. 1–6Google Scholar
  49. 49.
    A. Forte, G. Garcia-Donato, M. Steel, Methods and tools for Bayesian variable selection and model averaging in normal linear regression. Int. Stat. Rev./Rev. Int. de Stat. 86(2), 237–258 (2018)MathSciNetCrossRefGoogle Scholar
  50. 50.
    G. Csárdi, pkgsearch: Search CRAN R Packages. R package version 2.0.1. (2018).
  51. 51.
    H. Ishwaran, U.B. Kogalur, J.S. Rao, spikeslab: prediction and variable selection using spike and slab regression. R J. 2, 68–73 (2010)CrossRefGoogle Scholar
  52. 52.
    P. Carbonetto, M. Stephens, Scalable variational inference for Bayesian variable selection in regression, and its accuracy in genetic association studies. Bayesian Anal. 7, 73–108 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  53. 53.
    D. Rossell, J.D. Cook, D. Telesca, P. Roebuck, mombf: moment and inverse moment bayes factors. R Package Version 1. 0, vol. 3 (2008)Google Scholar

Copyright information

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Queensland University of TechnologyBrisbaneAustralia

Personalised recommendations