Abstract
This paper studies the variable selection problem in high dimensional linear regression, where there are multiple response vectors, and they share the same or similar subsets of predictor variables to be selected from a large set of candidate variables. In the literature, this problem is called multi-task learning, support union recovery or simultaneous sparse coding in different contexts. In this paper, we propose a Bayesian method for solving this problem by introducing two nested sets of binary indicator variables. In the first set of indicator variables, each indicator is associated with a predictor variable or a regressor, indicating whether this variable is active for any of the response vectors. In the second set of indicator variables, each indicator is associated with both a predicator variable and a response vector, indicating whether this variable is active for the particular response vector. The problem of variable selection can then be solved by sampling from the posterior distributions of the two sets of indicator variables. We develop the Gibbs sampling algorithm for posterior sampling and demonstrate the performances of the proposed method for both simulated and real data sets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Barbieri, M., Berger, J.O.: Optimal predictive model selection. Annals of Statistics 32, 870–897 (2004)
Chen, R.-B., Chu, C.H., Lai, T.Y., Wu, Y.N.: Stochastic Matching Pursuit for Bayesian Variable Selection. Statistics and Computing 21, 247–289 (2011)
Chen, R.B., Chu, C.H., Yuan, S., Wu, Y.N.: Bayesian sparse group selection (preprint 2014)
Dellaportas, P., Forster, J.J., Ntzoufras, I.: Bayesian variable selection using the gibbs sampler. In: Dey, D.K., Ghosh, S., Mallick, B. (eds.) Generalized Linear Models: A Bayesian Perspective, pp. 271–286. Marcel Dekker, New York (2000)
Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association 96, 1348–1360 (2001)
Farcomeni, A.: Bayesian constrained variable selection. Statistica Sinica 20(3), 1043–1062 (2010)
George, E.I., McCulloch, R.E.: Variable Selection via Gibbs Sampling. Journal of the American Statistical Association 88, 881–889 (1993)
Liu, J., Ji, S., Ye, J.: SLEP: Sparse Learning with Efficient Projections. Arizona State University (2009)
Lounici, K., Pontil, M., Tsybakov, A.B., van de Geer, S.: Taking advantage of sparsity in multi-task learning. In: Proceedings of the 22nd Conference on Information Theory, pp. 73–82 (2009)
Obozinski, G., Wainwright, M.J., Jordan, M.I.: Support union recovery in high-dimensional multivariate regression. The Annals of Statistics 39(1), 1–47 (2011)
Olshausen, B., Field, D.: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 (1996)
Raman, S., Fuchs, T.J., Wild, P.J., Dahl, E., Roth, V.: The Bayesian group-lasso for analyzing contingency tables. In: Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, pp. 881–888. ACM, New York (2009)
Simon, N., Friedman, J., Hastie, T., Tibshirani, R.: A sparse group lasso. Journal of Computational and Graphical Statistics 22(2), 231–245 (2013)
Tibshirani, R.: Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society. Series B (Methodological), 267–288 (1996)
Tipping, M.E.: Sparse Bayesian Learning and the Relevance Vector Machine. Journal of Machine Learning Research 1, 211–244 (2001)
Tropp, J.A.: Just relax: convex programming methods for identifying sparse signals in noise. IEEE Transactions on Information Theory 52(3), 1030–1051 (2006)
Yuan, M., Lin, Y.: Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. Ser. B Stat. Methodol. 68(1), 49–67 (2006)
Zhang, C.-H.: Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics 38(2), 894–942 (2010)
Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 67(2), 301–320 (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Chen, WP., Wu, Y.N., Chen, RB. (2014). Bayesian Variable Selection for Multi-response Linear Regression. In: Cheng, SM., Day, MY. (eds) Technologies and Applications of Artificial Intelligence. TAAI 2014. Lecture Notes in Computer Science(), vol 8916. Springer, Cham. https://doi.org/10.1007/978-3-319-13987-6_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-13987-6_8
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-13986-9
Online ISBN: 978-3-319-13987-6
eBook Packages: Computer ScienceComputer Science (R0)