Abstract
The automatic relevance determination (ARD) shows good performance in many applications. Recently, it has been applied to brain current estimation with the variational method. Although people who use the ARD tend to pay attention to one benefit of the ARD, sparsity, we, in this paper, focus on another benefit, generalization. In this paper, we clarify the generalization error of the ARD in the case that a class of prior distributions is used, and show that good generalization is caused by singularities of the ARD. Sparsity is not observed in that case, however, the mechanism that the singularities provide good generalization implies the mechanism that they also provide sparsity.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
MacKay, D.J.C.: Bayesian Non-linear Modeling for the Energy Prediction Competition. ASHRAE Transactions 100, 1053–1062 (1994)
Neal, R.M.: Bayesian Learning for Neural Networks. Springer, Heidelberg (1996)
Hinton, G.E., van Camp, D.: Keeping Neural Networks Simple by Minimizing the Description Length of the Weights. In: Proc. of COLT, pp. 5–13 (1993)
Attias, H.: Inferring Parameters and Structure of Latent Variable Models by Variational Bayes. In: Proc. of UAI (1999)
Sato, M., Yoshioka, T., Kajihara, S., Toyama, K., Goda, N., Doya, K., Kawato, M.: Hierarchical Bayesian Estimation for MEG inverse problem. Neuro Image 23, 806–826 (2004)
Osako, M., Yamashita, O., Hiroe, N., Sato, M.: Verification of Hierarchical Bayesian Estimation Combining MEG and fMRI: A Motor Task Analysis (in Japanese). In: Technical Report of IEICE, Tokyo, Japan, vol. NC2006-130, pp. 73–78 (2007)
Wipf, D., Ramirez, R., Palmer, J., Makeig, S., Rao, B.: Analysis of Empirical Bayesian Methods for Neuroelectromagnetic Source Localization. In: Advances in NIPS, vol. 19 (2006)
Watanabe, S.: Algebraic Analysis for Nonidentifiable Learning Machines. Neural Computation 13, 899–933 (2001)
Nakajima, S., Watanabe, S.: Variational Bayes Solution of Linear Neural Networks and its Generalization Performance. Neural Computation 19, 1112–1153 (2007)
James, W., Stein, C.: Estimation with Quadratic Loss. In: Proc. of the 4th Berkeley Symp. on Math. Stat. and Prob., pp. 361–379 (1961)
Efron, B., Morris, C.: Stein’s Estimation Rule and its Competitors—an Empirical Bayes Approach. J. of Am. Stat. Assoc. 68, 117–130 (1973)
Nakajima, S., Watanabe, S.: Analytic Solution of Hierarchical Variational Bayes in Linear Inverse Problem. In: Kollias, S., Stafylopatis, A., Duch, W., Oja, E. (eds.) ICANN 2006. LNCS, vol. 4132, pp. 240–249. Springer, Heidelberg (2006)
Sato, M.: Online Model Selection Based on the Variational Bayes. Neural Computation 13, 1649–1681 (2001)
Hamalainen, M., Hari, R., Ilmoniemi, R.J, Knuutila, J., Lounasmaa, O.V.: Magnetoencephalography — Theory, Instrumentation, and Applications to Noninvasive Studies of the Working Human Brain. Rev. Modern Phys. 65, 413–497 (1993)
Blankertz, B., Dornhege, G., Krauledat, M., Curio, G., Muller, K.R.: The Non-invasive Berlin Brain-Computer Interface: Fast Acquisition of Effective Performance in Untrained Subjects (2007) (to appear in Neuro Image)
Watanabe, S., Amari, S.: Learning Coefficients of Layered Models When the True Distribution Mismatches the Singularities. Neural Computation 15, 1013–1033 (2003)
Watanabe, S.: Algebraic Information Geometry for Learning Machines with Singularities. In: Advances in NIPS, vol. 13, pp. 329–336 (2001)
Stein, C.: Estimation of the Mean of a Multivariate Normal Distribution. Annals of Statistics 9, 1135–1151 (1981)
Wang, B., Titterington, D.M.: Convergence and Asymptotic Normality of Variational Bayesian Approximations for Exponential Family Models with Missing Values. In: Proc. of UAI, Banff, Canada, pp. 577–584 (2004)
Watanabe, K., Watanabe, S.: Stochastic Complexities of Gaussian Mixtures in Variational Bayesian Approximation. Journal of Machine Learning Research 7, 625–644 (2006)
Nakajima, S., Watanabe, S.: Generalization Error and Free Energy of Variational Bayes Approach of Linear Neural Networks. In: Proc. of ICONIP, Taipei, Taiwan, pp. 55–60 (2005)
Barber, D., Chiappa, S.: Unified Inference for Variational Bayesian Linear Gaussian State-Space Models. In: Advances in NIPS, vol. 19 (2006)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Nakajima, S., Watanabe, S. (2007). Generalization Error of Automatic Relevance Determination. In: de Sá, J.M., Alexandre, L.A., Duch, W., Mandic, D. (eds) Artificial Neural Networks – ICANN 2007. ICANN 2007. Lecture Notes in Computer Science, vol 4668. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-74690-4_1
Download citation
DOI: https://doi.org/10.1007/978-3-540-74690-4_1
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-74689-8
Online ISBN: 978-3-540-74690-4
eBook Packages: Computer ScienceComputer Science (R0)