Abstract
We study the properties of the eigenvalues of Gram matrices in a non-asymptotic setting. Using local Rademacher averages, we provide data-dependent and tight bounds for their convergence towards eigenvalues of the corresponding kernel operator. We perform these computations in a functional analytic framework which allows to deal implicitly with reproducing kernel Hilbert spaces of infinite dimension. This can have applications to various kernel algorithms, such as Support Vector Machines (SVM). We focus on Kernel Principal Component Analysis (KPCA) and, using such techniques, we obtain sharp excess risk bounds for the reconstruction error. In these bounds, the dependence on the decay of the spectrum and on the closeness of successive eigenvalues is made explicit.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Anderson, T.W.: Asymptotic theory for principal component analysis. Ann. Math. Stat. 34, 122–148 (1963)
Bartlett, P., Bousquet, O. Mendelson, S.: Localized Rademacher complexities (2003) (submitted) available at http://www.kyb.mpg.de/publications/pss/ps2000.ps
Bartlett, P.L., Mendelson, S.: Rademacher and Gaussian complexities: risk bounds and structural results. Journal of Machine Learning Research 3, 463–482 (2002)
Baxendale, P.: Gaussian measures on function spaces. Amer. J. Math. 98, 891–952 (1976)
Besse, P.: Etude descriptive d’un pocessus; approximation, interpolation. PhD thesis, Université de Toulouse (1979)
Besse, P.: Approximation spline de l’analyse en composantes principales d’une variable aléatoire hilbertienne. Ann. Fac. Sci. Toulouse (Math.) 12(5), 329–349 (1991)
Boucheron, S., Lugosi, G., Massart, P.: A sharp concentration inequality with applications. Random Structures and Algorithms 16, 277–292 (2000)
Dauxois, J., Pousse, A.: Les analyses factorielles en calcul des probabilités et en statistique: essai d’étude synthétique. PhD thesis
Dunford, N., Schwartz, J.T.: Linear Operators Part II: Spectral Theory, Self Adjoint Operators in Hilbert Space. Number VII in Pure and Applied Mathematics. John Wiley & Sons, New York (1963)
Hoeffding, W.: Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association 58, 13–30 (1963)
Koltchinskii, V.: Asymptotics of spectral projections of some random matrices approximating integral operators. Progress in Probability 43, 191–227 (1998)
Koltchinskii, V., Giné, E.: Random matrix approximation of spectra of integral operators. Bernoulli 6(1), 113–167 (2000)
Massart, P.: Some applications of concentration inequalities to statistics. Annales de la Faculté des Sciencies de Toulouse, IX pp. 245–303 (2000)
Mendelson, S.: Estimating the performance of kernel classes. Journal of Machine Learning Research 4, 759–771 (2003)
Ramsay, J.O., Dalzell, C.J.: Some tools for functional data analysis. Journal of the Royal Statistical Society, Series B 53(3), 539–572 (1991)
Schölkopf, B., Smola, A.J., Müller, K.-R.: Kernel principal component analysis. In: Schölkopf, B., Burges, C.J.C., Smola, A.J. (eds.) Advances in Kernel Methods - Support Vector Learning, pp. 327–352. MIT Press, Cambridge (1999); Short version appeared in Neural Computation 10, 1299–1319 (1998)
Shawe-Taylor, J., Williams, C., Cristianini, N., Kandola, J.: Eigenspectrum of the gram matrix and its relationship to the operator eigenspectrum. In: ALT 2002. LNCS, vol. 2533, pp. 23–40. Springer-, Heidelberg (2002), http://www.support-vector.net/papers/eigenspectrum.pdf
Torki, M.: Etude de la sensibilité de toutes les valeurs propres non nulles d’un opérateur compact autoadjoint. Technical Report LAO97-05, Université Paul Sabatier (1997), Available at http://mip.ups-tlse.fr/publi/rappLAO/97.05.ps.gz
Williamson, R.C., Shawe-Taylor, J., Schölkopf, B., Smola, A.J.: Sample-based generalization bounds. IEEE Transactions on Information Theory (1999) (submitted) Also: NeuroCOLT Technical Report NC-TR-99-055
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zwald, L., Bousquet, O., Blanchard, G. (2004). Statistical Properties of Kernel Principal Component Analysis. In: Shawe-Taylor, J., Singer, Y. (eds) Learning Theory. COLT 2004. Lecture Notes in Computer Science(), vol 3120. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-27819-1_41
Download citation
DOI: https://doi.org/10.1007/978-3-540-27819-1_41
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-22282-8
Online ISBN: 978-3-540-27819-1
eBook Packages: Springer Book Archive