Robustness of Kernel Based Regression: A Comparison of Iterative Weighting Schemes

  • Kris De Brabanter
  • Kristiaan Pelckmans
  • Jos De Brabanter
  • Michiel Debruyne
  • Johan A. K. Suykens
  • Mia Hubert
  • Bart De Moor
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5768)


It has been shown that Kernel Based Regression (KBR) with a least squares loss has some undesirable properties from robustness point of view. KBR with more robust loss functions, e.g. Huber or logistic losses, often give rise to more complicated computations. In this work the practical consequences of this sensitivity are explained, including the breakdown of Support Vector Machines (SVM) and weighted Least Squares Support Vector Machines (LS-SVM) for regression. In classical statistics, robustness is improved by reweighting the original estimate. We study the influence of reweighting the LS-SVM estimate using four different weight functions. Our results give practical guidelines in order to choose the weights, providing robustness and fast convergence. It turns out that Logistic and Myriad weights are suitable reweighting schemes when outliers are present in the data. In fact, the Myriad shows better performance over the others in the presence of extreme outliers (e.g. Cauchy distributed errors). These findings are then illustrated on toy example as well as on a real life data sets.


Least Squares Support Vector Machines Robustness Kernel methods Reweighting 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Edgeworth, F.Y.: On Observations Relating to Several Quantities. Hermathena 6, 279–285 (1887)Google Scholar
  2. 2.
    Tukey, J.W.: A survey of sampling from contaminated distributions. In: Olkin, I. (ed.) Contributions to Probability and Statistics, pp. 448–485. Stanford University Press, Stanford (1960)Google Scholar
  3. 3.
    Huber, P.J.: Robust Estimation of a Location Parameter. Ann. Math. Stat. 35, 73–101 (1964)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Hampel, F.R.: A General Definition of Qualitative Robustness. Ann. Math. Stat. 42, 1887–1896 (1971)CrossRefzbMATHGoogle Scholar
  5. 5.
    Huber, P.J.: Robust Statistics. Wiley, Chichester (1981)CrossRefzbMATHGoogle Scholar
  6. 6.
    Rousseeuw, P.J., Leroy, A.M.: Robust Regression and Outlier Detection. Wiley, Chichester (2003)zbMATHGoogle Scholar
  7. 7.
    Maronna, R., Martin, D., Yohai, V.: Robust Statistics. Wiley, Chichester (2006)CrossRefzbMATHGoogle Scholar
  8. 8.
    Hampel, F.R., Ronchetti, E.M., Rousseeuw, P.J., Stahel, W.A.: Robust Statistics: The Approach Beased on Influence Functions. Wiley, Chichester (1986)zbMATHGoogle Scholar
  9. 9.
    Christmann, A., Steinwart, I.: Consistency and Robustness of Kernel Based Regression in Convex Risk Minimization. Bernoulli 13(3), 799–819 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Debruyne, M., Christmann, A., Hubert, M., Suykens, J.A.K.: Robustness and Stability of Reweighted Kernel Based Regression. Technical Report 06-09, Department of Mathematics, K.U.Leuven, Leuven, Belgium (2008)Google Scholar
  11. 11.
    Debruyne, M., Hubert, M., Suykens, J.A.K.: Model Selection in Kernel Based Regression using the Influence Function. J. Mach. Learn. Res. 9, 2377–2400 (2008)MathSciNetzbMATHGoogle Scholar
  12. 12.
    Suykens, J.A.K., Van Gestel, T., De Brabanter, J., De Moor, B., Vandewalle, J.: Least Squares Support Vector Machines. World Scientific, Singapore (2002)CrossRefzbMATHGoogle Scholar
  13. 13.
    Vapnik, V.N.: Statistical Learning Theory. Wiley, Chichester (1999)zbMATHGoogle Scholar
  14. 14.
    Suykens, J.A.K., De Brabanter, J., Lukas, L., Vandewalle, J.: Weighted Least Squares Support Vector Machines: Robustness and Sparse Approximation. Neurocomputing 48(1-4), 85–105 (2002)CrossRefzbMATHGoogle Scholar
  15. 15.
    Arce, G.R.: Nonlinear Signal Processing: A Statistical Approach. Wiley, Chichester (2005)zbMATHGoogle Scholar
  16. 16.
    Gonzalez, J.G., Arce, G.R.: Weighted Myriad Filters: A Robust Filtering Framework derived from Alpha-Stable Distributions. In: Proceedings of the 1996 IEEE Conference one Acoustics (1996)Google Scholar
  17. 17.
    Hubert, M., Rousseeuw, P.J., Vanden Branden, K.: ROBPCA: a New Approach to Robust Principal Components Analysis. Technometrics 47, 64–79 (2005)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Kris De Brabanter
    • 1
  • Kristiaan Pelckmans
    • 1
  • Jos De Brabanter
    • 1
    • 2
  • Michiel Debruyne
    • 3
  • Johan A. K. Suykens
    • 1
  • Mia Hubert
    • 4
  • Bart De Moor
    • 1
  1. 1.KULeuven, ESAT-SCDLeuvenBelgium
  2. 2.Departement Ind. Ing.KaHo Sint-Lieven (Associatie K.U.Leuven)Gent
  3. 3.Department of Mathematics and Computer ScienceUniversiteit AntwerpenAntwerpenBelgium
  4. 4.Department of StatisticsKULeuvenLeuvenBelgium

Personalised recommendations