Robustness of Kernel Based Regression: A Comparison of Iterative Weighting Schemes
It has been shown that Kernel Based Regression (KBR) with a least squares loss has some undesirable properties from robustness point of view. KBR with more robust loss functions, e.g. Huber or logistic losses, often give rise to more complicated computations. In this work the practical consequences of this sensitivity are explained, including the breakdown of Support Vector Machines (SVM) and weighted Least Squares Support Vector Machines (LS-SVM) for regression. In classical statistics, robustness is improved by reweighting the original estimate. We study the influence of reweighting the LS-SVM estimate using four different weight functions. Our results give practical guidelines in order to choose the weights, providing robustness and fast convergence. It turns out that Logistic and Myriad weights are suitable reweighting schemes when outliers are present in the data. In fact, the Myriad shows better performance over the others in the presence of extreme outliers (e.g. Cauchy distributed errors). These findings are then illustrated on toy example as well as on a real life data sets.
KeywordsLeast Squares Support Vector Machines Robustness Kernel methods Reweighting
Unable to display preview. Download preview PDF.
- 1.Edgeworth, F.Y.: On Observations Relating to Several Quantities. Hermathena 6, 279–285 (1887)Google Scholar
- 2.Tukey, J.W.: A survey of sampling from contaminated distributions. In: Olkin, I. (ed.) Contributions to Probability and Statistics, pp. 448–485. Stanford University Press, Stanford (1960)Google Scholar
- 10.Debruyne, M., Christmann, A., Hubert, M., Suykens, J.A.K.: Robustness and Stability of Reweighted Kernel Based Regression. Technical Report 06-09, Department of Mathematics, K.U.Leuven, Leuven, Belgium (2008)Google Scholar
- 16.Gonzalez, J.G., Arce, G.R.: Weighted Myriad Filters: A Robust Filtering Framework derived from Alpha-Stable Distributions. In: Proceedings of the 1996 IEEE Conference one Acoustics (1996)Google Scholar