Kernel Methods

Abstract

In this chapter we describe a class of regression techniques that achieve flexibility in estimating the regression function f(X) over the domain ℝ^P by fitting a different but simple model separately at each query point x ₀. This is done by using only those observations close to the target point x ₀ to fit the simple model, and in such a way that the resulting estimated function $\hat f\left( X \right)$(X) is smooth in ℝ^P. This localization is achieved via a weighting function or kernel K _⋋(x ₀, x _i ), which assigns a weight to x _i based on its distance from x ₀. The kernels K _⋋ are typically indexed by a parameter ⋋ that dictates the width of the neighborhood. These memory-based methods require in principle little or no training; all the work gets done at evaluation time. The only parameter that needs to be determined from the training data is ⋋. The model, however, is the entire training data set.