Statistical Learning Theory and Kernel-Based Methods

Abstract

The basics of kernel methods and their position in the generalized data-driven fault diagnostic framework are reviewed. The review starts out with statistical learning theory, covering concepts such as loss functions, overfitting and structural and empirical risk minimization. This is followed by linear margin classifiers, kernels and support vector machines. Transductive support vector machines are discussed and illustrated by way of an example related to multivariate image analysis of coal particles on conveyor belts. Finally, unsupervised kernel methods, such as kernel principal component analysis, are considered in detail, analogous to the application of linear principal component analysis in multivariate statistical process control. Fault diagnosis in a simulated nonlinear system by the use of kernel principal component analysis is included as an example to illustrate the concepts.

Nomenclature

Symbol	Description
$\tilde{\mathrm{x}}$	Point satisfying all inequality constraints of an optimization problem
${K_{ij }}$	Element of Gram matrix in ith row and jth column
${N_{SV }}$	Number of support vectors in a training data set
${R_e}(f)$	Empirical risk of overfitting function f
${f_0}(\mathbf{x})$	Objective function in an optimization problem
${f_0}(\tilde{\mathbf{x}} )$	Objective function value at point where all inequality constraints of optimization problem are satisfied
$f_0^{*}$	Optimal value of objective function
${m_G}(\mathbf{x},y)$	Geometrical margin, i.e. the distance of a point x with associated label y from a separating hyperplane defined by parameters w and b
${\xi_i}$	ith slack variable of an optimization problem
C p *	Optimal parameter in a kernel function
C q	Parameter in a kernel function
D	Diameter of sphere enclosing a set of (training) data
D s	Diameter of smallest sphere enclosing a set of (training) data
$\mathcal{F}$	Function space, class of functions
f*	Function associated with lowest risk bound
h	Capacity parameter of a model, such as VC dimension
$\mathcal{H}$	Feature space
K	Gram matrix
L(y, f(x))	Loss function
M	Dimensionality of input space
m	Margin or shortest distance between two separating hyperplanes
P(x, y)	Joint probability distribution between x and y
Q	Arbitrary number of parameters
δ	Confidence
ξ	Vector of slack variables
ρ	Bias defining location of a hyperplane
σ	Width of Gaussian kernel
$\varPhi (\mathbf{x})$	Mapping function from space to space
C	Covariance matrix of mean-centred data matrix X
v	Arbitrary vector
$L(\tilde{\mathbf{x}}, \propto )$	Lagrangian function value at point where all inequality constraints of optimization problem are satisfied
$L(\mathbf{x},\propto )$	Lagrangian function
N	Number of samples in a training data set
R(f)	Risk of overfitting function f
C(h, N, δ)	Capacity of a model
$g(\propto )$	Lagrangian dual function
$k(\mathbf{x},{\mathbf{x}}^{\prime})$	Kernel function
$m(\mathbf{w},b)$	Margin of a separating hyperplane
	Principal component score space
	Vector space of x
∝	Vector of Lagrangian multipliers
θ	Angle
κ	Sigmoidal kernel parameter
v	Parameter in the optimization of a soft margin classifier
ρ	Parameter in the optimization of a soft margin classifier
$\vartheta$	Sigmoidal kernel parameter