Tree-Based Methods

Abstract

In this chapter, tree-based methods are discussed as another of the three major machine learning paradigms considered in the book. This includes the basic information theoretical approach used to construct classification and regression trees and a few simple examples to illustrate the characteristics of decision tree models. Following this is a short introduction to ensemble theory and ensembles of decision trees, leading to random forest models, which are discussed in detail. Unsupervised learning of random forests in particular is reviewed, as these characteristics are potentially important in unsupervised fault diagnostic systems. The interpretation of random forest models includes a discussion on the assessment of the importance of variables in the model, as well as partial dependence analysis to examine the relationship between predictor variables and the response variable. A brief review of boosted trees follows that of random forests, including discussion of concepts, such as gradient boosting and the AdaBoost algorithm. The use of tree-based ensemble models is illustrated by an example on rotogravure printing and the identification of defects in hot rolled steel plate.

Nomenclature

Symbol	Description
i(η)	Impurity measure of node η in a classification tree
p(k|η)	Proportion of samples in class k at node η in a classification tree
C	Number of classes in a classification problem
Δi(ς, η)	Decrease in impurity for a candidate split position ς at node η
ς	Split point index in a decision tree
ς*	Optimal split point index in a decision tree
η	Node index variable
	Input space
N	Sample size
p R	Proportion of samples reporting to the right descendent node after splitting in a classification tree
p L	Proportion of samples reporting to the left descendent node after splitting in a classification tree
c R	Prediction in right descendent node of a regression tree after splitting
c L	Prediction in left descendent node of a regression tree after splitting
η R	Index of right descendent node
η L	Index of left descendent node
X k	kth bootstrap sample of learning data set X
T	Set of ensemble trees
t k	kth tree in an ensemble of trees
K	Number of trees in an ensemble of classification or regression trees
t k (⋅)	Prediction of kth tree in an ensemble of classification or regression trees
m	Number of variables considered at each split point in a random forest tree
M	Total number of input variables
$\mathbf{X}_{{\rm OOB}(j)}^k$	Out-of-bag (OOB) input learning data for the kth tree in an ensemble of trees with variable j permuted
$\mathbf{y}_{\rm OOB}^k$	Out-of-bag (OOB) output learning data for the kth tree in an ensemble of trees
${\omega_j}({{t_k}})$	Importance measure for jth variable in kth tree in an ensemble of trees (random forest)
${\omega_j}$	Importance measure for jth variable in an ensemble of trees (random forest)
${{\mathbf{X}}_S}$	Subset of variables in X
X C	Subset of variables in X complementary to ${X_S}$
${X_{i,C }}$	Values of samples in X C
$\bar{f}({{{\mathbf{X}}_S}})$	Partial dependence of a predicted response to the subset of variables in ${{\mathbf{X}}_S}$
b j	jth scalar calculation point
S	Proximity matrix
D	Dissimilarity matrix
g	Unknown data density
g 0	Reference distribution
X 0	Synthetic data set obtained by random sampling from the product of marginal distributions in X
Z	Concatenated matrix
T	Scaling coordinate features
β	K × 1 weighting vector of trees in a boosted tree ensemble
w	N × 1 weighting vector of samples in a boosted tree ensemble
${\epsilon_k}$	Ensemble error
${\beta_k}$	Weight of kth tree in boosted tree ensemble
W k	Normalizing constant
$F(\mathbf{x})$	Output of ensemble of boosted classification or regression trees
L(y,f(x))	Loss function of a classifier or regressor
g k (x)	Gradient at x after at kth iteration
ρ k	Optimization search step size at kth iteration
θ k	Parameters of kth model in an ensemble
i w (η)	Weighted cross-entropy of node η
$Q\left( {k|\eta } \right)$	Sum of weights of samples in node η labelled as class k
W(η)	Sum of all sample weights present in node η
${{{\mathrm{X}}^{\prime}}_{\mathrm{t}}}$	Matrix of time series column vectors with mean centred columns
${{\widehat{\mathbf{X}}}_i}$	ith of d
${{\tilde{\mathbf{X}}}_i}$	ith trajectory matrix
$\rho_{p,q}^{(w) }$	Weighted or w-correlation between time series p and q
$\rho_{\max}^{(L,K) }$	Maximum of the absolute value of the correlations between the rows and between the columns of a pair of trajectory matrices ${{\tilde{\mathbf{X}}}_i}$ and ${{\tilde{\mathbf{X}}}_j}$
$\mathcal{N}(a,b)$	Normal distribution with mean a and standard deviation b
$\mathbf{u}(t)$	Input vector at time t
$\mathbf{y}(t)$	Vector of measured variables at time t
$\mathbf{v}(\mathrm{t})$	Gaussian noise with variance 0.01
$\mathbf{w}(t)$	Gaussian noise with variance 0.1