The voice of optimization

Abstract

We introduce the idea that using optimal classification trees (OCTs) and optimal classification trees with-hyperplanes (OCT-Hs), interpretable machine learning algorithms developed by Bertsimas and Dunn (Mach Learn 106(7):1039–1082, 2017), we are able to obtain insight on the strategy behind the optimal solution in continuous and mixed-integer convex optimization problem as a function of key parameters that affect the problem. In this way, optimization is not a black box anymore. Instead, we redefine optimization as a multiclass classification problem where the predictor gives insights on the logic behind the optimal solution. In other words, OCTs and OCT-Hs give optimization a voice. We show on several realistic examples that the accuracy behind our method is in the 90–100% range, while even when the predictions are not correct, the degree of suboptimality or infeasibility is very low. We compare optimal strategy predictions of OCTs and OCT-Hs and feedforward neural networks (NNs) and conclude that the performance of OCT-Hs and NNs is comparable. OCTs are somewhat weaker but often competitive. Therefore, our approach provides a novel insightful understanding of optimal strategies to solve a broad class of continuous and mixed-integer optimization problems.

Appendices

Appendix 1: Optimal classification trees

OCTs and OCT-Hs developed by Bertsimas and Dunn (2017, 2019) are a recently proposed generalization of classification and regression trees (CARTs) developed by Breiman et al. (1984) that construct decision trees that are near optimal with significantly higher prediction accuracy while retaining their interpretability. Bertsimas and Dunn (2019) have shown that given a NN (feedforward, convolutional and recurrent), we can construct an OCT-H that has the same in sample accuracy, that is OCT-Hs are at least as powerful as NNs. The constructions can sometimes generate OCT-Hs with high depth. However, they also report computational results that show that OCT-Hs and NNs have comparable performance in practice even for depths of OCT-Hs below 10.

Architecture

OCT recursively partition the feature space ${\mathbf{R}}^p$ to construct hierarchical disjoint regions. A tree can be defined as a set of nodes $t \in {\mathcal {T}}$ of two types ${\mathcal {T}}= {\mathcal {T}}_B \cup {\mathcal {T}}_L$:

Branch nodes:

Nodes $t \in {\mathcal {T}}_B$ at the tree branches describe a split of the form ${a_t^T\theta < b_t}$ where $a_t \in {\mathbf{R}}^p$ and $b_t \in {\mathbf{R}}$. They partition the space in two subsets: the points on the left branch satisfying the inequality and the remaining ones points on the right branch. If splits involve a single variable we denote them as parallel and we refer to the tree as optimal classification tree (OCT).. This is achieved by enforcing all components of $a_t$ to be all 0 except from one. Otherwise, if the components of $a_t$ can be freely nonzero, we denote the splits as hyperplanes and we refer to the tree as optimal classification tree with-hyperplanes (OCT-H).

Leaf nodes:

Nodes $t \in {\mathcal {T}}_L$ at the tree leaves make a class prediction for each point falling into that node.

An example OCT for the Iris dataset appears in Fig. 8 (Bertsimas and Dunn 2019). For each new data point it is straightforward to follow which hyperplanes are satisfied and to make a prediction. This characteristic makes OCTs and OCT-Hs highly interpretable. Note that the level of interpretability of the resulting trees can be also tuned by changing minimum sparsity of $a_t$. The two extremes are maximum sparsity OCTs and minimum sparsity OCT-Hs but we can specify anything in between.

Fig. 8

Example OCT for the Iris dataset (Bertsimas and Dunn 2019)

Learning

With the latest computational advances in MIO, Bertsimas and Dunn (2019) were able to exactly formulate the tree training as a MIO problem and solve it in a reasonable amount of time for problem sizes arising in real-world applications.

The OCT cost function is a tradeoff between the misclassification error at each leaf and the tree complexity

$$\begin{aligned} {\mathcal {L}}_{OCT} = \sum _{t \in {\mathcal {T}}_L} L_t + \alpha \sum _{t \in {\mathcal {T}}_B} \Vert a_t\Vert _1, \end{aligned}$$

where the $L_t$ is the misclassification error at node t and the second term represents the complexity of the tree measured as the sum of the norm of the hyperplane coefficients in all the splits. The parameter $\alpha > 0$ regulates the tradeoff. For more details about the cost function and the constraints of the problem, we refer the reader to Bertsimas and Dunn (2019, Sects. 2.2, 3.1).

Bertsimas and Dunn apply a local search method (Bertsimas and Dunn 2019, Sect. 2.3) that manages to solve OCT problems for realistic sizes in fraction of the time an off-the-shelf optimization solver would take. The algorithm proposed iteratively improves and refines the current tree until a local minimum is reached. By repeating this search from different random initialization points the authors compute several local minima and then take the best one as the resulting tree. This heuristic showed remarkable performance both in terms of computation time and quality of the resulting tree becoming the algorithm included in the OptimalTrees.jl Julia package (Bertsimas and Dunn 2019).

Appendix 2: Neural networks

NNs have recently become one of the most prominent machine learning techniques revolutionizing fields such as speech recognition (Hinton et al. 2012) and computer vision (Krizhevsky et al. 2012). The wide range of applications of these techniques recently extended to autonomous driving (Bojarski et al. 2016) and reinforcement learning (Silver et al. 2017). The popularity of neural networks is also due to the widely used open-source libraries learning on CPUs and GPUs coming from both academia and industry such as TensorFlow (Abadi et al. 2015), Caffe (Jia et al. 2014) and PyTorch (Paszke et al. 2017). We use feedforward neural networks which offer a good tradeoff between simplicity and accuracy without resorting to more complex architectures such as convolutional or recurrent neural networks (LeCun et al. 2015).

Architecture

Given L layers, a neural network is a composition of functions of the form

$$\begin{aligned} \hat{s} = f_L(f_{L-1}(\dots f_1(\theta ))), \end{aligned}$$

where each function consists of

$$\begin{aligned} y_{l} = f(y_{l-1}) = g(W_l y_{l-1} + b_l). \end{aligned}$$

(12)

The number of nodes in each layer is $n_l$ and corresponds to the dimension of the vector $y_{l} \in {\mathbf{R}}^{n_l}$. Layer $l=1$ is defined as the input layer and $l=L$ as the output layer. Consequently $y_{1} = \theta$ and $y_{L} = \hat{s}$. The linear transformation in each layer is composed of an affine transformation with parameters $W_{l} \in {\mathbf{R}}^{n_l \times n_{l-1}}$ and $b_l \in {\mathbf{R}}^{n_l}$. The activation function $g: {\mathbf{R}}^{n_l} \rightarrow {\mathbf{R}}^{n_l}$ models nonlinearities. We chose as activation function the rectified linear unit (ReLU) defined as

$$\begin{aligned} g(x) = \max (x, 0), \end{aligned}$$

for all the layers $l=1,\ldots ,L-1$. Note that the $\max$ operator is intended element-wise. We chose a ReLU because on the one hand it provides sparsity to the model since it is 0 for the negative components of x and because on the other hand it does not suffer from the vanishing gradient issues of the standard sigmoid functions (Goodfellow et al. 2016). An example neural network can be found in Fig. 9.

Fig. 9

Example feedforward neural network with functions $f_i,\;i=1,\ldots ,L$ defined in (12)

For the output layer we would like the network to output not only the predicted class, but also a normalized ranking between them according to how likely they are to be the right one. This can be achieved with a softmax activation function in the output layer defined as

$$\begin{aligned} g(x)_j = \frac{e^{x_j}}{\sum _{j=1}^{M} e^{x_j}}, \end{aligned}$$

where $j=1,\ldots ,M$ are the elements of g(x). Hence, $0 \le g(x) \le 1$ and the predicted class is ${\mathrm{argmax}}(\hat{s})$.

Learning

Before training the network, we rewrite the labels for the neural network learning using a one-hot encoding, i.e., $s_i \in {\mathbf{R}}^M$ where M is the total number of classes and all the elements of $s_i$ are 0 except the one corresponding to the class which is 1.

We define a smooth cost function amenable to algorithms such as gradient descent, i.e., the cross-entropy loss

$$\begin{aligned} {\mathcal {L}}_{{\mathrm{NN}}} = \sum _{i=1}^{N} -s_i^T\log (\hat{s}_i), \end{aligned}$$

where $\log$ is intended element-wise. The cross-entropy loss ${\mathcal {L}}$ can also be interpreted as the distance between the predicted probability density of the labels compared to the true one.

The actual training phase consists of applying stochastic gradient descent using the derivatives of the cost function using the back-propagation rule. This method works very well in practice and provides good out-of-sample performance with short training times.