Histogram-Based Algorithm for Building Gradient Boosting Ensembles of Piecewise Linear Decision Trees

Abstract

One of the most popular machine learning algorithms is gradient boosting over decision trees. This algorithm achieves high quality out of the box combined with comparably low training and inference time. However, modern machine learning applications require machine learning algorithms, that can achieve better quality in less inference time, which leads to an exploration of grading boosting algorithms over other forms of base learners. One of such advanced base learners is a piecewise linear tree, which has linear functions as predictions in leaves. This paper introduces an efficient histogram-based algorithm for building gradient boosting ensembles of such trees. The algorithm was compared with modern gradient boosting libraries on publicly available datasets and achieved better quality with a decrease in ensemble size and inference time. It was proven, that algorithm is independent of a linear transformation of individual features.