Alternating Direction Method of Multipliers for Hierarchical Basis Approximators

Abstract

Sparse grids have been successfully used for the mining of vast datasets with a moderate number of dimensions. Compared to established machine learning techniques like artificial neural networks or support vector machines, sparse grids provide an analytic approximant that is easier to analyze and to interpret. More important, they are based on a high-dimensional discretization of the feature space, are thus less data-dependent than conventional approaches, scale only linearly in the number of data points and are well-suited to deal with huge amounts of data. But with an increasing size of the datasets used for learning, computing times clearly can become prohibitively large for normal use, despite the linear scaling. Thus, efficient parallelization strategies have to be found to exploit the power of modern hardware. We investigate the parallelization opportunities for solving high-dimensional machine learning problems with adaptive sparse grids using the alternating direction method of multipliers (ADMM). ADMM allows us to split the initially large problem into smaller ones. They can then be solved in parallel while their reduced problem sizes can even be small enough for an explicitly assembly of the system matrices. We show the first results of the new approach using a set of problems and discuss the challenges that arise when applying ADMM to a hierarchical basis.