Efficient Dynamic Device Placement for Deep Neural Network Training on Heterogeneous Systems

Abstract

Deep Neural Networks (DNNs) based learning methods have brought revolutionary advances in computer vision and machine learning. However, training a DNN model often requires very intensive computational resources. For edge incremental learning, more energy efficient learning solutions are called for. Heterogeneous computing is more power efficient, and has been increasingly popular for embedded platforms. Therefore, how to deploy training models on heterogeneous platforms to support edge learning is a critical issue.

Due to the increasing size of DNNs, it is rather difficult to determine how to dispatch a large number of operations to proper devices. One state-of-art approach uses reinforcement learning to address this device placement issue, but is too costly to apply in an embedded setting. In this paper, our approach leverages the information available from the computational graph of the model, and the dynamic profiles of run time and communication time of each device, to more efficiently deploy operations on heterogeneous systems. We use Critical Earliest Finish Time (CEFT) algorithm together with the Partitioned Boolean Quadratic Assignment Problem (PBQP) solver to find a cost-effective placement, and dynamically adjust assignments during the training process, which makes our method more adaptive and effective for different computational environments. On AlexNet, VGG, Inception, ResNet, RNNLM and other well-known models, our approach significantly outperforms traditional algorithms and reinforcement learning based methods.