Automatic data decomposition for message-passing machines

Abstract

The data distribution problem is very complex, because it involves trade-off decisions between minimizing communication and maximizing parallelism. A common approach towards solving this problem is to break the data mapping into two stages: an alignment stage and a distribution stage. The alignment stage attempts to increase parallelism, while the distribution stage attempts to decrease communication overhead. As opposed to previous approaches, we consider the alignment and distribution problems in a unified framework, and attempt to simultaneously maximize parallelism and minimize communication overhead. The problem becomes harder if dynamic remapping, multi-dimensional distributions, array replications and control flow are taken into account. This paper formulates the full data decomposition problem that addresses all these issues and presents a simple new algorithm to find the optimal solution of the dynamic data distribution problem, given the number of processors and a partitioning of the input program into phases. The algorithm runs efficiently for small search spaces (several hundreds of data distributions).