An Algorithmic Framework for Geo-Distributed Analytics

Abstract

Large-scale cloud enterprises operate tens to hundreds of datacenters, running a variety of services that produce enormous amounts of data, such as search clicks and infrastructure operation logs. A recent research direction in both academia and industry is to attempt to process the “big data” in multiple datacenters, as the alternative of centralized processing might be too slow and costly (e.g., due to transferring all the data to a single location). Running such geo-distributed analytics jobs at scale gives rise to key resource management decisions: Where should each of the computations take place? Accordingly, which data should be moved to which location, and when? Which network paths should be used for moving the data, etc. These decisions are complicated not only because they involve the scheduling of multiple types of resources (e.g., compute and network), but also due to the complicated internal data flow of the jobs—typically structured as a DAG of tens of stages, each of which with up to thousands of tasks. Recent work [17, 22, 25] has dealt with the resource management problem by abstracting away certain aspects of the problem, such as the physical network connecting the datacenters, the DAG structure of the jobs, and/or the compute capacity constraints at the (possibly heterogeneous) datacenters. In this paper, we provide the first analytical model that includes all aspects of the problem, with the objective of minimizing the makespan of multiple geo-distributed jobs. We provide exact and approximate algorithms for certain practical scenarios and suggest principled heuristics for other scenarios of interest.

6 Appendix: Linearizing Multiplicative Constraints

Recall that our multiplicative constraints are of the form

$$ \forall i,j\in [n],e=(u,v)\in E\,:\,\sum _{t\le T}r_{i,j,e,t}=D_{e}\frac{OUT_{i,u,T}}{D_{OUT,u}}\cdot \frac{IN_{j,v,T}}{D_{IN,v}} $$

The Taylor series is expanded around the estimated values $\widehat{OUT}_{i,u,T},\widehat{IN}_{j,v,T}$. We solve the LP iteratively and use the values the LP found for the variables in the previous iteration as the estimated values. After several iterations, the values converge to their true value, and thus, the multiplications become more accurate.

Recall that the first-order Taylor expansion for the multiplication $x\cdot y$ expanded around the point $(\hat{x},\hat{y})$ is: $x\cdot y\simeq \hat{x}\cdot \hat{y}+(x-\hat{x})\cdot \hat{y}+(y-\hat{y})\cdot \hat{x}$. Dividing both sides by the constant $\frac{D_{e}}{D_{OUT,u}D_{IN,v}}$ and approximating the multiplication using first-order Taylor, we obtain $\forall i,j\in [n],e=(u,v)\in E$:

$$\begin{aligned}&\frac{D_{OUT,u}D_{IN,v}}{D_{e}}\sum _{t\le T}r_{i,j,e,t}\ge \widehat{OUT}_{i,u,T}\cdot \widehat{IN}_{j,v,T} \\&+(OUT_{i,u,T}-\widehat{OUT}_{i,u,T})\cdot \widehat{IN}_{j,v,T}+(IN_{j,v,T}-\widehat{IN}_{j,v,T})\cdot \widehat{OUT}_{i,u,T}. \end{aligned}$$

Note that we have turned this equality constraint into a non-equality, since we have other constraints for the total flow from previous sections. Our multiplication evaluating the flow might be slightly more or less than the true multiplication value. In case it is more than the true value, we simply send less flow, and no constraints are violated. In case it is less than the true value, we need to send more data than planned—which might violate capacity constraints. In this case, we simply use a little more time for the entire flow to be sent. As long as the approximation is reasonable, this extra-time will be small.

To obtain relatively accurate values for $\widehat{OUT}_{i,u,T},\widehat{IN}_{j,v,T}$, we solve the LP iteratively and use the previous values as our approximation. For the first iteration only we use $\widehat{OUT}_{i,u,T}=0,\widehat{IN}_{j,v,T}=0$. We use the same approach for our second set of multiplicative constraints (see Sect. 3.4) $\forall i,j\in [n],e=(u,v)\in E,t\in [T]\,:\,\frac{\sum _{t'\le t}r_{i,j,e,t'}}{\sum _{t'\le T}r_{i,j,e,t'}}\ge \frac{COMP_{j,v,t}}{COMP_{j,v,T}};$ details omitted for brevity. The resulting LP is then solved iteratively, as described, to obtain a nearly feasible solution. The small infeasibility translates into some extra-time required for completing flows that are larger than anticipated by the LP.