Collaborative Management of Correlated Shuffle Transfer

Abstract

The inter-flow aggregation of correlated Incast transfer is introduced in Chap. 7. This chapter considers how to realize the in-network aggregation of correlated Shuffle transfer, so that the consumed network resources can be considerably reduced. Specifically, the in-network aggregation problem of Shuffle transfer is formalized for a BCube data center. To tackle this NP-hard problem, we propose two approximate methods for efficiently constructing the shuffle aggregation subgraph, solely based on the labels of their members and the data center topology. The expected in-network aggregation can be effectively achieved through the collaborative transmission of traffic based on this subgraph. This chapter also introduces the scalable traffic forwarding mode based on Bloom filters, so as to achieve the desired in-network aggregation effect for a large number of coexisting Shuffle transfers. Although this chapter chooses BCube as the network topology, the concept of Shuffle in-network aggregation is applicable to other types of data center topologies.