The Deployment of Large-Scale Data Synchronization System for Cross-DC Networks

  • Yuchao Zhang
  • Ke Xu


Many important cloud services require replicating massive data from one datacenter (DC) to multiple DCs. While the performance of pair-wise inter-DC data transfers has been much improved, prior solutions are insufficient to optimize bulk-data multicast, as they fail to explore the rich inter-DC overlay paths that exist in geo-distributed DCs, as well as the remaining bandwidth reserved for online traffic under fixed bandwidth separation scheme. To take advantage of these opportunities, we present BDS+, a near-optimal network system for large-scale inter-DC data replication. BDS+ is an application-level multicast overlay network with a fully centralized architecture, allowing a central controller to maintain an up-to-date global view of data delivery status of intermediate servers, in order to fully utilize the available overlay paths. Furthermore, in each overlay path, it leverages dynamic bandwidth separation to make use of the remaining available bandwidth reserved for online traffic. By constantly estimating online traffic demand and rescheduling bulk-data transfers accordingly, BDS+ can further speed up the massive data multicast. Through a pilot deployment in one of the largest online service providers and large-scale real-trace simulations, we show that BDS+ can achieve 3–5× speedup over the provider’s existing system and several well-known overlay routing baselines of static bandwidth separation. Moreover, dynamic bandwidth separation can further reduce the completion time of bulk data transfers by 1.2 to 1.3 times.


  1. 1.
    Chu, Y.-H., Rao, S.G., Zhang, H.: A case for end system multicast. ACM SIGMETRICS Perform. Eval. Rev. 28(1), 1–12 (2000). ACM, New YorkGoogle Scholar
  2. 2.
    Datta, A.K., Sen, R.K.: 1-approximation algorithm for bottleneck disjoint path matching. Inf. Process. Lett. 55(1), 41–44 (1995)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Andreev, K., Maggs, B.M., Meyerson, A., Sitaraman, R.K.: Designing overlay multicast networks for streaming. In: Proceedings of the Fifteenth Annual ACM Symposium on Parallel Algorithms and Architectures, pp. 149–158 (2003)Google Scholar
  4. 4.
    Sripanidkulchai, K., Maggs, B., Zhang, H.: An analysis of live streaming workloads on the internet. In: IMC, pp. 41–54. ACM (2004)Google Scholar
  5. 5.
    Zhang, X., Liu, J., Li, B., Yum, Y.-S.: CoolStreaming/DONet: a data-driven overlay network for peer-to-peer live media streaming. In: INFOCOM, vol. 3, pp. 2102–2111. IEEE (2005)Google Scholar
  6. 6.
    Huang, T.Y., Johari, R., Mckeown, N., Trunnell, M., Watson, M.: A buffer-based approach to rate adaptation: evidence from a large video streaming service. In: SIGCOMM, pp. 187–198 (2014)Google Scholar
  7. 7.
    Repantis, T., Smith, S., Smith, S., Wein, J.: Scaling a monitoring infrastructure for the Akamai network. ACM SIGOPS Oper. Syst. Rev. 44(3), 20–26 (2010)CrossRefGoogle Scholar
  8. 8.
    Mukerjee, M.K., Hong, J., Jiang, J., Naylor, D., Han, D., Seshan, S., Zhang, H.: Enabling near real-time central control for live video delivery in CDNS. ACM SIGCOMM Comput. Commun. Rev. 44(4), 343–344 (2014). ACMGoogle Scholar
  9. 9.
    Gog, I., Schwarzkopf, M., Gleave, A., Watson, R.N.M., Hand, S.: Firmament: Fast, Centralized Cluster Scheduling at Scale. In: OSDI, pp. 99–115. USENIX Association, Savannah (2016). [Online]. Available:
  10. 10.
    Cohen, B.: Incentives build robustness in bittorrent. In: Proceedings of the First Workshop on the Economics of Peer-to-Peer Systems, pp. 1–1 (2003)Google Scholar
  11. 11.
    Garg, N., Vazirani, V.V., Yannakakis, M.: Primal-dual approximation algorithms for integral flow and multicut in trees. Algorithmica 18(1):3–20 (1997)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Garg, N., Koenemann, J.: Faster and simpler algorithms for multicommodity flow and other fractional packing problems. SIAM J. Comput. 37(2):630–652 (2007)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Reed, M.J.: Traffic engineering for information-centric networks. In: IEEE ICC, pp. 2660–2665 (2012)Google Scholar
  14. 14.
    Fleischer, L.K.: Approximating fractional multicommodity flow independent of the number of commodities. In: SIDMA, pp. 505–520 (2000)Google Scholar
  15. 15.
    Friedrich and Pukelsheim: The three sigma rule. Am. Stat. 48(2):88–91 (1994) [Online]. Available:
  16. 16.
    Adams, R.P., MacKay, D.J.: Bayesian online changepoint detection. arXiv preprint :0710.3742 (2007)Google Scholar
  17. 17.
    Roberts, S.: Control chart tests based on geometric moving averages. Technometrics 1(3):239–250 (1959)CrossRefGoogle Scholar
  18. 18.
    Lucas, J.M., Saccucci, M.S.: Exponentially weighted moving average control schemes: properties and enhancements. Technometrics 32(1):1–12 (1990)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Smith, A.: A Bayesian approach to inference about a change-point in a sequence of random variables. Biometrika 62(2):407–416 (1975)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Stephens, D.: Bayesian retrospective multiple-changepoint identification. Appl. Stat. 43, 159–178 (1994)CrossRefGoogle Scholar
  21. 21.
    Barry, D., Hartigan, J.A.: A Bayesian analysis for change point problems. J. Am. Stat. Assoc. 88(421):309–319 (1993)MathSciNetzbMATHGoogle Scholar
  22. 22.
    Green, P.J.: Reversible jump markov chain monte carlo computation and bayesian model determination. Biometrika 82(4):711–732 (1995)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Page, E.: A test for a change in a parameter occurring at an unknown point. Biometrika 42(3/4):523–527 (1955)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Desobry, F., Davy, M., Doncarli, C.: An online kernel change detection algorithm. IEEE Trans. Signal Process. 53(8):2961–2974 (2005)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Lorden, G., et al.: Procedures for reacting to a change in distribution. Ann. Math. Stat. 42(6):1897–1908 (1971)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Bayesian changepoint detection.
  27. 27.
    Kumar, A., Jain, S., Naik, U., Raghuraman, A., Kasinadhuni, N., Zermeno, E.C., Gunn, C.S., Björn Carlin, J.A., Amarandei-Stavila, M., et al.: BwE: flexible, hierarchical bandwidth allocation for WAN distributed computing. In: ACM SIGCOMM, pp. 1–14 (2015)Google Scholar
  28. 28.
    Wang, H., Li, T., Shea, R., Ma, X., Wang, F., Liu, J., Xu, K.: Toward cloud-based distributed interactive applications: measurement, modeling, and analysis. In: IEEE/ACM ToN (2017)Google Scholar
  29. 29.
    Chen, Y., Alspaugh, S., Katz, R.H.: Design insights for MapReduce from diverse production workloads. University of California, Berkeley, Department of Electrical Engineering & Computer Sciences, Technical Report (2012)Google Scholar
  30. 30.
    Kavulya, S., Tan, J., Gandhi, R., Narasimhan, P.: An analysis of traces from a production mapreduce cluster. In: CCGrid, pp. 94–103. IEEE (2010)Google Scholar
  31. 31.
    Mishra, A.K., Hellerstein, J.L., Cirne, W., Das, C.R.: Towards characterizing cloud backend workloads: insights from Google compute clusters. ACM SIGMETRICS PER 37(4):34–41 (2010)CrossRefGoogle Scholar
  32. 32.
    Reiss, C., Tumanov, A., Ganger, G.R., Katz, R.H., Kozuch, M.A.: Heterogeneity and dynamicity of clouds at scale: Google trace analysis. In: Proceedings of the Third ACM Symposium on Cloud Computing, p. 7. ACM (2012)Google Scholar
  33. 33.
    Lamport, L.: The part-time parliament. ACM TOCS 16(2):133–169 (1998)CrossRefGoogle Scholar
  34. 34.
    The go programming language:
  35. 35.
    Kostić, D., Rodriguez, A., Albrecht, J., Vahdat, A.: Bullet: high bandwidth data dissemination using an overlay mesh. ACM SOSP 37(5), 282–297 (2003). ACMGoogle Scholar
  36. 36.
    Solve linear programming problems – matlab linprog:
  37. 37.

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  • Yuchao Zhang
    • 1
  • Ke Xu
    • 2
  1. 1.Beijing University of Posts and TelecommBeijingChina
  2. 2.Tsinghua UniversityBeijingChina

Personalised recommendations