Advertisement

Cost-Efficient Task Scheduling for Geo-distributed Data Analytics

  • Linfeng Xie
  • Yang Dai
  • Yongjin Zhu
  • Xin LiEmail author
  • Xiangbo Li
  • Zhuzhong Qian
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11637)

Abstract

Geo-distributed data processing is affected by many factors, some countries or regions prohibit the transmission of original user data abroad. Therefore, it is necessary to adopt a non-centralized processing method for these data, but at the same time, many problems will arise. Firstly, it is unavoidable to transfer job’s intermediate data across regions, which will result in data transmission cost. Secondly, the WAN bandwidth is often much smaller than the bandwidth within clusters, which makes it easier to become the bottleneck of geo-distributed job. In addition, because the idle computing resources in the cluster may change with time, it will also cause some difficulties in task scheduling. Therefore, this paper considers the problem of task scheduling for big data jobs on geo-distributed data, considering the budget constraints on intermediate data trans-regional transmission, and without moving the original data. we design a budget-constrained task scheduling strategy CETS. Through the experimental analysis of different scenarios, the effectiveness of the proposed algorithm strategy is verified.

Keywords

Big data Cloud computing Cost efficient Geo-distributed data processing Task scheduling 

References

  1. 1.
    Pu, Q., et al.: Low latency geo-distributed data analytics. ACM SIGCOMM Comput. Commun. Rev. 45(4), 421–434 (2015)CrossRefGoogle Scholar
  2. 2.
    Vulimiri, A., Curino, C., Godfrey, P.B., Jungblut, T., Padhye, J., Varghese, G.: Global analytics in the face of bandwidth and regulatory constraints (2017)Google Scholar
  3. 3.
    Preguiça, K., Rodrigues, R.: Pixida: optimizing data parallel jobs in bandwidth-skewed environments. In: Proceedings of VLDB Endowment (2015)Google Scholar
  4. 4.
    Hu, Z., Li, B., Luo, J.: Flutter: scheduling tasks closer to data across geo-distributed datacenters. In: IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on Computer Communications, pp. 1–9. IEEE (2016)Google Scholar
  5. 5.
    Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. HotCloud 10(10–10), 95 (2010)Google Scholar
  6. 6.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
  7. 7.
    Jayalath, C., Eugster, P.: Efficient geo-distributed data processing with rout. In: 2013 IEEE 33rd International Conference on Distributed Computing Systems, pp. 470–480. IEEE (2013)Google Scholar
  8. 8.
    Olston, C., Reed, B., Srivastava, U., Kumar, R., Tomkins, A.: Pig Latin: a not-so-foreign language for data processing. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, pp. 1099–1110. ACM (2008)Google Scholar
  9. 9.
    Afrati, F., Dolev, S., Sharma, S., Ullman, J.D.: Meta-MapReduce: a technique for reducing communication in MapReduce computations arXiv preprint arXiv:1508.01171 (2015)
  10. 10.
    Gadre, H., Rodero, I., Parashar, M.: Investigating MapReduce framework extensions for efficient processing of geographically scattered datasets. ACM SIGMETRICS Perform. Eval. Rev. 39(3), 116–118 (2011)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Linfeng Xie
    • 1
  • Yang Dai
    • 1
  • Yongjin Zhu
    • 1
  • Xin Li
    • 2
    • 3
    Email author
  • Xiangbo Li
    • 3
  • Zhuzhong Qian
    • 3
  1. 1.JiangSu Frontier Electric Technology Co., LTDNanjingChina
  2. 2.CCSTNanjing University of Aeronautics and AstronauticsNanjingChina
  3. 3.State Key Laboratory for Novel Software TechnologyNanjing UniversityNanjingChina

Personalised recommendations