Abstract
Modern data centers face new scheduling challenges in optimizing job-level performance objectives, where a significant challenge is the scheduling of highly parallel data flows with a common performance goal (e.g., the shuffle operations in MapReduce applications). Chowdhury and Stoica [6] introduced the coflow abstraction to capture these parallel communication patterns, and Chowdhury et al. [8] proposed effective heuristics to schedule coflows efficiently. In our previous paper [18], we considered the strongly NP-hard problem of minimizing the total weighted completion time of coflows with release dates, and developed the first polynomial-time scheduling algorithms with O(1)-approximation ratios.
In this paper, we carry out a comprehensive experimental analysis on a Facebook trace and extensive simulated instances to evaluate the practical performance of several algorithms for coflow scheduling, including our approximation algorithms developed in [18]. Our experiments suggest that simple algorithms provide effective approximations of the optimal, and that the performance of the approximation algorithm of [18] is relatively robust, near optimal, and always among the best compared with the other algorithms, in both the offline and online settings.
Research partially supported by NSF grant CCF-1421161.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
In this paper, the term “flow” refers to data flows in computer networking, and is not to be confused with the notion of “flow time,” commonly used in the scheduling literature.
- 2.
Here “flow time” refers to the length of time from the release time of a coflow to its completion time, as in scheduling theory.
- 3.
These completion times depend on the scheduling rule used. Thus, ECT depends on the underlying scheduling algorithm. In Sect. 3.2, the scheduling algorithms are described in more detail.
References
Apache hadoop. http://hadoop.apache.org
Google dataflow. https://www.google.com/events/io
Alizadeh, M., Yang, S., Sharif, M., Katti, S., McKeown, N., Prabhakar, B., Shenker, S.: pfabric: Minimal near-optimal datacenter transport. SIGCOMM Comput. Commun. Rev. 43(4), 435–446 (2013)
Ballani, H., Costa, P., Karagiannis, T., Rowstron, A.: Towards predictable datacenter networks. SIGCOMM Comput. Commun. Rev. 41(4), 242–253 (2011)
Borthakur, D.: The hadoop distributed file system: Architecture and design. Hadoop Project Website (2007)
Chowdhury, M., Stoica, I.: Coflow: A networking abstraction for cluster applications. In: HotNets-XI, pp. 31–36 (2012)
Chowdhury, M., Zaharia, M., Ma, J., Jordan, M.I., Stoica, I.: Managing data transfers in computer clusters with orchestra. SIGCOMM Comput. Commun. Rev. 41(4), 98–109 (2011)
Chowdhury, M., Zhong, Y., Stoica, I.: Efficient coflow scheduling with Varys. In: SIGCOMM (2014)
Dean, J., Ghemawat, S.: Mapreduce: Simplified data processing on large clusters. In: OSDI, p. 10 (2004)
Dogar, F., Karagiannis, T., Ballani, H., Rowstron, A.: Decentralized task-aware scheduling for data center networks. Technical Report MSR-TR–96 2013
Hall, L.A., Schulz, A.S., Shmoys, D.B., Wein, J.: Scheduling to minimize average completion time: Off-line and on-line approximation algorithms. Math. Oper. Res. 22(3), 513–544 (1997)
Kang, N., Liu, Z., Rexford, J., Walker, D.: Optimizing the “one big switch” abstraction in software-defined networks. In: CoNEXT, pp. 13–24 (2013)
Leung, J.Y., Li, H., Pinedo, M.: Order scheduling in an environment with dedicated resources in parallel. J. Sched. 8(5), 355–386 (2005)
Phillips, C.A., Stein, C., Wein, J.: Minimizing average completion time in the presence of release dates. Math. Program. 82(1–2), 199–223 (1998)
Pinedo, M.: Scheduling: Theory, Algorithms, and Systems, 3rd edn. Springer, New York (2008)
Popa, L., Krishnamurthy, A., Ratnasamy, S., Stoica, I.: Faircloud: Sharing the network in cloud computing. In: HotNets-X, pp. 22:1–22:6 (2011)
Qiu, Z., Stein, C., Zhong, Y.: Experimental analysis of algorithms for coflow scheduling. arXiv (2016). http://arxiv.org/abs/1603.07981
Qiu, Z., Stein, C., Zhong, Y.: Minimizing the total weighted completion time of coflows in datacenter networks. In: ACM Symposium on Parallelism in Algorithms and Architectures, pp. 294–303 (2015)
Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The hadoop distributed file system. In: MSST, pp. 1–10 (2010)
Skutella, M.: List scheduling in order of \(\alpha \)-points on a single machine. In: Bampis, E., Jansen, K., Kenyon, C. (eds.) Efficient Approximation and Online Algorithms. LNCS, vol. 3484, pp. 250–291. Springer, Heidelberg (2006)
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M.J., Shenker, S., Stoica, I.: Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In: NSDI, p. 2 (2012)
Acknowledgment
Yuan Zhong would like to thank Mosharaf Chowdhury and Ion Stoica for numerous discussions on the coflow scheduling problem, and for sharing the Facebook data.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Qiu, Z., Stein, C., Zhong, Y. (2016). Experimental Analysis of Algorithms for Coflow Scheduling. In: Goldberg, A., Kulikov, A. (eds) Experimental Algorithms. SEA 2016. Lecture Notes in Computer Science(), vol 9685. Springer, Cham. https://doi.org/10.1007/978-3-319-38851-9_18
Download citation
DOI: https://doi.org/10.1007/978-3-319-38851-9_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-38850-2
Online ISBN: 978-3-319-38851-9
eBook Packages: Computer ScienceComputer Science (R0)