Cluster Computing

, Volume 22, Issue 4, pp 1431–1446 | Cite as

Flow-aware explicit congestion notification for datacenter networks

  • Pan Zhou
  • Hongfang YuEmail author
  • Gang Sun
  • Long Luo
  • Shouxi Luo
  • Zilong Ye


Explicit congestion notification (ECN) has been widely adopted by recent proposals to build up high-throughput and low-latency datacenter network transport. In these ECN-based proposals, when the queue length of a switch exceeds a pre-defined threshold, the switch would mark all arriving packets with ECN to explicitly notify their senders to slow down the rates. Such a design enables the network to eliminate congestions quickly. However, it marks packets without considering the flow state, which may overkill flows, especially those only send a few packets, thus resulting in significant throughput loss and long flow completion times. In this paper, we propose a novel flow-aware ECN marking approach (FECN), which can improve the throughput and flow completion time by taking flow states into consideration. By selectively marking packets respecting to their flow rates, FECN enables the network to precisely slow down the high-speed flows to avoid congestions without killing low-speed short flows. Moreover, FECN does not require switches to maintain per-flow state, which yields low overhead and thus makes FECN to be easily implemented and deployed in commodity switches. Simulations show that FECN can shorten the flow completion time by up to 44.7% and reduce the throughput loss by up to 40.3%, compared with prior flow-agnostic ECN marking approach.


Datacenter ECN Congestion Flow-aware 



This research was partially supported by the National Natural Science Foundation of China (61571098), Fundamental Research Funds for the Central Universities (ZYGX2016J217), the 111 Project (B14039), and Fundamental Research Funds for the Central Universities (2682019CX61).


  1. 1.
    Sun, G., Zhu, G., Yu, H., et al.: Cost-efficient service function chain orchestration for low-latency applications in NFV networks. IEEE Syst. J. (2018)Google Scholar
  2. 2.
    Alizadeh, M., Yang, S., Sharif, M., Katti, S. et al.: pFabric: minimal near-optimal datacenter transport. In: Proc. SIGCOMM, pp. 435446 (2013)Google Scholar
  3. 3.
    Hoff, T.: Latency is everywhere and it costs you sales how to crush it. is-everywhere-and-it-costs-you-sales-how-to-crush-it.html (2009)
  4. 4.
    Sun, G., Li, Y., Vasilakos, A., Guizani, M.: Energy-efficient and traffic-aware service function chaining orchestration in multi-domain networks. Future Gener. Comput. Syst. 91, 347–360 (2019)CrossRefGoogle Scholar
  5. 5.
    Sun, G., Yu, H.: A new technique for efficient live migration of multiple virtual machines. Future Gener. Comput. Syst. 55, 74–86 (2016)CrossRefGoogle Scholar
  6. 6.
    Sun, G., Liao, D., Yu, H.: Live migration for multiple correlated virtual machines in cloud-based data centers. IEEE Trans. Serv. Comput. 11(2), 279–291 (2018)CrossRefGoogle Scholar
  7. 7.
    Munir, A., Qazi, I.: Minimizing flow completion times in data centers, INFOCOM. Proc. IEEE IEEE 2013, 2157–2165 (2013)Google Scholar
  8. 8.
    Alizadeh, M., Greenberg, A., Maltz, D. et al.: Data center TCP (DCTCP). In: Proc. SIGCOMM, pp. 6374 (2010)Google Scholar
  9. 9.
    Luo, S., Hongfang, Y., Zhao, Y., Wang, S., Shui, Y., Li, L.: Towards practical and near-optimal coflow scheduling for data center networks. IEEE Trans. Parallel Distrib. Syst. 27(11), 3366–3380 (2016)CrossRefGoogle Scholar
  10. 10.
    Zhu, Y., Eran, H., Firestone, D., Guo, C. et al.: Congestion Control for Large-Scale RDMA Deployments. In: Proc. SIGCOMM (2015)Google Scholar
  11. 11.
    Wu, H., Ju, J., Lu, G., Guo, C., Xiong, Y., Zhang, Y.: Tuning ECN for data center networks. In: CoNEXT (2012)Google Scholar
  12. 12.
    Bai, W., Chen, L., Chen, K., Wu, H.: Enabling ECN in multi-service multi-queue data centers. In: Usenix Conference on Networked Systems Design and Implementation USENIX Association, pp. 537–549 (2016)Google Scholar
  13. 13.
    Floyd, S., Jacobson, V.: Random early detection gateways for congestion avoidance. IEEE/ACM Trans. Netw. 4, 397–413 (1993)CrossRefGoogle Scholar
  14. 14.
    Shan, D., Ren, F.: Improving ECN marking scheme with micro-burst traffic in data center networks. In: INFOCOM (2017)Google Scholar
  15. 15.
    The Network Simulator NS-3.
  16. 16.
    Lin, D., Morris, R.: Dynamics of random early detection. In: Proc. SIGCOMM, pp. 127–137 (1997)Google Scholar
  17. 17.
    Zhao, Z., Jiang, Z., Lu, C. et al.: Towards coordinated congestion control and load balancing in datacenter networks. In: Global Communications Conference (GLOBECOM), IEEE (2013)Google Scholar
  18. 18.
    Alizadeh, M., Kabbani, A., et al.: Less is more: trading a little bandwidth for ultra-low latency in the data center. In: Usenix Conference on Networked Systems Design and Implementation pp. 19–19 (2012)Google Scholar
  19. 19.
    Rong, P., Prabhakar, B., Psounis, K.: CHOKe—a stateless active queue management scheme for approximating fair bandwidth allocation. In: INFOCOM (2000)Google Scholar
  20. 20.
    Lakshman, T., Wong, L.: SRED: stabilized RED. In: Proceedings of INFOCOM pp. 1346–1355 (1999)Google Scholar
  21. 21.
    Mittal, R., Radhika, V., et al.: TIMELY: RTT-based congestion control for the datacenter. In: ACM Conference on Special Interest Group on Data Communication ACM, pp. 537–550 (2015)Google Scholar
  22. 22.
    Lee, C., Park, C.: DX: latency-based congestion control for datacenters. IEEE/ACM Trans. Netw. 25(1), 335–348 (2017)CrossRefGoogle Scholar
  23. 23.
    Zhao, Z., Li, Q., et al.: Reduce completion time and guarantee throughput by transport with slight congestion. In: IEEE International Conference on Communications IEEE pp. 1–6 (2016)Google Scholar
  24. 24.
    Bai, W., Chen, K., et al.: Enabling ECN over Generic Packet Scheduling. In: International on Conference on Emerging NETWORKING Experiments and Technologies ACM, pp. 191–204 (2016)Google Scholar
  25. 25.
    Wilson, C., Ballani, H.: Better never than late: meeting deadlines in datacenter networks. Acm Sigcomm Comput. Commun. Rev. 41(4), 50–61 (2011)CrossRefGoogle Scholar
  26. 26.
    Hong, C., Caesar, M., Godfrey, P.: Finishing flows quickly with preemptive scheduling. Acm Sigcomm Comput. Commun. Rev. 42(4), 127–138 (2012)CrossRefGoogle Scholar
  27. 27.
  28. 28.
    Nichols, K., Jacobson, V.: Controlling queue delay. Commun. ACM 55, 1–7 (2012)CrossRefGoogle Scholar
  29. 29.
    Yuanwei, L., et al.: Multi-Path Transport for RDMA in Datacenters. In: 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI) (2018)Google Scholar
  30. 30.
    Alizadeh, M., Yang, S., Sharif, M., Katti, S., McKeown, N., Prabhakar, B., Shenker, S.: pFabric: minimal near-optimal datacenter transport. In: ACM SIGCOMM (2013)Google Scholar
  31. 31.
    Perry, J., Ousterhout, A., Balakrishnan, H., Shah, D., Fugal, H.: Fastpass: A centralized zero-queue datacenter network. In: Proc. ACM SIGCOMM (2014)Google Scholar
  32. 32.
    Perry, J., Balakrishnan, H., Shah, D.: Flowtune: flowlet control for datacenter networks. In: NSDI (2017)Google Scholar
  33. 33.
    Vamanan, B., Hasan, J., Vijaykumar, T. N.: Deadline-aware datacenter TCP (D2TCP). In: Proc. ACM SIGCOMM (2012)Google Scholar
  34. 34.
    Gao, Chengxi, Lee, Victor C.S., Li, Keqin: DemePro: DEcouple packet marking from enqueuing for multiple services with PROactive congestion control. IEEE Trans. Cloud Comput. 1, 1–1 (2017)CrossRefGoogle Scholar
  35. 35.
    David, Z., et al.: DeTail: reducing the flow completion time tail in datacenter networks. In: Proceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication. ACM (2012)Google Scholar
  36. 36.
    Sun, G., Liao, D., Zhao, D., Sun, Z., Chang, V.: Towards provisioning hybrid virtual networks in federated cloud data centers. Future Gener. Comput. Syst. 87, 457–469 (2018)CrossRefGoogle Scholar
  37. 37.
    Alizadeh, M., Kabbani, A., Atikoglu, B., Prabhakar, B.: Stability analysis of QCN: the averaging principle. In: SIGMETRICS (2011)Google Scholar
  38. 38.
    Alizadeh, M., Javanmard, A., Prabhakar, B.: Analysis of DCTCP: Stability, convergence and fairness. In: SIGMETRICS (2011)Google Scholar
  39. 39.
    Cisco White Paper: Intelligent Buffer Management on Cisco Nexus 9000 Series Switches.
  40. 40.
    Lee, C., Nakagawa, Y., Hyoudou, K., Kobayashi, S., Shiraki, O., Shimizu, T.: Control, flow-aware congestion, to improve throughput under TCP incast in datacenter networks. In: IEEE 39th Annual Computer Software and Applications Conference. Taichung, pp. 155–162 (2015)Google Scholar
  41. 41.
    Sivaraman A., et al.: Programmable packet scheduling at line rate. In: Proc. ACMSIGCOMM Conf., pp. 4457 (2016)Google Scholar
  42. 42.
    Sharma, N. et al.: Approximating fair queueing on reconfigurable switches. In: USENIX Symposium on Networked Systems Design and Implementation (2018)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  • Pan Zhou
    • 1
  • Hongfang Yu
    • 1
    Email author
  • Gang Sun
    • 1
  • Long Luo
    • 1
  • Shouxi Luo
    • 2
  • Zilong Ye
    • 3
  1. 1.University of Electronic Science and Technology of ChinaChengduPeople’s Republic of China
  2. 2.Southwest Jiaotong UniversityChengduPeople’s Republic of China
  3. 3.California State UniversityLos AngelesUSA

Personalised recommendations