Data-Centric Task Scheduling Algorithm for Hybrid Tasks in Cloud Data Centers
With the development of big data, a demand for data analysis keeps increasing. This requirement has prompted a need for data-aware task scheduling approach that can simultaneously schedule various tasks such as batched tasks and real-time tasks in a data center efficiently. To this end, we propose a hybrid task scheduling strategy coupled with data migration in data center. Firstly, we translate the task scheduling problem into task selection problem, and give methods of selecting batched tasks and real-time tasks respectively. Then the method for scheduling both batched tasks and real-time tasks is introduced in detail. Finally, we integrate data migration into the hybrid scheduling strategy. Experimental results show that, compared to the traditional FIFO algorithm, the proposed task scheduling strategy greatly improves the data locality and data migration performs very well on reducing the job execution time. Our algorithm also guarantees an acceptable fairness for tasks.
KeywordsData analysis Data migration Batched task Real-time task Hybrid scheduling
This work is supported in part by the National Natural Science Foundation of China under Grant 61373015, in part by the Jiangsu Natural Science Foundation under Grant BK20160813 and BK20140832, in part by the National Key R&D Program of China under Grant 2018YFB1003902, in part by the Open Project Funded by State Key Laboratory of Computer Architecture under Grant CARCH201710, and in part by the Project Funded by China Postdoctoral Science Foundation.
- 1.Apache hadoop. http://hadoop.apache.org/
- 2.Apache pig. http://pig.apache.org/
- 3.Chen, Q., Zhang, D., Guo, M., Deng, Q., Guo, S.: SAMR: a self-adaptive mapreduce scheduling algorithm in heterogeneous environment. In: IEEE International Conference on Computer and Information Technology, pp. 2736–2743, June 2010Google Scholar
- 4.Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. In: Proceedings of USENIX OSDI, pp. 1–45 (2013)Google Scholar
- 6.Li, D., Wu, J., Chang, W.: Efficient cloudlet deployment: local cooperation and regional proxy. In: International Conference on Computing, Networking and Communications, pp. 757–761, March 2018Google Scholar
- 9.Li, X., Wu, J., Qian, Z., Tang, S., Lu, S.: Towards location-aware joint job and data assignment in cloud data centers with NVM. In: Proceedings of IEEE IPCCC, pp. 1–8, December 2017Google Scholar
- 11.Thomas, L., R, S.: Survey on mapreduce scheduling algorithms. Int. J. Comput. Appl. 95(23), 9–13 (2014)Google Scholar
- 12.Vavilapalli, V.K., et al.: Apache hadoop yarn: yet another resource negotiator. In: Proceedings of the 4th Annual Symposium on Cloud Computing, no. 5, October 2013Google Scholar
- 14.Yu, B., Pan, J.: Location-aware associated data placement for geo-distributed data-intensive applications. In: IEEE Conference on Computing Communications, pp. 603–611, April 2015Google Scholar
- 15.Zaharia, M., Borthakur, D., Sarma, J.S., Elmeleegy, K., Shenker, S., Stoica, I.: Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In: Proceedings of the 5th European Conference on Computer Systems, pp. 265–278. ACM (2010)Google Scholar