Advertisement

DDS: A Deadline Driven Workflow Scheduling Algorithm for Hybrid Amazon Instances

  • Zitai Ma
  • Jian CaoEmail author
  • Shiyou Qian
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9464)

Abstract

Workflows can orchestrate multiple applications that need resources to execute. The cloud computing has emerged as an on-demand resource provisioning paradigm, which can support workflow execution. In recent years, Amazon offers a new service option, i.e., EC2 spot instances, whose price is on average more than 75 % lower than the one of on-demand instances. Therefore, we can make use of spot instances to execute workflows in a cost-efficient way. However, the spot instances is cut off when their price increases and exceeds the customer’s bid, which will make the task failed and the execution time becomes unpredictable. We propose a deadline driven scheduling (DDS) algorithm which is able to use both on-demand and spot instances to reduce the cost while the deadline of workflows can also be guaranteed with a high probability. Especially, we use an attribute, called global weight, to represent the interdependency relations of tasks and schedule the tasks whose interdependent tasks need longer time first to reduce the whole execution time. The experimental results demonstrate that DDS algorithm is effective in reducing cost while satisfying the deadline constraints of workflows.

Keywords

Workflow Scheduling algorithm Spot instance Cloud computing 

Notes

Acknowledgements

This work is partially supported by China National Science Foundation (Granted Number 61272438, 61472253), Research Funds of Science and Technology Commission of Shanghai Municipality (Granted Number 15411952502, 12511502704).

References

  1. 1.
    Case Studies of Amazon AWS service. http://aws.amazon.com/solutions/case-studies
  2. 2.
    Yu, J., Buyya, R., Tham, C.K.: A cost-based scheduling of scientific workflow applications on utility grids. In: Proceedings of the 1st IEEE International Conference on e-Science and Grid Computing, vol. 8 p. 147 (2005)Google Scholar
  3. 3.
    Duan, R., Prodan, R., Fahringer, T.: Performance and cost optimization for multiple large-scale grid workflow applications. In: SC 2007 Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, 2007, p. 12. IEEE (2007)Google Scholar
  4. 4.
    Mao, M., Humphrey, M.: Auto-Scaling to minimize cost and meet application deadlines in cloud workflows. In: Proceedings of International Conference on High Performance Computing, Networking, Storage and Analysis (SC), pp. 49:1–49:12 (2011)Google Scholar
  5. 5.
    Byun, E.K., Kee, Y.S., Kim, J.S., et al.: Cost optimized provisioning of elastic resources for application workflows. Future Gener. Comput. Syst. 27(8), 1011–1026 (2011)CrossRefGoogle Scholar
  6. 6.
    Yi, S., Andrzejak, A., Kondo, D.: Monetary cost-aware checkpointing and migration on amazon cloud spot instances. IEEE Trans. Serv. Comput. 5, 512–524 (2011)CrossRefGoogle Scholar
  7. 7.
    Voorsluys, W., Buyya, R.: Reliable provisioning of spot instances for compute-intensive applications, pp. 542–549 (2012)Google Scholar
  8. 8.
    Zhong, H., Tao, K., Zhang, X.: An approach to optimized resource scheduling algorithm for open-source cloud systems. In: Fifth Annual China Grid Conference (2010)Google Scholar
  9. 9.
    Lin, C., Lu, S.: Scheduling scientific workflows elastically for cloud computing. In: IEEE 4th International Conference on Cloud Computing (2011)Google Scholar
  10. 10.
    Liu, H., Xu, D., Miao, H.: Ant colony optimization based service flow scheduling with various QoS requirements in cloud computing. In: 2011 First ACIS International Symposium on Software and Network Engineering (SSNE), pp. 53–58. IEEE (2011)Google Scholar
  11. 11.
    Yu, J., Buyya, R., Tham, C.K.: Cost-based scheduling of scientific workflow application on utility grids. In: First International Conference on e-Science and Grid Computing (2005)Google Scholar
  12. 12.
    Kllapi, H., Sitaridi, E., Tsangaris, M.M., Ioannidis, Y.: Schedule optimization for data processing flows on the cloud. In: Proceedings of the ACM SIGMOD International Conference on Management of Data (2011)Google Scholar
  13. 13.
    Wang, H., Jing, Q., Chen, R., He, B., Qian, Z., Zhou, L.: Distributed systems meet economics: pricing in the cloud. In: Proceedings of the Second USENIX Conference on Hot Topics in Cloud Computing (HotCloud), pp. 6–6 (2010)Google Scholar
  14. 14.
    Herodotou, H., Babu, S.: Profiling, what-if analysis, and cost-based optimization of mapreduce programs. Proc. VLDB Endowment 4(11), 1111–1122 (2011)Google Scholar
  15. 15.
    Zhou, A.C., He, B., Liu, C.: Probabilistic scheduling of scientific workflows in dynamic cloud environments. In: CoRR (2013)Google Scholar
  16. 16.
    Juve, G., Chervenak, A., Deelman, E., et al.: Characterizing and profiling scientific workflows. Future Gener. Comput. Syst. 29(3), 682–692 (2013)CrossRefGoogle Scholar
  17. 17.
    Garey M.R., Johnson, D.S.: Computers and intractability: a guide to the theory of NP-completeness. W.H. Freeman & Co., (2003)Google Scholar
  18. 18.
    Kwok, Y.K., Ahmad, I.: Static scheduling algorithms for allocating directed task graphs to multiprocessors. ACM Comput. Surv. 31(4), 406–471 (1999)CrossRefGoogle Scholar
  19. 19.
    Sakellariou, R., Zhao, H.: A hybrid heuristic for DAG scheduling on heterogeneous systems. In: The 13th Heterogeneous Computing Workshop (HCW 2004), Santa Fe, New, Mexico, USA, 26 April 2004Google Scholar
  20. 20.
    Topcuoglu, H., Hariri, S., Wu, M.-Y.: Task scheduling algorithms for heterogeneous processors. In: 8th Proceedings of Heterogeneous Computing Workshop (1999)Google Scholar
  21. 21.
    Javadi, B., Thulasiramy, R.K., Buyya, R.: Statistical modeling of spot instance prices in public cloud environments. In: 2011 Fourth IEEE International Conference on Utility and Cloud Computing, pp. 219–228. IEEE Computer Society (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Shanghai Jiao Tong UniversityShanghaiChina

Personalised recommendations