Stochastic Programming Approach for Resource Selection Under Demand Uncertainty

  • Tanveer Hossain Bhuiyan
  • Mahantesh HalappanavarEmail author
  • Ryan D. Friese
  • Hugh Medal
  • Luis de la Torre
  • Arun Sathanur
  • Nathan R. Tallent
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11332)


Cost-efficient selection and scheduling of a subset of geographically distributed resources to meet the demands of a scientific workflow is a challenging problem. The problem is exacerbated by uncertainties in demand and availability of resources. In this paper, we present a stochastic optimization based framework for robust decision making in the selection of distributed resources over a planning horizon under demand uncertainty. We present a novel two-stage stochastic programming model for resource selection, and implement an L-shaped decomposition algorithm to solve this model. A Sample Average Approximation algorithm is integrated to enable stochastic optimization to solve problems with a large number of scenarios. Using the metric of stochastic solution, we demonstrate up to 30% cost reduction relative to solutions without explicit consideration of demand uncertainty for a 24-month problem. We also demonstrate up to 54% cost reduction relative to a previously developed solution for a 36-month problem. We further argue that the composition of resources selected is superior to solutions computed without explicit consideration of uncertainties. Given the importance of resource selection and scheduling of complex scientific workflows, especially in the context of commercial cloud computing, we believe that our novel stochastic programming framework will benefit many researchers as well as users of distributed computing resources.



This work was supported by the Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows (IPPD) Project. IPPD is funded by the U.S. Department of Energy Awards FWP-66406 and DE-SC0012630 at the Pacific Northwest National Laboratory. The work of Luis de la Torre was supported in part by the U.S. Department of Energy, Office of Science, Office of Workforce Development for Teachers and Scientists (WDTS) under the Visiting Faculty Program (VFP).


  1. 1.
    Benders, J.F.: Partitioning procedures for solving mixed-variables programming problems. Numerische mathematik 4(1), 238–252 (1962)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Bhuiyan, T.H., Nandi, A.K., Medal, H., Halappanavar, M.: Minimizing expected maximum risk from cyber-attacks with probabilistic attack success. In: 2016 IEEE Symposium on Technologies for Homeland Security (HST), pp. 1–6. IEEE (2016)Google Scholar
  3. 3.
    Birge, J.R., Louveaux, F.: Introduction to Stochastic Programming, 2nd edn. Springer Publishing Company, New York (2011). Scholar
  4. 4.
    Chaisiri, S., Lee, B.S., Niyato, D.: Optimization of resource provisioning cost in cloud computing. IEEE Trans. Serv. Comput. 5(2), 164–177 (2012)CrossRefGoogle Scholar
  5. 5.
    Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)CrossRefGoogle Scholar
  6. 6.
    Friese, R.D., Halappanavar, M., Sathanur, A.V., Schram, M., Kerbyson, D.J., de la Torre, L.: Towards efficient resource allocation for distributed workflows under demand uncertainties. In: Klusáček, D., Cirne, W., Desai, N. (eds.) JSSPP 2017. LNCS, vol. 10773, pp. 103–121. Springer, Cham (2018). Scholar
  7. 7.
    Gurobi, O.: Gurobi optimizer reference manual (2015).
  8. 8.
    Halappanavar, M., Schram, M., de la Torre, L., Barker, K., Tallent, N.R., Kerbyson, D.J.: Towards efficient scheduling of data intensive high energy physics workflows. In: Proceedings of the 10th Workshop on Workflows in Support of Large-Scale Science, WORKS 2015, pp. 3:1–3:9. ACM, New York, USA (2015)Google Scholar
  9. 9.
    Hara, T.: Belle II: Computing and network requirements. In: Proceedings of the Asia-Pacific Advanced Network, pp. 115–122 (2014)CrossRefGoogle Scholar
  10. 10.
    Huang, Z.C., He, C., Gu, L., Wu, J.F.: On-demand service in grid: architecture, design and implementation. In: 2005 Proceedings of 11th International Conference on Parallel and Distributed Systems, vol. 2, pp. 674–678. IEEE (2005)Google Scholar
  11. 11.
    Jirutitijaroen, P., Singh, C.: Reliability constrained multi-area adequacy planning using stochastic programming with sample-average approximations. IEEE Trans. Power Syst. 23(2), 504–513 (2008)CrossRefGoogle Scholar
  12. 12.
    Johannes, A., Borhan, N., Liu, C., Ranjan, R., Chen, J.: A user demand uncertainty based approach for cloud resource management. In: 2013 IEEE 16th International Conference on Computational Science and Engineering (CSE), pp. 566–571. IEEE (2013)Google Scholar
  13. 13.
    Kleywegt, A.J., Shapiro, A., Homem-de Mello, T.: The sample average approximation method for stochastic discrete optimization. SIAM J. Optim. 12(2), 479–502 (2002)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Kusic, D., Kandasamy, N.: Risk-aware limited lookahead control for dynamic resource provisioning in enterprise computing systems. Cluster Comput. 10(4), 395–408 (2007)CrossRefGoogle Scholar
  15. 15.
    Mak, W.K., Morton, D.P., Wood, R.K.: Monte carlo bounding techniques for determining solution quality in stochastic programs. Oper. Res. Lett. 24(1), 47–56 (1999)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Medernach, E., Sanlaville, E.: Fair resource allocation for different scenarios of demands. Eur. J. Oper. Res. 218(2), 339–350 (2012)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Rodriguez, M.A., Buyya, R.: Deadline based resource provisioningand scheduling algorithm for scientific workflows on clouds. IEEE Trans. Cloud Comput. 2(2), 222–235 (2014)CrossRefGoogle Scholar
  18. 18.
    Santoso, T., Ahmed, S., Goetschalckx, M., Shapiro, A.: A stochastic programming approach for supply chain network design under uncertainty. Eur. J. Oper. Res. 167(1), 96–115 (2005)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Saravanan, B., Das, S., Sikri, S., Kothari, D.: A solution to the unit commitment problem-a review. Front. Energy 7(2), 223 (2013)CrossRefGoogle Scholar
  20. 20.
    Shapiro, A., Homem-de Mello, T.: A simulation-based approach to two-stage stochastic programming with recourse. Math. Program. 81(3), 301–325 (1998)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Van Slyke, R.M., Wets, R.: L-shaped linear programs with applications to optimal control and stochastic programming. SIAM J. Appl. Math. 17(4), 638–663 (1969)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Yang, J., Qiu, J., Li, Y.: A profile-based approach to just-in-time scalability for cloud applications. In: 2009 IEEE International Conference on Cloud Computing, CLOUD 2009, pp. 9–16. IEEE (2009)Google Scholar
  23. 23.
    Zhang, Q., Zhu, Q., Boutaba, R.: Dynamic resource allocation for spot markets in cloud computing environments. In: 2011 Fourth IEEE International Conference on Utility and Cloud Computing (UCC), pp. 178–185. IEEE (2011)Google Scholar
  24. 24.
    Zheng, Q.P., Wang, J., Pardalos, P.M., Guan, Y.: A decomposition approach to the two-stage stochastic unit commitment problem. Annal. Oper. Res. 210(1), 387–410 (2013)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Tanveer Hossain Bhuiyan
    • 1
  • Mahantesh Halappanavar
    • 2
    Email author
  • Ryan D. Friese
    • 2
  • Hugh Medal
    • 1
  • Luis de la Torre
    • 3
  • Arun Sathanur
    • 2
  • Nathan R. Tallent
    • 2
  1. 1.Mississippi State UniversityStarkvilleUSA
  2. 2.Pacific Northwest National LaboratoryRichlandUSA
  3. 3.Washington State UniversityPullmanUSA

Personalised recommendations