Sample average approximation and stability tests applied to energy system design

  • Pernille SeljomEmail author
  • Asgeir Tomasgard
Original Paper


This paper uses confidence intervals from sample average approximation (SAA) and stability tests to evaluate the quality of the solution of a long-term energy system model with stochastic wind power production. Using poorly designed scenarios can give stochastic model results that depend on the scenario representation rather than the actual underlying uncertainty. Nevertheless, there is little focus on the quality of the solutions of stochastic energy models in the applied literature. Our results demonstrate how too small a sample size can give a poor energy system design and misrepresent the value of the stochastic solution (VSS). We demonstrate how to evaluate the number of scenarios needed to ensure in-sample and out-of-sample stability. We also show how replication and testing of many candidate solutions using SAA iterations can provide a solution with a satisfactory confidence interval, including when the samples contain fewer scenarios than required for stability. An important observation, though, is that if SAA repeatedly solves the model with a sample size that satisfies in-sample and out-of-sample stability, the confidence interval is narrow, and the solutions are of high quality in terms of providing a tight bound for the optimal solution.



Billion euro


Combined heat and power


District heat


Expected value


Photovoltaic power


Sample average approximation


The integrated MARKAL-EFOM system


The value of stochastic solution



The authors thank the anonymous reviewer for contributing to improve the quality of the paper considerably with valuable comments and suggestions. The authors are grateful to the Research Council of Norway for funding this work through the following projects: “The future Norwegian energy system in a North-European context” (Grant number: 207067; and “Assessment of the value of flexibility services from the Norwegian energy system” (Grant number: 268097;


  1. 1.
    Loulou, R., Remme, U., Kanudia, A., Lehtila, A., Goldstein, G.: Documentation for the TIMES Model - Part I. Energy Technology Systems Analysis Program (2005). Accessed 22 Oct 2019
  2. 2.
    Loulou, R., Remme, U., Kanudia, A., Lehtila, A., Goldstein, G.: Documentation for the TIMES Model - Part II. Energy Technology Systems Analysis Programme (2005). Accessed 22 Oct 2019
  3. 3.
    Loulou, R., Lehtila, A., Kanudia, A., Remme, U., Goldstein, G.: Documentation for the TIMES Model - Part III. Energy Technology Systems Analysis Programme (2005). Accessed 22 Oct 2019
  4. 4.
    Loulou, R., Labriet, M.: ETSAP-TIAM: the TIMES integrated assessment model Part I: model structure. CMS 5, 7–40 (2008)zbMATHCrossRefGoogle Scholar
  5. 5.
    Loulou, R.: ETSAP-TIAM: the TIMES integrated assessment model Part II: mathematical formulation. Comput. Manag. Sci. 5, 41–66 (2008)MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Collins, S., et al.: Integrating short term variations of the power system into integrated energy system models: a methodological review. Renew. Sustain. Energy Rev. 76, 839–856 (2017)CrossRefGoogle Scholar
  7. 7.
    Connolly, D., et al.: A review of computer tools for analysing the integration of renewable energy into various energy systems. Appl. Energy 87(4), 1059–1082 (2010)CrossRefGoogle Scholar
  8. 8.
    Pfenninger, S., Hawkes, A., Keirstead, J.: Energy systems modeling for twenty-first century energy challenges. Renew. Sustain. Energy Rev. 33, 74–86 (2014)CrossRefGoogle Scholar
  9. 9.
    Kall, P., Wallace, S.W.: Stochastic programming. Wiley, Chichester (1994)zbMATHGoogle Scholar
  10. 10.
    Loulou, R., Lehtila, A.: Stochastic programming and tradeoff analysis in TIMES. TIMES Version 3.9 User Note. Energy Technology Systems Analysis Programme (2016). Accessed 22 Oct 2019
  11. 11.
    Nijs, W., Poncelet, K.: Integrating recurring uncertainties in ETSAP energy system models. VITO (2016).
  12. 12.
    Seljom, P., Tomasgard, A.: Short-term uncertainty in long-term energy system models—a case study of wind power in Denmark. Energy Econ. 49, 157–167 (2015)CrossRefGoogle Scholar
  13. 13.
    Seljom, P., et al.: The impact of Zero Energy Buildings on the Scandinavian energy system. Energy 118, 284–296 (2017)CrossRefGoogle Scholar
  14. 14.
    Seljom, P., Tomasgard, A.: The impact of policy actions and future energy prices on the cost-optimal development of the energy system in Norway and Sweden. Energy Policy 106, 85–102 (2017)CrossRefGoogle Scholar
  15. 15.
    Zhang, H., Chen, W., Huang, W.: TIMES modelling of transport sector in China and USA: comparisons from a decarbonization perspective. Appl. Energy 162, 1505–1514 (2016)CrossRefGoogle Scholar
  16. 16.
    Shi, J., Chen, W., Yin, X.: Modelling building’s decarbonization with application of China TIMES model. Appl. Energy 162, 1303–1312 (2016)CrossRefGoogle Scholar
  17. 17.
    Daly, H.E., et al.: Incorporating travel behaviour and travel time into TIMES energy system models. Appl. Energy 135, 429–439 (2014)CrossRefGoogle Scholar
  18. 18.
    Vaillancourt, K., et al.: A Canadian 2050 energy outlook: analysis with the multi-regional model TIMES-Canada. Appl. Energy 132, 56–65 (2014)CrossRefGoogle Scholar
  19. 19.
    Lind, A., et al.: Analysis of the EU renewable energy directive by a techno-economic optimisation model. Energy Policy 60, 364–377 (2013)CrossRefGoogle Scholar
  20. 20.
    Krook Riekkola, A., Ahlgren, E.O., Söderholm, P.: Ancillary benefits of climate policy in a small open economy: the case of Sweden. Energy Policy 39(9), 4985–4998 (2011)CrossRefGoogle Scholar
  21. 21.
    IEA: Nordic Energy Technology Perspectives. Cities, flexibility and pathways to carbon-neutrality. In: I.E. Agency (Eds.), Paris: Nordic Energy Research (2016)Google Scholar
  22. 22.
    Nagl, S., Fürsch, M., Dietmar, L.: The costs of electricity systems with a high share of fluctutating renewables: a stochastic investment and dispatch optimization Model for Europe. Energy J 34, 151–179 (2013)CrossRefGoogle Scholar
  23. 23.
    Spiecker, S., Vogel, P., Weber, C.: Evaluating interconnector investments in the north European electricity system considering fluctuating wind power penetration. Energy Econ. 37, 114–127 (2013)CrossRefGoogle Scholar
  24. 24.
    Sun, N., Ellersdorfer, I., Swider, D.J.: Model-based long-term electricity generation system planning under uncertainty. In: 3rd International Conference on Electric Utility Deregulation and Restructuring and Power Technologies, NanJing (2008)Google Scholar
  25. 25.
    Skar, C., Doorman, G., Tomasgard, A.: The future European power system under a climate policy regime. In: Energy Conference (ENERGYCON), 2014 IEEE International (2014)Google Scholar
  26. 26.
    Kleywegt, A.J., Shapiro, A., Homem-de-Mello, T.: The sample average approximation method for stochastic discrete optimization. SIAM J. Optim. 12(2), 479–502 (2001)MathSciNetzbMATHCrossRefGoogle Scholar
  27. 27.
    Shapiro, A.: Monte Carlo sampling methods. In: Ruszczynski, A., Shapiro, A. (eds.) Handbooks in operations research and management science, pp. 353–425. Elsevier, Amsterdam (2003)Google Scholar
  28. 28.
    Kaut, M., Wallace, S.W.: Evaluation of scenario-generation methods for stochastic programming. Pac. J. Optim. 3, 257–271 (2007)MathSciNetzbMATHGoogle Scholar
  29. 29.
    Glasserman, P.: Monte Carlo methods in financial engineering. Springer, New York (2013)zbMATHGoogle Scholar
  30. 30.
    Fleishman, A.I.: A method for simulating non-normal distributions. Psychometrika 43(4), 521–532 (1978)zbMATHCrossRefGoogle Scholar
  31. 31.
    Høyland, K., Kaut, M., Wallace, S.W.: A Heuristic for moment-matching scenario generation. Comput. Optim. Appl. 24(2), 169–185 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  32. 32.
    Ponomareva, K., Roman, D., Date, P.: An algorithm for moment-matching scenario generation with application to financial portfolio optimisation. Eur. J. Oper. Res. 240(3), 678–687 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  33. 33.
    Dupačová, J., Gröwe-Kuska, N., Römisch, W.: Scenario reduction in stochastic programming. Math. Program. 95(3), 493–511 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  34. 34.
    Heitsch, H., Römisch, W.: Scenario reduction algorithms in stochastic programming. Comput. Optim. Appl. 24(2), 187–206 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  35. 35.
    Heitsch, H., Römisch, W.: Scenario tree reduction for multistage stochastic programs. CMS 6(2), 117–133 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  36. 36.
    Heitsch, H., Römisch, W.: Stability and scenario trees for multistage stochastic programs. In: Dantzig, B., Infanger, G. (eds.) Stochastic programming: the state of the art in honor of George, pp. 139–164. Springer, New York (2011)zbMATHGoogle Scholar
  37. 37.
    Heitsch, H., Römisch, W.: Scenario tree modeling for multistage stochastic programs. Math. Program. 118(2), 371–406 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  38. 38.
    Pflug, G.C., Pichler, A.: Approximations for probability distributions and stochastic optimization problems. In: Bertocchi, M., Consigli, G., Dempster, M.A.H. (eds.) Stochastic optimization methods in finance and energy: new financial products and energy market strategies, pp. 343–387. Springer, New York (2011)zbMATHCrossRefGoogle Scholar
  39. 39.
    Hochreiter, R., Pflug, G.C.: Financial scenario generation for stochastic multi-stage decision processes as facility location problems. Ann. Oper. Res. 152(1), 257–272 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  40. 40.
    Pflug, G.C.: Scenario tree generation for multiperiod financial optimization by optimal discretization. Math. Program. 89(2), 251–271 (2001)MathSciNetzbMATHCrossRefGoogle Scholar
  41. 41.
    Quan, H., et al.: A computational framework for uncertainty integration in stochastic unit commitment with intermittent renewable energy sources. Appl. Energy 152, 71–82 (2015)CrossRefGoogle Scholar
  42. 42.
    Jin, S., et al.: Modeling and solving a large-scale generation expansion planning problem under uncertainty. Energy Syst. 2(3), 209–242 (2011)CrossRefGoogle Scholar
  43. 43.
    Munoz, F.D., Watson, J.-P.: A scalable solution framework for stochastic transmission and generation planning problems. CMS 12(4), 491–518 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  44. 44.
    Feng, Y., Ryan, S.M.: Scenario construction and reduction applied to stochastic power generation expansion planning. Comput. Oper. Res. 40(1), 9–23 (2013)zbMATHCrossRefGoogle Scholar
  45. 45.
    Fotouhi Ghazvini, M.A., et al.: Stochastic short-term maintenance scheduling of GENCOs in an oligopolistic electricity market. Appl. Energy 101, 667–677 (2013)CrossRefGoogle Scholar
  46. 46.
    Siahkali, H., Vakilian, M.: Stochastic unit commitment of wind farms integrated in power system. Electr. Power Syst. Res. 80(9), 1006–1017 (2010)CrossRefGoogle Scholar
  47. 47.
    Meibom, P., et al.: Stochastic optimization model to study the operational impacts of high wind penetrations in Ireland. Power Syst IEEE Trans 26(3), 1367–1379 (2011)CrossRefGoogle Scholar
  48. 48.
    Gil, E., et al.: Generation capacity expansion planning under hydro uncertainty using stochastic mixed integer programming and scenario reduction. IEEE Trans. Power Syst. 30(4), 1838–1847 (2015)CrossRefGoogle Scholar
  49. 49.
    Ottesen, S.O., Tomasgard, A.: A stochastic model for scheduling energy flexibility in buildings. Energy 88, 364–376 (2015)CrossRefGoogle Scholar
  50. 50.
    Jaehnert, S., et al.: Transmission expansion planning in Northern Europe in 2030—methodology and analyses. Energy Policy 61, 125–139 (2013)CrossRefGoogle Scholar
  51. 51.
    Ottesen, S.Ø., Tomasgard, A., Fleten, S.-E.: Prosumer bidding and scheduling in electricity markets. Energy 94, 828–843 (2016)CrossRefGoogle Scholar
  52. 52.
    Pappala, V.S., et al.: A stochastic model for the optimal operation of a wind-thermal power system. IEEE Trans. Power Syst. 24(2), 940–950 (2009)CrossRefGoogle Scholar
  53. 53.
    Baringo, L., Conejo, A.J.: Risk-constrained multi-stage wind power investment. Power Syst. IEEE Trans. 28(1), 401–411 (2013)CrossRefGoogle Scholar
  54. 54.
    Vaagen, H., Wallace, S.W.: Product variety arising from hedging in the fashion supply chains. Int. J. Prod. Econ. 114(2), 431–455 (2008)CrossRefGoogle Scholar
  55. 55.
    Fang, Y., Chen, L., Fukushima, M.: A mixed R&D projects and securities portfolio selection model. Eur. J. Oper. Res. 185(2), 700–715 (2008)zbMATHCrossRefGoogle Scholar
  56. 56.
    Hoff, A., et al.: A metaheuristic for stochastic service network design. J. Heuristic. 16(5), 653–679 (2010)zbMATHCrossRefGoogle Scholar
  57. 57.
    Qianfan, W., Jianhui, W., Yongpei, G.: Price-based unit commitment with wind power utilization constraints. Power Syst. IEEE Trans. 28(3), 2718–2726 (2013)CrossRefGoogle Scholar
  58. 58.
    Zhao, C., et al.: Expected value and chance constrained stochastic unit commitment ensuring wind power utilization. IEEE Trans. Power Syst. 29(6), 2696–2705 (2014)CrossRefGoogle Scholar
  59. 59.
    Schütz, P., Tomasgard, A., Ahmed, S.: Supply chain design under uncertainty using sample average approximation and dual decomposition. Eur. J. Oper. Res. 199(2), 409–419 (2009)zbMATHCrossRefGoogle Scholar
  60. 60.
    Santoso, T., et al.: A stochastic programming approach for supply chain network design under uncertainty. Eur. J. Oper. Res. 167(1), 96–115 (2005)MathSciNetzbMATHCrossRefGoogle Scholar
  61. 61.
    Verweij, B., et al.: The sample average approximation method applied to stochastic routing problems: a computational study. Comput. Optim. Appl. 24(2–3), 289–333 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  62. 62.
    Ünlü, Y., Rosetti, M.S.: Evaluating variance reduction techniques within a sample average approximation method for a constrained inventory policy optimization problem. In: Simulation Conference (WSC), Proceedings of the 2011 Winter (2011)Google Scholar
  63. 63.
    Long, Y., Lee, L.H., Chew, E.P.: The sample average approximation method for empty container repositioning with uncertainties. Eur. J. Oper. Res. 222(1), 65–75 (2012)zbMATHCrossRefGoogle Scholar
  64. 64.
    Swamy, C., Shmoys, D.B.: Sampling-based approximation algorithms for multistage stochastic optimization. SIAM J. Comput. 41(4), 975–1004 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  65. 65.
    Higle, L.J.: Stochastic programming: optimization when uncertainty matters. Emerging Theory, Methods, and Applications, pp. 30–53 (2005). Google Scholar
  66. 66.
    Birge, J.R.: The value of the stochastic solution in stochastic linear programs with fixed recourse. Math. Program. 24(1), 314–325 (1982)MathSciNetzbMATHCrossRefGoogle Scholar
  67. 67.
    Mak, W.-K., Morton, D.P., Wood, R.K.: Monte Carlo bounding techniques for determining solution quality in stochastic programs. Oper. Res. Lett. 24(1–2), 47–56 (1999)MathSciNetzbMATHCrossRefGoogle Scholar
  68. 68.
    Shapiro, A., Homem-de-Mello, T.: A simulation-based approach to two-stage stochastic programming with recourse. Math. Program. 81(3), 301–325 (1998)MathSciNetzbMATHCrossRefGoogle Scholar
  69. 69.
    Linderoth, J., Shapiro, A., Wright, S.: The empirical behavior of sampling methods for stochastic programming. Ann. Oper. Res. 142(1), 215–241 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  70. 70.
    Danish Energy Agency: Danmarks energifremskriving (2017). Accessed 22 Oct 2019
  71. 71.
    NVE: Kostnader i energisektoren. Kraft, varme og effektivisering (2015). The Norwegian Water Resources and Energy Directorate
  72. 72.
    Lind, A., Rosenberg, E.: TIMES-Norway model documentation. Institute for Energy Technology (2013). Accessed 22 Oct 2019
  73. 73.’s analyseforudsætninger 2015–2035 (2015). Accessed 22 Oct 2019
  74. 74.
    Kaut, M., et al.: Multi-horizon stochastic programming. CMS 11, 179–193 (2014)MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Industrial Economics and Technology ManagementNorwegian University of Science and Technology (NTNU)TrondheimNorway
  2. 2.Department of Renewable Energy SystemsInstitute for Energy Technology (IFE)KjellerNorway

Personalised recommendations