Skip to main content
SpringerLink
Log in

Stacked LSTM Snapshot Ensembles for Time Series Forecasting

  • Conference paper
  • First Online:
Theory and Applications of Time Series Analysis (ITISE 2018)

Part of the book series: Contributions to Statistics ((CONTRIB.STAT.))

Included in the following conference series:

Abstract

Ensembles of machine learning models have proven to improve the performance of prediction tasks in various domains. The additional computational costs for the performance increase are usually high since multiple models must be trained. Recently, snapshot ensembles (Huang et al. in Snapshot ensembles: train 1 get M for free, (2017) [16]) provide a comparably computationally cheap way of ensemble learning for artificial neural networks (ANNs). We extend snapshot ensembles to the application of time series forecasting, which comprises two essential steps. First, we show that determining reasonable selections for sequence lengths can be used to efficiently escape local minima. Additionally, combining the forecasts of snapshot LSTMs with a stacking approach greatly boosts the performance compared to the mean of the forecasts as used in the original snapshot ensemble approach. We demonstrate the effectiveness of the algorithm on five real-world datasets and show that the forecasting performance of our approach is superior to conservative ensemble architectures as well as a single, highly optimized LSTM.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.qlik.com/us/products/qlik-data-market, accessed June 19, 2019.

  2. 2.

    https://github.com/saschakrs/TS-SnapshotEnsemble, accessed June 1, 2018.

References

  1. Adhikari, R.: A neural network based linear ensemble framework for time series forecasting. Neurocomputing 157(2015), 231–242 (2015)

    Article  Google Scholar 

  2. Adhikari, R., Agrawal, R.K.: A linear hybrid methodology for improving accuracy of time series forecasting. Neural Comput. Appl. 25(2), 269–281 (2014)

    Article  Google Scholar 

  3. Aladag, C.H., Egrioglu, E., Kadilar, C.: Forecasting nonlinear time series with a hybrid methodology. Appl. Math. Lett. 22(9), 1467–1470 (2009)

    Article  Google Scholar 

  4. Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994)

    Article  Google Scholar 

  5. Borovykh, A., Bohte, S., Oosterlee, C.W.: Conditional time series forecasting with convolutional neural networks. J. Computat. Financ. (2018)

    Google Scholar 

  6. Cerqueira, V., et al.: Arbitrated ensemble for time series forecasting. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, Cham (2017)

    Chapter  Google Scholar 

  7. Elfeky, M.G., Aref, W.G., Elmagarmid, A.K.: Periodicity detection in time series databases. IEEE Trans. Knowl. Data Eng. 17(7), 875–887 (2005)

    Article  Google Scholar 

  8. Gers, F.A., Eck, D., Schmidhuber, J.: Applying LSTM to time series predictable through time-window approaches. In: Neural Nets WIRN Vietri-01, pp. 193–200. Springer (2002)

    Google Scholar 

  9. Gothwal, H., Kedawat, S., Kumar, R.: Cardiac arrhythmias detection in an ECG beat signal using fast fourier transform and artificial neural network. J. Biomedi. Sci. Eng. 4(04), 289 (2011)

    Article  Google Scholar 

  10. Hamilton, J.D.: Time Series Analysis, vol. 2. Princeton University Press, Princeton (1994)

    Google Scholar 

  11. He, Z., Gao, S., Xiao, L., Liu, D., He, H., Barber, D.: Wider and deeper, cheaper and faster: tensorized LSTMs for sequence learning. In: Advances in Neural Information Processing Systems, pp. 1–11 (2017)

    Google Scholar 

  12. Hipel, K.W., McLeod, A.I.: Time Series Modelling of Water Resources and Environmental Systems, vol. 45. Elsevier (1994)

    Google Scholar 

  13. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  14. Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2(5), 359–366 (1989)

    Article  Google Scholar 

  15. Hu, B., Lu, Z., Li, H., Chen, Q.: Convolutional neural network architectures for matching natural language sentences. In: Advances in Neural Information Processing Systems, pp. 2042–2050 (2014)

    Google Scholar 

  16. Huang, G., Li, Y., Pleiss, G., Li, Z., Hopcroft, J., Weinberger, K.: Snapshot ensembles: train 1 get M for free. In: Proceedings of the International Conference on Learning Representations (ICLR 2017) (2017)

    Google Scholar 

  17. Krstanovic, S., Paulheim, H.: Ensembles of recurrent neural networks for robust time series forecasting. In: International Conference on Innovative Techniques and Applications of Artificial Intelligence, pp. 34-46. Springer (2017)

    Google Scholar 

  18. Lichman, M.: UCI Machine Learning Repository (2013). http://archive.ics.uci.edu/ml

  19. Lngkvist, M., Karlsson, L., Loutfi, A.: A review of unsupervised feature learning and deep learning for time-series modeling. Pattern Recognit. Lett. 42, 11–24 (2014)

    Google Scholar 

  20. Malhotra, P., Vig, L., Shroff, G., Agarwal, P.: Long short term memory networks for anomaly detection in time series. In: Proceedings Presses Universitaires de Louvain, vol. 89 (2015)

    Google Scholar 

  21. Oliveira, M., Torgo, L.: Ensembles for time series forecasting. In: JMLR: Workshop and Conference Proceedings, vol. 39, pp. 360–370 (2014)

    Google Scholar 

  22. Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. In: International Conference on Machine Learning, pp. 1310–1318 (2013)

    Google Scholar 

  23. Pratt, H., et al.: FCNN: Fourier Convolutional Neural Networks. Machine Learning and Knowledge Discovery in Databases, Springer, Cham (2017)

    Chapter  Google Scholar 

  24. Sharma, D., Issac, B., Raghava, G.P.S., Ramaswamy, R.: Spectral Repeat Finder (SRF): identification of repetitive sequences using Fourier transformation. Bioinformatics 20(9), 1405–1412 (2004)

    Article  Google Scholar 

  25. Shumway, R.H., Stoffer, D.S.: Time Series Analysis and Its Applications: With R Examples. Springer, New York (2010)

    Google Scholar 

  26. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)

    Google Scholar 

  27. Van Den Oord, A., et al.: WaveNet: a generative model for raw audio. In: SSW (2016)

    Google Scholar 

  28. Wang, L., Zou, H., Su, J., Li, L., Chaudhry, S.: An ARIMA-ANN hybrid model for time series forecasting. Syst. Res. Behav. Sci. 30(3), 244–259 (2013)

    Article  Google Scholar 

  29. Welch, P.: The use of fast Fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Trans. Audio Electroacoust. 15(2), 70–73 (1967)

    Article  Google Scholar 

  30. Wen, R., Torkkola, K., Narayanaswamy, B.: A Multi-Horizon Quantile Recurrent Forecaster. NIPS 2017 Time Series Workshop (2017)

    Google Scholar 

  31. Zhang, L., Suganthan, P.N.: Benchmarking ensemble classifiers with novel co-trained Kernal Ridge regression and random vector functional link ensembles [Research Frontier]. IEEE Computat. Intell. Maga. 12(4), 61–72 (2017)

    Article  Google Scholar 

  32. Zhang, P.G.: Time series forecasting using a hybrid ARIMA and neural network model. Neurocomputing 50(2003), 159–175 (2003)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Research Group Data and Web Science, University of Mannheim, Mannheim, Germany

    Sascha Krstanovic & Heiko Paulheim

Authors
  1. Sascha Krstanovic
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Heiko Paulheim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sascha Krstanovic .

Editor information

Editors and Affiliations

  1. Faculty of Sciences, University of Granada, Granada, Spain

    Olga Valenzuela

  2. ETSIIT, CITIC-UGR, University of Granada, Granada, Spain

    Fernando Rojas

  3. ETSIIT, CITIC-UGR, University of Granada, Granada, Spain

    Héctor Pomares

  4. ETSIIT, CITIC-UGR, University of Granada, Granada, Spain

    Ignacio Rojas

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Krstanovic, S., Paulheim, H. (2019). Stacked LSTM Snapshot Ensembles for Time Series Forecasting. In: Valenzuela, O., Rojas, F., Pomares, H., Rojas, I. (eds) Theory and Applications of Time Series Analysis. ITISE 2018. Contributions to Statistics. Springer, Cham. https://doi.org/10.1007/978-3-030-26036-1_7

Download citation

Publish with us

Policies and ethics

Search

Navigation