Abstract
Accurate forecasting of a high variability time series has relevance in many applications such as supply-chain management, price prediction in stock markets and demand forecasting in the energy segment. Most often forecasts of such time series depend on many factors ranging from weather to socio-economic attributes such as GDP or average income. Dependence on such features can cause the underlying time series to be highly variable in nature and possess non-stationary shifts. Most traditional forecasting methods fail to capture such trend shifts and high variability present in the data. Further, for certain applications, it may be necessary to estimate the confidence of the forecasts. In this work, we propose two variants of recurrent mixture density network (RMDN), for time series forecasting, that have the ability to handle high-dimensional input features, capture trend shifts and high variability present in the data, and provide a confidence estimate of the forecast. To this end, we first pass the high-dimensional time series data through a feedforward layer, which performs dimensionality reduction or feature selection in an unsupervised manner by inducing sparsity on the weights of the feedforward layer. The resultant low-dimensional time series is then fed through recurrent layers to capture temporal patterns. These recurrent layers also aid in learning the latent representation of the input data. Thereafter, a mixture density network (MDN) is used to model the variability and trend shifts present in the input and it also estimates the confidence of the predictions. The models are trained in an end-to-end fashion and the efficacy of the proposed models is demonstrated on three publicly available datasets from energy markets.
Keywords
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
In case of ED, \(\mathbf {z}_{t}\) comes from decoder.
- 2.
- 3.
- 4.
References
Amarawickrama, H., Hunt, L.: Electricity demand for Sri Lanka: a time series analysis. Energy 33(5), 724–739 (2008). https://doi.org/10.1016/j.energy.2007.12.008
Bazzani, L., Larochelle, H., Torresani, L.: Recurrent mixture density network for spatiotemporal visual attention. arXiv preprint arXiv:1603.08199 (2016)
Beenstock, M., Goldin, E., Nabot, D.: The demand for electricity in Israel. Energy Econ. 21(2), 168–183 (1999). https://doi.org/10.1016/s0140-9883(98)00005-x
Bishop, C.M.: Mixture density networks. Technical report, Citeseer (1994)
Bouktif, S., Fiaz, A., Ouni, A., Serhani, M.: Optimal deep learning LSTM model for electric load forecasting using feature selection and genetic algorithm: comparison with machine learning approaches. Energies 11(7), 1636 (2018)
Cheng, Y., Xu, C., Mashima, D., Thing, V.L.L., Wu, Y.: PowerLSTM: power demand forecasting using long short-term memory neural network. In: Cong, G., Peng, W.-C., Zhang, W.E., Li, C., Sun, A. (eds.) ADMA 2017. LNCS (LNAI), vol. 10604, pp. 727–740. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-69179-4_51
Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)
DiPietro, R., Hager, G.D.: Unsupervised learning for surgical motion by learning to predict the future. arXiv preprint arXiv:1806.03318 (2018)
Douglas, A., Breipohl, A., Lee, F., Adapa, R.: Risk due to load forecast uncertainty in short term power system planning. IEEE Trans. Power Syst. 13(4), 1493–1499 (1998). https://doi.org/10.1109/59.736296
Gonzalez-Romera, E., Jaramillo-Moran, M., Carmona-Fernandez, D.: Monthly electric energy demand forecasting based on trend extraction. IEEE Trans. Power Syst. 21(4), 1946–1953 (2006). https://doi.org/10.1109/tpwrs.2006.883666
Graves, A.: Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850 (2013)
Graves, A., Rahman, M.A., Hinton, G.: Speech recognition with deep recurrent neural networks. In: IEEE International Conference on Acoustics, Speech and Signal processing (ICASSP), pp. 6645–6649. IEEE (2013)
Gugulothu, N., Malhotra, P., Vig, L., Shroff, G.: Sparse neural networks for anomaly detection in high-dimensional time series. In: AI4IOT Workshop in Conjunction with ICML, IJCAI and ECAI, July 2018
Gugulothu, N., TV, V., Malhotra, P., Vig, L., Agarwal, P., Shroff, G.: Predicting remaining useful life using time series embeddings based on recurrent neural networks. Int. J. Prognostics Health Manage. IJPHM. arXiv preprint arXiv:1709.01073 (2017)
Gugulothu, N., et al.: On practical aspects of using RNNs for fault detection in sparsely-labeled multi-sensor time series. In: PHM Society Conference, vol. 10 (2018)
Guo, Y., Liao, W., Wang, Q., Yu, L., Ji, T., Li, P.: Multidimensional time series anomaly detection: a GRU-based Gaussian mixture variational autoencoder approach. In: Asian Conference on Machine Learning, pp. 97–112 (2018)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Ketter, W., Collins, J., Weerdt, M.: The 2018 Power Trading Agent Competition (2017). https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3087096/
Malhotra, P., Ramakrishnan, A., Anand, G., Vig, L., Agarwal, P., Shroff, G.: LSTM-based encoder-decoder for multi-sensor anomaly detection. arXiv preprint arXiv:1607.00148 (2016)
Morales, F.J.O., Roggen, D.: Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. In: Sensors (2016)
Mukherjee, S., et al.: ARMDN: associative and recurrent mixture density networks for eRetail demand forecasting. arXiv preprint arXiv:1803.03800 (2018)
Oehmcke, S., Zielinski, O., Kramer, O.: Input quality aware convolutional LSTM networks for virtual marine sensors. Neurocomputing 275, 2603–2615 (2018). https://doi.org/10.1016/j.neucom.2017.11.027
Oğcu, G., Demirel, O.F., Zaim, S.: Forecasting electricity consumption with neural networks and support vector regression. Procedia Soc. Behav. Sci. 58, 1576–1585 (2012). https://doi.org/10.1016/j.sbspro.2012.09.1144
Polson, M., Sokolov, V.: Deep learning for energy markets. arXiv preprint arXiv:1808:05527 (2014)
Samouilidis, J.E., Mitropoulos, C.: Energy and economic growth in industrializing countries: the case of Greece. Energy Econ. 6(3), 191–201 (1984). https://doi.org/10.1016/0140-9883(84)90016-1
Saurav, S., et al.: Online anomaly detection with concept drift adaptation using recurrent neural networks. In: Proceedings of the ACM India Joint International Conference on Data Science and Management of Data, CoDS-COMAD 2018, pp. 78–87. ACM, New York (2018). https://doi.org/10.1145/3152494.3152501
Scardapane, S., Comminiello, D., Hussain, A., Uncini, A.: Group sparse regularization for deep neural networks. Neurocomputing 241, 81–89 (2017). https://doi.org/10.1016/j.neucom.2017.02.029
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)
Tibshirani, R.: Regression shrinkage and selection via the lasso. J. Roy. Stat. Soc. Ser. B (Methodol.) 58, 267–288 (1996)
Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, ICML 2008, pp. 1096–1103. ACM, New York (2008). https://doi.org/10.1145/1390156.1390294
Xu, J., Rahmatizadeh, R., Bölöni, L., Turgut, D.: Real-time prediction of taxi demand using recurrent neural networks. IEEE Trans. Intell. Transp. Syst. 19(8), 2572–2581 (2018). https://doi.org/10.1109/TITS.2017.2755684
Xu, Y., Kong, Q., Huang, Q., Wang, W., Plumbley, M.D.: Convolutional gated recurrent neural network incorporating spatial features for audio tagging. In: International Joint Conference on Neural Networks IJCNN, pp. 3461–3466 (2017). https://doi.org/10.1109/IJCNN.2017.7966291
Zhang, H., Heiden, E., Julian, R., He, Z., Lim, J.J., Sukhatme, G.S.: Auto-conditioned recurrent mixture density networks for complex trajectory generation. arXiv preprint arXiv:1810.00146 (2018)
Zhao, Y., Yang, R., Chevalier, G., Shah, R., Romijnders, R.: Applying deep bidirectional LSTM and mixture density network for basketball trajectory prediction. CoRR abs/1708.05824 (2017). http://arxiv.org/abs/1708.05824
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Gugulothu, N., Subramanian, E., Bhat, S.P. (2019). Sparse Recurrent Mixture Density Networks for Forecasting High Variability Time Series with Confidence Estimates. In: Tetko, I., Kůrková, V., Karpov, P., Theis, F. (eds) Artificial Neural Networks and Machine Learning – ICANN 2019: Deep Learning. ICANN 2019. Lecture Notes in Computer Science(), vol 11728. Springer, Cham. https://doi.org/10.1007/978-3-030-30484-3_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-30484-3_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-30483-6
Online ISBN: 978-3-030-30484-3
eBook Packages: Computer ScienceComputer Science (R0)