Advertisement

Industrial Prediction Intervals with Data Uncertainty

  • Jun Zhao
  • Wei Wang
  • Chunyang Sheng
Chapter
Part of the Information Fusion and Data Science book series (IFDS)

Abstract

Prediction intervals (PIs) construction is a comprehensive prediction technique that provides not only the point estimates of the industrial variables, but also the reliability of the prediction results indicated by an interval. Reviewing the conventional PIs construction methods (e.g., delta method, mean and variance-based estimation method, Bayesian method, and bootstrap technique), we provide some recently developed approaches in this chapter. Here, a bootstrapping-based ESN ensemble (BESNE) model is specially proposed to produce reliable PIs for industrial time series, in which a simultaneous training method based on Bayesian linear regression is developed. Besides, to cope with the error accumulation caused by the traditional iterative mode of time series prediction, a non-iterative granular ESN is also reported for PIs construction, where the network connections are represented by the interval-valued information granules. In addition, we present a mixed Gaussian kernel-based regression model to construct PIs, in which a gradient descent algorithm is derived to optimize the hyper-parameters of the mixed Gaussian kernel. In order to tackle the incomplete testing input problem, a kernel-based high order dynamic Bayesian network (DBN) model for industrial time series is then proposed, which directly deals with the missing points involved in the inputs. Finally, we provide some case studies to verify the effectiveness of these approaches.

Keywords

Prediction intervals Time series Reliability Uncertainty Delta method Mean and variance-based estimation Bootstrap technique ESNs ensemble Bayesian estimation Non-iterative prediction Interval-weighted Gamma test Mixed Gaussian kernel Noisy inputs ANNs Missing input DBNs Approximate inference 

References

  1. 1.
    Zapranis, A., & Livanis, E. (2005). Prediction intervals for neural network models. In Proceedings of the 9th WSEAS International Conference on Computers. World Scientific and Engineering Academy and Society (WSEAS)Google Scholar
  2. 2.
    De Veaux, R. D., Schumi, J., Schweinsberg, J., & Ungar, L. H. (1998). Prediction intervals for neural networks via nonlinear regression. Technometrics, 40(4), 273–282.MathSciNetCrossRefGoogle Scholar
  3. 3.
    Hwang, J. T. G., & Ding, A. A. (1997). Prediction intervals for artificial neural networks. Journal of the American Statistical Association, 92(438), 748–757.MathSciNetCrossRefGoogle Scholar
  4. 4.
    Nix, D. A., & Weigend, A. S. (1994). Estimating the mean and variance of the target probability distribution. In Proceedings of the IEEE International Conference on Neural Networks, Orlando, FL (Vol. 1, pp. 55–60).Google Scholar
  5. 5.
    Rivals, I., & Personnaz, L. (2000). Construction of confidence intervals for neural networks based on least squares estimation. Neural Networks, 13(4–5), 463.CrossRefGoogle Scholar
  6. 6.
    Ding, A., & He, X. (2003). Backpropagation of pseudo-errors: Neural networks that are adaptive to heterogeneous noise. IEEE Transactions on Neural Networks, 14(2), 253–262.CrossRefGoogle Scholar
  7. 7.
    Dybowski, R., & Roberts, S. (2000). Confidence intervals and prediction intervals for feed-forward neural networks. In R. Dybowski & V. Gant (Eds.), Clinical applications of artificial neural networks. Cambridge, U.K: Cambridge University Press.Google Scholar
  8. 8.
    Bishop, C. M. (1995). Neural networks for pattern recognition. London, UK: Oxford University Press.zbMATHGoogle Scholar
  9. 9.
    MacKay, D. J. C. (1989). The evidence framework applied to classification networks. Neural Computation, 4(5), 720–736.CrossRefGoogle Scholar
  10. 10.
    Hagan, M., & Menhaj, M. (2002). Training feedforward networks with the Marquardt algorithm. IEEE Transactions on Neural Networks, 5(6), 989–993.CrossRefGoogle Scholar
  11. 11.
    Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Annals of Statistics, 7(1), 1–26.MathSciNetCrossRefGoogle Scholar
  12. 12.
    Heskes, T. (1997). Practical confidence and prediction intervals. In T. P. M. Mozer & M. Jordan (Eds.), Neural information processing systems (Vol. 9, pp. 176–182). Cambridge, MA: MIT Press.Google Scholar
  13. 13.
    Sheng, C., Zhao, J., Wang, W., et al. (2013). Prediction intervals for a noisy nonlinear time series based on a bootstrapping reservoir computing network ensemble. IEEE Transactions on Neural Networks & Learning Systems, 24(7), 1036–1048.CrossRefGoogle Scholar
  14. 14.
    Tibshirani, R. (1996). A comparison of some error estimates for neural network models. Neural Computation, 8(1), 152–163.MathSciNetCrossRefGoogle Scholar
  15. 15.
    Khosravi, A., Nahavandi, S., Creighton, D., et al. (2011). Comprehensive review of neural network-based prediction intervals and new advances. IEEE Transactions on Neural Networks, 22(9), 1341–1356.CrossRefGoogle Scholar
  16. 16.
    Anguita, D., Ghio, A., Oneto, L., et al. (2012). In-sample and out-of-sample model selection and error estimation for support vector machines. IEEE Transactions on Neural Networks & Learning Systems, 23(9), 1390.CrossRefGoogle Scholar
  17. 17.
    Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York, USA: Chapman & Hall.CrossRefGoogle Scholar
  18. 18.
    Efron, B., & Tibshirani, R (1995). Cross-validation and the bootstrap: Estimating the error rate of a prediction rule. Dept. Stat., Stanford Univ., Stanford, CA, USA, Tech. Rep. TR-477.Google Scholar
  19. 19.
    Arlot, S., & Celisse, A. (2010). A survey of cross-validation procedures for model selection. Statistics Surveys, 4, 40–79.MathSciNetCrossRefGoogle Scholar
  20. 20.
    Efron, B., & Tibshirani, R. (1997). Improvements on cross-validation: The .632+ bootstrap method. Journal of the American Statistical Association, 92(438), 548–560.MathSciNetzbMATHGoogle Scholar
  21. 21.
    Xue, Y., Yang, L., & Haykin, S. (2007). Decoupled echo state networks with lateral inhibition. Neural Networks, 20(3), 365–376.CrossRefGoogle Scholar
  22. 22.
    Sheng, C., Zhao, J., & Wang, W. (2017). Map-reduce framework-based non-iterative granular echo state network for prediction intervals construction. Neurocomputing, 222, 116–126.CrossRefGoogle Scholar
  23. 23.
    Dong, R., & Pedrycz, W. (2008). A granular time series approach to long-term forecasting and trend forecasting. Physica A: Statistical Mechanics and its Applications, 387(13), 3253–3270.CrossRefGoogle Scholar
  24. 24.
    Song, M., & Pedrycz, W. (2013). Granular neural networks: Concepts and development schemes. IEEE Transactions on Neural Networks & Learning Systems, 24(4), 542–553.CrossRefGoogle Scholar
  25. 25.
    Cimino, A., Lazzerini, B., Marcelloni, F., et al. (2011). Granular data regression with neural networks, Fuzzy logic and applications. Lecture Notes in Computer Science, 6857, 172–179.CrossRefGoogle Scholar
  26. 26.
    Jaeger, H., & Haas, H. (2004). Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science, 304, 78–80.CrossRefGoogle Scholar
  27. 27.
    Jaeger, H. (2002). Tutorial on training recurrent neural networks, covering BPTT, RTRL, EKF and echo state network approach. German National Research Center for Information Technology, GMD Rep. 159.Google Scholar
  28. 28.
    Zhao, J., Wang, W., Liu, Y., et al. (2011). A two-stage online prediction method for a blast furnace gas system and its application. IEEE Transactions on Control Systems Technology, 19(3), 507–520.CrossRefGoogle Scholar
  29. 29.
    Zhao, J., Liu, Q., Wang, W., et al. (2012). Hybrid neural prediction and optimized adjustment for coke oven gas system in steel industry. IEEE Transactions on Neural Networks and Learning Systems, 23(3), 439–450.CrossRefGoogle Scholar
  30. 30.
    Liu, Y., Liu, Q., Wang, W., et al. (2012). Data-driven based model for flow prediction of steam system in steel industry. Information Sciences, 193, 104–114.CrossRefGoogle Scholar
  31. 31.
    Pedrycz, W., & Homenda, W. (2013). Building the fundamentals of granular computing: A principle of justifiable granularity. Applied Soft Computing, 13(10), 4209–4218.CrossRefGoogle Scholar
  32. 32.
    Jones, J. A., Evans, D., & Kemp, S. E. (2007). A note on the Gamma test analysis of noisy input/output data and noisy time series. Physica D: Nonlinear Phenomena, 229(1), 1–8.MathSciNetCrossRefGoogle Scholar
  33. 33.
    Liitiainen, E., Verleysen, M., Corona, F., & Lendasse, A. (2009). Residual variance estimation in machine learning. Neurocomputing, 72(16–18), 3692–3703.CrossRefGoogle Scholar
  34. 34.
    Evans, D., & Jones, A. J. (2002). A proof of the gamma test. Society of London. Series A, 458, 2759–2799.MathSciNetCrossRefGoogle Scholar
  35. 35.
    Khosravi, A., Nahavandi, S., Creighton, D., et al. (2011). Lower upper bound estimation method for construction of neural network-based prediction intervals. IEEE Transactions on Neural Networks, 22(3), 337–346.CrossRefGoogle Scholar
  36. 36.
    Kennedy, J., & Eberhart, R. (1995). Particle swarm optimization. In Proceedings of the IEEE International Conference on Neural Networks (pp. 1942–1948). Piscataway: IEEE Service Center.Google Scholar
  37. 37.
    Mackey, M. C., & Glass, L. (1977). Oscillation and chaos in physiological control systems. Science, 197(4300), 287–289.CrossRefGoogle Scholar
  38. 38.
    De, B. K., De, B. J., Suykens, J. A., et al. (2011). Approximate confidence and prediction intervals for least squares support vector regression. IEEE Transactions on Neural Networks, 22(1), 110–120.CrossRefGoogle Scholar
  39. 39.
    Bishop, C. M. (2006). Pattern recognition and machine learning. New York: Springer Press.zbMATHGoogle Scholar
  40. 40.
    Vapnik, V. (1995). The nature of statistical learning theory. New York: Springer.CrossRefGoogle Scholar
  41. 41.
    Boyd, V., & Faybusovich, L. (2006). Convex optimization. IEEE Transactions on Automatic Control, 51(11), 1859–1859.CrossRefGoogle Scholar
  42. 42.
    Wright, W. A. (1999). Bayesian approach to neural-network modeling with input uncertainty. IEEE Transactions on Neural Networks, 10(6), 1261.CrossRefGoogle Scholar
  43. 43.
    Chen, L., Liu, Y., Zhao, J., Wang, W., & Liu, Q. (2016). Prediction intervals for industrial data with incomplete input using kernel-based dynamic Bayesian networks. Artificial Intelligence Review, 46, 307–326.CrossRefGoogle Scholar
  44. 44.
    Fung, R., & Chang, K. C. (1990). Weighting and integrating evidence for stochastic simulation in Bayesian networks. In P. P. Bonissone, M. Henrion, L. N. Kanal, & J. F. Lemmer (Eds.), Uncertainty in artificial intelligence (Vol. 5, pp. 208–219). North Holland: Elsevier.Google Scholar
  45. 45.
    Tipping, M. E. (2001). Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research, 1, 211–244.MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Jun Zhao
    • 1
  • Wei Wang
    • 1
  • Chunyang Sheng
    • 2
  1. 1.Dalian University of TechnologyDalianChina
  2. 2.Shandong University of Science and TechnologyQingdaoChina

Personalised recommendations