Abstract
To estimate the quality of the induced predictive model we generally use measures of averaged prediction accuracy, such as the relative mean squared error on test data. Such evaluation fails to provide local information about reliability of individual predictions, which can be important in risk-sensitive fields (medicine, finance, industry etc.). Related work presented several ways for computing individual prediction reliability estimates for single-target regression models, but has not considered their use with multi-target regression models that predict a vector of independent target variables. In this paper we adapt the existing single-target reliability estimates to multi-target models. In this way we try to design reliability estimates, which can estimate the prediction errors without knowing true prediction errors, for multi-target regression algorithms, as well. We approach this in two ways: by aggregating reliability estimates for individual target components, and by generalizing the existing reliability estimates to higher number of dimensions. The results revealed favorable performance of the reliability estimates that are based on bagging variance and local cross-validation approaches. The results are consistent with the related work in single-target reliability estimates and provide a support for multi-target decision making.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Kukar, M., Kononenko, I.: Reliable classifications with machine learning. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) ECML 2002. LNCS (LNAI), vol. 2430, pp. 219–231. Springer, Heidelberg (2002). doi:10.1007/3-540-36755-1_19
Pevec, D., Kononenko, I.: Input dependent prediction intervals for supervised regression. Intell. Data Anal. 18(5), 873–887 (2014)
Bosnić, Z., Kononenko, I.: Comparison of approaches for estimating reliability of individual regression predictions. Data Knowl. Eng. 67(3), 504–516 (2008)
Bosnić, Z., Kononenko, I.: Estimation of individual prediction reliability using the local sensitivity analysis. Appl. Intell. 29(3), 187–203 (2008)
Kocev, D., Džeroski, S., White, M.D., Newell, G.R., Griffioen, P.: Using single-and multi-target regression trees and ensembles to model a compound index of vegetation condition. Ecol. Model. 220(8), 1159–1168 (2009)
Kocev, D., Vens, C., Struyf, J., Džeroski, S.: Ensembles of multi-objective decision trees. In: Kok, J.N., Koronacki, J., Mantaras, R.L., Matwin, S., Mladenič, D., Skowron, A. (eds.) ECML 2007. LNCS (LNAI), vol. 4701, pp. 624–631. Springer, Heidelberg (2007). doi:10.1007/978-3-540-74958-5_61
Breiman, L., Friedman, J., Stone, C.J., Olshen, R.A.: Classification and Regression Trees. CRC Press, New York (1984)
Segal, M.R.: Tree-structured methods for longitudinal data. J. Am. Stat. Assoc. 87(418), 407–418 (1992)
De’Ath, G.: Multivariate regression trees: a new technique for modeling species-environment relationships. Ecology 83(4), 1105–1117 (2002)
Struyf, J., Džeroski, S.: Constraint based induction of multi-objective regression trees. In: Bonchi, F., Boulicaut, J.-F. (eds.) KDID 2005. LNCS, vol. 3933, pp. 222–233. Springer, Heidelberg (2006). doi:10.1007/11733492_13
Blockeel, H., De Raedt, L., Ramon, J.: Top-down induction of clustering trees (2000). arXiv preprint: arXiv:cs/0011032
Segal, M., Xiao, Y.: Multivariate random forests. Wiley Interdisc. Rev.: Data Min. Knowl. Discov. 1(1), 80–87 (2011)
Džeroski, S., Ženko, B.: Stacking with multi-response model trees. In: Roli, F., Kittler, J. (eds.) MCS 2002. LNCS, vol. 2364, pp. 201–211. Springer, Heidelberg (2002). doi:10.1007/3-540-45428-4_20
Golub, G., Kahan, W.: Calculating the singular values and pseudo-inverse of a matrix. J. Soc. Ind. Appl. Math. Ser. B: Numer. Anal. 2(2), 205–224 (1965)
Tsoumakas, G., Katakis, I., Vlahavas, I.: Mining multi-label data. In: Maimon, O., Rokach, L. (eds.) Data Mining and Knowledge Discovery Handbook, pp. 667–685. Springer, New York (2009)
Lichman, M.: UCI machine learning repository. School of Information and Computer Sciences, University of California, Irvine (2013). http://archive.ics.uci.edu/ml
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Jakomin, M., Bosnić, Z. (2017). Reliability Estimation of Individual Multi-target Regression Predictions. In: Wagner, M., Li, X., Hendtlass, T. (eds) Artificial Life and Computational Intelligence. ACALCI 2017. Lecture Notes in Computer Science(), vol 10142. Springer, Cham. https://doi.org/10.1007/978-3-319-51691-2_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-51691-2_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-51690-5
Online ISBN: 978-3-319-51691-2
eBook Packages: Computer ScienceComputer Science (R0)