Abstract
We present a new methodology for assessing when data-based predictive models can be trusted. Particularly, we propose to learn a model from experimentation that determines, for a given labeled data set and a learning technique, when the model generated by the respective technique on the given data can be trusted to perform within specified accuracy limits. That is to say, we apply machine learning to machine learning: We repeatedly use a technique to generate models, referred as primary model, for a supervised regression problem. Based on the resulting model performance on a hold-out validation set, we then learn when the trained primary model can be expected to perform well and when there is a concern regarding the trustworthiness of that model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., Mané, D.: Concrete problems in ai safety. arXiv preprint arXiv:1606.06565 (2016)
Bhattacharyya, S.: Confidence in predictions from random tree ensembles. Knowl. Inf. Syst. 35(2), 391–410 (2013)
Bosnić, Z., Kononenko, I.: Estimation of individual prediction reliability using the local sensitivity analysis. Appl. Intell. 29(3), 187–203 (2008)
Bosnić, Z., Kononenko, I.: Automatic selection of reliability estimates for individual regression predictions. Knowl. Eng. Rev. 25(1), 27–47 (2010)
Bosnić, Z., Kononenko, I.: Correction of regression predictions using the secondary learner on the sensitivity analysis outputs. Comput. Inform. 29(6), 929–946 (2010)
Christiano, P.F., Leike, J., Brown, T., Martic, M., Legg, S., Amodei, D.: Deep reinforcement learning from human preferences. In: Advances in Neural Information Processing Systems, pp. 4299–4307 (2017)
Dietvorst, B.J., Simmons, J.P., Massey, C.: Algorithm aversion: people erroneously avoid algorithms after seeing them err. J. Exp. Psychol. Gen. 144(1), 114–126 (2015)
Asuncion, A., Newman, D.: UCI Machine Learning Repository. University of California, School of Information and Computer Science, Irvine, CA (2007)
Groce, A., et al.: You are the only possible oracle: effective test selection for end users of interactive machine learning systems. IEEE Trans. Software Eng. 40(3), 307–323 (2014)
Heskes, T.: Practical confidence and prediction intervals. In: Advances in Neural Information Processing Systems, pp. 176–182 (1997)
Jiang, H., Kim, B., Guan, M.Y., Gupta, M.: To trust or not to trust a classifier. In: 32nd International Conference on Neural Information Processing Systems, pp. 5546–5557 (2018)
Liaw, A., Wiener, M., et al.: Classification and regression by randomforest. R News 2(3), 18–22 (2002)
Lipton, Z.C., Wang, Y.X., Smola, A.: Detecting and correcting for label shift with black box predictors. arXiv preprint arXiv:1802.03916 (2018)
Matsumoto, E.Y., Del-Moral-Hernandez, E.: Improving regression predictions using individual point reliability estimates based on critical error scenarios. Inf. Sci. 374, 65–84 (2016)
Nushi, B., Kamar, E., Horvitz, E.: Towards accountable AI: hybrid human-machine analyses for characterizing system failure. In: Sixth AAAI Conference on Human Computation and Crowdsourcing (2018)
Papadopoulos, H., Vovk, V., Gammerman, A.: Regression conformal prediction with nearest neighbours. J. Artif. Intell. Res. 40, 815–840 (2011)
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z.B., Swami, A.: The limitations of deep learning in adversarial settings. In: 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 372–387. IEEE (2016)
Pevec, D., Kononenko, I.: Prediction intervals in supervised learning for model evaluation and discrimination. Appl. Intell. 42(4), 790–804 (2014). https://doi.org/10.1007/s10489-014-0632-z
Prahl, A., Van Swol, L.: Understanding algorithm aversion: when is advice from automation discounted? J. Forecast. 36(6), 691–702 (2017)
Shafer, G., Vovk, V.: A tutorial on conformal prediction. J. Mach. Learn. Res. 9, 371–421 (2008)
Suykens, J.A., Vandewalle, J.: Least squares support vector machine classifiers. Neural Process. Lett. 9(3), 293–300 (1999)
Yeomans, M., Shah, A., Mullainathan, S., Kleinberg, J.: Making sense of recommendations. J. Behav. Decis. Making 32(4), 403–414 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Castro, M.P., Sellmann, M., Yang, Z., Virani, N. (2020). Empirical Confidence Models for Supervised Machine Learning. In: Goutte, C., Zhu, X. (eds) Advances in Artificial Intelligence. Canadian AI 2020. Lecture Notes in Computer Science(), vol 12109. Springer, Cham. https://doi.org/10.1007/978-3-030-47358-7_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-47358-7_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-47357-0
Online ISBN: 978-3-030-47358-7
eBook Packages: Computer ScienceComputer Science (R0)