Abstract
In Chap. 8 we argued for complementing a prediction rule, i.e., a trained prediction algorithm, with a system for testing deviations from exchangeability. As soon as serious violations of exchangeability are detected, we start retraining the prediction algorithm or take other appropriate measures. In this short chapter we suggest a shortcut: we test directly the predictions output by the prediction rule, and a successful way of testing then automatically translates to improved (“protected”) predictions. In this way we apply methods developed in the first two chapters of this part outside conformal prediction. Our procedures of protected prediction can be used from the very beginning, or after detecting lack of exchangeability (but before retraining is complete). We consider separately the case of regression, in which we combine the probability integral transformation and betting martingales, and classification, in which we move even further from conformal prediction.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Anscombe, F.J.: Rejection of outliers. Technometrics 2, 123–147 (1960)
Brier, G.W.: Verification of forecasts expressed in terms of probability. Month. Weather Rev. 78, 1–3 (1950)
Cox, D.R.: Two further applications of a model for binary regression. Biometrika 45, 562–565 (1958)
Dawid, A.P.: Calibration-based empirical probability (with discussion). Ann. Stat. 13, 1251–1285 (1985)
Dawid, A.P.: Probability forecasting. In: Kotz, S., Johnson, N.L., Read, C.B. (eds.) Encyclopedia of Statistical Sciences, vol. 7, pp. 210–218. Wiley, New York (1986). Reprinted in the second edition (2006) on pp. 6445–6452 (Volume 10)
Dawid, A.P., Vovk, V.: Prequential probability: principles and properties. Bernoulli 5, 125–162 (1999)
Gama, J., Medas, P., Castillo, G., Rodrigues, P.: Learning with drift detection. In: Bazzan, A.L.C., Labidi, S. (eds.) Advances in Artificial Intelligence: SBIA 2004, pp. 286–295. Springer, Berlin (2004)
Gneiting, T., Raftery, A.E.: Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc. 102, 359–378 (2007)
Gneiting, T., Balabdaoui, F., Raftery, A.E.: Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. B 69, 243–268 (2007)
Good, I.J.: Rational decisions. J. R. Stat. Soc. B 14, 107–114 (1952)
Harries, M.: Splice-2 comparative evaluation: electricity pricing. Tech. Rep. UNSW-CSE-TR-9905, Artificial Intelligence Group, School of Computer Science and Engineering, University of New South Wales (1999)
Herbster, M., Warmuth, M.K.: Tracking the best expert. Mach. Learn. 32, 151–178 (1998)
Huber, P.J., Ronchetti, E.M.: Robust Statistics, 2nd edn. Wiley, Hoboken (2009). First edition (by Huber): 1981
Kahneman, D., Slovic, P., Tversky, A. (eds.): Judgment under Uncertainty: Heuristics and Biases. Cambridge University Press, Cambridge, England (1982)
Kolmogorov, A.N.: Sur la notion de la moyenne. Atti della Reale Accademia Nazionale dei Lincei. Classe di scienze fisiche, matematiche, e naturali. Rendiconti Serie VI 12(9), 388–391 (1930)
Kolmogorov, A.N.: Grundbegriffe der Wahrscheinlichkeitsrechnung. Springer, Berlin (1933). Published in English as Foundations of the Theory of Probability (Chelsea, New York). First edition, 1950; second edition, 1956
Lévy, P.: Théorie de l’addition des variables aléatoires. Gauthier-Villars, Paris (1937). Second edition: 1954
Littlestone, N., Warmuth, M.K.: The weighted majority algorithm. Inf. Comput. 108, 212–261 (1994)
Rosenblatt, M.: Remarks on a multivariate transformation. Ann. Math. Stat. 23, 470–472 (1952)
Shafer, G., Vovk, V.: Game-Theoretic Foundations for Probability and Finance. Wiley, Hoboken (2019)
Torres-Sospedra, J., Montoliu, R., Martínez-Usó, A., Avariento, J.P, Arnau, T.J., Benedito-Bordonau, M., Huerta, J: UJIIndoorLoc: a new multi-building and multi-floor database for WLAN fingerprint-based indoor localization problems. In: 2014 International Conference on Indoor Positioning and Indoor Navigation (IPIN 2014), pp. 261–270. Institute of Electrical and Electronics Engineers (2014)
Vovk, V.: Aggregating strategies. In: Fulk, M., Case, J. (eds.) Proceedings of the Third Annual Workshop on Computational Learning Theory, pp. 371–383. Morgan Kaufmann, San Mateo (1990)
Vovk, V.: Universal forecasting algorithms. Inf. Comput. 96, 245–277 (1992)
Vovk, V.: Derandomizing stochastic prediction strategies. Mach. Learn. 35, 247–282 (1999)
Vovk, V.: Competitive on-line statistics. Int. Stat. Rev. 69, 213–248 (2001)
Vovk, V.: The fundamental nature of the log loss function. In: Beklemishev, L.D., Blass, A., Dershowitz, N., Finkbeiner, B., Schulte, W. (eds.) Fields of Logic and Computation II: Essays Dedicated to Yuri Gurevich on the Occasion of His 75th Birthday. Lecture Notes in Computer Science, vol. 9300, pp. 307–318. Springer, Cham (2015)
Vovk, V.: Protected probabilistic regression. Tech. Rep. arXiv:2105.08669 [cs.LG], arXiv.org e-Print archive (2021)
Vovk, V., Zhdanov, F.: Prediction with expert advice for the Brier game. J. Mach. Learn. Res. 10, 2445–2471 (2009)
Vovk, V., Petej, I., Gammerman, A.: Protected probabilistic classification. Tech. Rep. arXiv:2107.01726 [cs.LG], arXiv.org e-Print archive (2021). Short version published as poster extended abstract: Proceedings of Machine Learning Research 152, 297–299 (2021). COPA 2021
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Vovk, V., Gammerman, A., Shafer, G. (2022). Non-conformal Shortcut. In: Algorithmic Learning in a Random World. Springer, Cham. https://doi.org/10.1007/978-3-031-06649-8_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-06649-8_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-06648-1
Online ISBN: 978-3-031-06649-8
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)