Abstract
Our question in this chapter is how much we can potentially lose when relying on conformal test martingales as compared with unrestricted testing of either exchangeability or statistical randomness. We will see that at a crude scale we do not lose much when restricting our attention to testing statistical randomness with conformal test martingales. Following the old tradition of the algorithmic theory of randomness, in this chapter we consider the binary case only; this makes our results less relevant in practical applications, but they are still reassuring.
We consider two very different settings of the problem. In the first, we are given an arbitrary critical region for testing exchangeability of finite binary sequences of a given length, and the question is whether there exists a conformal test martingale emulating this critical region. In the second, we are given a natural alternative, simple or composite, to the hypothesis of exchangeability, and the question is whether there exists a conformal test martingale competitive with the likelihood ratio of the alternative to the null hypothesis (suitably defined).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Arbuthnott, J.: An argument for divine Providence, taken from the constant regularity observ’d in the births of both sexes. Philos. Trans. R. Soc. Lond. 27, 186–190 (1710–1712)
Bernoulli, J.: Ars Conjectandi. Thurnisius, Basel (1713). English translation, with an introduction and notes, by Edith Dudley Sylla: The Art of Conjecturing, together with Letter to a Friend on Sets in Court Tennis. Johns Hopkins University Press, Baltimore (2006). Russian translation (second edition, with commentaries by Oscar B. Sheynin and Yurii V. Prokhorov):
, Nauka, Moscow (1986)Cournot, A.-A.: Exposition de la théorie des chances et des probabilités. Hachette, Paris (1843)
Cover, T.M., Thomas, J.A.: Elements of Information Theory, 2nd edn. Wiley, Hoboken (2006)
Grünwald, P.D.: The Minimum Description Length Principle. MIT Press, Cambridge (2007)
Jeffreys, H.: Theory of Probability, 3rd edn. Oxford University Press, Oxford (1961)
Kelly, J.L.: A new interpretation of information rate. Bell Syst. Tech. J. 35, 917–926 (1956)
Kolmogorov, A.N.: Grundbegriffe der Wahrscheinlichkeitsrechnung. Springer, Berlin (1933). Published in English as Foundations of the Theory of Probability (Chelsea, New York). First edition, 1950; second edition, 1956
Kolmogorov, A.N.: On tables of random numbers. Sankhya Ind. J. Stat. A 25, 369–376 (1963)
Kolmogorov, A.N.: Three approaches to the quantitative definition of information. Problems Inf. Transm. 1, 1–7 (1965)
Kolmogorov, A.N.: Logical basis for information theory and probability theory. IEEE Trans. Inf. Theory IT-14, 662–664 (1968)
Kolmogorov, A.N.: Combinatorial foundations of information theory and the calculus of probabilities. Russian Math. Surv. 38, 29–40 (1983). This article was written in 1970 in connection with Kolmogorov’s talk at the International Mathematical Congress in Nice
Lehmann, E.L., Romano, J.P.: Testing Statistical Hypotheses, 4th edn. Springer, Cham (2022). First edition (by Lehmann): 1959
Martin-Löf, P.: The definition of random sequences. Inf. Control 9, 602–619 (1966)
Norris, J.R.: Markov Chains. Cambridge University Press, Cambridge (1997)
Ramdas, A., Ruf, J., Larsson, M., Koolen, W.: Testing exchangeability: fork-convexity, supermartingales and e-processes. Int. J. Approx. Reason. 141, 83–109 (2022)
Robbins, H.: A remark on Stirling’s formula. Am. Math. Month. 62, 26–29 (1955)
Shafer, G.: The language of betting as a strategy for statistical and scientific communication (with discussion). J. R. Stat. Soc. A 184, 407–478 (2021)
Shafer, G., Vovk, V.: The sources of Kolmogorov’s Grundbegriffe. Stat. Sci. 21, 70–98 (2006). A greatly expanded version is published as arXiv report 1802.06071 under the title “The origins and legacy of Kolmogorov’s Grundbegriffe”
Shafer, G., Vovk, V.: Game-Theoretic Foundations for Probability and Finance. Wiley, Hoboken (2019)
Takeuchi, J., Kawabata, T., Barron, A.R.: Properties of Jeffreys mixture for Markov sources. IEEE Trans. Inf. Theory 59, 438–457 (2013)
Vovk, V.: On the concept of the Bernoulli property. Russian Math. Surv. 41, 247–248 (1986). Russian original:
. Another English translation with proofs: arXiv report 1612.08859
Vovk, V.: A logic of probability, with application to the foundations of statistics (with discussion). J. R. Stat. Soc. B 55, 317–351 (1993)
Vovk, V.: Conformal testing in a binary model situation. Proc. Mach. Learn. Res. 152, 131–150 (2021). COPA 2021
Vovk, V.: Testing randomness online. Stat. Sci. 36, 595–611 (2021)
Vovk, V., V’yugin, V.V.: On the empirical validity of the Bayesian method. J. R. Stat. Soc. B 55, 253–266 (1993)
Vovk, V., Wang, R.: E-values: calibration, combination, and applications. Ann. Stat. 49, 1736–1754 (2021)
Vovk, V., Nouretdinov, I., Gammerman, A.: Conformal testing: binary case with Markov alternatives. Proc. Mach. Learn. Res. 179, 207–218 (2022). COPA 2022
Wald, A.: Sequential Analysis. Wiley, New York (1947)
Wald, A., Wolfowitz, J.: Optimum character of the sequential probability ratio test. Ann. Math. Stat. 19, 326–339 (1948)
, Nauka, Moscow (1986)
. Another English translation with proofs: Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Vovk, V., Gammerman, A., Shafer, G. (2022). Efficiency of Conformal Testing. In: Algorithmic Learning in a Random World. Springer, Cham. https://doi.org/10.1007/978-3-031-06649-8_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-06649-8_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-06648-1
Online ISBN: 978-3-031-06649-8
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)