Abstract
Probability theory as logic is founded on three simple desiderata: that degrees of belief should be represented by real numbers, that one should reason consistently, and that the theory should reduce to Aristotelian logic when the truth values of the hypotheses are known. Because this theory represents a probability as a state of knowledge, not a state of nature, hypotheses such as “The frequency of oscillation of a sinusoidal signal had value ω when the data were taken,” or “Model x is a better description of the data than model y” make perfect sense. Problems of the first type are generally thought of as parameter estimation problems, while problems of the second type are thought of as model selection problems. However, in probability theory there is no essential distinction between these two types of problems. They are both solved by application of the sum and product rules of probability theory. Model selection problems are conceptually more difficult, because the models may have different functional forms. Consequently, conceptual difficulties enter the problem that are not present in parameter estimation. This paper is a tutorial on model selection. The conceptual problems that arise in model selection will be illustrated in such a way as to automatically avoid any difficulties. A simple example is worked in detail. This example, (radar target identification) illustrates all of the points of principle that must be faced in more complex model selection problems, including how to handle nuisance parameters, uninformative prior probabilities, and incomplete sets of models.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
William of Ockham, ca 1340.
Jeffreys, H., Theory of Probability, Oxford University Press, London, 1939; Later editions, 1948, 1961.
Jaynes, E. T., JASA,Sept. 1979, p. 740, review of “Inference, Methods, and Decision: Towards a Bayesian Philosophy of Science.” by R. D. Rosenkrantz, D. Reidel Publishing Co., Boston.
Gull, S. F., “Bayesian Inductive Inference and Maximum Entropy,” in Maximum Entropy and Bayesian Methods in Science and Engineering“ 1, pp. 53–75, G. J. Erickson and C. R. Smith Eds, Kluwer Academic Publishers, Dordrecht the Netherlands, 1988.
Bretthorst, G. Larry, “Bayesian Spectrum Analysis and Parameter Estimation,” in Lecture Notes in Statistics 48, Springer-Verlag, New York, New York, 1988.
Bretthorst, G. Larry, “Bayesian Analysis I: Parameter Estimation Using Quadrature NMR Models,” J. Magn. Reson, 88, pp. 533–551 (1990).
Bretthorst, G. Larry, “Bayesian Analysis II: Model Selection,” J. Magn. Reson, 88, pp. 552–570 (1990).
Bretthorst, G. Larry, “Bayesian Analysis III: Spectral Analysis,” J. Magn. Reson, 88, pp. 571–595 (1990).
Tribus, M., Rational Descriptions, Decisions and Designs, Pergamon Press, Oxford, 1969.
Zellner, A., An Introduction to Bayesian Inference in Econometrics, John Wiley and Sons, New York, 1971. Second edition (1987); R. E. Krieger Pub. Co., Malabar, Florida.
Jaynes, E. T., “Probability Theory — The Logic of Science,” in preparation. Copies of this TeX manuscript are available by anonymous FTP from “bayes.wustl.edu”
Jaynes, E. T., “How Does the Brain do Plausible Reasoning?” unpublished Stanford University Microwave Laboratory Report No. 421 (1957); reprinted in Maximum-Entropy and Bayesian Methods in Science and Engineering 1, pp. 1–24, G. J. Erickson and C. R. Smith Eds, 1988.
Bretthorst, G. Larry, An Introduction to Parameter Estimation Using Bayesian Probability Theory,“ in Maximum Entropy and Bayesian Methods, Dartmouth College 1989, P. Fougère ed, Kluwer Academic Publishers, Dordrecht the Netherlands, 1990.
Bayes, Rev. T., “An Essay Toward Solving a Problem in the Doctrine of Chances,” Philos. Trans. R. Soc. London 53, pp. 370–418 (1763); reprinted in Biornetrika 45, pp. 293–315 (1958), and Facsimiles of Two Papers by Bayes, with commentary by W. Edwards Deming, New York, Hafner, 1963.
Laplace, P. S., A Philosophical Essay on Probabilities, unabridged and unaltered reprint of Truscott and Emory translation, Dover Publications, Inc., New York, 1951, original publication date 1814.
Jaynes, E. T., “Prior Probabilities,” IEEE Transactions on Systems Science and Cybernetics,SSC-4, pp. 227–241 (1968); reprinted in [20].
Shore J. E., R. W. Johnson, IEEE Trans. on Information Theory, IT-26, No. 1, pp. 26–37, 1981.
Shore J. E., R. W. Johnson, IEEE Trans. on Information Theory, IT-27, No. 4, pp. 472–482, 1980.
Jaynes, E. T., “Where Do We Stand On Maximum Entropy?” in The Maximum Entropy Formalism,R. D. Levine and M. Tribus Eds,pp. 15–118, Cambridge: MIT Press, 1978; Reprinted in [20].
Jaynes, E. T., Papers on Probability, Statistics and Statistical Physics, a reprint collection, D. Reidel, Dordrecht the Netherlands, 1983; second edition Kluwer Academic Publishers, Dordrecht the Netherlands, 1989.
Jaynes, E. T., “Marginalization and Prior Probabilities,” in Bayesian Analysis in Econometrics and Statistics,A. Zellner, ed,North-Holland Publishing Company, Amsterdam, 1980; reprinted in [20].
Shannon, C. E., “A Mathematical Theory of Communication,” Bell Syst. Tech. J 27, pp. 379–423 (1948).
Jaynes, E. T., 1989, “The Theory of Radar Target Discrimination,” in MICOM Technical Report RD-AS-91–6, Feb. 1991.
Bretthorst, G. Larry, “Radar Target Identification The Information Processing Aspects,” Contract number DAAL03–92-c-0034, TCN 92060.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1996 Springer Science+Business Media Dordrecht
About this chapter
Cite this chapter
Bretthorst, G.L. (1996). An Introduction to Model Selection Using Probability Theory as Logic. In: Heidbreder, G.R. (eds) Maximum Entropy and Bayesian Methods. Fundamental Theories of Physics, vol 62. Springer, Dordrecht. https://doi.org/10.1007/978-94-015-8729-7_1
Download citation
DOI: https://doi.org/10.1007/978-94-015-8729-7_1
Publisher Name: Springer, Dordrecht
Print ISBN: 978-90-481-4407-5
Online ISBN: 978-94-015-8729-7
eBook Packages: Springer Book Archive