Inductive Reasoning

  • Ming Li
  • Paul Vitányi
Part of the Graduate Texts in Computer Science book series (TCS)


The Oxford English Dictionary defines induction as “the process of inferring a general law or principle from the observations of particular instances.” This defines precisely what we would like to call inductive inference. On the other hand, we regard inductive reasoning as a more general concept than inductive inference, namely, as a process of reassigning a probability (or credibility) to a law or proposition from the observation of particular instances.


Inductive Reasoning Inductive Inference Minimum Description Length Kolmogorov Complexity Short Program 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

History and References

  1. E. Asmis [Epicurus Scientific Method, Cornell University Press, 1984].Google Scholar
  2. J.G. Kemeny [Phil. Rev., 62(1953), 391–408]CrossRefGoogle Scholar
  3. [Phil. Trans. Roy. Soc. 25 (1763) 376-398. (Ibid., 54(1764) 298-310, R. Price (Ed.))].Google Scholar
  4. B. de Finetti [Probability, Induction, and Statistics, John Wiley & Sons, 1972].Google Scholar
  5. I.J. Good [Good Thinking, University of Minnesota Press, 1983].Google Scholar
  6. P.S. Laplace [Ibid.], R.von Mises [Probability, Statistics and Truth, Macmillan, 1939].Google Scholar
  7. T.L. Fine [Theories of Probability, Academic Press, 1973].Google Scholar
  8. R.J. Solomonoff [Inform. Contr., 7(1964), 1–22, 224-254].MathSciNetzbMATHCrossRefGoogle Scholar
  9. R.J. Solomonoff [IEEE Trans. Inform. Theory, IT-24(1978), 422–432].MathSciNetCrossRefGoogle Scholar
  10. see also [T.M. Cover, ‘Universal gambling schemes and the complexity measures of Kolmogorov and Chaitin,’ Tech. Rept. 12, 1974, Statistics Dept, Stanford Univ.].Google Scholar
  11. P. Gács [private communication of October 5, 1989].Google Scholar
  12. [P.M.B. Vitányi and M. Li, ‘Minimum Description Length Induction, Bayesianism, and Kolmogorov Complexity,’ Manuscript, CWI, Amsterdam, September 1996].Google Scholar
  13. [T.M. Cover and J.A. Thomas, Elements of Information Theory, Wiley, 1991].Google Scholar
  14. [M. Li and P.M.B. Vitányi, J. Comput. System Sci., 44:2(1992), 343–384].MathSciNetzbMATHCrossRefGoogle Scholar
  15. [T.M. Cover, ‘Universal gambling schemes and the complexity measures of Kolmogorov and Chaitin,’ Tech. Rept. 12, 1974, Statistics Dept, Stanford Univ.; The impact of processing techniques on communication, Nijhoff Publishers, 1985, pp. 23-33].Google Scholar
  16. [Problems Inform. Transmission, 25(1989), 285-292; Inform. Comput., 96(1992), 245-277].Google Scholar
  17. [A. DeSantis, G. Markowsky, and M. Wegman, Proc. 29th IEEE Symp. Found. Comput. Sci., 1988, pp. 110-119; N. Little-stone and M. Warmuth, Proc. 30th IEEE Symp. Found. Comput. Sci., 1989, pp. 256-261].Google Scholar
  18. E.M. Gold [Inform. Contr., 37(1967), 447–474].MathSciNetCrossRefGoogle Scholar
  19. D. Angluin and C. Smith [Comput. Surveys, 16(1983), 239–269]MathSciNetGoogle Scholar
  20. D. Angluin [Proc. 24th Symp. Theory Comput, 1992, pp. 351-369].Google Scholar
  21. L.G. Valiant [Comm. ACM, 27(1984), 1134–1142]zbMATHCrossRefGoogle Scholar
  22. V.N. Vapnik and A.Ya. Chervonenkis [Theory Probab. Appl., 16:2(1971), 264–280]MathSciNetzbMATHCrossRefGoogle Scholar
  23. J. Pearl [Int. J. Gen. Syst., 4(1978), 255–264]MathSciNetzbMATHCrossRefGoogle Scholar
  24. [D. Angluin, Ibid.]Google Scholar
  25. [B.K. Natarajan, Machine Learning: A Theoretical Approach, Morgan Kaufmann, 1991; M. Anthony and N. Biggs, Computational Learning Theory, Cambridge University Press, 1992; M. Kearns and U. Vazirani, Introduction to Computational Learning Theory, MIT Press, 1994].Google Scholar
  26. A. Blumer, A. Ehrenfeucht, D. Haussier, M. Warmuth [Inform. Process. Lett, 24(1987), 377–380; J. ACM, 35:4(1989), 929-965].MathSciNetzbMATHCrossRefGoogle Scholar
  27. [M. Li, Proc. 31st IEEE Symp. Found. Comput. Sci., 1990, pp. 125-134; revised version: T. Jiang and M. Li, Math. Systems Theory, to appear].Google Scholar
  28. R. Rivest [Machine Learning, 2:3(1987), 229–246].Google Scholar
  29. [R. Gavaldà, Ph.D. Thesis, Universitat Politécnica de Catalunya, 1992]Google Scholar
  30. M. Li and P.M.B. Vitányi [SIAM J. Comput, 20:5(1991), 915–935].CrossRefGoogle Scholar
  31. [V. Chvátal, Math. Oper. Res., 4:3(1979), 233–235MathSciNetzbMATHCrossRefGoogle Scholar
  32. D.S. Johnson, J. Comput. System Sci., 9(1974), 256–276.MathSciNetzbMATHCrossRefGoogle Scholar
  33. L. Lovász, Discrete Math., 13(1975), 383–390].MathSciNetzbMATHCrossRefGoogle Scholar
  34. G. Benedek and A. Itai, Theoret. Comput. Sci., 86:2(1991), 377–390.MathSciNetzbMATHCrossRefGoogle Scholar
  35. [M. Li and P.M.B. Vitányi, SIAM J. Comput., 20:5(1991), 915–935].CrossRefGoogle Scholar
  36. J. J. Rissanen in [Automatica, 14(1978), 465–471]zbMATHCrossRefGoogle Scholar
  37. C.S. Wallace and D.M. Boulton [Computing Journal, 11(1968), 185–195].zbMATHCrossRefGoogle Scholar
  38. [C.S. Wallace and P.R. Freeman, J. Royal Stat. Soc. B, 49:3(1987), 240–252, 252-265; J. Royal Stat Soc. B, 54:1(1992), 195-209]MathSciNetzbMATHGoogle Scholar
  39. [G.J. Chaitin, Scientific American, 232:5(1975), 47–52].CrossRefGoogle Scholar
  40. [Ann. Stat, 11(1982), 416-431.Google Scholar
  41. Ann. Stat, 14:3(1986), 1080-1100.Google Scholar
  42. Encyclopedia Stat Sci., V, S. Kotz and N.L. Johnson (Eds.), Wiley, 1986; J. Royal Stat Soc., 49(1987), 223-239, Discussion 252-265; and as a monograph Stochastic Complexity and Statistical Inquiry, World Scientific, 1989].Google Scholar
  43. [J.J. Rissanen, IEEE Trans. Inform. Theory, IT-42:1(1996), 40–47].MathSciNetCrossRefGoogle Scholar
  44. [M. Li and P.M.B. Vitányi, J. Comput. Syst. Sci., 44(1992), 343–384.zbMATHCrossRefGoogle Scholar
  45. M. Li and P.M.B. Vitányi, Computer Science Today, J. van Leeuwen, Ed., Lecture Notes in Computer Science, Vol. 1000, Springer-Verlag, 1995, 518-535.Google Scholar
  46. P.M.B. Vitányi and M. Li, Proc. ISIS: Information, Statistics and Induction in Science, World Scientific, Singapore, 1996, 282-291. and in particular P.M.B. Vitányi and M. Li, ‘Minimum Description Length Induction, Bayesianism, and Kolmogorov Complexity,’ Manuscript, CWI, Amsterdam, September 1996].Google Scholar
  47. [K. Yamanishi, A Randomized Approximation of the MDL for Stochastic Models with Hidden Variables, Proc. 9th ACM Conf. Comput. Learning Theory, 1996; and V. Vovk, Learning about the parameter of the Bernoulli Model, J. Comput. System Sci., to appear].Google Scholar
  48. [K. Yamanishi, Machine Learning, 9(1993), 165–203].Google Scholar
  49. [Ann. Stat., 14(1986), 1080-1100] and [’stochastic complexity and the maximum entropy principle,’ unpublished].Google Scholar
  50. [J.R. Quinlan and R. Rivest, Inform. Comput., 80(1989), 227–248].MathSciNetzbMATHCrossRefGoogle Scholar
  51. [personal communication with M. Wax, 1988].Google Scholar
  52. [Wallace C.S., Patrick J.D., Machine Learning, 11(1993), 7–22].zbMATHCrossRefGoogle Scholar
  53. [J.J. Rissanen,’ stochastic complexity in learning’ J. Comput. Syst. Th., To appear].Google Scholar
  54. [Q. Gao and M. Li, 11th IJCAI, 1989, pp. 843-848].Google Scholar
  55. [E.P.D. Pednault, 11th IJCAI, 1989, pp. 1603-1609].Google Scholar
  56. [H. Mamitsuka and K. Yamanishi, Comput. Appl. Biosciences (CABIOS), 11:4(1995), 399–411].Google Scholar
  57. [P. Cheeseman and R. Kanefsky, Working Notes, AAAI Spring Symposium Series, Stanford University, 1990].Google Scholar
  58. [L. Allison, C.S. Wallace, and C.N. Yee, Int. Symp. Artific. Intell. and Math., January 1990; pattern recognition; smoothing of planar curvesGoogle Scholar
  59. [S. Itoh, IEEE ISIT, January 1990]Google Scholar
  60. [A.R. Barron, Nonparametric Functional Estimation and Related Topics, G. Roussas, Ed., Kluwer, 1991, pp. 561-576].Google Scholar
  61. [A.R. Barron and T.M. Cover, IEEE Trans. Inform. Theory, IT-37 (1991), 1034–1054 (Correction Sept. 1991)].MathSciNetCrossRefGoogle Scholar
  62. [B. Yu, ‘Minimum description length principle: a review,’ Manuscript, Dept. of Stat., Univ. of Calif. Berkeley, 1996].Google Scholar
  63. [L. Allison, C.S. Wallace, C.N. Yee J. Mol. Evol., 35(1992), 77–89].CrossRefGoogle Scholar
  64. [D.P. McKenzie, P.D. McGorry, C.S. Wallace, L.H. Low, D.L. Copolov, B.S. Singh, Meth. Inform. Medicine, 32:2(1993), 161–166].Google Scholar
  65. [D.L. Dowe, J.J. Oliver, T.I. Dix, L. Allison, and C.S. Wallace, Proc. 26th Hawaii Int. Conf. Syst. Sciences, 1993, 669-678].Google Scholar
  66. [D.L. Dowe, D.L., L. Allison, T.I. Dix, L. Hunter, C.S. Wallace, and T. Edgoose, Proc. 1st Pacif. Symp. Biocomput. (PSB-1), Hawaii, U.S.A., 1996, 242-255].Google Scholar
  67. [Proc. 7th Australian Joint Conf. Artific. Intel., Armidale, NSW, Australia, 1994, 37-44.Google Scholar
  68. [D.L. Dowe and K.B. Korb, pp. 212-223 in Proc. Inform. Stat. and Induction in Science (ISIS) Conf., World Scientific, Singapore, 1996.Google Scholar
  69. A. Milosavljevic and J. Jurka, [Machine Learning, 12(1993), 69–87; Proc. 1st Int’l Conf. Intelligent Systems for Molecular Biology, AAAI Press, 1993, 284-291; CABIOS, 9:4(1993), 407-411].Google Scholar
  70. R.A. Fisher in [Phil. Trans. Royal Soc. London, Ser. A, 222(1925), 309–368].CrossRefGoogle Scholar
  71. [IEEE Trans. Syst. Sci. Cyb., SSC-4(1968), 227-241; Proc. IEEE, 70(1982), 939-952].Google Scholar
  72. [E.T. Jaynes, Papers on Probability, Statistics, and Statistical Physics, 2nd edition, Kluwer Academic Publishers, 1989].Google Scholar
  73. J.J. Rissanen [Ann. Stat., 14(1986), 1080–1100]MathSciNetzbMATHCrossRefGoogle Scholar
  74. M. Feder [IEEE Trans. Inform. Theory, IT-32(1986), 847–849].MathSciNetzbMATHCrossRefGoogle Scholar
  75. Rissanen [Ann. Stat., 11(1982), 416–431]MathSciNetCrossRefGoogle Scholar
  76. [H.A. Keuzenkamp and M. McAleer, The Economic Journal, 105(1995), 1–21].CrossRefGoogle Scholar
  77. [N. Chater, Psychological Review, 103(1996), 566–581].CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 1997

Authors and Affiliations

  • Ming Li
    • 1
  • Paul Vitányi
    • 2
  1. 1.Department of Computer ScienceUniversity of WaterlooWaterlooCanada
  2. 2.Centrum voor Wiskunde en InformaticaSJ AmsterdamThe Netherlands

Personalised recommendations