Abstract
The Probably Approximately Correct (PAC) learning model, which has received much attention recently in the machine learning community, attempts to formalize the notion of learning from examples. In this paper, we review several extensions to the basic PAC model with a focus on the information complexity of learning. The extensions discussed are learning over a class of distributions, learning with queries, learning functions, and learning from generalized samples.
This work was supported in part by the U.S. Army Research Office under grants DAAL03-86-K-0171 and DAAL03-92-G-0320 and by the National Science Foundation under grant IRI-9457645.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Alexandrov, A.D. and Yu.G. Reshetnyak, General Theory of Irregular Curves, Mathematics and Its Applications (Soviet Series) Vol. 29, Kluwer Academic Publishers, 1989.
Amsterdam, J., “Extending the Valiant learning model,” Proc. 5th Int. Conf, on Machine Learning, pp. 381–394, 1988.
Angluin, D. and P. Laird, “Learning from noisy examples,” Machine Learning, Vol. 2, pp. 343–370, 1988.
Angluin, D., “Queries and Concept Learning,” Machine Learning, Vol. 2, pp.319–342, 1988.
Ben-David, S., G.M. Benedek, Y. Mansour, “A parameterization scheme for classifying models of learnability,” Proc. Second Annual Workshop on Computational Learning Theory, pp. 285–302, 1989.
Ben-David, S., A. Itai, E. Kushilevitz, “Learning by distances,” Proc. of Third Annual Workshop on Computational Learning Theory, pp. 232–245, 1990.
Benedek, G.M. and A. Itai, “Learnability with respect to fixed distributions,” Theoretical Computer Science, Vol. 86(2), pp. 377–390, 1991.
Benedek, G.M. and A. Itai, “Nonuniform learnability,” ICALP, pp. 82–92, 1988.
Buescher, K. and P.R. Kumar, “Simultaneous learning and estimation for classes of probabilities,” Proc. Fourth Annual Workshop on Computational Learning Theory, Santa Cruz, CA, Aug 1991.
Blumer, A., A. Ehrenfeucht, D. Haussler, M. Warmuth, “Occam’s razor,” Info. Proc. Let., Vol. 24, pp. 377–380, 1987.
Blumer, A., A. Ehrenfeucht, D. Haussler, M. Warmuth, “Learnability and the Vapnik-Chervonenkis dimension,” J. ACM, Vol. 6, No. 4, pp. 929–965, 1989.
Dudley, R.M., “Central limit theorems for empirical measures,” Ann. Probability, Vol. 6, No. 6, pp. 899–929, 1978.
Dudley, R.M., “Metric entropy of some classes of sets with differentiate boundaries,” J. Approx. Theory, Vol. 10, No. 3, pp. 227–236, 1974.
Dudley, R.M., S.R. Kulkarni, T.J. Richardson, and O. Zeitouni, “A metric entropy bound is not sufficient for learnability,” IEEE Trans. Information Theory, Vol. 40, No. 3, pp. 883–885, 1994.
Ehrenfeucht, A., D. Haussler, M. Kearns, and L. Valiant, “A general lower bound on the number of examples needed for learning,” Information and Computation, Vol. 82, No. 3, pp. 247–251, 1989.
Eisenberg, B. and R.L. Rivest, “On the Sample Complexity of Pac-Learning Using Random and Chosen Examples,” Proc. Third Annual Workshop on Computational Learning Theory, pp. 154–162, 1990.
Elias, P., “Universal codeword sets and representations of the integers,” IEEE Trans, on Info. Theory, Vol. IT-21, No. 2, pp. 194–203, 1975.
Gallager, R.G., Information Theory and Reliable Communication, Wiley & Sons, 1968.
Gasarch, W.I and C.H. Smith, “Learning via queries,” Proc. of the 29th IEEE Symp. on Foundations of Computer Science, 1988.
Gold, I.M, “Language identification in the limit,” Information and Control, Vol. 10, pp. 447–474, 1967.
Haussler, D., M. Kearns, N. Littlestone, M.K. Warmuth, “Equivalence of models for polynomial learnability,” Proc. First Workshop on Computational Learning Theory, pp. 42–55, 1988.
Haussler, D., M. Kearns, R. Schapire, “Bounds on the sample complexity of Bayesian learning using information theory and the VC dimension,” Proc. Fourth Annual Workshop on Computational Learning Theory, Santa Cruz, CA, Aug 1991.
Haussler, D., “Decision theoretic generalizations of the PAC model for neural net and other learning applications,” Information and Computation, Vol. 100, pp. 78–150, 1992.
Karl, W.C., “Reconstructing objects from projections,” Ph.D. Thesis, Dept. of EECS, Massachusetts Institute of Technology, February, 1991.
Kearns, M. and M. Li, “Learning in the presence of malicious errors,” Proc. 20th ACM Symp. on Theory of Comp., Chicago, Illinois, pp. 267–279, 1988.
Kearns, M.J. and R.E. Schapire, “Efficient distribution-free learning of probabilistic concepts,” Proc. of the 31th IEEE Symp. on Foundations of Computer Science, 1990.
Kinber, E.B., “Some problems of learning with an oracle,” Proc. Third Workshop on Computational Learning Theory, pp. 178–186, 1990.
Kolmogorov, A.N. and V.M. Tihomirov, “£-Entropy and €-capacity of sets in functional spaces,” Amer. Math. Soc. Transi., Vol. 17, pp. 277–364, 1961.
Kulkarni, S.R., “Problems of computational and information complexity in machine vision and learning,” Ph.D. thesis, Dept. of Electrical Engineering and Computer Science, M.I.T., June, 1991.
Kulkarni, S.R., “Applications of PAC learning to problems in geometric reconstruction,” Proc. 27th Annual Conf. on Info. Sciences and Systems, Johns Hopkins University, March, 1993.
Kulkarni, S.R., S.K. Mitter, J.N. Tsitsiklis, “Active learning using arbitrary binary valued queries,” Machine Learning, Vol. 11, pp. 23–35, 1993.
Kulkarni, S.R., S.K. Mitter, J.N. Tsitsiklis, and 0. Zeitouni, “PAC learning with generalized samples and an application to stochastic geometry,” IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 15, No. 9, pp. 1–10, 1993.
Kulkarni, S.R. and D.N.C. Tse, “A paradigm for class identification problems,” IEEE Trans. Information Theory, Vol. 40, No. 3, pp. 696–705, 1994.
Kulkarni, S.R. and M. Vidyasagar, “Learning decision rules for pattern classification under a family of probability measures,” submitted to IEEE Trans. Information Theory.
Lele, A.S., S.R. Kulkarni, and A.S. Willsky, “Convex set estimation from support line measurements and applications to target reconstruction from laser radar data,” J. Optical Soc. of Amer. A, Vol. 9, No. 10, 1992.
Linial, N., Y. Mansour, R.L. Rivest, “Results on learnability and the Vapnik-Chervonenkis dimension,” Proc. First Workshop on Computational Learning Theory, pp. 56–68, 1988.
Littlestone, N., “Mistake bounds and logarithmic linear-threshold learning,” Ph.D. thesis, U.C. Santa Cruz, March, 1989.
Littlestone, N., “Learning when irrelevant attributes abound: A new linear-threshold algorithm,” Machine Learning, Vol. 2, pp. 285–318, 1988.
Moran, P.A.P., “Measuring the length of a curve,” Biometrika, Vol. 53, pp. 359–364, 1966.
Natarajan, B.K., “Probably approximate learning over classes of distributions,” Carnegie-Mellon Univ., unpublished manuscript, 1989.
Natarajan, B.K. and P.T. Tadepalli, “Two new frameworks for learning,” Proc. 5th Int. Conf. on Machine Learning, pp. 402–415, 1988.
Pollard, D., Convergence of Stochastic Processes, Springer-Verlag, 1984.
Prince, J.L. and A.S. Willsky, “Estimating convex sets from noisy support line measurements,” IEEE Trans. PAMI, Vol. 12, pp. 377–389, 1990.
Rissanen, J., “A universal prior for the integers and estimation by minimum description length,” Annals of Statistics, Vol. 11, No. 2, pp.416–431, 1983.
Rissanen, J., Stochastic Complexity in Statistical Inquiry, Series in Computer Science Vol. 15, World Scientific, 1989.
Rivest, R.L., A.R. Meyer, D.J. Kleitman, K. Winklmann, and J. Spencer, “Coping with errors in binary search procedures,” J. of Computer and System Sciences, Vol. 20, pp. 396–404, 1980.
Salzberg, S., A. Delcher, D. Heath, and S. Kasif, “Learning with a helpful teacher,” Technical Report, Dept. of Computer Science, Johns Hopkins University, 1990.
Santalo, L.A., Integral Geometry and Geometric Probability. Volume 1 of Encyclopedia of Mathematics and its Applications, Addison-Wesley, Reading, MA, 1976.
Schapire, R., “The strength of weak learnability,” Machine Learning, Vol. 5, pp.197–227, 1990.
Skiena, S.S., “Geometric probing,” Ph.D. thesis, Dept. of Computer Science, Univ. of Illinois at Urbana-Champaign, (report no. UIUCDCS-R-88–1425), April, 1988.
Sloan, R., “Types of noise in data for concept learning,” Proc. First Workshop on Computational Learning Theory, pp. 91–96, 1988.
Steinhaus, H., “Length, shape, and area,” Colloquium Mathematicum, Vol. 3, pp. 1–13, 1954.
Tikhomirov, V.M., “Kolmogorov’s work on e-entropy of functional classes and the superposition of functions,” Russian Math. Surveys, Vol. k8, pp. 51–75, 1963.
Valiant, L.G., “A theory of the learnable,” Comm. ACM, Vol. 27, No. 11, pp. 1134–1142, 1984.
Vapnik, V. N. and A. Ya. Chervonenkis, “On the uniform convergence of relative frequencies to their probabilities,” Theory of Prob. and its Appl., Vol. 16, No. 2, pp. 264–280, 1971.
Vapnik, V. N. and A. Ya. Chervonenkis, “Necessary and and sufficient conditions for the uniform convergence of means to their expectations,” Theory of Prob. and its Appl., Vol. 26, No. 3, pp. 532–553, 1981.
Vapnik, V.N., Estimation of Dependences Based on Empirical Data, Springer-Verlag, 1982.
Yamanishi, K., “A learning criterion for stochastic rules,” Proc. Third Workshop on Computational Learning Theory, pp. 67–81, 1990.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1996 Birkhäuser Boston
About this chapter
Cite this chapter
Kulkarni, S.R. (1996). A Review of Some Extensions to the PAC Learning Model. In: Kueker, D.W., Smith, C.H. (eds) Learning and Geometry: Computational Approaches. Progress in Computer Science and Applied Logic, vol 14. Birkhäuser Boston. https://doi.org/10.1007/978-1-4612-4088-4_3
Download citation
DOI: https://doi.org/10.1007/978-1-4612-4088-4_3
Publisher Name: Birkhäuser Boston
Print ISBN: 978-1-4612-8646-2
Online ISBN: 978-1-4612-4088-4
eBook Packages: Springer Book Archive