Skip to main content
SpringerLink
Log in

On Learnability, Complexity and Stability

  • Chapter
  • First Online:
Empirical Inference

  • 3914 Accesses

Abstract

We considerStability—( Learnability—( the fundamental question of learnability of a hypothesis class in the supervised learningSupervised learning setting and in the general learningGeneral learning setting introduced by Vladimir Vapnik. We survey classic results characterizing learnability in terms of suitable notions of complexity, as well as more recent results that establish the connection between learnability and stability of a learning algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    ConsistencyConsistency can be defined with respect to other convergence notions for random variables. If the loss function is bounded, convergence in probability is equivalent to convergence in expectation.

  2. 2.

    We say that a learning algorithm A is symmetric if it does not depend on the order of the points in z n .

  3. 3.

    Note that this construction is not possible in classification or in regression with the square loss.

References

  1. Alon, N., Ben-David, S., Cesa-Bianchi, N., Haussler, D.: Scale-sensitive dimensions, uniform convergence, and learnability. J. ACM 44(4), 615–631 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  2. Anthony, M., Bartlett, P.L.: Neural network learning: theoretical foundations. Cambridge University Press, Cambridge (1999)

    Book  MATH  Google Scholar 

  3. Bartlett, P., Long, P., Williamson, R.: Fat-shattering and the learnability of real-valued functions. J. Comput. Syst. Sci. 52, 434–452 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  4. Bousquet, O., Elisseeff, A.: Stability and generalization. J. Mach. Learn. Res. 2, 499–526 (2002)

    MathSciNet  MATH  Google Scholar 

  5. Daniely, A., Sabato, S., Ben-David, S., Shalev-Shwartz, S.: Multiclass learnability and the ERM principle. J. Mach. Learn. Res. Proc. Track 19, 207–232 (2011)

    Google Scholar 

  6. Devroye, L., Györfi, L., Lugosi, G.: A Probabilistic Theory of Pattern Recognition. Applications of Mathematics 31. Springer, New York (1996)

    Google Scholar 

  7. Dudley, R., Giné, E., Zinn, J.: Uniform and universal Glivenko-Cantelli classes. J. Theor. Prob. 4, 485–510 (1991)

    Article  MATH  Google Scholar 

  8. Engl, H.W., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Mathematics and Its Applications, vol. 375. Kluwer, Dordrecht (1996)

    Google Scholar 

  9. Kearns, M.J., Schapire, R.E.: Efficient distribution-free learning of probabilistic concepts. In: Computational Learning Theory and Natural Learning Systems. Bradford Books, vol. I, pp. 289–329. MIT, Cambridge (1994)

    Google Scholar 

  10. Kutin, S., Niyogi, P.: Almost-everywhere algorithmic stability and generalization error. Technical report TR-2002-03, Department of Computer Science, The University of Chicago (2002)

    Google Scholar 

  11. Mukherjee, S., Niyogi, P., Poggio, T., Rifkin, R.: Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization. Adv. Comput. Math. 25(1–3), 161–193 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  12. Poggio, T., Rifkin, R., Mukherjee, S., Niyogi, P.: General conditions for predictivity in learning theory. Nature 428, 419–422 (2004)

    Article  Google Scholar 

  13. Rakhlin, A., Sridharan, K., Tewari, A.: Online learning: beyond regret. J. Mach. Learn. Res. Proc. Track 19, 559–594 (2011)

    Google Scholar 

  14. Shalev-Shwartz, S., Shamir, O., Srebro, N., Sridharan, K.: Learnability, stability and uniform convergence. J. Mach. Learn. Res. 11, 2635–2670 (2010)

    MathSciNet  MATH  Google Scholar 

  15. Steinwart, I., Christmann, A.: Support Vector Machines. Information Science and Statistics. Springer, New York (2008)

    MATH  Google Scholar 

  16. Vapnik, V.: The Nature of Statistical Learning Theory. Springer, New York (1995)

    Book  MATH  Google Scholar 

  17. Vapnik, V.N., Chervonenkis, A.Y.: Theory of uniform convergence of frequencies of events to their probabilities and problems of search for an optimal solution from empirical data. Avtomatika i Telemekhanika 2, 42–53 (1971)

    MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Laboratory for Computational and Statistical Learning, Istituto Italiano di Tecnologia and Massachusetts Institute of Technology, MIT Building 46-5155, 77 Massachusetts Avenue, Cambridge, MA, 02139, USA

    Silvia Villa & Lorenzo Rosasco

  2. Dipartimento di Informatica, Istituto Italiano di Tecnologia, Bioingegneria, Robotica e Ingegneria dei Sistemi, Universit‘a di Genova, Via Dodecaneso 35, 16146, Genova, Italy

    Lorenzo Rosasco

  3. Center for Biological and Computational Learning, Massachusetts Institute of Technology, Building 46-5155, 77 Massachusetts Avenue, Cambridge, MA, 02139, USA

    Tomaso Poggio

Authors
  1. Silvia Villa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lorenzo Rosasco
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tomaso Poggio
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Silvia Villa .

Editor information

Editors and Affiliations

  1. Max Planck Institute for Intelligent Systems, Tübingen, Germany

    Bernhard Schölkopf

  2. Dept. of Computer Science, Royal Holloway, University of London, Egham, Surrey, United Kingdom

    Zhiyuan Luo

  3. Department of Computer Science, Royal Holloway, University of London, Egham, Surrey, United Kingdom

    Vladimir Vovk

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Villa, S., Rosasco, L., Poggio, T. (2013). On Learnability, Complexity and Stability. In: Schölkopf, B., Luo, Z., Vovk, V. (eds) Empirical Inference. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-41136-6_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-41136-6_7

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-41135-9

  • Online ISBN: 978-3-642-41136-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation