Advertisement

Neural Processing Letters

, Volume 34, Issue 2, pp 133–154 | Cite as

On the Curse of Dimensionality in Supervised Learning of Smooth Regression Functions

  • Elia Liitiäinen
  • Francesco Corona
  • Amaury Lendasse
Article

Abstract

In this paper, the effect of dimensionality on the supervised learning of infinitely differentiable regression functions is analyzed. By invoking the Van Trees lower bound, we prove lower bounds on the generalization error with respect to the number of samples and the dimensionality of the input space both in a linear and non-linear context. It is shown that in non-linear problems without prior knowledge, the curse of dimensionality is a serious problem. At the same time, we speculate counter-intuitively that sometimes supervised learning becomes plausible in the asymptotic limit of infinite dimensionality.

Keywords

Supervised learning Van Trees Minimax Analytic function Nonparametric regression High dimensional 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Tsitsiklis JN, Bertsekas DP (1996) Neuro-dynamic programming. Optimization and neural computation, vol 3. Athena Scientific, BelmontGoogle Scholar
  2. 2.
    Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. Adaptive computation and machine learning. MIT Press, CambridgeGoogle Scholar
  3. 3.
    Stone CJ (1982) Optimal global rates of convergence for nonparametric regression. Ann Stat 10(4): 1040–1053MATHCrossRefGoogle Scholar
  4. 4.
    Van Trees H (2001) Detection, estimation, and modulation theory, part I. Wiley-Interscience, New YorkCrossRefGoogle Scholar
  5. 5.
    Bobrovsky BZ, Mayer-Wolf E, Zakai M (1987) Some classes of global Cramer–Rao bounds. Ann Stat 15(4): 1421–1438MathSciNetMATHCrossRefGoogle Scholar
  6. 6.
    Golubev YK, Levit BY, Tsybakov AB (1996) Some classes of global Cramer–Rao bounds. Bernoulli 2(2): 167–181MathSciNetMATHCrossRefGoogle Scholar
  7. 7.
    Guerre E, Tsybakov AB (1998) Exact asymptotic minimax constants for the estimation of analytic functions in l p. Probab Theory Relat Fields 2(112): 33–51MathSciNetCrossRefGoogle Scholar
  8. 8.
    Ibragimov I (2001) Estimation of analytic functions. Lect Notes Monogr Ser 36: 359–383CrossRefGoogle Scholar
  9. 9.
    Artiles LM, Levit BY (2003) Adaptive regression on the real line in classes of smooth functions. Aust J Stat 32(1–2): 99–129Google Scholar
  10. 10.
    Belitser E, Levit B (2003) On the empirical Bayes approach to adaptive filtering. Math Methods Stat 12(2): 131–154MathSciNetGoogle Scholar
  11. 11.
    Barron A (1993) Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans Inf Theory 39(3): 930–945MathSciNetMATHCrossRefGoogle Scholar
  12. 12.
    Verleysen M (2003) Learning high-dimensional data. In: Ablameyko S, Goras L, Gori M, Piuri V (eds) Limitations and future trends in neural computation. IOS Press, Amsterdam, pp 141–162Google Scholar
  13. 13.
    Verleysen M, Francois D (2005) The curse of dimensionality in data mining and time series prediction. In: Cabestany J, Prieto A, Sandoval F (eds) Computational intelligence and bioinspired systems. Lecture notes in computer science, vol 3512. Springer, Berlin, pp 758–770CrossRefGoogle Scholar
  14. 14.
    Cramer H (1946) Mathematical methods of statistics. Princeton University Press, PrincetonMATHGoogle Scholar
  15. 15.
    Boucheron S, Bousquet O, Lugosi G (2004) Concentration inequalities. In: Bousquet O, Luxburg UV, Rätsch G (eds) Advanced lectures in machine learning. Springer, Berlin, pp 208–240CrossRefGoogle Scholar
  16. 16.
    Rasmussen C, Williams C (2006) Gaussian processes for machine learning. MIT Press, CambridgeMATHGoogle Scholar
  17. 17.
    Sorjamaa A, Hao J, Reyhani N, Ji Y, Lendasse A (2007) Methodology for long-term prediction of time series. Neurocomputing 70(16–18): 2861–2869CrossRefGoogle Scholar
  18. 18.
    Guillén A, Sovilj D, Mateo F, Rojas I, Lendasse A (2008) Minimizing the Delta test for variable selection in regression problems. Int J High Perform Syst Archit 1(4): 269–281CrossRefGoogle Scholar
  19. 19.
    Kreyszig E (1989) Introductory functional analysis with applications, 1 edn. Wiley, New YorkMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC. 2011

Authors and Affiliations

  • Elia Liitiäinen
    • 1
  • Francesco Corona
    • 1
  • Amaury Lendasse
    • 1
  1. 1.Department of Information and Computer ScienceAalto University School of Science and TechnologyAaltoFinland

Personalised recommendations