Advertisement

Regression Efficacy and the Curse of Dimensionality

Chapter

Abstract

This chapter gives a geometric representation of a class of nonparametric regression estimators that includes series expansions (Fourier, wavelet, Tchebyshev, and others), kernels and other locally weighted regressions, splines, and artificial neural networks. For any estimator having this geometric representation, there is no curse of dimensionality—asymptotically, the error goes to \(0\) at the parametric rate. Regression efficacy measures the amount of variation in the conditional mean of the dependent variable, \(Y\), that can be achieved by moving the explanatory variables across their whole range. The dismally slow, dimension-dependent rates of convergence are calculated using a class of target functions in which efficacy is infinite, and the analysis allows for the possibility that the dependent variable, \(Y\), may be an ever-receding target.

Keywords

Nonparametric Regression Dense Class Compact Domain Nonparametric Technique Monotonic Path 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgments

We owe many thanks to Xiaohong Chen, Graham Elliot, Jinyong Hahn, James Hamilton, Qi Li, Dan Slesnick, Hal White, Paul Wilson, and an anonymous referee for numerous insights, questions, references, conversations and corrections.

References

  1. Adams, C. and J. A. Clarkson (1933). On Definitions of Bounded Variation for Functions of Two Variables. Transactions of the American Mathematical Society 35(4), 824–854.CrossRefGoogle Scholar
  2. Anderson, R. and W. Zame’s (2001). Genericity with Infinitely Many Parameters. Advances in Theoretical Economics: Vol. 1: No. 1, Article 1.Google Scholar
  3. Aronszajn, N. (1976). Differentiability of Lipschitzian Mappings Between Banach Spaces. Studia Mathematica LVII, 147–190.Google Scholar
  4. Barron, A. (1993). Universal Approximation Bounds for Superpositions of a Sigmoidal Function. IEEE Transactions on Information Theory 39(3), 930–945.CrossRefGoogle Scholar
  5. Benyamini, Y. and J. Lindenstrauss (2000). Geometric Nonlinear Functional Analysis. Providence, R.I.: American Mathematical Society, Colloquium publications (American Mathematical Society) v. 48.Google Scholar
  6. Billingsley, P. (2008). Probability and Measure. John Wiley and Sons, New York.Google Scholar
  7. Breiman, L. (2001). Statistical Modeling: The Two Cultures. Statistical Science 16(3), 199–231.CrossRefGoogle Scholar
  8. Brown, L. D., M. G. Low, and L. H. Zhao (1997). Superefficiency in Nonparametric Function Estimation. Annals of Statistics 25(6), 2607–2625.CrossRefGoogle Scholar
  9. Chen, X. (2007). Large Sample Sieve Estimation of Nonparametric Models. Handbook of Econometrics 6, 5549–5632.CrossRefGoogle Scholar
  10. Chen, X. and H. White (1999). Improved Rates and Asymptotic Normality for Nonparametric Neural Network Estimators. IEEE Tran. Information Theory 45, 682–691.CrossRefGoogle Scholar
  11. Cohen, A., R. DeVore, and G. Kerkyacharian (2001). Maximal Spaces with Given Rate of Convergence for Thresholding Algorithms. Applied and Computational Harmonic Analysis 11, 167–191.CrossRefGoogle Scholar
  12. Davidson, R. and J. G. McKinnon (1987). Implicit Alternatives and the Local Power of Test Statistics. Econometrica 55(6), 1305–1329.CrossRefGoogle Scholar
  13. Dellacherie, C. and P.-A. Meyer (1978). Probabilities and Potential, vol. 29 of North-Holland Mathematics Studies. North-Holland Publishing Co., Amsterdam.Google Scholar
  14. Feller, W. (1971). An Introduction to Probability Theory and its Applications, v. II. John Wiley and Sons, New York.Google Scholar
  15. Hornik, K. (1993). Some New Results on Neural Network Approximation. Neural Networks 6(8), 1069–1072.CrossRefGoogle Scholar
  16. Hornik, K, M. Stinchcombe and H. White (1989). Multi-layer Feedforward Networks are Universal Approximators. Neural Networks 2, 359–366.CrossRefGoogle Scholar
  17. Hornik, K, M. Stinchcombe and H. White (1990). Universal Approximation of an Unknown Mapping and its Derivatives using Multilayer Feedforward Networks. Neural Networks 3, 551–560.CrossRefGoogle Scholar
  18. Lehmann, E. L. and G. Casella (1998). Theory of Point Estimation, 2nd ed. Springer-Verlag, New York.Google Scholar
  19. Mhaskar, H. N. and C. A. Michelli (1995). Degree of Approximation by Neural and Translation Networks with a Single Hidden Layer. Advances in Applied Mathematics 16, 151–183.CrossRefGoogle Scholar
  20. Newey, W. (1996). Convergence Rates and Asymptotic Normality for Series Estimators. Journal of Econometrics 79, 147–168.CrossRefGoogle Scholar
  21. Niyogi, P. and F. Girosi (1999). Generalization bounds for function approximation from scattered noisy data. Advances in Computational Mathematics 10, 51–80.CrossRefGoogle Scholar
  22. Stinchcombe, M. (1999). Neural Network Approximation of Continuous Functionals and Continuous Functions on Compactifications. Neural Networks 12, 467–477.CrossRefGoogle Scholar
  23. Stinchcombe, M. (2001). The Gap Between Probability and Prevalence: Loneliness in Vector Spaces. Proceedings of the American Mathematical Society 129, 451–457.CrossRefGoogle Scholar
  24. Stinchcombe, M. and H. White (1990). Approximating and Learning Unknown Mappings Using Multilayer Feedforward Networks with Bounded Weights. Proceedings of the International Joint Conference on Neural Networks, Washington, D. C., III, 7–16. San Diego, CA.: SOS Printing.Google Scholar
  25. Stinchcombe, M. and H. White (1998). Consistent Specification Testing with Nuisance Parameters Present Only Under the Alternative. Econometric Theory 14, 295–325.CrossRefGoogle Scholar
  26. Stone, C. J. (1982). Optimal global rates of convergence for nonparametric regression. Annals of Statistics 10, 1040–1053.CrossRefGoogle Scholar
  27. Tibshirani, R. (1996). Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society, Series B, 58(1), 267–288.Google Scholar
  28. Yukich, J., M. Stinchcombe, and H. White (1995). Sup-Norm Approximation Bounds for Networks through Probabilistic Methods. IEEE Transactions on Information Theory 41(4), 1021–1027.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Department of EconomicsU.T. AustinAustinUSA
  2. 2.StataCorpCollege StationUSA

Personalised recommendations