Regression Efficacy and the Curse of Dimensionality
This chapter gives a geometric representation of a class of nonparametric regression estimators that includes series expansions (Fourier, wavelet, Tchebyshev, and others), kernels and other locally weighted regressions, splines, and artificial neural networks. For any estimator having this geometric representation, there is no curse of dimensionality—asymptotically, the error goes to \(0\) at the parametric rate. Regression efficacy measures the amount of variation in the conditional mean of the dependent variable, \(Y\), that can be achieved by moving the explanatory variables across their whole range. The dismally slow, dimension-dependent rates of convergence are calculated using a class of target functions in which efficacy is infinite, and the analysis allows for the possibility that the dependent variable, \(Y\), may be an ever-receding target.
We owe many thanks to Xiaohong Chen, Graham Elliot, Jinyong Hahn, James Hamilton, Qi Li, Dan Slesnick, Hal White, Paul Wilson, and an anonymous referee for numerous insights, questions, references, conversations and corrections.
- Anderson, R. and W. Zame’s (2001). Genericity with Infinitely Many Parameters. Advances in Theoretical Economics: Vol. 1: No. 1, Article 1.Google Scholar
- Aronszajn, N. (1976). Differentiability of Lipschitzian Mappings Between Banach Spaces. Studia Mathematica LVII, 147–190.Google Scholar
- Benyamini, Y. and J. Lindenstrauss (2000). Geometric Nonlinear Functional Analysis. Providence, R.I.: American Mathematical Society, Colloquium publications (American Mathematical Society) v. 48.Google Scholar
- Billingsley, P. (2008). Probability and Measure. John Wiley and Sons, New York.Google Scholar
- Dellacherie, C. and P.-A. Meyer (1978). Probabilities and Potential, vol. 29 of North-Holland Mathematics Studies. North-Holland Publishing Co., Amsterdam.Google Scholar
- Feller, W. (1971). An Introduction to Probability Theory and its Applications, v. II. John Wiley and Sons, New York.Google Scholar
- Lehmann, E. L. and G. Casella (1998). Theory of Point Estimation, 2nd ed. Springer-Verlag, New York.Google Scholar
- Stinchcombe, M. and H. White (1990). Approximating and Learning Unknown Mappings Using Multilayer Feedforward Networks with Bounded Weights. Proceedings of the International Joint Conference on Neural Networks, Washington, D. C., III, 7–16. San Diego, CA.: SOS Printing.Google Scholar
- Tibshirani, R. (1996). Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society, Series B, 58(1), 267–288.Google Scholar