Orthogonalization and Thresholding Method for a Nonparametric Regression Problem

Abstract

In this article, we proposed training methods for improving the generalization capability of a learning machine that is defined by a weighted sum of many fixed basis functions and is used as a nonparametric regression method. In the basis of the proposed methods, vectors of basis function outputs are orthogonalized and coefficients of the orthogonal vectors are estimated instead of weights. The coefficients are set to zero if those are less than predetermined threshold levels which are theoretically reasonable under the assumption of Gaussian noise. We then obtain a resulting weight vector by transforming the thresholded coefficients. When we apply an eigen-decomposition based orthogonalization procedure, it yields shrinkage estimators of weights. If we employ the Gram-Schmidt orthogonalization scheme, it produces a sparse representation of a target function in terms of basis functions. A simple numerical experiment showed the validity of the proposed methods by comparing with other alternative methods including the leave-one-out cross validation.