Sparse Regression: Utilizing the Higher-order Structure of Data for Prediction
Independent component analysis and the closely related method of sparse coding model multidimensional data as linear combinations of independent components that have nongaussian, usually sparse, distributions. Such a modelling approach is especially suitable in large dimensions, as it avoids the curse of dimensionality. It also seems to represent important properties of sensory data. In this paper we show how to use these models for regression. If the joint density of two random vectors is modelled by independent component analysis, it is possible to obtain simple algorithms to compute the maximum likelihood predictor of one of the vectors when the other vector is observed. The obtained predictors are nonlinear, but in contrast to such nonparametric methods as MLP, the nonlinearities are not chosen ad hoc: They are directly determined by the density approximation.
Unable to display preview. Download preview PDF.
- A. Hyvärinen. Sparse code shrinkage: Denoising of nongaussian data by maximum likelihood estimation. Technical report, Helsinki University of Technology, Laboratory of Computer and Information Science, 1998.Google Scholar
- A. Hyvärinen, P. Hoyer, and E. Oja. Sparse code shrinkage for image denoising. In Proc. IEEE Int. Joint Conf. on Neural Networks, pages 859–864, Anchorage, Alaska, 1998.Google Scholar
- D.-T. Pham, P. Garrat, and C. Jutten. Separation of a mixture of independent sources through a maximum likelihood approach. In Proc. EUSIPCO, pages 771–774, 1992.Google Scholar