Greedy Kernel Principal Component Analysis

Abstract

This contribution discusses one aspect of statistical learning and generalization. The theory of learning is very relevant to cognitive systems including cognitive vision.

A technique allowing to approximate a huge training set is proposed. The approach aims to represent data in a low dimensional space with possibly minimal representation error which is similar to the Principal Component Analysis (PCA). In contrast to the PCA, the basis vectors of the low dimensional space used for data representation are properly selected vectors from the training set and not as their linear combinations. The basis vectors can be selected by a simple algorithm which has low computational requirements and allows on-line processing of huge data sets. As the computations in the proposed algorithm appear in the form of a dot product, kernel methods can be used to cope with non-linear problems.

The proposed method was tested to approximate training sets of the Support Vector Machines and the Kernel Fisher Linear Discriminant which are known methods for learning classifiers. The experiments show that the proposed approximation can significantly reduce the complexity of the found classifiers while retaining their accuracy. On the other hand, the method is not very suitable for denoising.

The authors were supported by the European Union projects ICA 1-CT-2000-70002, IST-2001-32184 ActIPret, by the Czech Ministry of Education under project MSM 212300013, by the Grant Agency of the Czech Republic project 102/03/0440.