Advertisement

Abstract

Polytope Faces Pursuit (PFP) is a greedy algorithm that approximates the sparse solutions recovered by ℓ1 regularised least-squares (Lasso) [4,10] in a similar vein to (Orthogonal) Matching Pursuit (OMP) [16]. The algorithm is based on the geometry of the polar polytope where at each step a basis function is chosen by finding the maximal vertex using a path-following method. The algorithmic complexity is of a similar order to OMP whilst being able to solve problems known to be hard for (O)MP. Matching Pursuit was extended to build kernel-based solutions to machine learning problems, resulting in the sparse regression algorithm, Kernel Matching Pursuit (KMP) [17]. We develop a new algorithm to build sparse kernel-based solutions using PFP, which we call Kernel Polytope Faces Pursuit (KPFP). We show the usefulness of this algorithm by providing a generalisation error bound [7] that takes into account a natural regression loss and experimental results on several benchmark datasets.

Keywords

Polytope Faces Pursuit Orthogonal Matching Pursuit Pseudo-dimension Sample Compression Bounds Regression Kernel methods 

References

  1. 1.
    Anthony, M., Bartlett, P.: Neural Network Learning: Theoretical Foundations. Cambridge University Press, Cambridge (1999)CrossRefzbMATHGoogle Scholar
  2. 2.
    Chen, S.: Basis Pursuit. PhD thesis, Department of Statistics, Stanford University (November 1995)Google Scholar
  3. 3.
    Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing 20(1), 33–61 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Donoho, D.: Neighborly polytopes and sparse solution of underdetermined linear equations. Technical report, Department of Statistics, Stanford Univ., Stanford, CA (2005)Google Scholar
  5. 5.
    Floyd, S., Warmuth, M.: Sample compression, learnability, and the Vapnik-Chervonenkis dimension. Machine Learning 21(3), 269–304 (1995)Google Scholar
  6. 6.
    Graepel, T., Herbrich, R., Shawe-Taylor, J.: Generalisation error bounds for sparse linear classifiers. In: Proceedings of the Thirteenth Annual Conference on Computational Learning Theory, pp. 298–303 (2000)Google Scholar
  7. 7.
    Hussain, Z., Shawe-Taylor, J.: Theory of matching pursuit. In: Koller, D., Schuurmans, D., Bengio, Y., Bottou, L. (eds.) Advances in Neural Information Processing Systems, vol. 21, pp. 721–728 (2009)Google Scholar
  8. 8.
    Mallat, S., Zhang, Z.: Matching pursuit with time-frequency dictionaries. IEEE Transactions on Signal Processing 41(12), 3397–3415 (1993)CrossRefzbMATHGoogle Scholar
  9. 9.
    Plumbley, M.D.: Polar polytopes and recovery of sparse representations (2005)Google Scholar
  10. 10.
    Plumbley, M.D.: Recovery of sparse representations by polytope faces pursuit. In: Rosca, J.P., Erdogmus, D., Príncipe, J.C., Haykin, S. (eds.) ICA 2006. LNCS, vol. 3889, pp. 206–213. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  11. 11.
    Shawe-Taylor, J., Anthony, M., Biggs, N.L.: Bounding sample size with the Vapnik-Chervonenkis dimension. Discrete Applied Mathematics 42(1), 65–73 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge (2004)CrossRefzbMATHGoogle Scholar
  13. 13.
    Smola, A.J., Bartlett, P.: Sparse greedy gaussian process regression. Advances in neural information processing systems 13 (2001)Google Scholar
  14. 14.
    Smola, A.J., Schölkopf, B.: Sparse greedy matrix approximation for machine learning. In: Proceedings of 17th International Conference on Machine Learning, pp. 911–918. Morgan Kaufmann, San Francisco (2000)Google Scholar
  15. 15.
    Tibshirani, R.: Regression selection and shrinkage via the lasso. Technical report, Department of Statistics, University of Toronto (June 1994), ftp://utstat.toronto.edu/pub/tibs/lasso.ps
  16. 16.
    Tropp, J.A., Gilbert, A.C., Strauss, M.J.: Algorithms for simultaneous sparse approximation. part i: Greedy pursuit. Signal Processing, special issue Sparse approximations in signal and image processing 86, 572–588 (2006)zbMATHGoogle Scholar
  17. 17.
    Vincent, P., Bengio, Y.: Kernel matching pursuit. Machine Learning 48(1-3), 165–187 (2002)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Tom Diethe
    • 1
  • Zakria Hussain
    • 1
  1. 1.Department of Computer ScienceUniversity College LondonUK

Personalised recommendations