Weighted Coordinate-Wise Pegasos

  • Vilen Jumutc
  • Johan A. K. Suykens
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8251)


Pegasos is a popular and reliable machine learning algorithm for making linear Support Vector Machines solvable at the larger scale. It benefits from the strongly convex optimization objective, faster convergence rates and lower computational and memory costs. In this paper we devise a new weighted formulation of the Pegasos algorithm which favors from the different coordinate-wise λ i regularization parameters. Together with the proposed extension we give a brief theoretical justification of its convergence to an optimal solution and analyze at a glance its computational costs. We conclude our paper with the numerical results obtained for UCI datasets and demonstrate the merits and the importance of our approach for achieving a better classification accuracy and convergence rates in the partially or fully stochastic setting.


Support Vector Machine Relevance Vector Machine Linear Support Vector Machine Hinge Loss Projection Step 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Joachims, T.: Training linear SVMs in linear time. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2006, pp. 217–226. ACM, New York (2006)Google Scholar
  2. 2.
    Shalev-Shwartz, S., Singer, Y., Srebro, N.: Pegasos: Primal Estimated sub-GrAdient SOlver for SVM. In: Proceedings of the 24th International Conference on Machine Learning, ICML 2007, New York, NY, USA, pp. 807–814 (2007)Google Scholar
  3. 3.
    Chapelle, O.: Training a support vector machine in the primal. Neural Computation 19, 1155–1178 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Nemirovski, A., Juditsky, A., Lan, G., Shapiro, A.: Robust stochastic approximation approach to stochastic programming. SIAM J. on Optimization 19(4), 1574–1609 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    MacKay, D.J.: Comparison of approximate methods for handling hyperparameters. Neural Computation 11, 1035–1068 (1999)CrossRefGoogle Scholar
  6. 6.
    Tipping, M.E.: Sparse bayesian learning and the relevance vector machine. J. Mach. Learn. Res. 1, 211–244 (2001)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Chu, W., Keerthi, S.S., Ong, C.J.: Bayesian trigonometric support vector classifier. Neural Comput. 15(9), 2227–2254 (2003)CrossRefzbMATHGoogle Scholar
  8. 8.
    Xavier-De-Souza, S., Suykens, J.A.K., Vandewalle, J., Bollé, D.: Coupled simulated annealing. IEEE Trans. Sys. Man Cyber. Part B 40(2), 320–335 (2010)CrossRefGoogle Scholar
  9. 9.
    Vapnik, V.: Statistical learning theory, 1st edn. Wiley (September 1998)Google Scholar
  10. 10.
    Suykens, J.A.K., Van Gestel, T., De Brabanter, J., De Moor, B., Vandewalle, J.: Least Squares Support Vector Machines. World Scientific, Singapore (2002)CrossRefzbMATHGoogle Scholar
  11. 11.
    Shalev-Shwartz, S., Singer, Y.: Logarithmic regret algorithms for strongly convex repeated games. Technical report, The Hebrew University (2007)Google Scholar
  12. 12.
    Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, New York (2004)CrossRefzbMATHGoogle Scholar
  13. 13.
    Wipf, D., Nagarajan, S.: A New View of Automatic Relevance Determination. In: Platt, J.C., Koller, D., Singer, Y., Roweis, S. (eds.) Advances in Neural Information Processing Systems 20, pp. 1625–1632. MIT Press, Cambridge (2008)Google Scholar
  14. 14.
    MacKay, D.J.: Bayesian interpolation. Neural Computation 4, 415–447 (1991)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Neal, R.M.: Bayesian Learning for Neural Networks. Springer-Verlag New York, Inc., Secaucus (1996)CrossRefzbMATHGoogle Scholar
  16. 16.
    Nelder, J.A., Mead, R.: A simplex method for function minimization. Computer Journal 7, 308–313 (1965)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Vilen Jumutc
    • 1
  • Johan A. K. Suykens
    • 1
  1. 1.Katholieke Universiteit LeuvenLeuvenBelgium

Personalised recommendations