Stochastic Subgradient Estimation Training for Support Vector Machines

  • Sangkyun LeeEmail author
  • Stephen J. Wright
Conference paper
Part of the Springer Proceedings in Mathematics & Statistics book series (PROMS, volume 30)


Subgradient algorithms for training support vector machines have been successful in solving many large-scale and online learning problems. However, for the most part, their applicability has been restricted to linear kernels and strongly convex formulations. This paper describes efficient subgradient approaches without such limitations. Our approaches make use of randomized low-dimensional approximations to nonlinear kernels, and minimization of a reduced primal formulation using an algorithm based on robust stochastic approximation, which does not require strong convexity. Experiments illustrate that our approaches produce solutions of comparable prediction accuracy with the solutions acquired from existing SVM solvers, but often in much shorter time.


Support Vector Machine (SVM) Subgradient Estimates Robust Stochastic Approximation Strong Convexity Online Learning Problem 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The authors acknowledge the support of NSF Grants DMS-0914524 and DMS-0906818, and in part by Deutsche Forschungsgemeinschaft (DFG) within the Collaborative Research Center SFB 876 “Providing Information by Resource-Constrained Analysis,” project C1.


  1. 1.
    Bordes, A., Ertekin, S., Weston, J., Bottou, L.: Fast kernel classifiers with online and active learning. J. Mach. Learn. Res. 6, 1579–1619 (2005)MathSciNetzbMATHGoogle Scholar
  2. 2.
    Boser, B.E., Guyon, I.M., Vapnik, V.N.: A training algorithm for optimal margin classifiers. In: Proceedings of the Fifth Annual Workshop on Computational Learning Theory, pp. 144–152. ACM, New York (1992)Google Scholar
  3. 3.
    Bottou, L.: SGD: Stochastic gradient descent, (2005). Accessed 30 Mar 2012
  4. 4.
    Chapelle, O.: Training a support vector machine in the primal. Neural Comput. 19, 1155–1178 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  5. 5.
    Drineas, P., Mahoney, M.W.: On the Nystrom method for approximating a gram matrix for improved kernel-based learning. J. Mach. Learn. Res. 6, 2153–2175 (2005)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Fan, R.-E., Chen, P.-H., Lin, C.-J.: Working set selection using second order information for training SVM. J. Mach. Learn. Res. 6, 1889–1918 (2005)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Franc, V., Sonnenburg, S.: Optimized cutting plane algorithm for support vector machines. In: Proceedings of the 25th International Conference on Machine Learning, pp. 320–327. ACM, New York (2008)Google Scholar
  8. 8.
    Joachims, T.: Making large-scale support vector machine learning practical. In: Advances in Kernel Methods – Support Vector Learning, pp. 169–184. MIT, Cambridge (1999)Google Scholar
  9. 9.
    Joachims, T.: Training linear SVMs in linear time. In: International Conference on Knowledge Discovery and Data Mining, pp. 217–226. ACM, New York (2006)Google Scholar
  10. 10.
    Joachims, T., Yu, C.-N.J.: Sparse kernel SVMs via cutting-plane training. Mach. Learn. 76(2–3), 179–193 (2009)CrossRefGoogle Scholar
  11. 11.
    Joachims, T., Finley, T., Yu, C.-N.: Cutting-plane training of structural SVMs. Mach. Learn. 77(1), 27–59 (2009)zbMATHCrossRefGoogle Scholar
  12. 12.
    Kimeldorf, G., Wahba, G.: A correspondence between bayesian estimation on stochastic processes and smoothing by splines. Ann. Math. Statist. 41, 495–502 (1970)MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Lewis, D.D., Yang, Y., Rose, T.G., Dietterich, G., Li, F., Li, F.: RCV1: A new benchmark collection for text categorization research. J. Mach. Learn. Res. 5, 361–397 (2004)Google Scholar
  14. 14.
    Nemirovski, A., Yudin, D.B.: Problem Complexity and Method Efficiency in Optimization. Wiley, New York (1983)Google Scholar
  15. 15.
    Nemirovski, A., Juditsky, A., Lan, G., Shapiro, A.: Robust stochastic approximation approach to stochastic programming. SIAM J. Optim. 19(4), 1574–1609 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    Rahimi, A., Recht, B.: Random features for large-scale kernel machines. In: Advances in Neural Information Processing Systems, vol. 20, pp. 1177–1184. MIT, Cambridge (2008)Google Scholar
  17. 17.
    Scholkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT, Cambridge (2001)Google Scholar
  18. 18.
    Shalev-Shwartz, S., Singer, Y., Srebro, N.: Pegasos: Primal estimated sub-GrAdient SOlver for SVM. In: Proceedings of the 24th International Conference on Machine Learning, pp. 807–814. ACM, New York (2007)Google Scholar
  19. 19.
    Shalev-Shwartz, S., Singer, Y., Srebro, N., Cotter, A.: Pegasos: Primal estimated sub-GrAdient SOlver for SVM. Math. Program. Ser. B 127(1), 3–30 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  20. 20.
    Zinkevich, M.: Online convex programming and generalized infinitesimal gradient ascent. In: Proceedings of the 20th International Conference on Machine Learning, pp. 928–936. ACM, New York (2003)Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Fakultät für Informatik, LS VIIITechnische Universität DortmundDortmundGermany
  2. 2.Computer Sciences DepartmentUniversity of WisconsinMadisonUSA

Personalised recommendations