Bundle Methods for Structured Output Learning — Back to the Roots
Discriminative methods for learning structured output classifiers have been gaining popularity in recent years due to their successful applications in fields like computer vision, natural language processing, etc. Learning of the structured output classifiers leads to solving a convex minimization problem, still hard to solve by standard algorithms in real-life settings. A significant effort has been put to development of specialized solvers among which the Bundle Method for Risk Minimization (BMRM)  is one of the most successful. The BMRM is a simplified variant of bundle methods well known in the filed of non-smooth optimization. In this paper, we propose two speed-up improvements of the BMRM: i) using the adaptive prox-term known from the original bundle methods, ii) starting optimization from a non-trivial initial solution. We combine both improvements with the multiple cutting plane model approximation . Experiments on real-life data show consistently faster convergence achieving speedup up to factor of 9.7.
KeywordsStructured Output Learning Bundle Methods Risk Minimization Structured Output SVM
- 2.Uřičář, M., Franc, V.: Efficient Algorithm for Regularized Risk Minimization. In: CVWW 2012: Proceedings of the 17th Computer Vision Winter Workshop, pp. 57–64 (February 2012)Google Scholar
- 6.Shwartz, S., Singer, Y., Srebro, N.: Pegasos: Primal Estimated sub-GrAdient SOlver for SVM. In: Proceedings of International Conference on Machine Learning (ICML), pp. 807–814. ACM Press (2007)Google Scholar
- 10.Lemaréchal, C.: Nonsmooth optimization and descend methods. Technical report, IIASA, Laxenburg, Austria (1978)Google Scholar
- 11.Uřičář, M., Franc, V., Hlaváč, V.: Detector of facial landmarks learned by the structured output svm. In: VISAPP, vol. (1), pp. 547–556. SciTe Press (2012)Google Scholar
- 12.Sonnenburg, S., Rätsch, G., Henschel, S., Widmer, C., Behr, J., Zien, A., Bona, F.D., Binder, A., Gehl, C., Franc, V.: The shogun machine learning toolbox. J. Mach. Learn. Res. 99, 1799–1802 (2010)Google Scholar