Abstract
Automatic tuning of hyperparameter and parameter is an essential ingredient and important process for learning and applying Support Vector Machines (SVM). Previous tuning methods choose hyperparameter and parameter separately in different iteration processes, and usually search exhaustively in parameter spaces. In this paper we propose and implement a new tuning algorithm that chooses hyperparameter and parameter for SVM simultaneously and search the parameter space efficiently with a deliberate initialization of a pair of starting points. First we derive an approximate but effective radius margin bound for soft margin SVM. Then we combine multiparameters of SVM into one vector, converting the two separate tuning processes into one optimization problem. Further we discuss the implementation issue about the new tuning algorithm, and that of choosing initial points for iteration. Finally we compare the new tuning algorithm with old gradient based method and cross validation on five benchmark data sets. The experimental results demonstrate that the new tuning algorithm is effective, and usually outperforms those classical tuning algorithms.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Anguita, D., et al.: Theoretical and Practical Model Selection Methods for Support Vector Classifiers. In: Wang, L. (ed.) Support Vector Machines: Theory and Applications. Studies in Fuzziness and Soft Computing, pp. 159–181. Springer, Heidelberg (2005)
Ayat, N., Cheriet, M., Suen, C.: Automatic model selection for the optimization of the SVM kernels. Pattern Recognition Journal (2005), Avaliable at http://www.livia.etsmtl.ca/publications/2005/Ayat_pr2005.pdf
Chapelle, O., Vapnik, V.: Model Selection for Support Vector Machines. In: Advances in Neural Information Processing Systems, vol. 12, MIT Press, Cambridge (1999)
Charles, A.M., Pontil, M.: Learning the kernel function via regularization. Journal of Machine Learing Reasearch 6, 1099–1125 (2005)
Frauke, F., Christian, I.: Evolutionary tuning of multiple SVM parameters. Neurocomputing 64(1-4), 107–117 (2005), http://dx.doi.org/10.1016/j.neucom.2004.11.022
Soares, C., Brazdil, P.B.: A meta-learning method to select the kernel width in support vector regression. Machine Learning 54, 195–209 (2004)
Chapelle, O., et al.: Choosing multiple parameters for support vector machines. Machine Learning 46, 131–159 (2002)
Keerthi, S.S.: Efficient tuning of SVM hyperparameters using radius/margin bound and iterative algorithms. IEEE Transactions on Neural Networks 13(5), 1225–1229 (2002)
McCormick, G.P.: The projective SUMT method for convex programming. Math. Oper. Res. 14, 203–223 (1989)
Davidon, W.: Variable metric algorithms for minimization. Technical Report ANL-5990, Argonne National Lab (1959)
Vapnik, V.N.: The Nature of Statistical Learning Theory. Springer, New York (1995)
Burges, C.J.C.: A tutorial on support vector machine for pattern recgnition. Data Mining and Knowledge Discovery 2, 121–167 (1998)
Cristianini, N., Campbell, C., Shawe-Taylor, J.: Dynamically adapting kernel in support vector machines. In: Advances in Neural Information Processing Systems, vol. 11, pp. 204–210 (1998)
Shölkopf, B., et al.: Kernel-dependent support vector error bounds. Artifical Neural Networks 7, 103–108 (1999)
Chung, K.M., et al.: Radius margin bounds for support vector machines with RBF kernel. Neural Comput. 15, 2643–2681 (2003)
Newman, D., et al.: UCI Repository of Machine Learning Databases. Dept. of Information and Computer Sciences, University of California, Irvine (1998)
Platt, J.: Fast training of support vector machines using sequential minimal optimization. In: Advances in Kernel Methods - Support Vector Learning, MIT Press, Cambridge (1998)
King, R.: Statlog Databases. Dept. of Statist. Modeling Scie, University of Strathclyde, Glasgow, U.K. (1992)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer Berlin Heidelberg
About this paper
Cite this paper
Liao, S., Jia, L. (2007). Simultaneous Tuning of Hyperparameter and Parameter for Support Vector Machines. In: Zhou, ZH., Li, H., Yang, Q. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2007. Lecture Notes in Computer Science(), vol 4426. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-71701-0_18
Download citation
DOI: https://doi.org/10.1007/978-3-540-71701-0_18
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-71700-3
Online ISBN: 978-3-540-71701-0
eBook Packages: Computer ScienceComputer Science (R0)