Advertisement

Parameter investigation of support vector machine classifier with kernel functions

  • Alaa TharwatEmail author
Regular Paper
  • 113 Downloads

Abstract

Support vector machine (SVM) is one of the well-known learning algorithms for classification and regression problems. SVM parameters such as kernel parameters and penalty parameter have a great influence on the complexity and performance of predicting models. Hence, the model selection in SVM involves the penalty parameter and kernel parameters. However, these parameters are usually selected and used as a black box, without understanding the internal details. In this paper, the behavior of the SVM classifier is analyzed when these parameters take different values. This analysis consists of illustrative examples, visualization, and mathematical and geometrical interpretations with the aim of providing the basics of kernel functions with SVM and to show how it works to serve as a comprehensive source for researchers who are interested in this field. This paper starts by highlighting the definition and underlying principles of SVM in details. Moreover, different kernel functions are introduced and the impact of each parameter in these kernel functions is explained from different perspectives.

Keywords

Support vector machine (SVM) Kernel functions Radial basis function Polynomial kernel Gaussian kernel Parameter optimization Linear kernel 

Notes

References

  1. 1.
    Melki G, Kecman V, Ventura S, Cano A (2018) OLLAWV: online learning algorithm using worst-violators. Appl Soft Comput 66:384–393Google Scholar
  2. 2.
    Melki G, Cano A, Ventura S (2018) MIRSVM: multi-instance support vector machine with bag representatives. Pattern Recognit 79:228–241Google Scholar
  3. 3.
    Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297zbMATHGoogle Scholar
  4. 4.
    Byvatov E, Schneider G (2002) Support vector machine applications in bioinformatics. Appl Bioinform 2(2):67–77Google Scholar
  5. 5.
    Vatsa M, Singh R, Noore A (2005) Improving biometric recognition accuracy and robustness using DWT and SVM watermarking. IEICE Electron. Express 2(12):362–367Google Scholar
  6. 6.
    Moulin L, Da Silva AA, El-Sharkawi M, Marks RJ (2004) Support vector machines for transient stability analysis of large-scale power systems. IEEE Trans Power Syst 19(2):818–825Google Scholar
  7. 7.
    Doucet JP, Barbault F, Xia H, Panaye A, Fan B (2007) Nonlinear SVM approaches to QSPR/QSAR studies and drug design. Curr Comput Aided Drug Des 3(4):263–289Google Scholar
  8. 8.
    Wang L (2005) Support vector machines: theory and applications, vol 177. Springer, BerlinzbMATHGoogle Scholar
  9. 9.
    Melki G, Cano A, Kecman V, Ventura S (2017) Multi-target support vector regression via correlation regressor chains. Inf Sci 415:53–69MathSciNetGoogle Scholar
  10. 10.
    Friedrichs F, Igel C (2005) Evolutionary tuning of multiple SVM parameters. Neurocomputing 64:107–117Google Scholar
  11. 11.
    Tharwat A, Hassanien AE (2018) Chaotic antlion algorithm for parameter optimization of support vector machine. Appl Intell 48(3):670–686Google Scholar
  12. 12.
    Chapelle O, Vapnik V, Bousquet O, Mukherjee S (2002) Choosing multiple parameters for support vector machines. Mach Learn 46(1–3):131–159zbMATHGoogle Scholar
  13. 13.
    Wu CH, Tzeng GH, Lin RH (2009) A novel hybrid genetic algorithm for kernel function and parameter optimization in support vector regression. Expert Syst Appl 36(3):4725–4735Google Scholar
  14. 14.
    Subasi A (2013) Classification of EMG signals using PSO optimized SVM for diagnosis of neuromuscular disorders. Comput Biol Med 43(5):576–586Google Scholar
  15. 15.
    Lin SW, Ying KC, Chen SC, Lee ZJ (2008) Particle swarm optimization for parameter determination and feature selection of support vector machines. Expert Syst Appl 35(4):1817–1824Google Scholar
  16. 16.
    Zhang X, Chen X, He Z (2010) An ACO-based algorithm for parameter optimization of support vector machines. Expert Syst Appl 37(9):6618–6628Google Scholar
  17. 17.
    Tharwat A, Hassanien AE, Elnaghi BE (2017) A BA-based algorithm for parameter optimization of support vector machine. Pattern Recognit Lett 93:13–22Google Scholar
  18. 18.
    Ali S, Smith K (2003) Automatic parameter selection for polynomial kernel. In: Proceedings of IEEE international conference on information reuse and integration (IRI 2003), Lens, France, October 27–29, IEEE, pp 243–249Google Scholar
  19. 19.
    Abu-Mostafa YS, Magdon-Ismail M, Lin HT (2012) Learning from data, vol 4. AML Book, New YorkGoogle Scholar
  20. 20.
    Tharwat A (2016) Linear vs. quadratic discriminant analysis classifier: a tutorial. Int J Appl Pattern Recognit 3(2):145–180Google Scholar
  21. 21.
    Scholköpf B, Smola AJ (2001) Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT Press, CambridgeGoogle Scholar
  22. 22.
    Nilsson NJ (1965) Learning machines: foundations of trainable pattern-classifying systems, 1st edn. McGraw-Hill, New YorkzbMATHGoogle Scholar
  23. 23.
    Schölkopf B, Burges CJ (1999) Advances in kernel methods: support vector learning. MIT Press, CambridgezbMATHGoogle Scholar
  24. 24.
    Ben-Hur A, Weston J (2010) A user’s guide to support vector machines. In: Carugo O, Eisenhaber F (eds) Data mining techniques for the life sciences. Methods in molecular biology (Methods and protocols), vol 609. Humana PressGoogle Scholar
  25. 25.
    Kecman V (2001) Learning and soft computing: support vector machines, neural networks, and fuzzy logic models. MIT Press, CambridgezbMATHGoogle Scholar
  26. 26.
    Guyon I, Boser B, Vapnik V (1993) Automatic capacity tuning of very large VC-dimension classifiers. In: Hanson SJ, Cowan JD, Giles CL (eds) Advances in neural information processing systems 5. Morgan-KaufmannGoogle Scholar
  27. 27.
    Smola AJ, Schölkopf B (2004) A tutorial on support vector regression. Stat Comput 14(3):199–222MathSciNetGoogle Scholar
  28. 28.
    Keerthi SS, Lin CJ (2003) Asymptotic behaviors of support vector machines with Gaussian kernel. Neural Comput 15(7):1667–1689zbMATHGoogle Scholar
  29. 29.
    Ben-Hur A, Ong CS, Sonnenburg S, Schölkopf B, Rätsch G (2008) Support vector machines and kernels for computational biology. PLoS Comput Biol 4(10):e1000173Google Scholar
  30. 30.
    Lin CJ (2001) Formulations of support vector machines: a note from an optimization point of view. Neural Comput 13(2):307–317zbMATHGoogle Scholar
  31. 31.
    Burges CJ (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Discov 2(2):121–167Google Scholar
  32. 32.
    Smits GF, Jordaan EM (2002) Improved SVM regression using mixtures of kernels. In: Proceedings of the 2002 international joint conference on neural networks (IJCNN’02), vol 3. IEEE, pp 2785–2790Google Scholar
  33. 33.
    Chen F, Tang B, Song T, Li L (2014) Multi-fault diagnosis study on roller bearing based on multi-kernel support vector machine with chaotic particle swarm optimization. Measurement 47:576–590Google Scholar
  34. 34.
    Cortes C, Mohri M, Rostamizadeh A (2012) Algorithms for learning kernels based on centered alignment. J Mach Learn Res 13(Mar):795–828MathSciNetzbMATHGoogle Scholar
  35. 35.
    Tsuda K, Uda S, Kin T, Asai K (2004) Minimizing the cross validation error to mix kernel matrices of heterogeneous biological data. Neural Process Lett 19(1):63–72Google Scholar
  36. 36.
    Yeh CY, Huang CW, Lee SJ (2011) A multiple-kernel support vector regression approach for stock market price forecasting. Expert Syst Appl 38(3):2177–2186Google Scholar
  37. 37.
    Devos O, Ruckebusch C, Durand A, Duponchel L, Huvenne JP (2009) Support vector machines (SVM) in near infrared (NIR) spectroscopy: focus on parameters optimization and model interpretation. Chemom Intell Lab Syst 96(1):27–33Google Scholar
  38. 38.
    Arana-Daniel N, Gallegos AA, López-Franco C, Alanís AY, Morales J, López-Franco A (2016) Support vector machines trained with evolutionary algorithms employing kernel adatron for large scale classification of protein structures. Evolut Bioinform Online 12:285Google Scholar
  39. 39.
    Melgani F, Bazi Y (2008) Classification of electrocardiogram signals with support vector machines and particle swarm optimization. IEEE Trans Inf Technol Biomed 12(5):667–677Google Scholar
  40. 40.
    Tharwat A, Moemen YS, Hassanien AE (2017) Classification of toxicity effects of biotransformed hepatic drugs using whale optimized support vector machines. J Biomed Inform 68:132–149Google Scholar
  41. 41.
    Taie SA, Ghonaim W (2017) CSO-based algorithm with support vector machine for brain tumor’s disease diagnosis. In: Proceedings of IEEE international conference on pervasive computing and communications workshops (PerCom Workshops), IEEE, pp 183–187Google Scholar
  42. 42.
    Yuan SF, Chu FL (2007) Fault diagnostics based on particle swarm optimisation and support vector machines. Mech Syst Signal Process 21(4):1787–1798Google Scholar
  43. 43.
    Tang X, Zhuang L, Cai J, Li C (2010) Multi-fault classification based on support vector machine trained by chaos particle swarm optimization. Knowl Based Syst 23(5):486–490Google Scholar
  44. 44.
    Li X, Zhang X, Li C, Zhang L et al (2013) Rolling element bearing fault detection using support vector machine with improved ant colony optimization. Measurement 46(8):2726–2734Google Scholar
  45. 45.
    Can H, Jianchun X, Ruide Z, Juelong L, Qiliang Y, Liqiang X (2013) A new model for software defect prediction using particle swarm optimization and support vector machine. In: 25th conference Chinese control and decision conference (CCDC), IEEE, pp 4106–4110Google Scholar
  46. 46.
    Wei J, Jian-qi Z, Xiang Z (2011) Face recognition method based on support vector machine and particle swarm optimization. Expert Syst Appl 38(4):4390–4393Google Scholar
  47. 47.
    Xiao T, Ren D, Lei S, Zhang J, Liu X (2014) Based on grid-search and PSO parameter optimization for support vector machine. In: Proceedings of the 11th world congress on intelligent control and automation (WCICA), IEEE, pp 1529–1533Google Scholar
  48. 48.
    Abdulameer MH, Abdullah S, Huda SN, Othman ZA (2014) Support vector machine based on adaptive acceleration particle swarm optimization. Sci World J 2014:1–8Google Scholar
  49. 49.
    Li L, Yan Shi D, Xu J (2013) Color image segmentation based-on SVM using mixed features and combined kernel. In: International conference on intelligent science and big data engineering, Springer, pp 401–409Google Scholar
  50. 50.
    Kharrat A, Abid M (2014) Toward efficient segmentation of brain tumors based on support vector machine classifier through optimized RBF kernel parameters and optimal texture features. Int J Cognit Inf Nat Intell (IJCINI) 8(2):15–33Google Scholar
  51. 51.
    Ye Z, Ma L, Wang M, Chen H, Zhao W (2015) Texture image classification based on support vector machine and bat algorithm. In: Proceedings of IEEE 8th international conference on Intelligent data acquisition and advanced computing systems: technology and applications (IDAACS), vol 1., IEEE, pp 309–314Google Scholar
  52. 52.
    Blake CL (1998) UCI Repository of machine learning databases. Irvine, University of California. http://www.ics.uci.edu/~mlearn/MLRepository.html
  53. 53.
    Tharwat A (2016) Principal component analysis—a tutorial. Int J Appl Pattern Recognit 3(3):197–240Google Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.Faculty of Computer Science and EngineeringFrankfurt University of Applied SciencesFrankfurt am MainGermany

Personalised recommendations