Neural Computing and Applications

, Volume 29, Issue 9, pp 497–509 | Cite as

A divide-and-conquer method for large scale ν-nonparallel support vector machines

Original Article
  • 97 Downloads

Abstract

Recently, nonparallel support vector machine (NPSVM), a branch of support vector machines (SVMs), is developed and has attracted considerable interest. A kind of developed NPSVM, ν-nonparallel support vector machine (ν-NPSVM), which inherits the advantages of ν-support vector machine (ν-SVM) enables us to achieve higher classification accuracy and less time to tune for parameters. However, applications of ν-NPSVM to large data sets are seriously hampered by their excessive training time. In this paper, we use divide-and-conquer technique for large scale ν-nonparallel support vector machine (DC-νNPSVM) aiming at overcoming this burden. In the division step, we first divide the whole samples into several smaller subsamples and solve smaller subproblems using ν-NPSVM independently. We theoretically and experimentally show that objective function value, solutions, and support vectors solved by (DC-νNPSVM) are likely to those of the whole ν-NPSVM. In the conquer step, subsolutions from subproblems are used as an initial coordinate descent solver which converges to the optimal solution quickly. Moreover, multi-level (DC-νNPSVM) is adopted to balance the accuracy and efficiency. (DC-νNPSVM) can achieve higher accuracy by tuning the parameters in a smaller range and control the number of support vectors efficiently. Quantities of experiments show our (DC-νNPSVM) outperforms state-of-the-art SVM methods for large scale data sets.

Keywords

Support vector machines ν-Nonparallel support vector machines Large scale data sets Divide-and-conquer 

Notes

Acknowledgments

This work has been partially supported by grants from National Natural Science Foundation of China (Nos.61472390, 11271361, 71331005), Major International (Regional) Joint Research Project (No.71110107026).

References

  1. 1.
    Cortes C, Vapnik VN (1995) Support-vector networks. Mach Learn 20(3):273–297MATHGoogle Scholar
  2. 2.
    Vapnik VN (1996) The nature of statistical learning theory. Springer, New YorkMATHGoogle Scholar
  3. 3.
    Vapnik VN (1998) Statistical learning theory. Wiley, New YorkMATHGoogle Scholar
  4. 4.
    Trafalis TB, Ince H (2000) Support vector machine for regression and applications to financial forecasting. In: Proceedings of the IEEE-INNS-ENNS international joint conference on neural networks, vol 6, pp 348–353Google Scholar
  5. 5.
    Schölkopf B, Smola AJ, Williamson RC, Bartlett PL (2000) New support vector algorithms. Neural Comput 12:1207–1245CrossRefGoogle Scholar
  6. 6.
    Tian YJ, Zhang Q, Liu DL (2014) ν-Nonparallel support vector machine for pattern classification. Neural Comput Appl 25:1007–1020CrossRefGoogle Scholar
  7. 7.
    Platt JC (1998) Fast training of support vector machines using sequential minimal optimization. In: Schölkopf B, Burges CJC, Smola AJ (eds) Advances in Kernel Methods-Support Vector Learning. MIT Press, CambridgeGoogle Scholar
  8. 8.
    Chang CC, Lin CJ (2011) Libsvm: a linrary for support vector machines. ACM Trans Intell Syst Tecnol 2(27):1–27Google Scholar
  9. 9.
    Hsieh CJ, Chang KW, Lin CJ, Keerthi SS, Sundararajan S (2008) A dual coordinate descent method for large-scale linear svm. In: Proceedings of the 25th international conference on machine learning, pp 408–415Google Scholar
  10. 10.
    Fine S, Scheinberg K (2001) Efficient svm training using low-rank kernel representations. J Mach Learn Res 2:243–264MATHGoogle Scholar
  11. 11.
    Zhang K, Lan L, Wang Z, Moerchen F (2012) Scaling up kernel SVM on Limited resources: a low-rank linearization approach. In: Proceedings of 15th internation conference on artificial intelligence and statistics (AISTATS)Google Scholar
  12. 12.
    Le Q, Sarlós T, Smola A (2013) Fastfood-approximating kernel expansions in loglinear time. In: Proceedings of the 30th international conference on machine learningGoogle Scholar
  13. 13.
    John M, Christian JD (1989) Fast learning in networks of locally-tuned processing units. Neural Comput 1:281–294CrossRefGoogle Scholar
  14. 14.
    Yu H, Yang J, Han J, Li X (2005) Making SVMs scalable to large data sets using hierarchical cluster indexing. Data Min Knowl Discov 11(3):295–321MathSciNetCrossRefGoogle Scholar
  15. 15.
    Zhang K, Tsang IW, Kwok JT (2008) Improved Nyström low rank approximation and error analysis. In: Proceedings of the 25st international conference on machine learningGoogle Scholar
  16. 16.
    Jacobs RA, Jordan MI, Nowlan SJ, Hinton GE (1991) Adaptive mixtures of local experts. Neural Comput 3(1):79–87CrossRefGoogle Scholar
  17. 17.
    Tresp V (2000) A bayesian committee machine. Neural Comput 12:2719–2741CrossRefGoogle Scholar
  18. 18.
    Graf HP, Cosatto E, Bottou L, Durdanovic I, Vapnik VN (2005) Parallel support vector machines: the cascade SVM. In: Proceedings of conference on neural information processing systems (NIPS)Google Scholar
  19. 19.
    Hsieh CJ, Si S, Dhillon IS (2014) A divide-and-conquer solver for kernel support vector machines. In: Proceedings of the 31st international conference on machine learningGoogle Scholar
  20. 20.
    Tian YJ, Ju XC, Shi Y (2016) A divide-and-combine method for large scale nonparallel support vector machines. Neural Netw 75:12–21CrossRefGoogle Scholar
  21. 21.
    Tsang IW, Kwok JT, Cheung PM (2005) Core vector machines: fast svm training on very large data sets. J Mach Learn Res 6:363–392MathSciNetMATHGoogle Scholar
  22. 22.
    Tsang IW, Kocsor A, Kwok JT (2007) Simpler core vector machines with enclosing balls. In: Proceedings of the 24th international conference on machine learningGoogle Scholar
  23. 23.
    Wang D, Zhang B, Zhang P, Qiao H (2010) An online core vector machine with adaptive meb adjustment. Pattern Recognit 42(10):3468–3482CrossRefMATHGoogle Scholar
  24. 24.
    Rahimi A, Recht B (2007) Random features for large-scale kernel machines. In: Advances in neural information processing systemsGoogle Scholar
  25. 25.
    Kelley J (1960) The cutting-plane method for solving convex programs. J Soc Ind Appl Math 8(4):703–712MathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    Joachims T (2006) Training linear svms in linear time. In: Proceedings of the 12th ACM SIGKDD international conference on knowledge discovery and data miningGoogle Scholar
  27. 27.
    Franc V, Sonnenburg S (2008) Optimized cutting plane algorithm for support vector machines. In: Proceedings of the 25th international conference on machine learning, pp 320–327Google Scholar
  28. 28.
    Teo CH, Vishwanathan S, Smola A, Le QV (2010) Bundle methods for regularized risk minimization. J Mach Learn Res 11:311–365MathSciNetMATHGoogle Scholar
  29. 29.
    Zhang T (2004) Solving large scale linear prediction problems using stochastic gradient descent algorithms. In: Proceedings of the 21st international conference on machine learningGoogle Scholar
  30. 30.
    Shalev-Shwartz S, Singer Y, Srebro N, Cotter A (2011) Pegasos: primal estimated sub-gradient solver for SVM. Math Program 127:3–30MathSciNetCrossRefMATHGoogle Scholar
  31. 31.
    Bordes A, Ertekin S, Weston J, Bottou L (2005) Fast kernel classifiers with online and active learning. J Mach Learn Res 6:1579–1619MathSciNetMATHGoogle Scholar
  32. 32.
    Fan RE, Chen PH, Lin CJ (2005) Working set selection using second order information for training SVM. J Mach Learn Res 6:1889–1918MathSciNetMATHGoogle Scholar
  33. 33.
    Radha C, Rong J, Timothy CH, Anil KJ (2011) Approximate kernel k-means: solution to large scale kernel clustering. In: ACM SIGKDD conference on knowledge discovery and data miningGoogle Scholar
  34. 34.
    Tian YJ, Qi ZQ, Ju XC, Shi Y, Liu XH (2013) Nonparallel support vector machines for pattern classification. IEEE Trans Cybern 44(7):1067–1079CrossRefGoogle Scholar

Copyright information

© The Natural Computing Applications Forum 2016

Authors and Affiliations

  1. 1.School of Economics and ManagementTsinghua UniversityBeijingChina
  2. 2.Postdoctoral Programme of Agricultural Bank of ChinaBeijingChina
  3. 3.Research Center on Fictitious Economy and Data ScienceChinese Academy of SciencesBeijingChina
  4. 4.Key Laboratory of Big Data Mining and Knowledge ManagementChinese Academy of SciencesBeijingChina

Personalised recommendations