Generalization improvement for regularized least squares classification
- 97 Downloads
In the past decades, regularized least squares classification (RLSC) is a commonly used supervised classification method in the machine learning filed because it can be easily resolved through the simple matrix analysis and achieve a close-form solution. Recently, some studies conjecture that the margin distribution is more crucial to the generalization performance. Moreover, from the view of margin distribution, RLSC only considers the first-order statistics (i.e., margin mean) and does not consider the actual higher-order statistics of margin distribution. In this paper, we propose a novel RLSC which takes into account the actual second-order (i.e., variance) information of margin distribution. It is intuitively expected that small margin variance will improve the generalization performance of RLSC from a geometric view. We incorporate the margin variance into the objective function of RLSC and achieve the optimal classifier by minimizing the margin variance. To evaluate the performance of our algorithm, we conduct a series of experiments on several benchmark datasets in comparison with RLSC, kernel minimum squared error, support vector machine and large margin distribution machine. And the empirical results verify the effectiveness of our algorithm and indicate that the margin distribution is helpful to improve the classification performance.
KeywordsRegularized least squares classification Margin distribution Second-order statistic
The work was supported by National Natural Science Foundation of China under Grant Nos. 61601162, 61501154 and 61671197, and Open Foundation of first level Zhejiang key in key discipline of Control Science and Engineering.
Compliance with ethical standards
Conflict of interest
The authors declare that they have no conflict of interest.
- 4.Hsieh CJ, Chang KW, Lin CJ, Keerthi SS, Sundararajan S (2008) A dual coordinate descent method for large-scale linear SVM. In: Proceedings of the 25th international conference on machine learning, ACM, pp 408–415Google Scholar
- 5.Lichman M (2013) UCI machine learning repository. http://archive.ics.uci.edu/ml
- 7.Reyzin L, Schapire RE (2006) How boosting the margin can also boost classifier complexity. In: Proceedings of the 23rd international conference on Machine learning, ACM, pp 753–760Google Scholar
- 8.Rifkin R, Yeo G, Poggio T (2003) Regularized least-squares classification. Nato Sci Ser Sub Ser III Comput Syst Sci 190:131–154Google Scholar
- 12.Xu J, Zhang X, Li Y (2001) Kernel MSE algorithm: a unified framework for KFD, LS-SVM and KRR. In: Proceedings of international joint conference on neural networks, IEEE, pp 1486–1491Google Scholar
- 15.Zhang P, Peng J (2004) SVM vs regularized least squares classification. In: Proceedings of the 17th international conference on pattern recognition, IEEE, pp 176–179Google Scholar
- 16.Zhou ZH (2014) Large margin distribution learning. In: Artificial neural networks in pattern recognition, Springer, pp 1–11Google Scholar