Annals of Operations Research

, Volume 276, Issue 1–2, pp 191–210 | Cite as

Relaxed support vector regression

  • Orestis P. Panagopoulos
  • Petros XanthopoulosEmail author
  • Talayeh Razzaghi
  • Onur Şeref
S.I.: Computational Biomedicine


Datasets with outliers pose a serious challenge in regression analysis. In this paper, a new regression method called relaxed support vector regression (RSVR) is proposed for such datasets. RSVR is based on the concept of constraint relaxation which leads to increased robustness in datasets with outliers. RSVR is formulated using both linear and quadratic loss functions. Numerical experiments on benchmark datasets and computational comparisons with other popular regression methods depict the behavior of our proposed method. RSVR achieves better overall performance than support vector regression (SVR) in measures such as RMSE and \(R^2_{adj}\) while being on par with other state-of-the-art regression methods such as robust regression (RR). Additionally, RSVR provides robustness for higher dimensional datasets which is a limitation of RR, the robust equivalent of ordinary least squares regression. Moreover, RSVR can be used on datasets that contain varying levels of noise.


Regression Relaxed support vector regression Outliers Relaxed support vector machines Support vector regression 


  1. Alcalá-Fdez, J., Fernández, A., Luengo, J., Derrac, J., García, S., Sánchez, L., et al. (2011). Keel data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework. Journal of Multiple-Valued Logic and Soft Computing, 17, 255–287.Google Scholar
  2. Behnke, A. R., & Wilmore, J. H. (1974). Evaluation and regulation of body build and composition. Englewood Cliffs: Prentice-Hall.Google Scholar
  3. Bishop, C. M. (2006). Pattern recognition and machine learning. New York: Springer.Google Scholar
  4. Cao, G., Guo, Y., & Bouman, C. (2010). High dimensional regression using the sparse matrix transform (SMT). In 2010 IEEE international conference on acoustics speech and signal processing (ICASSP) (pp. 1870–1873). IEEE.Google Scholar
  5. Cao, F., Ye, H., & Wang, D. (2015). A probabilistic learning algorithm for robust modeling using neural networks with random weights. Information Sciences, 313, 62–78.Google Scholar
  6. Cauwenberghs, G., & Poggio, T. (2001). Incremental and decremental support vector machine learning. In Advances in Neural Information Processing Systems. NIPS’00 Proceedings of the 13th International Conference on Neural Information Processing Systems, Denver, CO, 2000 (pp. 409–415). Cambridge, MA: MIT Press.Google Scholar
  7. Chang, C. C., & Lin, C. J. (2011). Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3), 27.Google Scholar
  8. Cifarelli, C., Guarracino, M. R., Seref, O., Cuciniello, S., & Pardalos, P. M. (2007). Incremental classification with generalized eigenvalues. Journal of Classification, 24(2), 205–219.Google Scholar
  9. Diehl, C. P., & Cauwenberghs, G. (2003). SVM incremental learning, adaptation and optimization. In Proceedings of the international joint conference on neural networks, 2003 (Vol. 4, pp. 2685–2690). IEEE.Google Scholar
  10. Dulá, J., & López, F. (2013). Dea with streaming data. Omega, 41(1), 41–47.Google Scholar
  11. D’Urso, P., Massari, R., & Santoro, A. (2011). Robust fuzzy regression analysis. Information Sciences, 181(19), 4154–4174.Google Scholar
  12. Eubank, R. L. (1999). Nonparametric regression and spline smoothing. Boca Raton: CRC Press.Google Scholar
  13. Guarracino, M. R., Cuciniello, S., & Feminiano, D. (2009). Incremental generalized eigenvalue classification on data streams. In International workshop on data stream management and mining (pp. 1–12).Google Scholar
  14. Guvenir, H. A., & Uysal, I. (2000). Bilkent University function approximation repository. Accessed August 10, 2015.Google Scholar
  15. Harrison, D., & Rubinfeld, D. L. (1978). Hedonic housing prices and the demand for clean air. Journal of Environmental Economics and Management, 5(1), 81–102.Google Scholar
  16. Hawkins, D. M. (1980). Identification of outliers (Vol. 11). New York: Springer.Google Scholar
  17. Hirst, J. D., King, R. D., & Sternberg, M. J. (1994). Quantitative structure-activity relationships by neural networks and inductive logic programming. II. The inhibition of dihydrofolate reductase by triazines. Journal of Computer-Aided Molecular Design, 8(4), 421–432.Google Scholar
  18. Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), 55–67.Google Scholar
  19. Huang, C. M., Lee, Y. J., Lin, D. K., & Huang, S. Y. (2007). Model selection for support vector machines via uniform design. Computational Statistics and Data Analysis, 52(1), 335–346.Google Scholar
  20. IBM. (2013). IBM ILOG CPLEX: High-performance mathematical programming engine. Accessed 9 Apr 2018.
  21. Johansen, S. (1988). Statistical analysis of cointegration vectors. Journal of Economic Dynamics and Control, 12(2), 231–254.Google Scholar
  22. Kibler, D., Aha, D. W., & Albert, M. K. (1989). Instance-based prediction of real-valued attributes. Computational Intelligence, 5(2), 51–57.Google Scholar
  23. Kirts, S., Panagopoulos, O. P., Xanthopoulos, P., & Nam, B. H. (2017). Soil-compressibility prediction models using machine learning. Journal of Computing in Civil Engineering, 32(1), 04017067.Google Scholar
  24. Kohavi, R., et al. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. IJCAI, 14, 1137–1145.Google Scholar
  25. Levinson, N. (1947). The wiener rms (root mean square) error criterion in filter design and prediction. Institute of Electrical and Electronics Engineers, 1(3), 129–148.Google Scholar
  26. Lichman, M. (2013). UCI machine learning repository.
  27. Lin, C. J., Hsu, C. W., & Chang, C. C. (2003). A practical guide to support vector classification. National Taiwan University.
  28. Panagopoulos, A. A. (2013). A novel method for predicting the power output of distributed renewable energy resources. Ph.D. thesis, Diploma thesis, Technical University of Crete.Google Scholar
  29. Panagopoulos, A. A. (2016). Efficient control of domestic space heating systems and intermittent energy resources. Ph.D. thesis, University of Southampton.Google Scholar
  30. Panagopoulos, A. A., Chalkiadakis, G., & Koutroulis, E. (2012). Predicting the power output of distributed renewable energy resources within a broad geographical region. In Proceedings of the 20th European conference on artificial intelligence (pp. 981–986). IOS Press.Google Scholar
  31. Panagopoulos, A. A., Maleki, S., Rogers, A., Venanzi, M., & Jennings, N. R. (2017). Advanced economic control of electricity-based space heating systems in domestic coalitions with shared intermittent energy resources. ACM Transactions on Intelligent Systems and Technology (TIST), 8(4), 59.Google Scholar
  32. Panagopoulos, O. P., Pappu, V., Xanthopoulos, P., & Pardalos, P. M. (2016). Constrained subspace classifier for high dimensional datasets. Omega,. Scholar
  33. Pang, S., Ozawa, S., & Kasabov, N. (2005). Incremental linear discriminant analysis for classification of data streams. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 35(5), 905–914.Google Scholar
  34. Pappu, V., Panagopoulos, O. P., Xanthopoulos, P., & Pardalos, P. M. (2015). Sparse proximal support vector machines for feature selection in high dimensional datasets. Expert Systems with Applications,. Scholar
  35. Peters, G., & Lacic, Z. (2012). Tackling outliers in granular box regression. Information Sciences, 212, 44–56.Google Scholar
  36. Rousseeuw, P. J., & Leroy, A. M. (2005). Robust regression and outlier detection (Vol. 589). New York: Wiley.Google Scholar
  37. Şeref, O., Chaovalitwongse, W. A., & Brooks, J. P. (2014). Relaxing support vectors for classification. Annals of Operations Research, 216(1), 229–255.Google Scholar
  38. Smith, M. R., & Martinez, T. (2011). Improving classification accuracy by identifying and removing instances that should be misclassified. In The 2011 international joint conference on neural networks (IJCNN) (pp. 2690–2697). IEEE.Google Scholar
  39. Smola, A. J., & Schölkopf, B. (2004). A tutorial on support vector regression. Statistics and Computing, 14(3), 199–222.Google Scholar
  40. Street, J. O., Carroll, R. J., & Ruppert, D. (1988). A note on computing robust regression estimates via iteratively reweighed least squares. The American Statistician, 42(2), 152–154.Google Scholar
  41. Theil, H. (1959). Economic forecasts and policy. The American Economic Review, 49(4), 711–716.Google Scholar
  42. Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B (Methodological), 58, 267–288.Google Scholar
  43. Vapnik, V. (2000). The nature of statistical learning theory. New York: Springer.Google Scholar
  44. Vapnik, V. N., & Vapnik, V. (1998). Statistical learning theory (Vol. 1). New York: Wiley.Google Scholar
  45. Wolters, R., & Kateman, G. (1989). The performance of least squares and robust regression in the calibration of analytical methods under non-normal noise distributions. Journal of Chemometrics, 3(2), 329–342.Google Scholar
  46. Xanthopoulos, P., Guarracino, M. R., & Pardalos, P. M. (2014). Robust generalized eigenvalue classifier with ellipsoidal uncertainty. Annals of Operations Research, 216(1), 327–342.Google Scholar
  47. Xanthopoulos, P., Panagopoulos, O. P., Bakamitsos, G. A., & Freudmann, E. (2016). Hashtag hijacking: What it is, why it happens and how to avoid it. Journal of Digital and Social Media Marketing, 3(4), 353–362.Google Scholar
  48. Xanthopoulos, P., Pardalos, P., & Trafalis, T. B. (2012). Robust data mining. New York: Springer.Google Scholar
  49. Yang, E., Lozano, A., & Ravikumar, P. (2014). Elementary estimators for high-dimensional linear regression. In: Proceedings of the 31st international conference on machine learning (ICML-14) (pp. 388–396).Google Scholar
  50. Yeh, I. C. (2007). Modeling slump flow of concrete using second-order regressions and artificial neural networks. Cement and Concrete Composites, 29(6), 474–480.Google Scholar
  51. Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2), 301–320.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer Information SystemsCalifornia State University, StanislausTurlockUSA
  2. 2.Department of Decision and Information SciencesStetson UniversityDeLandUSA
  3. 3.Department of Industrial EngineeringNew Mexico State UniversityLas CrucesUSA
  4. 4.Department of Business Information TechnologyVirginia Polytechnic Institute and State UniversityBlacksburgUSA

Personalised recommendations