Abstract
Multivariate adaptive regression spline (MARS) denotes a modern methodology from statistical learning which is important in both classification and regression. It is very useful for high-dimensional problems and shows a great promise for fitting nonlinear multivariate functions by using its ability to estimate the contributions of the basis functions so that both the additive and the interactive effects of the predictors are allowed to determine the response variable. The MARS algorithm for estimating the model function consists of two sub-algorithms. In our paper, we propose not to use second algorithm. Instead, we construct a penalized residual sum of squares (PRSS) for MARS as a higher-order Tikhonov regularization problem which is also known as ridge regression that shrinks coefficients and make them more stable. But it cannot perform variable selection in the model and, hence, does not give an easily interpretable model (especially, if the number of variable p is large). For this reason, we change the Tikhonov penalty function with the generalized Lasso penalty for solving the problem PRSS, taking an advantage for feature selection. We treat this problem using continuous optimization techniques which we consider to become an important complementary technology and model-based alternative to the concept of the backward stepwise algorithm. In particular, we apply the elegant framework of conic quadratic programming (CQP), and we call the solution as CG-Lasso. Here, we gain from an area of convex optimization whose programs are very well-structured, herewith, resembling linear programming and, hence, permitting the use of powerful interior point methods (IPMs).
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Breiman, L., Friedman, J.H., Olshen, R., Stone, C.: Classification and Regression Trees. Wadsworth, Belmont (1984)
Friedman, J.H.: Multivariate adaptive regression splines. Ann. Statist. 19(1), 1–141 (1991)
Aster, R.C., Borchers, B., Thurber, C.H.: Parameter Estimation and Inverse Problems. Academic Press, NewYork (2013)
Tibshirani, R.J., Taylor, J.: The solution path of the generalized Lasso. Ann. Statist. 39(3), 1335–1371 (2011)
Nemirovski, A.: Lectures on Modern Convex Optimization. Israel Institute Technology (2002)
Nash, G., Sofer, A.: Linear and Nonlinear Programming. McGraw-Hill, New York (1996)
Nemirovskii, A.S., Todd, M.J.: Interior point methods in convex programming. Acta Numer. 17, 191–234 (2008)
Bagirov, A., Karmitsa, N., Mäkelä, M.M.: Introduction to Nonsmooth Optimization: Theory, Practice and Software. Springer, New York (2014)
Shor, N.Z.: Minimization Methods for Non-differentiable Functions. Springer, Berlin (1985)
Hoerl, A.E., Kennard, R.W.: Ridge regression iterative estimation of the biasing parameter. Comm. Statist. Theory Methods. 5(1), 77–88 (1976)
Renegar, J.: A Mathematical View of Interior-Point Methods in Convex Optimization MOS-SIAM Series on Optimization. SIAM, Philadelphia (1987)
Karmarkar, N.: A new polynomial-time algorithm for linear programming. Combinatorica. 4, 373–395 (1984)
Ben-Tal, A., Nemirovski, A.: Lectures on Modern Convex Optimization: Analysis, Algorithms and Engineering Applications MOS-SIAM Series on Optimization. SIAM, Philadelphia (2001)
Lobo, M., Vandenberghe, S.L., Boyd, S., Lebret, H.: Applications of second-order cone programming. Linear Algebra Appl. 284, 193–228 (1998)
Weisberg, S.: Applied Linear Regression. Wiley, Hoboken (2005)
Hansen, P.C.: Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Line-ear Inversion. SIAM, Philadelphia (1998)
Tibshirani, R.: Regression shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 58, 267–288 (1996)
Taylan, P., Weber, G.W., Yerlikaya, F.: A new approach to multivariate adaptive regression splines by using Tikhonov regularization and continuous optimization. J. TOP. 18(2), 377–395 (2010)
Craven, P., Wahba, G.: Smoothing noisy data with spline functions: estimating the correct degree of smoothing by the method of generalized cross-validation. Numer. Math. 31, 377–403 (1979)
Weber, G.W., Batmaz, I., Köksal, G., et al.: CMARS: a new contribution to nonparametric regression with multivariate adaptive regression splines supported by continuous optimization. Inverse Probl. Sci. Eng. 20(3), 371–400 (2012)
Fletcher, R.: Practical Methods of Optimization. Wiley, New York (1987)
Schmidt, M., Fung, G., Rosales, R.: Optimization Methods for L1-Regularization. UBC Technical Report TR-2009-19 (2009)
Pringle, R.M., Rayner, A.A.: Generalized Inverse Matrices with Applications to Statistics. Hafner Publishing, NewYork (1971)
Aitchison, P.W.: Generalized inverse matrices and their applications. Int. J. Math. Educ. Sci. Technol. 13(1), 99–109 (1982)
Hastie, T., Tibshirani, R., Friedman, J.H.: The Element of Statistical Learning. Springer, New York (2001)
Blumschein, P., Hung, W., Jonassen, D.: Model-Based Approaches to Learning: Using Systems Models and Simulations to Improve Understanding and problem Solving in Complex Domains. Sense Publishers, Rotterdam (2009)
Taylan, P., Weber, G.W., Beck, A.: New approaches to regression by generalized additive models and continuous optimization for modern applications in finance, science and technology. Optimization. 56(5–6), 675–698 (2007)
IÅŸcanoÄŸlu, A., Weber, G.W., Taylan, P.: Predicting Default Probabilities with Generalized Additive Models for Emerging Markets Graduate Summer School on Recent Advances in Statistics. METU, Ankara (2007)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer International Publishing AG, part of Springer Nature
About this chapter
Cite this chapter
Taylan, P., Weber, G.W. (2019). CG-Lasso Estimator for Multivariate Adaptive Regression Spline. In: TaÅŸ, K., Baleanu, D., Machado, J. (eds) Mathematical Methods in Engineering. Nonlinear Systems and Complexity, vol 24. Springer, Cham. https://doi.org/10.1007/978-3-319-90972-1_9
Download citation
DOI: https://doi.org/10.1007/978-3-319-90972-1_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-90971-4
Online ISBN: 978-3-319-90972-1
eBook Packages: EngineeringEngineering (R0)