Abstract
An efficient sparse model is very significant to handle the highly or super-highly dimensional data. The optimization algorithms in solving the sparsity constraint problem have been progressively improved. In this paper, we propose a new quadratic approximation greedy pursuit algorithm (QAGP) for \(\ell _0\)-constrained minimization with linear models. Our method first constructs an upper bound to the linear model at each iteration once the cost function of the model is L-Smooth, then we apply greedy pursuit to search the feasible solution of the upper bound. Compared to the Newton-type methods, our method does not need to calculate the Hessian matrix. We analyze the convergence of our method and verify the efficiency of it in sparse logistic regression and sparse \(L_2\)-SVMs tasks on synthetic and real datasets. The results demonstrate that the performance of our method is superior to other methods.
Fanfan Ji is currently working toward the Master degree in the School of Automation, Nanjing University of Information Science and Technology.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Agarwal, N., Bullins, B., Hazan, E.: Second-order stochastic optimization in linear time. Statistics 1050, 15 (2016)
Bahmani, S., Raj, B., Boufounos, P.T.: Greedy sparsity-constrained optimization. J. Mach. Learn. Res. 14, 807–841 (2013)
Bertsekas, D.P.: Nonlinear programming. J. Oper. Res. Soc. 48(3), 334 (1997)
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Cham (2006)
Blumensath, T., Davies, M.E.: Iterative hard thresholding for compressed sensing. Appl. Comput. Harmonic Anal. 27(3), 265–274 (2009)
Byrd, R.H., Lu, P., Nocedal, J., Zhu, C.: A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Comput. 16(5), 1190–1208 (1995)
Chen, J., Gu, Q.: Fast newton hard thresholding pursuit for sparsity constrained nonconvex optimization. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 757–766. ACM (2017)
Donoho, D.L., et al.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)
Foucart, S.: Hard thresholding pursuit: an algorithm for compressive sensing. SIAM J. Numer. Anal. 49(6), 2543–2563 (2011)
Johnson, R., Zhang, T.: Accelerating stochastic gradient descent using predictive variance reduction. In: Advances in Neural Information Processing Systems, pp. 315–323 (2013)
Lang, K.: Newsweeder: learning to filter netnews. In: Machine Learning Proceedings 1995, pp. 331–339. Elsevier (1995)
Lewis, D.D., Yang, Y., Rose, T.G., Li, F.: RCV1: a new benchmark collection for text categorization research. J. Mach. Learn. Res. 5, 361–397 (2004)
Li, X., Zhao, T., Arora, R., Liu, H., Haupt, J.: Stochastic variance reduced optimization for nonconvex sparse learning. In: International Conference on Machine Learning, pp. 917–925 (2016)
Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24(2), 227–234 (1995)
Needell, D., Tropp, J.A.: CoSaMP: iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmonic Anal. 26(3), 301–321 (2009)
Shalev-Shwartz, S., Srebro, N., Zhang, T.: Trading accuracy for sparsity in optimization problems with sparsity constraints. SIAM J. Optim. 20(6), 2807–2832 (2010)
Yuan, X.T., Li, P., Zhang, T.: Gradient hard thresholding pursuit. J. Mach. Learn. Res. 18, 1–43 (2018)
Yuan, X.T., Liu, Q.: Newton-type greedy selection methods for \(\ell _0 \)-constrained minimization. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2437–2450 (2017)
Yuan, X., Li, P., Zhang, T.: Gradient hard thresholding pursuit for sparsity-constrained optimization. In: International Conference on Machine Learning, pp. 127–135 (2014)
Zhang, T.: Adaptive forward-backward greedy algorithm for sparse learning with linear models. In: Advances in Neural Information Processing Systems, pp. 1921–1928 (2009)
Zhou, P., Yuan, X., Feng, J.: Efficient stochastic gradient hard thresholding. In: Advances in Neural Information Processing Systems, pp. 1988–1997 (2018)
Acknowledgements
This work was supported by the National Natural Science Foundation of China under Grant Numbers: 61876090.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Ji, F., Shuai, H., Yuan, XT. (2019). Quadratic Approximation Greedy Pursuit for Cardinality-Constrained Sparse Learning. In: Lin, Z., et al. Pattern Recognition and Computer Vision. PRCV 2019. Lecture Notes in Computer Science(), vol 11857. Springer, Cham. https://doi.org/10.1007/978-3-030-31654-9_29
Download citation
DOI: https://doi.org/10.1007/978-3-030-31654-9_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-31653-2
Online ISBN: 978-3-030-31654-9
eBook Packages: Computer ScienceComputer Science (R0)