A coordinate gradient descent method for ℓ 1-regularized convex minimization
In applications such as signal processing and statistics, many problems involve finding sparse solutions to under-determined linear systems of equations. These problems can be formulated as a structured nonsmooth optimization problems, i.e., the problem of minimizing ℓ 1-regularized linear least squares problems. In this paper, we propose a block coordinate gradient descent method (abbreviated as CGD) to solve the more general ℓ 1-regularized convex minimization problems, i.e., the problem of minimizing an ℓ 1-regularized convex smooth function. We establish a Q-linear convergence rate for our method when the coordinate block is chosen by a Gauss-Southwell-type rule to ensure sufficient descent. We propose efficient implementations of the CGD method and report numerical results for solving large-scale ℓ 1-regularized linear least squares problems arising in compressed sensing and image deconvolution as well as large-scale ℓ 1-regularized logistic regression problems for feature selection in data classification. Comparison with several state-of-the-art algorithms specifically designed for solving large-scale ℓ 1-regularized linear least squares or logistic regression problems suggests that an efficiently implemented CGD method may outperform these algorithms despite the fact that the CGD method is not specifically designed just to solve these special classes of problems.
KeywordsCoordinate gradient descent Q-linear convergence ℓ1-Regularization Compressed sensing Image deconvolution Linear least squares Logistic regression Convex optimization
Unable to display preview. Download preview PDF.
- 5.Chang, C.-C., Lin, C.-J.: LIBSVM—A library for support vector machines. http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/
- 12.Figueiredo, M., Nowak, R.: A bound optimization approach to wavelet-based image deconvolution. In: IEEE Int. Conf. on Image Processing—ICIP’05, 2005 Google Scholar
- 16.Golub, T.R., Slonim, D.K., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J.P., Coller, H., Loh, M.L., Downing, J.R., Caligiuri, M.A., Bloomfield, C.D., Lander, E.S.: Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286, 531–537 (1999) CrossRefGoogle Scholar
- 17.Hale, E.T., Yin, W., Zhang, Y.: A fixed-point continuation method for ℓ 1-regularized minimization with applications to compressed sensing. CAAM Technical report TR07-07, Department of Computational and Applied Mathematics, Rice University (July 2007) Google Scholar
- 20.Lee, S., Lee, H., Abeel, P., Ng, A.: Efficient ℓ 1-regularized logistic regression. In: Proceedings of the 21st National Conference on Artificial Intelligence, 2006 Google Scholar
- 21.Lewis, D.D., Yang, Y., Rose, T.G., Li, F.: RCV1: A new benchmark collection for text categorization research. J. Mach. Learn. Res. 5, 361–397 (2004) Google Scholar
- 25.Ng, A.Y.: Feature selection, ℓ 1 vs. ℓ 2 regularization, and rotational invariance, In: Proceedings of the 21st International Conference on Machine Learning, 2004 Google Scholar
- 34.Wang, L.: Efficient regularized solution path algorithms with applications in machine learning and data mining. Ph.D. thesis, University of Michigan (2008) Google Scholar
- 35.Wright, S.J., Nowak, R., Figueiredo, M.: Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. (2007, to appear) Google Scholar