Stochastic extra-gradient based alternating direction methods for graph-guided regularized minimization
- 17 Downloads
In this study, we propose and compare stochastic variants of the extra-gradient alternating direction method, named the stochastic extra-gradient alternating direction method with Lagrangian function (SEGL) and the stochastic extra-gradient alternating direction method with augmented Lagrangian function (SEGAL), to minimize the graph-guided optimization problems, which are composited with two convex objective functions in large scale. A number of important applications in machine learning follow the graph-guided optimization formulation, such as linear regression, logistic regression, Lasso, structured extensions of Lasso, and structured regularized logistic regression. We conduct experiments on fused logistic regression and graph-guided regularized regression. Experimental results on several genres of datasets demonstrate that the proposed algorithm outperforms other competing algorithms, and SEGAL has better performance than SEGL in practical use.
Key wordsStochastic optimization Graph-guided minimization Extra-gradient method Fused logistic regression Graph-guided regularized logistic regression
Unable to display preview. Download preview PDF.
- Azadi S, Sra S, 2014. Towards an optimal stochastic alternating direction method of multipliers. Int Conf on Machine Learning, p.620–628.Google Scholar
- Hsieh CJ, Sustik MA, Dhillon IS, et al., 2013. BIG & QUIC: sparse inverse covariance estimation for a million variables. Advances in Neural Information Processing Systems, p.3165–3173.Google Scholar
- Johnson R, Zhang T, 2013. Accelerating stochastic gradient descent using predictive variance reduction. Advances in Neural Information Processing Systems, p.315–323.Google Scholar
- Ouyang H, He N, Tran L, et al., 2013. Stochastic alternating direction method of multipliers. Int Conf on Machine Learning, p.80–88.Google Scholar
- Suzuki T, 2013. Dual averaging and proximal gradient descent for online alternating direction multiplier method. Int Conf on Machine Learning, p.392–400.Google Scholar
- Wang H, Banerjee A, 2013. Online alternating direction method (longer version). arXiv Preprint, 1306.3721.Google Scholar
- Zhao P, Yang J, Zhang T, et al., 2015. Adaptive stochastic alternating direction method of multipliers. Int Conf on Machine Learning, p.69–77.Google Scholar
- Zhong W, Kwok JT, 2013. Fast stochastic alternating direction method of multipliers. Int Conf on Machine Learning, p.46–54.Google Scholar