Abstract
In recent years, classification with big data sets has become one of the latest research topic in machine learning. Distributed classification have received much attention from industry and academia. Recently, the Alternating Direction Method of Multipliers (ADMM) is a widely-used method to solve learning problems in a distributed manner due to its simplicity and scalability. However, distributed ADMM usually converges slowly and thus suffers from expensive time cost in practice. To overcome this limitation, we propose a novel distributed stochastic ADMM (DS-ADMM) algorithm for big data classification based on the MPI framework. By formulating the original problem as a series of sub-problems through a cluster of multiple computers (nodes). In particular, we exploit a stochastic method for sub-problem optimization in parallel to further improve time efficiency. The experimental results show that our proposed distributed algorithm is suitable to enhance the performance of ADMM, and can be effectively applied for big data classification.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The datasets are available at http://www.csie.ntu.edu.tw/~cjlin/ libsvmtools/datasets/binary.html.
References
Yuan, G.-X., Ho, C.-H., Lin, C.-J.: Recent advances of large-scale linear classification. Proc. IEEE 100(9), 2584–2603 (2012)
Zaharia, M., et al.: Apache spark: a unified engine for big data processing. Commun. ACM 59(11), 56–65 (2016)
Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends ® Mach. Learn. 3(1), 1–122 (2011)
Agarwal, N., Suresh, A.T., Yu, F.X.X., Kumar, S., McMahan, B.: cpSGD: Communication-efficient and differentially-private distributed SGD. In: Advances in Neural Information Processing Systems, pp. 7564–7575 (2018)
Haddadpour F., Kamani, M.M., Mahdavi M., Cadambe V.: Trading redundancy for communication: speeding up distributed SGD for non-convex optimization. In: International Conference on Machine Learning, pp. 2545–2554 (2019)
Reddi, S.J., Hefny, A., Sra, S., Poczos, B., Smola, A.J.: On variance reduction in stochastic gradient descent and its asynchronous variants. In: Advances in Neural Information Processing Systems, pp. 2647–2655 (2015)
Suzuki, T.: Stochastic dual coordinate ascent with alternating direction method of multipliers. In: Proceedings of the 31st International Conference on Machine Learning, pp. 736–744 (2014)
Wang, H., Gao, Y., Shi, Y., Wang, R.: Group-based alternating direction method of multipliers for distributed linear classification. IEEE Trans. Cybern. 47(11), 3568–3582 (2017)
Zheng, S., Kwok, J.T.: Stochastic variance-reduced admm. arXiv preprint arXiv:1604.07070 (2016)
Dajun, D., Xue, L., Wenting, L., Rui, C., Minrui, F., Lei, W.: Admm-based distributed state estimation of smart grid under data deception and denial of service attacks. IEEE Trans. Syst. Man Cybern. Syst. 49(8), 1698–1711 (2019)
Forero, P.A., Cano, A., Giannakis, G.B.: Consensus-based distributed support vector machines. J. Mach. Learn. Res. 11, 1663–1707 (2010)
Zhang, C., Lee, H., Shin, K.: Efficient distributed linear classification algorithms via the alternating direction method of multipliers. In: Artificial Intelligence and Statistics, pages 1398–1406 (2012)
Lee, C.P., Chang, K.W., Upadhyay, S., Roth, D.: Distributed training of structured SVM. arXiv preprint arXiv:1506.02620 (2015)
Lee, C.P., Roth, D.: Distributed box-constrained quadratic optimization for dual linear SVM. In: International Conference on Machine Learning, pp. 987–996 (2015)
Shai, S.-S., Yoram, S., Nathan, S., Andrew, C.: Pegasos: primal estimated sub-gradient solver for SVM. Math. Program. 127(1), 3–30 (2011). https://doi.org/10.1007/s10107-010-0420-4
Xiaohe, W., Wangmeng, Z., Liang, L., Wei, J., Zhang, D.: F-SVM: Combination of feature transformation and SVM learning via convex relaxation. IEEE Trans. Neural Netw. Learn. Syst. 29(11), 5185–5199 (2018)
Hsieh, C.J., Chang, K.W., Lin, C.J., Keerthi, S.S., Sundararajan, S.: A dual coordinate descent method for large-scale linear SVM. In: Proceedings of the 25th international conference on Machine learning, pp. 408–415 (2008)
He, B., Yuan, X.: On the o(1/n) convergence rate of the douglas-rachford alternating direction method. SIAM J. Numer. Anal. 50(2), 700–709 (2012)
Liu Y., Shang F., Cheng J.: Accelerated variance reduced stochastic ADMM. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pp. 2287–2293 (2017)
Yu Y., Huang L.: Fast stochastic variance reduced ADMM for stochastic composition optimization. In: Proceedings of International Joint Conferences on Artifical Intelligence, pp. 3364–3370 (2017)
Shi, W., Ling, Q., Yuan, K., Gang, W., Yin, W.: On the linear convergence of the ADMM in decentralized consensus optimization. IEEE Trans. Signal Process. 62(7), 1750–1761 (2014)
Chen, C., He, B., Ye, Y., Yuan, X.: The direct extension of admm for multi-block convex minimization problems is not necessarily convergent. Math. Program. 155(1–2), 57–79 (2016)
Lin, C.J., Weng, R.C., Keerthi, S.S.: Trust region newton method for logistic regression. J. Mach. Learn. Res. 9, 627–650 (2008)
He, B., Ma, F., Yuan, X.: On the step size of symmetric alternating directions method of multipliers (2015). http://www.optimization-online.org
Acknowledgement
This work is partially supported by the Fundamental Research Funds for the Central Universities, No. 30918014108, and the National Science Foundation of China No. 61806096.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Wang, H., Li, X., Chen, X., Qi, L., Xu, X. (2020). Distributed Stochastic Alternating Direction Method of Multipliers for Big Data Classification. In: Zhang, X., Liu, G., Qiu, M., Xiang, W., Huang, T. (eds) Cloud Computing, Smart Grid and Innovative Frontiers in Telecommunications. CloudComp SmartGift 2019 2019. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 322. Springer, Cham. https://doi.org/10.1007/978-3-030-48513-9_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-48513-9_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-48512-2
Online ISBN: 978-3-030-48513-9
eBook Packages: Computer ScienceComputer Science (R0)