Skip to main content
SpringerLink
Log in

Distributed Stochastic Alternating Direction Method of Multipliers for Big Data Classification

  • Conference paper
  • First Online:
Book cover Cloud Computing, Smart Grid and Innovative Frontiers in Telecommunications (CloudComp 2019, SmartGift 2019)

  • 876 Accesses

Abstract

In recent years, classification with big data sets has become one of the latest research topic in machine learning. Distributed classification have received much attention from industry and academia. Recently, the Alternating Direction Method of Multipliers (ADMM) is a widely-used method to solve learning problems in a distributed manner due to its simplicity and scalability. However, distributed ADMM usually converges slowly and thus suffers from expensive time cost in practice. To overcome this limitation, we propose a novel distributed stochastic ADMM (DS-ADMM) algorithm for big data classification based on the MPI framework. By formulating the original problem as a series of sub-problems through a cluster of multiple computers (nodes). In particular, we exploit a stochastic method for sub-problem optimization in parallel to further improve time efficiency. The experimental results show that our proposed distributed algorithm is suitable to enhance the performance of ADMM, and can be effectively applied for big data classification.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The datasets are available at http://www.csie.ntu.edu.tw/~cjlin/ libsvmtools/datasets/binary.html.

References

  1. Yuan, G.-X., Ho, C.-H., Lin, C.-J.: Recent advances of large-scale linear classification. Proc. IEEE 100(9), 2584–2603 (2012)

    Article  Google Scholar 

  2. Zaharia, M., et al.: Apache spark: a unified engine for big data processing. Commun. ACM 59(11), 56–65 (2016)

    Article  Google Scholar 

  3. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends ® Mach. Learn. 3(1), 1–122 (2011)

    MATH  Google Scholar 

  4. Agarwal, N., Suresh, A.T., Yu, F.X.X., Kumar, S., McMahan, B.: cpSGD: Communication-efficient and differentially-private distributed SGD. In: Advances in Neural Information Processing Systems, pp. 7564–7575 (2018)

    Google Scholar 

  5. Haddadpour F., Kamani, M.M., Mahdavi M., Cadambe V.: Trading redundancy for communication: speeding up distributed SGD for non-convex optimization. In: International Conference on Machine Learning, pp. 2545–2554 (2019)

    Google Scholar 

  6. Reddi, S.J., Hefny, A., Sra, S., Poczos, B., Smola, A.J.: On variance reduction in stochastic gradient descent and its asynchronous variants. In: Advances in Neural Information Processing Systems, pp. 2647–2655 (2015)

    Google Scholar 

  7. Suzuki, T.: Stochastic dual coordinate ascent with alternating direction method of multipliers. In: Proceedings of the 31st International Conference on Machine Learning, pp. 736–744 (2014)

    Google Scholar 

  8. Wang, H., Gao, Y., Shi, Y., Wang, R.: Group-based alternating direction method of multipliers for distributed linear classification. IEEE Trans. Cybern. 47(11), 3568–3582 (2017)

    Article  Google Scholar 

  9. Zheng, S., Kwok, J.T.: Stochastic variance-reduced admm. arXiv preprint arXiv:1604.07070 (2016)

  10. Dajun, D., Xue, L., Wenting, L., Rui, C., Minrui, F., Lei, W.: Admm-based distributed state estimation of smart grid under data deception and denial of service attacks. IEEE Trans. Syst. Man Cybern. Syst. 49(8), 1698–1711 (2019)

    Article  Google Scholar 

  11. Forero, P.A., Cano, A., Giannakis, G.B.: Consensus-based distributed support vector machines. J. Mach. Learn. Res. 11, 1663–1707 (2010)

    MathSciNet  MATH  Google Scholar 

  12. Zhang, C., Lee, H., Shin, K.: Efficient distributed linear classification algorithms via the alternating direction method of multipliers. In: Artificial Intelligence and Statistics, pages 1398–1406 (2012)

    Google Scholar 

  13. Lee, C.P., Chang, K.W., Upadhyay, S., Roth, D.: Distributed training of structured SVM. arXiv preprint arXiv:1506.02620 (2015)

  14. Lee, C.P., Roth, D.: Distributed box-constrained quadratic optimization for dual linear SVM. In: International Conference on Machine Learning, pp. 987–996 (2015)

    Google Scholar 

  15. Shai, S.-S., Yoram, S., Nathan, S., Andrew, C.: Pegasos: primal estimated sub-gradient solver for SVM. Math. Program. 127(1), 3–30 (2011). https://doi.org/10.1007/s10107-010-0420-4

    Article  MathSciNet  MATH  Google Scholar 

  16. Xiaohe, W., Wangmeng, Z., Liang, L., Wei, J., Zhang, D.: F-SVM: Combination of feature transformation and SVM learning via convex relaxation. IEEE Trans. Neural Netw. Learn. Syst. 29(11), 5185–5199 (2018)

    Article  Google Scholar 

  17. Hsieh, C.J., Chang, K.W., Lin, C.J., Keerthi, S.S., Sundararajan, S.: A dual coordinate descent method for large-scale linear SVM. In: Proceedings of the 25th international conference on Machine learning, pp. 408–415 (2008)

    Google Scholar 

  18. He, B., Yuan, X.: On the o(1/n) convergence rate of the douglas-rachford alternating direction method. SIAM J. Numer. Anal. 50(2), 700–709 (2012)

    Article  MathSciNet  Google Scholar 

  19. Liu Y., Shang F., Cheng J.: Accelerated variance reduced stochastic ADMM. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pp. 2287–2293 (2017)

    Google Scholar 

  20. Yu Y., Huang L.: Fast stochastic variance reduced ADMM for stochastic composition optimization. In: Proceedings of International Joint Conferences on Artifical Intelligence, pp. 3364–3370 (2017)

    Google Scholar 

  21. Shi, W., Ling, Q., Yuan, K., Gang, W., Yin, W.: On the linear convergence of the ADMM in decentralized consensus optimization. IEEE Trans. Signal Process. 62(7), 1750–1761 (2014)

    Article  MathSciNet  Google Scholar 

  22. Chen, C., He, B., Ye, Y., Yuan, X.: The direct extension of admm for multi-block convex minimization problems is not necessarily convergent. Math. Program. 155(1–2), 57–79 (2016)

    Article  MathSciNet  Google Scholar 

  23. Lin, C.J., Weng, R.C., Keerthi, S.S.: Trust region newton method for logistic regression. J. Mach. Learn. Res. 9, 627–650 (2008)

    MathSciNet  MATH  Google Scholar 

  24. He, B., Ma, F., Yuan, X.: On the step size of symmetric alternating directions method of multipliers (2015). http://www.optimization-online.org

Download references

Acknowledgement

This work is partially supported by the Fundamental Research Funds for the Central Universities, No. 30918014108, and the National Science Foundation of China No. 61806096.

Author information

Authors and Affiliations

  1. Key Laboratory of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, Nanjing University of Science and Technology, Nanjing, China

    Huihui Wang & Xiaolong Xu

  2. School of Computer Science and Technology, School of Software, Nanjing University of Posts and Telecommunications, Nanjing, China

    Xinwen Li & Xingguo Chen

  3. School of Information Science and Engineering, Qufu Normal University, Jining, China

    Lianyong Qi

Authors
  1. Huihui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xinwen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xingguo Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lianyong Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaolong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaolong Xu .

Editor information

Editors and Affiliations

  1. Macquarie University, Sydney, NSW, Australia

    Xuyun Zhang

  2. Macquarie University, Sydney, NSW, Australia

    Guanfeng Liu

  3. Pace University, New York, NY, USA

    Meikang Qiu

  4. James Cook University, Cairns, QLD, Australia

    Wei Xiang

  5. James Cook University, Smithfield, QLD, Australia

    Tao Huang

Rights and permissions

Reprints and permissions

Copyright information

© 2020 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, H., Li, X., Chen, X., Qi, L., Xu, X. (2020). Distributed Stochastic Alternating Direction Method of Multipliers for Big Data Classification. In: Zhang, X., Liu, G., Qiu, M., Xiang, W., Huang, T. (eds) Cloud Computing, Smart Grid and Innovative Frontiers in Telecommunications. CloudComp SmartGift 2019 2019. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 322. Springer, Cham. https://doi.org/10.1007/978-3-030-48513-9_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-48513-9_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-48512-2

  • Online ISBN: 978-3-030-48513-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation