General Parallel Execution Model for Large Matrix Workloads

  • Song DengEmail author
  • Xueke Xu
  • Fan Zhou
  • Haojing Weng
  • Wen Luo
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 905)


Large scale statistical computing is crucial for extracting useful information from huge amount of data for both large companies and research scientists. The Solutions developed by high-performance communities have been more limited to clusters or high-end machines for decades. The cost of maintaining such dedicated clusters are prohibiting, people start to look at cloud computing where we can rent a cluster by time and pay-as-we-go. In a cloud setting, system features including fault tolerance and scalability become important. In this paper, we proposed a simple and universal parallel execution model for large matrix workloads. We implement the model in Hadoop MapReduce framework using map-only jobs. Because of the superiority of the model, experiments show that our Hadoop-based execution engine can reduce the execution time of matrix multiplication by half comparing with previous works.


General parallel execution Model Large matrix workloads Hodoop 



Our research was supported by the Natural Science Foundation of China under grant No: 61462037 and the Natural Science Foundation of Jiangxi under grant No: 20142BAB217014.


  1. 1.
    Castaldo, A.M., Whaley, R.C.: Scaling LAPACK panel operations using parallel cache assignment. In: ACM Sigplan Symposium on Principles and Practice of Parallel Programming, vol. 45, no. 5, pp. 223–232 (2010)CrossRefGoogle Scholar
  2. 2.
    Toledo, S., Gustavson, F.G.: The design and implementation of SOLAR, a portable library for scalable out-of-core linear algebra computations. In: Proceedings of the Fourth Workshop on I/O in Parallel and Distributed Systems, pp. 28–40 (1996)Google Scholar
  3. 3.
    Wu, P., Chen, Z.Z.: FT-ScaLAPACK: correcting soft errors on-line for ScaLAPACK cholesky, QR and LU factorization routines. In: Proceedings of the 23rd International Symposium on High-performance Parallel and Distributed Computing (HPDC 2014), Vancouver, BC, Canada (2014)Google Scholar
  4. 4.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: 6th Conference on Symposium on Operating Systems Design & Implementation (OSDI 2004), Berkeley, CA, USA, vol. 6, pp. 107–113 (2004)Google Scholar
  5. 5.
    Li, J., Ma, X., Yoginath, S.B., Kora, G., Samatova, N.F.: Transparent runtime parallelization of the R scripting language. J. Parallel Distrib. Comput. 71(2), 157–168 (2011)CrossRefGoogle Scholar
  6. 6.
    Das, S., Sismanis, Y., Beyer, K.S., Gemulla, R., Haas, P.J., McPherson, J.: Ricardo: integrating R and Hadoop. In: Proceedings of the 2010 International Conference on Management of Data, Indianapolis, Indiana, USA, pp. 987–998 (2010)Google Scholar
  7. 7.
    Boehm, M., Tatikonda, S., Reinwald, B., Sen, P., Tian, Y., Burdick, D. R., Vaithyanathan, S.: Hybrid parallelization strategies for large-scale machine learning in SystemML. In: Proceedings of the VLDB Endowment, Hangzhou, China, vol. 7, no. 7 (2014)CrossRefGoogle Scholar
  8. 8.
    Lee, D.D., Seung, H.S.: Algorithms for non-negative matrix factorization. In: Proceedings of the NIPS, Denver, CO, USA (2000)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Song Deng
    • 1
    Email author
  • Xueke Xu
    • 1
  • Fan Zhou
    • 1
  • Haojing Weng
    • 1
  • Wen Luo
    • 1
  1. 1.School of Software and Internet of Things EngineeringJiangxi University of Finance and EconomicsNanchangChina

Personalised recommendations