Journal of Signal Processing Systems

, Volume 90, Issue 1, pp 53–67 | Cite as

A Hybrid CPU-GPU Multifrontal Optimizing Method in Sparse Cholesky Factorization

  • Yong Chen
  • Hai Jin
  • Ran Zheng
  • Yuandong Liu
  • Wei Wang


In many scientific computing applications, sparse Cholesky factorization is used to solve large sparse linear equations in distributed environment. GPU computing is a new way to solve the problem. However, sparse Cholesky factorization on GPU is hardly to achieve excellent performance due to the structure irregularity of matrix and the low GPU resource utilization. A hybrid CPU-GPU implementation of sparse Cholesky factorization is proposed based on multifrontal method. A large sparse coefficient matrix is decomposed into a series of small dense matrices (frontal matrices) in the method, and then multiple GEMM (General Matrix-matrix Multiplication) operations are computed on them. GEMMs are the main operations in sparse Cholesky factorization, but they are hardly to perform better in parallel on GPU. In order to improve the performance, the scheme of multiple task queues is adopted to perform multiple GEMMs parallelized with multifrontal method; all GEMM tasks are scheduled dynamically on GPU and CPU based on computation scales for load balance and computing-time reduction. Experimental results show that the approach can outperform the implementations of cuBLAS, achieving up to 1.98× speedup on GTX460 (Fermi micro-architecture) and 3.06× speedup on K20m (Kepler micro-architecture), respectively.


Multifrontal method Multiple task queues scheme Task allocation GPU acceleration 



This work is supported by the National Natural Science Foundation of China (grant No. 61133008) and the National Basic Research Program (973 Program) (grant No. 2013CB2282036).


  1. 1.
    Heath, MT, Ng, E, & Peyton, BW (1991). Parallel algorithms for sparse linear systems. SIAM Review, 33(3), 420–460.MathSciNetCrossRefMATHGoogle Scholar
  2. 2.
    Demmel, JW, Gilbert, JR, & Li, XS (1999). An asynchronous parallel supernodal algorithm for sparse gaussian elimination. SIAM Journal on Matrix Analysis and Applications, 20(4), 915–952.MathSciNetCrossRefMATHGoogle Scholar
  3. 3.
    Avron, H, Shklarski, G, & Toledo, S (2008). Parallel unsymmetric-pattern multifrontal sparse LU with column preorderin. ACM Transactions on Mathematical Software (TOMS), 34(2), 8.CrossRefMATHGoogle Scholar
  4. 4.
    Liu, JWH (1992). The multifrontal method for sparse matrix solution: Theory and practice. SIAM Review, 34(1), 82–109.MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Li, SG, Hu, CJ, Zhang, JC, & et al. (2015). Automatic tuning of sparse matrix-vector multiplication on multicore clusters. Science China Information Sciences, 58(9), 1–14.Google Scholar
  6. 6.
  7. 7.
    Dongarra, JJ, Du Croz, J, Hammarling, S, & et al. (1990). A set of level 3 basic linear algebra subprograms. ACM Transactions on Mathematical Software (TOMS), 16(1), 1–17.CrossRefMATHGoogle Scholar
  8. 8.
    Li, X, Li, F, & Clark, JM (2013). Exploration of multifrontal method with GPU in power flow computation. In Power and Energy Society General Meeting (PES), 2013 IEEE (pp. 1–5). IEEE.Google Scholar
  9. 9.
    Sao, P, Vuduc, R, & Li, XS (2014). A distributed, CPU-GPU sparse direct solver. In Euro-par 2014 parallel processing (pp. 487–498). Springer International Publishing.Google Scholar
  10. 10.
    Schenk, O, Christen, M, & Burkhart, H (2008). Algorithmic performance studies on graphics processing units. Journal of Parallel and Distributed Computing, 68(10), 1360–1369.CrossRefGoogle Scholar
  11. 11.
  12. 12.
  13. 13.
  14. 14.
    Toledo, S, Chen, D, & Rotkin, V. Taucs: a library of sparse linear solvers.
  15. 15.
    George, T, Saxena, V, Gupta, A, & et al. (2011). Multifrontal factorization of sparse SPD matrices on GPUs. Parallel & Distributed Processing Symposium (IPDPS), 2011, IEEE International (pp. 372–383). IEEE.Google Scholar
  16. 16.
    Gupta, A. (2000). WSMP watson sparse matrix package (Part-I: direct solution of symmetric sparse systems). Yorktown Heights: IBM TJ Watson Research Center. Tech. Rep RC.Google Scholar
  17. 17.
    Li, XS, & Demmel, JW (2003). SuperLU_DIST: a scalable distributed-memory sparse direct solver for unsymmetric linear systems. ACM Transactions on Mathematical Software (TOMS), 29(2), 110–140.CrossRefMATHGoogle Scholar
  18. 18.
    Lucas, RF, Wagenbreth, G, Davis, DM, & et al. (2010). Multifrontal computations on GPUs and their multi-core hosts. High Performance Computing for Computational ScienceCVECPAR 2010, (pp. 71–82). Berlin: Springer.MATHGoogle Scholar
  19. 19.
    Yu, CD, Wang, W, & Pierce, D (2011). A CPU-GPU hybrid approach for the unsymmetric multifrontal method. Parallel Computing, 37(12), 759–770.CrossRefGoogle Scholar
  20. 20.
    Davis, TA (2004). Algorithm 832: UMFPACK V4. 3—an unsymmetric-pattern multifrontal method. ACM Transactions on Mathematical Software (TOMS), 30(2), 196–199.MathSciNetCrossRefMATHGoogle Scholar
  21. 21.
    Lebedev, S, Akhmedzhanov, D, Kozinov, E, & et al. (2015). Dynamic parallelization strategies for multifrontal sparse cholesky factorization. Parallel computing technologies (pp. 68–79). Springer International Publishing.Google Scholar
  22. 22.
    Rennich, SC, Stosic, D, & Davis, TA (2014). Accelerating sparse cholesky factorization on GPUs. Proceedings of the Fourth Workshop on Irregular Applications: Architectures and Algorithms (pp. 9–16). IEEE Press.Google Scholar
  23. 23.
    Kim, K, & Eijkhout, V (2013). Scheduling a parallel sparse direct solver to multiple GPUs. Parallel and Distributed Processing Symposium Workshops & PhD Forum (IPDPSW), 2013 IEEE 27th International. IEEE, (pp. 1401–1408).Google Scholar
  24. 24.
    Zou, D, Dou, Y, Guo, S, & et al. (2014). Supernodal sparse cholesky factorization on graphics processing units. Concurrency and Computation: Practice and Experience, 26(16), 2713–2726.CrossRefGoogle Scholar
  25. 25.
    Yeralan, SN, Davis, TA, & Ranka, S (2013). Sparse QR factorization on gpu architectures. University of Florida. Tech. Rep.Google Scholar
  26. 26.
    Ren, L, Chen, X, Wang, Y, & et al. (2012). Sparse LU factorization for parallel circuit simulation on GPU. In Proceedings of the 49th Annual Design Automation Conference (pp. 1125–1130), ACM.Google Scholar
  27. 27.
    MIT CSAIL Supertech Research Group. Cilk: A linguistic and runtime technology for algorithmic multithreaded programming.
  28. 28.
    Chen, L, Villa, O, Krishnamoorthy, S, & et al. (2010). Dynamic load balancing on single-and multi-GPU systems. In IEEE international symposium on parallel & distributed processing (IPDPS), 2010 (pp. 1–12). IEEE.Google Scholar
  29. 29.
    Davis, TA, & Hu, Y. The University of Florida sparse matrix collection.
  30. 30.
    Wang, H, Wang, R, Luan, ZZ, & et al. (2015). Improving multiprocessor performance with fine-grain coherence bypass. Science China Information Sciences, 58(1), 1–15.Google Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  • Yong Chen
    • 1
  • Hai Jin
    • 1
  • Ran Zheng
    • 1
  • Yuandong Liu
    • 1
  • Wei Wang
    • 1
  1. 1.Services Computing Technology and System Lab, Big Data Technology and System Lab, Cluster and Grid Computing Lab, School of Computer Science and TechnologyHuazhong University of Science and TechnologyWuhanChina

Personalised recommendations