Using ESPRESO as Linear Solver Library for Third Party FEM Tools for Solving Large Scale Problems

  • Ondřej MecaEmail author
  • Lubomír Říha
  • Alexandros Markopoulos
  • Tomáš Brzobohatý
  • Tomáš Kozubek
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11087)


ESPRESO is a FEM package that includes a Hybrid Total FETI (HTFETI) linear solver targeted at solving large scale engineering problems. The scalability of the solver was tested on several of the world’s largest supercomputers. To provide our scalable implementation of HTFETI algorithms to all potential users, a simple C API was developed and is presented. The paper describes API methods, compilation and linking process.

As a proof of concept we interfaced ESPRESO with the CSC ELMER solver and compared its performance with the ELMER FETI solver. HTFETI performs two level decomposition, which significantly improves both memory utilization and solver performance. To select optimal second level decomposition we have developed a performance model that controls decomposition automatically. This is a major simplification for all users that ensures optimal solver settings.

We show that the ESPRESO HTFETI solver is up to 3.7 times faster than the ELMER FETI solver when running on 13 500 MPI processes (the 614 compute nodes of the Salomon supercomputer) and solving 1.5 billion unknown problems of 3D linear elasticity.


Total FETI Hybrid Total FETI ESPRESO ELMER Automatic tunning model Multi-level decomposition 



This work was supported by The Ministry of Education, Youth and Sports from the Large Infrastructures for Research, Experimental Development and Innovations project “IT4Innovations National Supercomputing Center – LM2015070”.


  1. 1.
    Farhat, C., Roux, F.X., Oden, J.T.: Implicit Parallel Processing in Structural Mechanics. Elsevier Science, Amsterdam (1994)zbMATHGoogle Scholar
  2. 2.
    Farhat, C., Mandel, J., Roux, F.X.: Optimal convergence properties of the FETI domain decomposition method. Comput. Method Appl. Mech. Eng. 115, 365–385 (1994)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Farhat, C., Lesoinne, M., LeTallec, P., Pierson, K., Rixen, D.: FETI-DP: a dual-primal unified FETI method, part I: a faster alternative to the two-level FETI method. Int. J. Numer. Meth. Eng. 50(7), 1523–1544 (2001)CrossRefGoogle Scholar
  4. 4.
    Klawonn, A., Widlund, O.B., Dryja, M.: Dual-primal FETI methods for three-dimensional elliptic problems with heterogeneous coefficients. SIAM J. Numer. Anal. 40, 159–179 (2002)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Dostál, Z., Horák, D., Kučera, R.: Total FETI - an easier implementable variant of the FETI method for numerical solution of elliptic PDE. Commun. Numer. Methods Eng. 196, 1155–1162 (2006)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Klawonn, A., Rheinbach, R.: Highly scalable parallel domain decomposition methods with an application to biomechanics. ZAMM 1, 5–32 (2010)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Klawonn, A., Lanser, M., Rheinbach, O.: Toward extremely scalable nonlinear domain decomposition methods for elliptic partial differential equations. SIAM J. Sci. Comput. 37(6), C667–C696 (2015)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Brzobohatý, T., Jarošová, M., Kozubek, T., Menšík, M., Markopoulos, A.: The hybrid total FETI method. In: Proceedings of the Third International Conference on Parallel, Distributed, Grid and Cloud Computing for Engineering. Civil-Comp, Ltd. (2011)Google Scholar
  9. 9.
    Intel: Math kernel library.
  10. 10.
    Petra, C.G., Schenk, O., Lubin, M., Gärtner, K.: An augmented incomplete factorization approach for computing the Schur complement in stochastic optimization. SIAM J. Sci. Comput. 36(2), C139–C162 (2014)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Petra, C.G., Schenk, O., Anitescu, M.: Real-time stochastic optimization of complex energy systems on high-performance computers. IEEE Comput. Sci. Eng. 16(5), 32–42 (2014)CrossRefGoogle Scholar
  12. 12.
    Suzuki, A., Roux, F.X.: A dissection solver with kernel detection for symmetric finite element matrices on shared memory computers. Int. J. Numer. Methods Eng. 100(2), 136–164 (2014)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Karypis, G., Kumar, V.: A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM J. Sci. Comput. 20(1), 359–392 (1998)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Říha, L., Brzobohatý, T., Markopoulos, A., Meca, O., Kozubek, T.: Massively parallel hybrid total FETI (HTFETI) solver. In: Proceedings of the Platform for Advanced Scientific Computing Conference, PASC 2016, pp. 7:1–7:11. ACM, New York (2016)Google Scholar
  15. 15.
    ESPRESO: Public repository.
  16. 16.
    ESPRESO: Documentation.
  17. 17.
    Elmer: CSC - IT Center for Science.
  18. 18.
    Elmer: Public repository.

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Ondřej Meca
    • 1
    Email author
  • Lubomír Říha
    • 1
  • Alexandros Markopoulos
    • 1
  • Tomáš Brzobohatý
    • 1
  • Tomáš Kozubek
    • 1
  1. 1.IT4InnovationsVŠB - Technical University of OstravaOstrava-PorubaCzech Republic

Personalised recommendations