Abstract
This paper presents a performance analysis of the finite element assembly process of the Albany Land Ice solver. The analysis shows that a speedup over traditional MPI-only simulations is achieved on multiple architectures including Intel Haswell CPUs, Intel Xeon Phi Knights Landing and IBM POWER8/NVIDIA P100 platforms. A scalability study also shows that performance remains reasonably close among all architectures. These results are obtained on a single codebase without architecture-dependent code optimizations by utilizing abstractions in shared memory parallelism from the Kokkos library and is part of an ongoing process of achieving performance portability for the Albany Land Ice code.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The interested reader is referred to Appendix A of [32] for the nonlinear Stokes flow equations.
- 2.
Available on github: https://github.com/SNLComputation/Albany.
- 3.
Available on github: https://github.com/trilinos/Trilinos.
References
Anzt, H., Augustin, W., Baumann, M., Bockelmann, H., Gengenbach, T., Hahn, T., Heuveline, V., Ketelaer, E., Lukarski, D., Otzen, A., et al.: Hiflow3–a flexible and hardware-aware parallel finite element package. Preprint Series of the Engineering Mathematics and Computing Lab, 6 (2010)
Baker, C.G., Heroux, M.A.: Tpetra, and the use of generic programming in scientific computing. Sci. Program. 20(2), 115–128 (2012)
Blatter, H.: Velocity and stress fields in grounded glaciers: a simple algorithm for including deviatoric stress gradients. J. Glaciol. 41(138), 333–344 (1995)
Brædstrup, C.F., Damsgaard, A., Egholm, D.L.: Ice-sheet modelling accelerated by graphics cards. Comput. Geosci. 72, 210–220 (2014)
CISM/The Community Ice Sheet Model. https://cism.github.io/index.html. Accessed 30 April 2018
Cornford, S.L., Martin, D.F., Graves, D.T., Ranken, D.F., A.M., Brocq, L., Gladstone, R.M., Payne, A.J., Ng, E.G., Lipscomb, W.H.: Adaptive mesh, finite volume modeling of marine ice sheets. J. Comput. Phys. 232(1), 529–549 (2013)
Demeshko, I., Watkins, J., Tezaur, I.K., Guba, O., Spotz, W.F., Salinger, A.G., Pawlowski, R.P., Heroux, M.A.: Toward performance portability of the Albany finite element analysis code using the Kokkos library. Int. J. High Perform. Comput. Appl. 33(2), 332–352 (2019)
Dukowicz, J.K., Price, S.F., Lipscomb, W.H.: Consistent approximations and boundary conditions for ice-sheet dynamics from a principle of least action. J. Glaciol. 56(197), 480–496 (2010)
Edwards, H.C., Trott, C.R., Sunderland, D.: Kokkos: enabling manycore performance portability through polymorphic memory access patterns. J. Parallel Distrib. Comput. 74(12), 3202–3216 (2014)
Evans, K.J., Kennedy, J.H., Lu, D., Forrester, M.M., Price, S., Fyke, J., Bennett, A.R., Hoffman, M.J., Tezaur, I., Zender, C.S., Vizcaíno, M.: LIVVkit 2.1: automated and extensible ice sheet model validation. Geosci. Model Dev. 12(3), 1067–1086 (2019)
Gagliardini, O., Zwinger, T., Gillet-Chaulet, F., Durand, G., Favier, L., de Fleurian, B., Greve, R., Malinen, M., Martín, C., Råback, P., et al.: Capabilities and performance of Elmer/Ice, a new-generation ice sheet model. Geosci. Model Dev. 6(4), 1299–1318 (2013)
Heroux, M.A., Bartlett, R.A., Howle, V.E., Hoekstra, R.J., Hu, J.J., Kolda, T.G., Lehoucq, R.B., Long, K.R., Pawlowski, R.P., Phipps, E.T., et al.: An overview of the Trilinos project. ACM Trans. Math. Softw. 31(3), 397–423 (2005)
Hoffman, M.J., Perego, M., Price, S.F., Lipscomb, W.H., Zhang, T., Jacobsen, D., Tezaur, I., Salinger, A.G., Tuminaro, R. and Bertagna, L.: MPAS-Albany Land Ice (MALI): a variable-resolution ice sheet model for Earth system modeling using Voronoi grids. Geosci. Model Dev. 11(9), 3747-3780 (2018)
Hornung, R.D., Keasler, J.A.: The RAJA portability layer: overview and status. Technical report, Lawrence Livermore National Lab. (LLNL), Livermore (2014)
Larour, E., Seroussi, H., Morlighem, M., Rignot, E.: Continental scale, high order, high spatial resolution, ice sheet modeling using the Ice Sheet System Model (ISSM). J. Geophys. Res. Earth Surf. 117(F1) (2012)
Markall, G.R., Slemmer, A., Ham, D.A., Kelly, P.H.J., Cantwell, C.D., Sherwin, S.J.: Finite element assembly strategies on multi-core and many-core architectures. Int. J. Numer. Methods Fluids 71(1), 80–97 (2013)
Medina, D.S., St-Cyr, A., Warburton, T.: OCCA: a unified approach to multi-threading languages. arXiv preprint arXiv:1403.0968 (2014)
MPAS-Albany Land Ice. https://mpas-dev.github.io/land_ice/land_ice.html. Accessed 30 April 2018
Neely, J.R.: DOE centers of excellence performance portability meeting. Technical report, Lawrence Livermore National Lab. (LLNL), Livermore (2016)
Pattyn, F.: A new three-dimensional higher-order thermomechanical ice sheet model: basic sensitivity, ice stream development, and ice flow across subglacial lakes. J. Geophys. Res. Solid Earth 108(B8) (2003)
Pawlowski, R.P., Phipps, E.T., Salinger, A.G.: Automating embedded analysis capabilities and managing software complexity in multiphysics simulation, part I: template-based generic programming. Sci. Program. 20(2), 197–219 (2012)
Pennycook, S.J., Sewall, J.D., Lee, V.W.: A metric for performance portability. arXiv preprint arXiv:1611.07409 (2016)
Pennycook, S.J., Sewall, J.D., Lee, V.W.: Implications of a metric for performance portability. Futur. Gener. Comput. Syst. 92, 947–958 (2017)
Perego, M., Price, S., Stadler, G.: Optimal initial conditions for coupling ice sheet models to Earth system models. J. Geophys. Res. Earth Surf. 119(9), 1894–1917 (2014)
Price, S.F., Hoffman, M.J., Bonin, J.A., Howat, I.M., Neumann, T., Saba, J., Tezaur, I., Guerber, J., Chambers, D.P., Evans, K.J., et al.: An ice sheet model validation framework for the Greenland ice sheet. Geosci. Model Dev. 10(1), 255–270 (2017)
Rathgeber, F., Markall, G.R., Mitchell, L., Loriant, N., Ham, D.A., Bertolli, C., Kelly, P.H.J.: PyOP2: a high-level framework for performance-portable simulations on unstructured meshes. In: 2012 SC Companion High Performance Computing, Networking, Storage and Analysis (SCC), pp. 1116–1123. IEEE, Piscataway (2012)
Rathgeber, F., Ham, D.A., Mitchell, L., Lange, M., Luporini, Fabio, A., McRae, T.T., Bercea, G.-T., Markall, G.R., Kelly, P.H.J.: Firedrake: automating the finite element method by composing abstractions. ACM Trans. Math. Softw. 43(3), 24 (2017)
Rutt, I.C., Hagdorn, M., Hulton, N.R.J., Payne, A.J.: The Glimmer community ice sheet model. J. Geophys. Res. Earth Surf. 114(F2) (2009)
Salinger, A.G., Bartlett, R.A., Bradley, A.M., Chen, Q., Demeshko, I.P., Gao, X., Hansen, G.A., Mota, A., Muller, R.P., Nielsen, E., et al.: Albany: using component-based design to develop a flexible, generic multiphysics analysis code. Int. J. Multiscale Comput. Eng. 14(4), 415–438 (2016)
Schoof, C., Hindmarsh, R.C.A.: Thin-film flows with wall slip: an asymptotic analysis of higher order glacier flow models. Q. J. Mech. Appl. Math. 63(1), 73–114 (2010)
Solomon, S.: Climate Change 2007-the Physical Science Basis: Working Group I Contribution to the Fourth Assessment Report of the IPCC, vol. 4. Cambridge University Press, Cambridge (2007)
Tezaur, I.K., Perego, M., Salinger, A.G., Tuminaro, R.S., Price, S.F.: Albany/FELIX: a parallel, scalable and robust, finite element, first-order Stokes approximation ice sheet solver built for advanced analysis. Geosci. Model Dev. 8(4), 1197 (2015)
Tezaur, I.K., Tuminaro, R.S., Perego, M., Salinger, A.G., Price, S.F.: On the scalability of the Albany/FELIX first-order Stokes approximation ice sheet solver for large-scale simulations of the Greenland and Antarctic ice sheets. Proc. Comput. Sci. 51, 2026–2035 (2015)
TOP500 Project: November 2017 TOP500 list. https://www.top500.org/lists/2017/11/. Accessed 5 April 2018
Tuminaro, R., Perego, M., Tezaur, I., Salinger, A., Price, S.: A matrix dependent/algebraic multigrid approach for extruded meshes with applications to ice sheet modeling. SIAM J. Sci. Comput. 38(5), C504–C532 (2016)
Winkelmann, R., Martin, M.A., Haseloff, M., Albrecht, T., Bueler, E., Khroulev, C., Levermann, A.: The Potsdam parallel ice sheet model (PISM-PIK)-part 1: model description. Cryosphere 5(3), 715 (2011)
Wright, S., Nocedal, J.: Numerical optimization. Springer Science, vol. 35, pp. 67–68. Springer, Berlin (1999)
Acknowledgements
This work was supported under the Biological and Environmental Research (BER) Scientific Discovery through Advanced Computing (SciDAC) Partnership: a collaboration between the Advanced Scientific Computing Research (ASCR) and BER programs funded by the U.S. Department of Energy’s Office of Science. The refactoring of Albany to a Kokkos programming model was also supported by Frameworks, Algorithms and Scalable Technologies for Mathematics (FASTMath) SciDAC Institute.
This research used resources of the National Energy Research Scientific Computing Center, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231.
Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia LLC, a wholly owned subsidiary of Honeywell International Inc. for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Watkins, J., Tezaur, I., Demeshko, I. (2020). A Study on the Performance Portability of the Finite Element Assembly Process Within the Albany Land Ice Solver. In: van Brummelen, H., Corsini, A., Perotto, S., Rozza, G. (eds) Numerical Methods for Flows. Lecture Notes in Computational Science and Engineering, vol 132. Springer, Cham. https://doi.org/10.1007/978-3-030-30705-9_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-30705-9_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-30704-2
Online ISBN: 978-3-030-30705-9
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)