Skip to main content
SpringerLink
Log in

EDGE: Extreme Scale Fused Seismic Simulations with the Discontinuous Galerkin Method

  • Conference paper
  • First Online:
High Performance Computing (ISC High Performance 2017)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 10266))

Included in the following conference series:

Abstract

This article introduces EDGE, a solver package for fused seismic simulations. Fused seismic simulations are a novel technique addressing one of the grand challenges of computational seismology: large ensemble runs of geometrically similar forward simulations. Application fields include, but are not limited to: uncertainty quantification in the context of seismic hazard analysis or the accurate derivation of velocity models through tomographic inversion. For efficient and accurate handling of complex model geometries (topography, fault geometries, material heterogeneities), EDGE utilizes the Discontinuous Galerkin (DG) method for spatial and Arbitrary high order DERivatives (ADER) for time discretization, implemented for unstructured tetrahedral meshes. EDGE’s ADER-DG scheme requires sparse and dense matrix-matrix multiplications at the kernel level. By choosing a sufficient memory layout and relying on runtime code generation and specialization, both, sparse and dense operations, can be efficiently vectorized on wide-SIMD machines. We present a convergence study of single and fused seismic simulations, code verification in an established benchmark, as well as a detailed performance assessment for different discretization orders. As target architecture we select the recently released Intel Xeon Phi processor, which powers the Theta and Cori-II supercomputers. For a single sixth order seismic forward simulation we achieved 10.4 PFLOPS of hardware performance and 5.0 PFLOPS for fused simulations in fourth order, both occupying 9,000 nodes of Cori-II. From a throughput perspective, fused seismic simulations can outperform a single forward simulation by 1.8\(\times \) to 4.6\(\times \), depending on the chosen order of the method.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Notes

  1. 1.

    EDGE is available under the 3-clause BSD license at http://dial3343.org.

References

  1. Reinders, J., Jeffers, J., Sodani, A. (eds.) Intel Xeon Phi Processor High Performance Programming Knights Landing Edition (2016). Ch. 4 and Ch. 6

    Google Scholar 

  2. Aoi, S., Fujiwara, H.: 3D finite-difference method using discontinuous grids. Bull. Seismol. Soc. Am. 89(4), 918–930 (1999)

    Google Scholar 

  3. Appelö, D., Petersson, N.A.: A stable finite difference method for the elastic wave equation on complex geometries with free surfaces. Commun. Comput. Phys. 5(1), 84–107 (2009)

    MathSciNet  MATH  Google Scholar 

  4. Bastian, P., et al.: Hardware-based efficiency advances in the EXA-DUNE project. In: Bungartz, H.-J., Neumann, P., Nagel, W.E. (eds.) Software for Exascale Computing - SPPEXA 2013-2015. LNCSE, vol. 113, pp. 3–23. Springer, Cham (2016). doi:10.1007/978-3-319-40528-5_1

    Chapter  Google Scholar 

  5. Benjemaa, M., Glinsky-Olivier, N., Cruz-Atienza, V.M., Virieux, J.: 3D dynamic rupture simulations by a finite volume method. Geophys. J. Int. 178, 541–560 (2009)

    Article  Google Scholar 

  6. Breuer, A., Heinecke, A., Rettenberger, S., Bader, M., Gabriel, A.-A., Pelties, C.: Sustained petascale performance of seismic simulations with SeisSol on SuperMUC. In: Kunkel, J.M., Ludwig, T., Meuer, H.W. (eds.) ISC 2014. LNCS, vol. 8488, pp. 1–18. Springer, Cham (2014). doi:10.1007/978-3-319-07518-1_1

    Chapter  Google Scholar 

  7. Chaljub, E., Komatitsch, D., Vilotte, J.-P., Capdeville, Y., Valette, B., Festa, G.: Spectral-element analysis in seismology. Adv. Geophys. 48, 365–419 (2007). Advances in Wave Propagation in Heterogenous Earth, http://www.sciencedirect.com/science/article/pii/S0065268706480079

    Article  Google Scholar 

  8. Chaljub, E., Maufroy, E., Moczo, P., Kristek, J., Hollender, F., Bard, P.-Y., Priolo, E., Klin, P., de Martin, F., Zhang, Z., Zhang, W., Chen, X.: 3-D numerical simulations of earthquake ground motion in sedimentary basins: testing accuracy through stringent models. Geophys. J. Int. 201(1), 90–111 (2015)

    Article  Google Scholar 

  9. Dumbser, M., Käser, M.: An arbitrary high-order discontinuous galerkin method for elastic waves on unstructured meshes - II. The three-dimensional isotropic case. Geophys. J. Int. 167(1), 319–336 (2006)

    Article  Google Scholar 

  10. Duru, K., Dunham, E.M.: Dynamic earthquake rupture simulations on nonplanar faults embedded in 3D geometrically complex, heterogeneous elastic solids. J. Comput. Phys. 305, 185–207 (2016)

    Article  MathSciNet  Google Scholar 

  11. Etienne, V., Chaljub, E., Virieux, J.: An hp-adaptive discontinuous Galerkin finite-element method for 3-D elastic wave modelling. Geophys. J. Int. 183(2), 941–962 (2010)

    Article  Google Scholar 

  12. Geuzaine, C., Remacle, J.-F.: Gmsh: A 3-D finite element mesh generator with built-in pre-and post-processing facilities. Int. J. Numer. Methods Eng. 79(11), 1309–1331 (2009)

    Article  MathSciNet  Google Scholar 

  13. Heinecke, A., Breuer, A., Bader, M., Dubey, P.: High order seismic simulations on the intel Xeon Phi processor (Knights Landing). In: Kunkel, J.M., Balaji, P., Dongarra, J. (eds.) ISC High Performance 2016. LNCS, vol. 9697, pp. 343–362. Springer, Cham (2016). doi:10.1007/978-3-319-41321-1_18

    Chapter  Google Scholar 

  14. Heinecke, A., Breuer, A., Rettenberger, S., Bader, M., Gabriel, A.-A., Pelties, C., Bode, A., Barth, W., Liao, X.-K., Vaidyanathan, K., et al.: Petascale high order dynamic rupture earthquake simulations on heterogeneous supercomputers. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (2014)

    Google Scholar 

  15. Heinecke, A., Henry, G., Hutchinson, M., Pabst, H.: LIBXSMM: accelerating small matrix multiplications by runtime code generation. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (2016)

    Google Scholar 

  16. Ichimura, T., Fujita, K., Quinay, P., Maddegedara, L., Hori, M., Tanaka, S., Shizawa, Y., Kobayashi, H., Minami, K.: Implicit nonlinear wave simulation with 1.08T DOF and 0.270T unstructured finite elements to enhance comprehensive earthquake simulation (2015)

    Google Scholar 

  17. Ichimura, T., Hori, M., Bielak, J.: A hybrid multiresolution meshing technique for finite element three-dimensional earthquake ground motion modelling in basins including topography. Geophys. J. Int. 177(3), 1221–1232 (2009)

    Article  Google Scholar 

  18. George, K., Vipin, K.: MeTis: Unstructured Graph Partitioning and Sparse Matrix Ordering System, version 4.0 (2009)

    Google Scholar 

  19. Kang, T.-S., Baag, C.-E.: An efficient finite-difference method for simulating 3D seismic response of localized basin structures. Bull. Seismol. Soc. Am. 94(5), 1690–1705 (2004)

    Article  Google Scholar 

  20. Käser, M., Dumbser, M., Puente, J., Igel, H.: An arbitrary high-order discontinuous galerkin method for elastic waves on unstructured meshes - III. viscoelastic attenuation. Geophys. J. Int. 168(1), 224–242 (2007)

    Article  Google Scholar 

  21. Komatitsch, D., Tromp, J.: Spectral-element simulations of global seismic wave propagation-II. Three-dimensional models, oceans, rotation and self-gravitation. Geophys. J. Int. 150(1), 303–318 (2002)

    Article  Google Scholar 

  22. Moczo, P., Ampuero, J.P., Kristek, J., Day, S.M., Kristekova, M., Pazak, P., Galis, M., Igel, H.: Comparison of numerical methods for seismic wave propagation and source dynamics - the SPICE code validation. In: Third International Symposium on the Effects of Surface Geology on Seismic Motion. Actes des journées scientifiques du LCPC. Laboratoire central des ponts et chaussées, Paris, France, pp. 1–10 (2006). ISBN 9782720824654

    Google Scholar 

  23. Kristeková, M., Kristek, J., Moczo, P.: Time-frequency misfit and goodness-of-fit criteria for quantitative comparison of time signals. Geophys. J. Int. 178(2), 813–825 (2009)

    Article  Google Scholar 

  24. Kristeková, M., Kristek, J., Moczo, P., Day, S.M.: Misfit criteria for quantitative comparison of seismograms. Bull. Seismol. Soc. Am. 96(5), 1836–1850 (2006)

    Article  Google Scholar 

  25. Liu, J., Chandrasekaran, B., Wu, J., Jiang, W., Kini, S., Yu, W., Buntinas, D., Wyckoff, P., Panda, D.K.: Performance comparison of MPI implementations over InfiniBand, Myrinet and Quadrics. In: Proceedings of the 2003 ACM/IEEE Conference on Supercomputing (2003)

    Google Scholar 

  26. Ma, S., Liu, P.: Modeling of the perfectly matched layer absorbing boundaries and intrinsic attenuation in explicit finite-element methods. Bull. Seismol. Soc. Am. 96(5), 1779–1794 (2006)

    Article  Google Scholar 

  27. Malas, T., Kurth, T., Deslippe, J.: Optimization of the sparse matrix-vector products of an IDR Krylov iterative solver in EMGeo for the Intel KNL manycore processor. In: Taufer, M., Mohr, B., Kunkel, J.M. (eds.) ISC High Performance 2016. LNCS, vol. 9945, pp. 378–389. Springer, Cham (2016). doi:10.1007/978-3-319-46079-6_27

    Chapter  Google Scholar 

  28. Moczo, P., Kristek, J., Vavryčuk, V.: 3D heterogeneous staggered-grid finite-difference modeling of seismic motion with volume harmonic and arithmetic averaging of elastic moduli and densities. Bull. Seismol. Soc. Am. 92(8), 3042–3066 (2002)

    Article  Google Scholar 

  29. Moczo, P., Robertsson, J.O.A., Eisner, L.: The finite-difference time-domain method for modeling of seismic wave propagation. Adv. Geophys. 48, 421–516 (2007). Advances in Wave Propagation in Heterogenous Earth, http://www.sciencedirect.com/science/article/pii/S0065268706480080

    Article  Google Scholar 

  30. Modave, A., St-Cyr, A., Warburton, T.: GPU performance analysis of a nodal discontinuous Galerkin method for acoustic and elastic models. Comput. Geosci. 91, 64–76 (2016)

    Article  Google Scholar 

  31. Peter, D., Komatitsch, D., Luo, Y., Martin, R., Goff, N., Casarotti, E., Loher, P., Magnoni, F., Liu, Q., Blitz, C., Nissen-Meyer, T., Basini, P., Tromp, J.: Forward and adjoint simulations of seismic wave propagation on fully unstructured hexahedral meshes. Geophys. J. Int. 186(2), 721–739 (2011)

    Article  Google Scholar 

  32. Pitarka, A.: 3D elastic finite-difference modeling of seismic motion using staggered grids with nonuniform spacing. Bull. Seismol. Soc. Am. 89(1), 54–68 (1999)

    Article  MathSciNet  Google Scholar 

  33. Shepherd, J.F., Johnson, C.R.: Hexahedral mesh generation constraints. Eng. Comput. 24(3), 195–213 (2008)

    Article  Google Scholar 

  34. Symes, W.W., Vdovina, T.: Interface error analysis for numerical wave propagation. Comput. Geosci. 13(3), 363–371 (2009)

    Article  Google Scholar 

  35. Taborda, R., Bielak, J.: Large-Scale earthquake simulation: computational seismology and complex engineering systems. Comput. Sci. Eng. 13(4), 14–27 (2011)

    Article  Google Scholar 

  36. Cruz-Atienza, V.M., Virieux, J., Aochi, H.: 3D finite-difference dynamic-rupture modeling along nonplanar faults. Geophysics 72(5), 123–137 (2007)

    Article  Google Scholar 

Download references

Acknowledgements

Only the great support of experts at NERSC and ALCF made our extreme-scale results possible. In particular, we thank J. Deslippe, S. Dosanjh, R. Gerber, and K. Kumaran. This work was supported by the Southern California Earthquake Center (SCEC) through contribution #16247. This research used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. This research used resources of the Argonne Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC02-06CH11357. This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1053575.

Optimization Notice: Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance. Intel, Xeon, and Intel Xeon Phi are trademarks of Intel Corporation in the U.S. and/or other countries.

Author information

Authors and Affiliations

  1. University of California, San Diego, 9500 Gilman Drive, La Jolla, CA, 92093, USA

    Alexander Breuer & Yifeng Cui

  2. Intel Corporation, 2200 Mission College Blvd., Santa Clara, CA, 95054, USA

    Alexander Heinecke

Authors
  1. Alexander Breuer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexander Heinecke
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yifeng Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexander Breuer .

Editor information

Editors and Affiliations

  1. Deutsches Klimarechenzentrum (DKRZ), Hamburg, Germany

    Julian M. Kunkel

  2. Tokyo Institute of Technology, Tokyo, Japan

    Rio Yokota

  3. Argonne National Laboratory, Argonne, IL, USA

    Pavan Balaji

  4. KAUST, Thuwal, Saudi Arabia

    David Keyes

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Breuer, A., Heinecke, A., Cui, Y. (2017). EDGE: Extreme Scale Fused Seismic Simulations with the Discontinuous Galerkin Method. In: Kunkel, J.M., Yokota, R., Balaji, P., Keyes, D. (eds) High Performance Computing. ISC High Performance 2017. Lecture Notes in Computer Science(), vol 10266. Springer, Cham. https://doi.org/10.1007/978-3-319-58667-0_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-58667-0_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-58666-3

  • Online ISBN: 978-3-319-58667-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation