Optimization and Local Time Stepping of an ADER-DG Scheme for Fully Anisotropic Wave Propagation in Complex Geometries
- 148 Downloads
Abstract
We present an extension of the earthquake simulation software SeisSol to support seismic wave propagation in fully triclinic anisotropic materials. To our best knowledge, SeisSol is one of the few open-source codes that offer this feature for simulations at petascale performance and beyond. We employ a Discontinuous Galerkin (DG) method with arbitrary high-order derivative (ADER) time stepping. Here, we present a novel implementation of fully physical anisotropy with a two-sided Godunov flux and local time stepping. We validate our implementation on various benchmarks and present convergence analysis with respect to analytic solutions. An application example of seismic waves scattering around the Zugspitze in the Bavarian Alps demonstrates the capabilities of our implementation to solve geophysics problems fast.
1 Introduction
To successfully model earthquakes and perform seismic simulations, accurate models for the source dynamics and the propagation of seismic waves are needed. For seismic wave propagation, acoustic, isotropic and anisotropic elastic, attenuating and poroelastic materials are the most important rheologies [21]. Seismic anisotropy affects speed and scattering of seismic waves depending on the direction of propagation and can be found on all scales in the solid Earth. Anisotropy can stem from finely layered or cracked materials, the internal crystal structure of minerals or the alignment of ice crystals in glaciers. Anisotropic material behavior is observed in fault zones [16, 17] and accounted for on global scale in refinements of the Preliminary Reference Earth Model [8]. Effective anisotropy on the scales of seismic wavelengths can be modeled by assuming homogeneous materials with directional dependent properties.
Anisotropy is one of the key seismic properties next to velocity and intrinsic attenuation. Locally, at the scale of earthquake fault zones, large variations in anisotropy reflect the strong material contrasts, extreme strains, and high dissipation of damaged rock. At the global scale, variations in anisotropy provide snapshots of our planet’s interior that inform our understanding of plate tectonics. Imaging of anisotropy is also crucial in industry contexts such as exploration or geothermal reservoir development and maintenance. All these applications require efficient forward solvers, ideally accounting for physical anisotropy together with the geometrical complexity of the geological subsurface. High-order accuracy is crucial to resolve small variations of anisotropy, which are often within only a few percent variation of isotropic material, depending on tectonic context.
Anisotropic material behavior has been successfully included in Finite Difference schemes [11, 22], pseudo-spectral methods [5], Spectral Element codes [13] and Discontinuous Galerkin (DG) schemes for the velocity–stress formulation [20] and for the velocity–strain formulation [26]. Only few open-source codes exist which are able to simulate seismic wave propagation in anisotropic materials and which are also tailored to run efficiently on supercomputers. The DG ansatz on unstructured grids allows us to include full physical anisotropy as we do not encounter geometrical restrictions. The DG software SeisSol has undergone end-to-end performance optimization over the last years [3, 10, 25]. However, anisotropic effects have been neglected thus far.
In this paper, we present a novel implementation of fully anisotropic wave propagation that exploits SeisSol’s high-performance implementation of element-local matrix operations and supports local time stepping. We first lay out the physical model and state the governing equations (Sect. 2). In Sect. 3 these equations are discretized using the DG method combined with arbitrary high-order derivative time stepping (ADER-DG). Our main numerics contribution is to introduce a two-sided numerical flux of the Godunov type in conjunction with a free-surface boundary condition based on solving an inverse Riemann problem. We highlight details of how we implemented theses features into the existing code base, and extended it to make use of local time stepping. Here, the key novelty is a general approach to integrate a numerical eigenvalue solver in SeisSol that replaces analytically derived formulas in the respective precomputation steps. In Sect. 5 we verify our implementation against various analytical solutions and community benchmark problems. We also present an updated reference solution for the AHSP1 benchmark [18], since our implementation revealed physical inconsistencies in the previous community reference solution. To demonstrate the capability of our code to solve real-world geophysical problems we model anisotropically scattering seismic waves radiating from a point source under the strong topography contrasts of Mount Zugspitze in the Bavarian Alps.
2 Physical Model
3 Numerical Approximation
De la Puente et al. [20] presented the numerics of including anisotropic material effects into ADER-DG seismic wave propagation simulations. We here improve the numerical scheme by a two-sided Godunov flux and a free-surface boundary condition, as well as adaptions necessary for local time stepping with anisotropy. A two-sided flux is physically more accurate and allows for coupling between different rheologies. Local time stepping improves performance drastically.
3.1 Spatial Discretization
3.2 Flux and Boundary Conditions
In DG schemes continuity across element boundaries is only enforced in a weak sense, via the flux term \((n_d A^d_{pq} Q_q)^*\) in Eq. (6). Hence, a proper numerical flux, which also takes the underlying physics into account, is essential. De la Puente et al. [20] demonstrated anisotropy with one-sided Rusanov flux and discussed an extension to Godunov fluxes for ADER-DG. Two-sided fluxes capture the correct jump conditions of the Rankine-Hugoniot condition on both sides of the inter-element boundaries. They have been introduced to SeisSol in [23] for acoustic and (visco)elastic materials. In the following we give an overview over the most important aspects of using two-sided flux formulations and on generalizing the isotropic flux to a two-sided formulation for the anisotropic case.
In Eq. (6), the surface integral over \(\partial \tau ^m\) can be dispatched into four integrals over the four triangular faces of each element. We evaluate the flux for each face individually, so we need to transform the quantities Q as well as the stress–strain relation into face-aligned coordinate systems. For anisotropic materials the stress–strain relation is represented by the matrix \(\mathcal {H}\), see Eq. (2). We can express the constitutive behavior in the face-aligned coordinate system via the matrix \(\tilde{\mathcal {H}} = N \cdot \mathcal {H} \cdot N^T\) (cf. [4]), where N is the so-called Bond matrix (cf. [20]). We define the matrix \(\tilde{A}\) to have the same structure as the matrix \(A^1\) but with entries \(\tilde{c}_{ij}\) from the matrix \(\tilde{\mathcal {H}}\). At each face we have the Jacobians and approximations on the inside \(\tilde{A}^-, Q^-\) and on the outside \(\tilde{A}^+, Q^+\).
The Godunov flux approximates the solution at the element boundary by solving a Riemann problem across the element interfaces. First the equations are transformed to a face-aligned coordinate system. The Rankine-Hugoniot condition states that discontinuities travel with wave speeds given via the eigenvalues of the Jacobian. The differences between the quantities are the corresponding eigenvectors. A detailed derivation can be found in [23, 26].
3.3 Time Discretization
To integrate Eq. (6) in time SeisSol employs the ADER method [6]. ADER time stepping expands each element solution locally in time as a Taylor series up to a certain order. The time derivatives are replaced by the discretized spatial derivatives following the Cauchy–Kowalewski theorem. To update one element we therefore only need the values of the element itself and its four neighbors, which fosters efficient parallelization.
ADER time stepping inherently supports local time stepping [7]: Smaller elements or elements with high wave speeds will be updated more often than large elements or elements with low wave speeds. Local time stepping is thus crucial for performant applications that use strong adaptive mesh refinement or where meshes suffer from badly shaped elements. Setups with a heterogeneous material can also benefit substantially from local time stepping.
Each element has to satisfy the stability criterion \(\varDelta t^m < \frac{1}{2N+1} \frac{l^m}{v^m}\) for the time step size \(\varDelta t^m\), where \(l^m\) and \(v^m\) denote the in-sphere diameter and maximum wave speed of element \(\tau ^m\), N is the order of the method.
In anisotropic materials the wave speeds depend on the direction of propagation. This has not been considered in previous work (e.g. [7]). For a fixed direction d we define the matrix \(M(d)_{ij} = d_k c_{iklj} d_l\). We calculate the wave speeds in direction d from the eigenvalues \(\lambda _i\) of the matrix M(d) via \(v_i = \sqrt{\lambda _i / \rho }\) resulting in a primary and two secondary waves (cf. [4]). The element-wise maximum wave speed is the maximum of these speeds over all directions d.
4 Implementation
SeisSol’s ADER-DG discretization is implemented via element-local matrix chain multiplications, which allows for high performance on modern CPUs [10, 25]. All required matrices are precomputed in the initialization phase and optimized kernels are generated for the matrix chain operations [24]. In the following we present the most important choices we made for our implementation of anisotropy. Concerning the matrices, the compute kernels of the isotropic case can be reused, just the assembly of the Jacobians and flux matrices differs.
4.1 Storage Requirements
For each element we store the material values. In comparison to isotropic materials 22 instead of 3 values have to be stored. This overhead is negligible compared to the storage required for the degrees of freedoms. For example, a discretization of order 6 requires 504 degrees of freedom per element.
Concerning the storage of the precomputed matrices, only the Jacobians A and the flux matrices \(G^+\) and \(G^-\) change between the isotropic and the anisotropic case. Based on the sparsity pattern of a matrix, the biggest rectangular non-zero block of the matrix is stored. We store the matrices A, \(G^+\) and \(G^-\) as full \(9\times 9\) matrices for the isotropic as well as for the anisotropic case, thus no overhead is produced. The underlying data structures are not changed.
4.2 Calculation of the Flux Term
- (i)
Even if an analytic expression for the eigenvalues is available it is lengthy and hence the implementation is error-prone.
- (ii)
From a software engineering point of view the use of the numerical eigenvalue solver replaces a lot of code that was previously needed for each material model individually. We unified the formulation of the Riemann solver for all material models (isotropic, anisotropic and viscoelastic). We use templating to distinguish the assembly of the Jacobians for each model. From then on we can use the same code to precalculate \(G^+\) and \(G^-\). We expect that these software engineering choices make it easy to include additional material models into SeisSol in the future. Also coupling between different material models within the same simulation can be obtained with little overhead.
- (iii)
The question of accuracy and stability of the numerical solver may arise. But stability of an analytically derived formula is also not guaranteed. Round-off errors and cancellation could drastically influence the accuracy of the derived eigenvectors. With our choice for using a stable numerical solver instead, we circumvent this problem.
4.3 Maximal Wave Speeds for Local Time Stepping
Local time stepping is implemented with a clustered scheme to meet the requirements of modern supercomputers [3]. To cluster the elements the required time step for each element has to be known in advance. To obtain the maximum wave speed for one element, we would have to find the maximum wave speed over all directions. This boils down to solving an optimization problem which involves the calculation of eigenvalues of arbitrary matrices. Solving this optimization problem analytically results in lengthy calculations. In practice, the time step is relaxed by a security factor to meet the CFL condition, so the maximum wave speed does not have to be computed exactly. We sample the wave speeds for several directions d and take their maximum as the maximum wave speed \(v^m\).
5 Validation and Performance
5.1 Convergence Analysis
5.2 Isotropy via Anisotropy: Layer over Halfspace (LOH1)
The community benchmark LOH1 [18] is designed for isotropic elastic materials. We here use it to validate backwards compatibility, as isotropic elasticity is a special case of anisotropic elasticity. The setup consists of a layered half space with a free surface boundary on top. The top layer is \(1\,\mathrm{km}\) thick with material parameters \(\rho = 2600\,\frac{\mathrm{kg}}{\mathrm{m}^3}\), \(\mu = 10.4\,\mathrm{GPa} \) and \(\lambda = 20.8\,\mathrm{GPa} \). The half space below has material parameters \(\rho = 2700\,\frac{\mathrm{kg}}{\mathrm{m}^3} \), \(\mu = \lambda = 32.4\,\mathrm{GPa} \). The source is a double couple point source with the only non-zero moment \(M_{xy}\) located in \(2\,\mathrm{km}\) depth. The moment rate time history is given by the function \(M_{xy}(t) = M_0 \frac{t}{T^2} \exp \left( -\frac{t}{T^2}\right) \) with maximal moment \(M_0 = 10^{18}\,\mathrm{Nm}\) and onset time \(t = 0.1\,\mathrm{s}\).
We compare the velocities at the free surface to the given reference solutions. There are nine receivers located along three different lines through the origin. The domain is the cuboid \([-40000,40000] \times [-40000,40000] \times [0, 32500]\,\mathrm{m}^3\). On all other surfaces than the free surface (\(x_3 = 0\)) we impose absorbing boundary conditions. The mesh is refined around the source and coarsened away from it. The characteristic length is approximately \(300\,\mathrm{m}\) in the vicinity of the source and grows up to \(2000\,\mathrm{m}\) towards the boundary. In total the mesh consists of 2.57 million cells. The mesh is large enough that waves do not leave the domain during the computational time and we do not observe problems with artificial reflections. At the same time we keep the computational effort reasonable due to the coarsening towards the absorbing boundaries.
Envelope and pulse misfits for the different velocity components
Receiver | EM | PM | ||||
---|---|---|---|---|---|---|
x | y | z | x | y | z | |
3 | \(1.32\%\) | \(0.06\%\) | \(0.06\%\) | \(0.15\%\) | \(0.00\%\) | \(0.00\%\) |
6 | \(0.79\%\) | \(0.79\%\) | \(1.04\%\) | \(0.12\%\) | \(0.10\%\) | \(0.12\%\) |
9 | \(0.99\%\) | \(1.04\%\) | \(0.83\%\) | \(0.14\%\) | \(0.10\%\) | \(0.10\%\) |
5.3 Anisotropic Homogeneous Space (AHSP, SISMOWINE)
5.4 Tilted Transversally Isotropic Medium
We compare our solution obtained at the receiver with the analytic solution using envelope and pulse misfit. For the horizontal source we obtained a maximal envelope misfit of \(2.09\%\) and a pulse misfit of \(0.49\%\). For the vertical source the misfits were \(2.16\%\) and \(0.28\%\) respectively. For both source types the numerical solution fits the analytic solution very well.
5.5 Application Example: Mount Zugspitze
In Fig. 3 snapshots of the vertical velocity field on the free surface are plotted. A circular shape for the isotropic example and an elliptic shape for the anisotropic part illustrate the effects of anisotropic materials on wave propagation under strong velocity contrast. We compare this simulation with and without local time stepping: moving from global to local time stepping drastically reduced the execution time from \(5990\,\mathrm{s}\) to \(210\,\mathrm{s}\). The simulation was run on 50 nodes of SuperMUC-NG. The computational more intense version with global time stepping achieved 1.414 TFlop/s, the version with local time stepping achieved 1.015 TFlop/s. This shows that local time stepping is crucial to obtain fast simulations when the element sizes vary a lot, such as in the case of surface topography.
6 Conclusion
The earthquake simulation code SeisSol has been successfully extended to take general anisotropic materials into account. A two-sided Godunov flux for anisotropic media has been derived and implemented. Together with the formulation of the free-surface boundary condition as the solution of an inverse Riemann problem it fits well with the other rheological models. Necessary changes to include local time stepping have been described and implemented.
The scheme has been validated against various benchmarks. The expected convergence rates are demonstrated in comparison to analytic solutions. The mismatch between our results on a community benchmark have been discussed with the maintainers and led to an update of the reference solution.
As anisotropy is non-neglectable to describe the Earth’s subsurface structure we expect a wide range of applications. Besides the importance of seismic anisotropy for exploration purposes, earthquake fault zones may be characterised by pervasive anisotropy. Earthquake monitoring and forecasting can be built upon this observation.
Notes
Acknowledgments
The research leading to these results has received funding from European Union Horizon 2020 research and innovation program (ENERXICO, grant agreement No. 828947), KAUST-CRG (FRAGEN, grant no. ORS-2017-CRG6 3389.02) and the European Research Council (TEAR, ERC Starting grant no. 852992). Computing resources were provided by the Leibniz Supercomputing Centre (project no. pr45fi on SuperMUC-NG).
References
- 1.Copernicus EU-DEM. https://land.copernicus.eu/pan-european/satellite-derived-products/eu-dem/eu-dem-v1.1. Accessed 14 Mar 2018
- 2.Aki, K., Richards, P.: Quantitative Seismology, 2nd edn. University Science Books, Sausalito (2002)Google Scholar
- 3.Breuer, A., Heinecke, A., Bader, M.: Petascale local time stepping for the ADER-DG finite element method. In: IPDPS, pp. 854–863 (2016)Google Scholar
- 4.Carcione, J.M.: Wave Fields in Real Media, 3rd edn. Elsevier, Oxford (2015)Google Scholar
- 5.Carcione, J.M., Kosloff, D., Behle, A., Seriani, G.: A spectral scheme for wave propagation simulation in 3-D elastic-anisotropic media. Geophysics 57(12), 1593–1607 (1992)CrossRefGoogle Scholar
- 6.Dumbser, M., Käser, M.: An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes - II. The three-dimensional isotropic case. Geophys. J. Int. 167(1), 319–336 (2006)CrossRefGoogle Scholar
- 7.Dumbser, M., Käser, M., Toro, E.F.: An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes - V. Local time stepping and p-adaptivity. Geophys. J. Int. 171(2), 695–717 (2007)CrossRefGoogle Scholar
- 8.Dziewonski, A.M., Anderson, D.L.: Preliminary reference earth model. Phys. Earth Planet. Inter. 25, 297–356 (1981)CrossRefGoogle Scholar
- 9.Guennebaud, G., Jacob, B., et al.: Eigen v3 (2010)Google Scholar
- 10.Heinecke, A., et al.: Petascale high order dynamic rupture earthquake simulations on heterogeneous supercomputers. In: Proceedings of SC 2014, pp. 3–14 (2014)Google Scholar
- 11.Igel, H., Mora, P., Riollet, B.: Anisotropic wave propagation through finite-difference grids. Geophysics 60(4), 1203–1216 (1995)CrossRefGoogle Scholar
- 12.Jaeken, J.W., Cottenier, S.: Solving the christoffel equation: phase and group velocities. Comput. Phys. Commun. 207, 445–451 (2016)MathSciNetCrossRefGoogle Scholar
- 13.Komatitsch, D., Barnes, C., Tromp, J.: Simulation of anisotropic wave propagation based upon a spectral element method. Geophysics 65(4), 1251–1260 (2000)CrossRefGoogle Scholar
- 14.Kristeková, M., Kristek, J., Moczo, P.: Time-frequency misfit and goodness-of-fit criteria for quantitative comparison of time signals. Geophys. J. Int. 178(2), 813–825 (2009)CrossRefGoogle Scholar
- 15.Käser, M., Hermann, V., de la Puente, J.: Quantitative accuracy analysis of the discontinuous Galerkin method for seismic wave propagation. Geophys. J. Int. 173(3), 990–999 (2008)CrossRefGoogle Scholar
- 16.Leary, P.C., Li, Y.G., Aki, K.: Observation and modelling of fault-zone fracture seismic anisotropy - I. P, SV and SH travel times. Geophys. J. Int. 91(2), 461–484 (1987)CrossRefGoogle Scholar
- 17.Licciardi, A., Eken, T., Taymaz, T., Piana Agostinetti, N., Yolsal-Çevikbilen, S.: Seismic anisotropy in central north Anatolian fault zone and its implications on crustal deformation. Phys. Earth Planet. Inter. 277, 99–112 (2018)CrossRefGoogle Scholar
- 18.Moczo, P., et al.: Comparison of numerical methods for seismic wave propagation and source dynamics - the SPICE code validation. In: 3rd International Symposium on the Effects of Surface Geology on Seismic Motion, pp. 1–10 (2006)Google Scholar
- 19.de la Puente, J., Dumbser, M., Käser, M., Igel, H.: Discontinuous Galerkin methods for wave propagation in poroelastic media. Geophysics 73(5), T77–T97 (2008)CrossRefGoogle Scholar
- 20.de la Puente, J., Käser, M., Dumbser, M., Igel, H.: An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes - IV. Anisotropy. Geophys. J. Int. 169(3), 1210–1228 (2007)CrossRefGoogle Scholar
- 21.Stein, S., Wysession, M.: An Introduction to Seismology, Earthquakes, and Earth Structure, 1st edn. Wiley-Blackwell, Malden (2002)Google Scholar
- 22.Sun, Y.C., Zhang, W., Chen, X.: 3D seismic wavefield modeling in generally anisotropic media with a topographic free surface by the curvilinear grid finite-difference method 3D seismic wavefield modeling in generally anisotropic media. Bull. Seismol. Soc. Am. 108(3A), 1287–1301 (2018)CrossRefGoogle Scholar
- 23.Uphoff, C.: Flexible model extension and optimisation for earthquake simulations at extreme scale. PhD thesis, Technical University of Munich (2020)Google Scholar
- 24.Uphoff, C., Bader, M.: Yet another tensor toolbox for discontinuous Galerkin methods and other applications. Submitted to ACM TOMS (2019)Google Scholar
- 25.Uphoff, C., et al.: Extreme scale multi-physics simulations of the tsunamigenic 2004 Sumatra megathrust earthquake. In: Proceedings of SC 2017, pp. 21:1–21:16 (2017)Google Scholar
- 26.Zhan, Q., et al.: Full-anisotropic poroelastic wave modeling: a discontinuous Galerkin algorithm with a generalized wave impedance. Comput. Methods Appl. Mech. Eng. 346, 288–311 (2019)MathSciNetCrossRefGoogle Scholar