Current and Emerging Time-Integration Strategies in Global Numerical Weather and Climate Prediction
- 2.4k Downloads
- 2 Citations
Abstract
The continuous partial differential equations governing a given physical phenomenon, such as the Navier–Stokes equations describing the fluid motion, must be numerically discretized in space and time in order to obtain a solution otherwise not readily available in closed (i.e., analytic) form. While the overall numerical discretization plays an essential role in the algorithmic efficiency and physically-faithful representation of the solution, the time-integration strategy commonly is one of the main drivers in terms of cost-to-solution (e.g., time- or energy-to-solution), accuracy and numerical stability, thus constituting one of the key building blocks of the computational model. This is especially true in time-critical applications, including numerical weather prediction (NWP), climate simulations and engineering. This review provides a comprehensive overview of the existing and emerging time-integration (also referred to as time-stepping) practices used in the operational global NWP and climate industry, where global refers to weather and climate simulations performed on the entire globe. While there are many flavors of time-integration strategies, in this review we focus on the most widely adopted in NWP and climate centers and we emphasize the reasons why such numerical solutions were adopted. This allows us to make some considerations on future trends in the field such as the need to balance accuracy in time with substantially enhanced time-to-solution and associated implications on energy consumption and running costs. In addition, the potential for the co-design of time-stepping algorithms and underlying high performance computing hardware, a keystone to accelerate the computational performance of future NWP and climate services, is also discussed in the context of the demanding operational requirements of the weather and climate industry.
1 Introduction
Over the past few decades, operational NWP and climate models have evolved tremendously thanks to the continuous improvement of computing technologies and of the underlying algorithms at the foundations of these models. Today, these algorithms are facing a major challenge, as NWP and climate models are transitioning to more sophisticated Earth System Models (ESM), that will incorporate more components, and towards more accurate representation of the flow physics. In addition, the hardware used to perform the simulations is also undergoing a dramatic change, with many-core architectures and co-processors—e.g., graphical processing units (GPU) and Intel’s Many Integrated Core processor architecture (MIC)—becoming the prevailing technologies. Therefore, it is necessary to review the core algorithmic strategies—in particular, the numerical discretizations and the time-integration strategies—currently adopted in the industry, to understand their potential in the future landscape.
To begin the discussion, we first need to characterize in more detail the models adopted in weather and climate applications. A numerical weather or climate model is constituted by a set of prognostic partial differential equations (PDEs) governing the fluid motion in the atmosphere (i.e., the geophysical flow) and by all those physical processes acting at a subgrid scale, whose statistical effects on the mean flow are expressed as a function of resolved-scale quantities [64]. The former is known as the dynamical core and represents the scale-resolved part of the model, whereas the latter are referred to as the physical parameterizations and include the under-resolved processes. The dynamical core is usually described by the laws of thermodynamics and Newton’s law of motion for the fluid air (e.g., compressible Navier–Stokes or Euler equations). Typical examples of physical parameterizations are convective processes, cloud microphysics, solar radiation, boundary layer turbulence and drag processes, which are handled by vertically-columnar submodels. The overall model is then discretized in space and time via suitable algorithmic blocks to approximate the continuous system of equations, thereby providing a solution otherwise not readily achievable.
In the short description above, we did not distinguish between a weather and a climate model, but we treated them as though they were identical. This deserves an explanation. While there are certainly overlaps between them, generally the time-scale and scope of a weather model differ substantially from those of a climate model. For example, the typical forecast window of weather models spans a range up to several days ahead, whilst for climate models, the forecast range is several months to years ahead. However, the set of PDEs defining the dynamical core are similar, if not identical; in fact, weather models are also referred to as the “higher-resolution siblings of the climate models’ atmospheric component” [85], since they typically have higher spatial and temporal resolution than climate models. Also, they include a smaller number of physical processes, although both weather and climate models are evolving towards Earth System Models ESMs, that will include dynamic oceans, cryospheres and biochemical cycles [85]. Yet, despite these differences, in both weather and climate the “numerical engine” used to solve the PDEs constituting the model must be effective under several evaluation metrics in order to provide a ‘fast’ and ‘quantitatively satisfactory’ forecast. Hereafter, we will make no distinctions between NWP and climate models, as the scope of this review targets a shared aspect of both, despite the usually different operational and scientific objectives of weather and climate applications. In addition, we will refer to NWP operational constraints omitting those of climate simulations, as they are commonly the most severe.
- (1)
Solution accuracy,
- (2)
Effectiveness of uncertainty quantification,
- (3)
Time-to-solution,
- (4)
Energy- or money-to solution, and
- (5)
Robustness (e.g., numerical stability).
Today there are a number of highly successful strategies that have emerged as the method of choice for the temporal (and spatial) discretization of the set of PDEs underlying NWP and climate models. Yet, the evolution of high-performance computing architectures requires a careful review of these strategies. From this perspective, the development of novel mathematical algorithms and their combination with existing successful methods, as well as hardware–software co-design are becoming an essential activity undertaken by many practitioners in the weather industry. In this work, we provide a broad overview on the different numerical time-integration strategies used to solve the PDEs arising in global NWP and climate models, emphasizing the most prominent techniques adopted in the weather community operationally in the past few decades, and describing the emerging solutions that operational weather centers (and the European Centre for Medium Range Weather Forecasts (ECMWF) in particular) are considering for the future. This review also aims to clearly categorize the currently adopted and emerging time-integration strategies under a more structured nomenclature.
The rest of the review is organized as follows. In Sect. 2, we introduce the set of equations used in weather and climate models. In Sect. 3, we categorize the most prominent time-integration approaches used in the industry. In Sect. 4, we highlight the time-stepping strategies adopted by the main operational and research models. In Sect. 5, we introduce three time-integration schemes that are being considered as potentially competitive for the future. Finally, in Sect. 6, we discuss the possible evolution that the weather and climate industry might undergo in the near to long-term future.
2 Equations Modeling the Atmosphere
Operational global NWP and climate models employ the compressible Euler equations to describe the fluid motion in the atmosphere, where the missing viscous stresses, and the sensible and latent heat fluxes as a result of diabatic processes are modeled as part of the physical parameterizations, and represented on the right-hand side of the equations as forcing terms. These equations are usually written in spherical coordinates—or coordinate mappings to a tangential plane for limited-area studies—and include the effects of gravity and the Earth rotation (i.e., the Coriolis force).
The formulation of the Euler equations can be either conservative (also referred to as flux-type through including the action of the continuity equation) or non-conservative (also referred to as advective-type). The form chosen has implications on the formal accuracy of global integrals such as mass, momentum and energy, important for climate projections, as well as local conservation properties important for very high-resolution simulations of clouds and convection. The chosen (type-) formulation of the equations influences the numerical schemes that can be most efficiently employed.
3 Time-Integration Strategies in Global NWP and Climate Modeling
- A.
Eulerian-based time-integration (EBTI), where spatial and temporal discretizations are viewed as independent from each other; and
- B.
Lagrangian or path-based time-integration (PBTI), where space and time are solved together, or where temporal derivatives may be expressed as spatial derivatives.
3.1 Eulerian-Based Time-Integration (EBTI)
- i.
multistage Runge–Kutta methods, that use multiple stages between two consecutive time-levels (or time-steps), discarding information from earlier time-steps; and
- ii.
linear multistep methods, that use information from mulitple earlier time-steps.
- 1.
solutions to the continuous model (7) comprise fast and slow modes, i.e. \({\mathcal {R}}_{f}\) (for the fast modes) and \({\mathcal {R}}_{g}\) (for the slow modes) can describe processes that differ by orders of magnitude with respect to the time-scale of their propagation; and in addition,
- 2.
in the discretized model, as already highlighted, the grid-spacings used to resolve the horizontal and vertical directions are highly anisotropic (\(z_h \ll s_h\)), reflecting the different scales that characterize the important processes in each direction. Since the terms on the right-hand side of (7) include spatial gradients, then, if \({\mathcal {R}}_{f}\) represents contributions from the vertical direction and \({\mathcal {R}}_{g}\) from the horizontal, the mesh-anisotropy leads to a separation of scales between the two terms.
In the rest of this subsection, two EBTI approaches will be discussed—both are formally horizontally-explicit, vertically-implicit approaches, but the “split-explicit” approach, discussed in Sect. 3.1.1, adds an additional level of complexity in its use of sub-steps to handle the integration of fast processes. The “HEVI” approaches discussed in Sect. 3.1.2 highlight more recent developments, motivated by high-resolution global atmospheric models, which do not use sub-stepping.
3.1.1 Split-Explicit Schemes
The SE approach gains efficiency by computing the contributions from \({\mathcal {R}}_{g}\) only on the longer time-step, \(\varDelta t\). The terms in \({\mathcal {R}}_{g}\) include advection and mixing terms that require a relatively large stencil of data and are therefore more computationally expensive. Meanwhile, the contributions from \({\mathcal {R}}_{f}\) (due to the fast waves) are computed every sub-step, \(\varDelta \tau \), but involve only immediate neighbour data-points to calculate local gradients. The latter aspect is particularly attractive for emerging computing technologies, given the reduced communication-to-flop ratio required, thus favoring co-processors, accelerators and many-core architectures.
Based on a stability analysis of the KW78 SE approach, [87] argued that greater efficiency could be gained by handling the buoyancy terms with the implicit scheme on the sub-step alongside the vertically-propagating acoustic waves. Under this approach, the longer time-step \(\varDelta t\) is limited by the maximum speed of advection. Meanwhile, the sub-step \(\varDelta \tau \) continues to be limited by the horizontally-propagating acoustic waves. In addition, it was found that the SE approach needs some damping in its formulation to ensure an acceptable stability region. Skamarock and Klemp [87] proposed a “divergence damping” term to filter the acoustic modes in their analysis of the leapfrog-based KW78 approach. More recently, [35] has demonstrated the importance of an isotropic application of the divergence damping to the acoustic waves. Alternatively, they proposed using simple off-centering for the sub-step implicit solver. Baldauf [6] includes a comprehensive stability analysis of RK-based SE approaches, where the free parameters associated with the various components of the method are optimised, in terms of accuracy and stability. In particular, optimal values are proposed for the off-centering of the implicit solution of the acoustic and buoyancy terms, and the magnitude of the divergence damping applied to the acoustic waves.
The precise integration schemes used in a SE approach are open to choice: simple, efficient methods for the fast components; and a method with good accuracy and an acceptable window of stability (in terms of time-step length) for the slow components. A SE approach has been adopted in a number of active research and operational limited-area atmospheric models: the JMA’s non-hydrostatic Mesoscale Model (JMA-NHM) continues to use a leapfrog-based approach for the long time-step integrations, but includes some low-order advection components in the short time-step computations to improve the computational stability [79, 80]. Other groups have moved towards single-time-level, multistage explicit Runge–Kutta schemes to integrate the slow components—both the COSMO [6, 7] and WRF [56, 88] models use a 3-stage 3rd-order Runge–Kutta (RK) scheme for integrating the slow components, retaining the forward–backward and trapezoidal schemes (previously described) for the fast components (following the analyses of RK methods for time-splitting in [108, 109]). Figure 3b illustrates the 3-stage RK-based approach.
With the recent developments of global models with “very” high resolution (grid-spacings \(\le \mathcal {O}\left( 10\right)\, \mathrm {km}\)), SE methods are now being used also for global NWP: the MPAS model [89] has adopted the SE approach (as described in [56]) based directly on the successful experiences with the 3-stage 3rd-order RK-based SE approach in WRF. The high resolution global non-hydrostatic model, NICAM, also uses a RK-based SE approach (with options for 2nd- or 3rd-order RK schemes) [82, 83].
3.1.2 Horizontally-Explicit Vertically-Implicit (HEVI) Schemes
Similarly to SE approaches, HEVI schemes are becoming more and more attractive due to the latest advancements in computing that are driving the development of “very” high-resolution global NWP models. For global NWP, the stratosphere plays a significant role in the global circulation [48, 50, 71]. Inclusion of a well-represented stratosphere has implications for the chosen time-integration methods, since the stratospheric polar jet (which contributes via the advection term) reaches speeds exceeding \(100\;\mathrm {m\;s}^{-1}\), i.e., the advective Courant number approaches the acoustic one. As highlighted in [34], in this context the efficiency gains from the SE approach become less clear: the horizontal splitting (which defines the sub-stepping) in SE schemes is only relevant when there is a scale-separation between fast insignificant and slow significant processes. With the acoustic and advective Courant numbers being similar, the sub-step \(\varDelta \tau \) and the model time-step \(\varDelta t\) are constrained by similar stability limits and little efficiency can be gained from sub-stepping. In addition, as already noted, SE models require artificial damping to ensure stabilization, with atmospheric models typically employing divergence damping (see, e.g., [6]). HEVI-based alternatives can be efficiently used for global non-hydrostatic equation sets and do not have the drawbacks affecting SE schemes.
Similar to the SE approach, the aim is to optimize the computational cost by selecting a relatively cheap (due to its local nature) but appropriately accurate (say, 3rd order) conditionally stable explicit scheme, which places a limit on the time-step \(\varDelta t\); and a less accurate (say, 2nd order) unconditionally stable implicit scheme to handle the large Courant number vertical processes. The loss of accuracy in computing the vertical processes is offset by their lesser physical importance. As for the SE case, the implicit problem is also cheap (e.g., a tridiagonal system), since vertical grid columns remain complete on each compute node.
Expressing the RK-HEVI approach as a double Butcher tableau makes it efficient to explore many alternative combinations of schemes—both through linear analyses [23, 40, 63] and numerical simulations [23, 40, 106]. The analyses focus on the accuracy and stability implied for components of simple linear systems (acoustic and gravity waves, and advection); and the performance of numerical implementations for idealised (dry) atmospheric tests, often compared to solutions from a very high-resolution high-order explicit RK method. Substantially different approaches have been recommended: some completely splitting the vertical and horizontal integrations using Strang-type splitting [8, 97, 100]; others proposing schemes that keep the vertical and horizontal solutions balanced in time, by integrating over the same time-interval, at each predictor-stage [40, 63, 106]. In keeping with the semi-implicit approach (Sect. 3.2), [106] stresses the importance of ending the integration with a stage that includes an implicit integration, thereby ensuring a balanced final solution.
Colavolpe et al. [23] proposes a further extension to the double Butcher tableau approach—a quadruple Butcher tableau, whereby the horizontal pressure-gradient and divergence terms are treated separately to the horizontal advection. Under their scheme, all four solutions are balanced in time at each predictor-stage, but in addition, a forward–backward type operation is introduced for the pressure-gradient and divergence terms (based on [68]) that alternates the forward/backward operations for the windspeed/pressure solutions at two predictor stages. They demonstrate that the additional splitting brings greater stability and accuracy at no extra computational cost.
Only one operational global NWP model but a number of global non-hydrostatic research models (cf. Table 1) have adopted HEVI time-integration methods. This trend seems to indicate that HEVI schemes are being considered as a valuable alternative to more commonly adopted time-integration strategies (such as the semi-implicit semi-Lagrangian method), especially for high-resolution models.
3.2 Path-based time-integration (PBTI)
The trajectory integral of \({\mathcal {R}}\) can be approximated using a weighted average of the integrand values from the physical space at a past time \(t_{\text {D}}\), the “departure time” where the state of the system is known, and the values from the physical space at a future time \(t_{\text {A}}\), the “arrival time”, where the solution is sought. In the PBTI approach when the approximation solely relies on the past values, the integration scheme is explicit, while when the approximation depends on the future (unknown) values of the integrand \({\mathcal {R}}\), the integration scheme is implicit. In the case of explicit integration schemes, the solution of the system is fairly straightforward but the approximation can become numerically unstable, for time-steps exceeding the Eulerian CFL condition of the fast processes, leading to failure of the simulation. On the other hand, in the case of implicit integration schemes, such as the commonly used trapezoidal rule (semi-implicit Crank–Nicholson), the approximation is guaranteed to be unconditionally stable but the resulting system of equations, usually in the form of a BVP, becomes more complicated and its numerical solution more difficult.
PBTI techniques have been very successful in NWP (as indicated in Table 1 with adoption of the technique as late as 2014). The most common PBTI strategy employed in NWP is the Semi-Lagrangian (SL) scheme that revolutionized the field two decades ago [94, 98]. Pure Lagrangian approaches, where the exact solution at the next time step is sought by translating the flow information on the mesh at the current time level along the trajectory integrals, with remapping only for postprocessing purposes, have never been adopted into operational NWP models. This has been mainly due to the initial mesh being significantly deformed in a few time-steps which results in large spatial truncation errors. Having a mesh with vertically aligned grid-points is essential for resolving complex diabatic processes and vertically-propagating gravity waves in a NWP model. In contrast to the pure Lagrangian approach, there is no mesh deformation in a semi-Lagrangian scheme. The reason is that backward trajectories are calculated at each time-step; these end at the model grid-points while they start from locations between mesh grid-points that must be determined.
More recently, forward-in-time finite volume (FTFV) integrators, that can be written in a congruent manner as the SL scheme, have also emerged [91] with applications in NWP and climate. Furthermore, vertical Lagrangian coordinates have been successfully applied in hydrostatic models [46].
3.2.1 The semi-implicit semi-Lagrangian scheme (SISL)
The SISL approach is currently the most popular option for operational global NWP models while it is also often used in limited area modeling. As shown in Table 1 the vast majority of the listed global NWP centers are using a model with a SISL dynamical core. A typical example is the ECMWF forecast model IFS, which has used a SISL approach since 1991. As discussed in [86], the change from Eulerian to semi-Lagrangian numerics improved the efficiency of IFS by a factor of six thus enabling a significant resolution upgrade at that time. Since 1991, further successful upgrades followed and currently the (high resolution) global forecast model is run at 9 km resolution in grid-point space, to this date the highest in the world.
- (i)
- SL advection and calculation of the departure points All operational SL codes work “backwards” in the sense that at a given discrete point in time t and with a model time-step of \(\varDelta t\) an air-pracel will start from a point in space between grid-points and will terminate at a given mesh grid-point. The latter are called “arrival points” and coincide with the model mesh grid points while the former are called “departure points” and they must be found as they are not known a priori. There is a unique departure point associated with each grid-point to be computed (this assumes that characteristics do not intersect, i.e., no discontinuities are permitted). Therefore, for a simple passive scalar advection of a generic field y without forcing, the solution at a new time step is:This means that to compute the field y values at the new time-step \(t+\varDelta t\), it suffices to compute a departure point “D” for each model grid point and then interpolate the transported field y at these departure points. The interpolation method uses the known y-values at time t, at a set of grid points nearest to “D”; the number and location of these grid points depend on the order of interpolation method used. For the more general problem (13), the forcing terms should also be interpolated at the departure point. To compute the location of the departure points the following trajectory equation must be solved:$$ y^{t+\varDelta t}_{\text {A}} = y_{\text {D}}^{t}.$$(16)where \(\mathbf {r}\) denotes the coordinates of a moving fluid parcel, for example \(\mathbf {r}=(x,y,z)\) if a Cartesian system is used. By integrating Eq. (17), we obtain:$$ \frac{\text {D}\mathbf {r}}{\text {D}t}={\mathbf {u}}(\mathbf {r},t), $$(17)The right-hand side integral of (18) is usually approximated using a 2nd order scheme such as the midpoint rule, resulting in an implicit equation of the form$$ \mathbf {r}_{\text {A}} - \mathbf {r}_{\text {D}} = \int _{t}^{t+\varDelta t}{\mathbf {u}}(\mathbf {r},t)\text {d}t.$$(18)which is solved iteratively (for details see [27]). The accuracy with which the departure points are computed influences greatly the overall accuracy of the model as shown in [27].$$ \mathbf {r}-\mathbf {r}_{\text {D}} = \varDelta t \,{\mathbf {u}}\left( \frac{\mathbf {r}+\mathbf {r}_{\text {D}}}{2},t+\frac{\varDelta t}{2}\right),$$(19)
In addition, the method employed to interpolate the terms of Eq. (13) to the departure points has also important implications in the model accuracy. From this perspective, it is common practice in operational SISL models to use a cubic interpolation formula most often based on tri-cubic Lagrange interpolation followed by formulae based on cubic Hermite or cubic spline polynomials. The interpolation is directional, i.e., it is performed separately in each of the three spatial coordinates. There is an intriguing interplay between the spatial and time truncation error in the SL advection method. Following the convergence analysis in [31], verified experimentally in [112] using the Navier–Stokes system, the leading order truncation error term for a SL method solving a 1D constant wind advection equation with an interpolation formula of order p on a grid with constant spacing \(\varDelta x\) and a time-integration method for the departure point of order k with time-step \(\varDelta t\) is \(\mathcal {O}(\varDelta t^k + \varDelta x^{p+1}/\varDelta t)\). This suggests that reducing the time-step only without refining the mesh resolution may not improve the overall solution accuracy as it increases the contribution from the error term which has \(\varDelta t\) in the denominator. However, with a shorter time-step the accuracy of the departure point calculation improves and a higher order interpolation scheme improves the accuracy of spatial structures such as waves [29].
- (ii)
- Semi-implicit time-discretization of forcing terms Eq. (15) is expensive and complex to solve due to its large dimension, its implicitness and in general its nonlinear form (right hand-side \({\mathcal {R}}\) includes nonlinear terms). For this reason, an approach commonly used in NWP is to extract fast terms from the right-hand side and linearise them around a constant reference profile. For example, in the IFS model the right-hand forcing term is split as follows:where \(\mathcal {L}\) contains the linear and linearised fast terms which are integrated implicitly and \(\mathcal {N}\) the remaining nonlinear terms \(\mathcal {N}={\mathcal {R}} - \mathcal {L}\) which are integrated explicitly. A two-time-level second order SISL discretization of (15) can be written as follows:$$ {\mathcal {R}}=\mathcal {N}+\mathcal {L} $$The slowly varying nonlinear terms at \(t+\varDelta t/2\) can be “safely” approximated by a second order extrapolation formula such as$$\begin{aligned} \frac{{\mathbf {y}}^{t+\varDelta t}_{\text {A}}-{\mathbf {y}}^t_{\text {D}}}{\varDelta t}=\frac{1}{2}\left( \mathcal {L}^t_{\text {D}} + \mathcal {L}^{t+\varDelta t}_{\text {A}}\right) + \frac{1}{2} \left( \mathcal {N}^{t+\varDelta t/2}_{\text {D}} + \mathcal {N}^{t+\varDelta t/2}_{\text {A}} \right) . \end{aligned}$$(20)or alternatives such as SETTLS [49], which are less prone to generate numerical noise. The latter aspect—i.e., the numerical noise issue—is particularly relevant in the stratosphere where large vertically stable areas occur and any small scale oscillations appearing due to the three time-level form of the extrapolation formula used may be amplified. Using “iterative semi-implicit” schemes [11, 26, 111], in which a future model state is predicted with 2-iterations (or more) with the first serving as a predictor and the second as a corrector, is the most effective method for solving the noise issue. However, it is more costly due to its iterative nature.$$ \mathcal {N}^{t+\varDelta t/2} =\frac{3}{2} \mathcal {N}^t - \frac{1}{2}\mathcal {N}^{t-\varDelta t}$$There is considerable variation in the implementation of the SI time-stepping by different models. An alternative, iterative, approach to the SI method (20) is followed by the UK Met Office (UKMO) Unified Model, where there is no separate treatment between linear and nonlinear terms. Here, the standard off-centerd semi-implicit discretization is used:The weight \(\alpha \) is either 0.5 or slightly larger to avoid non-physical numerical oscillations (noise) which may arise due to spurious orographic resonance [76]. To tackle the implicitness of Eq. (21) an iterative method with an outer and inner loop is used. This functions as a predictor-corrector two-time-level scheme. As stated in [66], in the outer iteration loop, the departure-point locations are updated using the latest available estimates of the winds at the next time step. In the inner loop the nonlinear terms, together with the Coriolis terms, are evaluated using estimates of the prognostic variables obtained at the previous iteration. Further details on the iterative approach followed by the UM can be found in [66, 111]. Iterative SI schemes are expensive algorithms, however, they are used by most non-hydrostatic SISL global models as in practice the cheaper non-iterative schemes based on time-extrapolation become unstable when long time steps are used.$$\begin{aligned} \frac{{\mathbf {y}}^{t+\varDelta t}_{\text {A}}-{\mathbf {y}}^t_{\text {D}}}{\varDelta t}=(1-\alpha ){\mathcal {R}}^t_{\text {D}} +\alpha {\mathcal {R}}^{t+\varDelta t}_{\text {A}}. \end{aligned}$$(21)
- (iii)
- Helmholtz solver Once the right-hand side of (20) has been evaluated the semi-implicit system can be solved. To avoid solving simultaneously all implicit equations in (20), it is common practice to derive a Helmholtz equation from these. The form of the Helmholtz equation depends on the type of space discretization. In spectral transform methods, such as the one used in IFS [104], the specific form of the semi-implicit system is derived from subtracting a system of equations linearised around a horizontally homogeneous reference state. The solution of this system is greatly accelerated by the separation of the horizontal and the vertical part, which matches the large anisotropy of horizontal to vertical grid dimensions prevalent in atmospheric models. In spectral transform methods, one uses the special property of the horizontal Laplacian operator in spectral space on the spherewhere \(\psi \) symbolises a prognostic variable, a is the Earth radius, and (n, m) are the total and zonal wavenumbers of the spectral discretization [104]. This conveniently transforms the 3D Helmholtz problem into a 2D matrix operator inversion with dimension of the vertical levels only, resulting in a very cheap direct solve [75]. Even in the non-hydrostatic context, formulated in mass-based vertical coordinates [60], only the solution of essentially two coupled Helmholtz problems allow the reduction of the system in a similar way [12, 14, 115]. This technique requires a transformation from grid-point space to spectral space and vice versa at each time-step, an aspect that increases the associated computational cost although the spectral space computations are based on FFTs and matrix–matrix multiplications that are well suited for modern computing architectures. One disadvantage of this technique is the need for a somewhat simple reference state that does not allow, by definition, the inclusion of horizontal variability (as would be desirable for terms involving orography). The relaxation of this constraint and some alternatives are discussed, for example, in [17].$$ \nabla ^2\psi _n^m = {n(n+1) \over a^2} \psi _n^m,$$(22)
For grid point models using finite differences, such as the UKMO Unified Model, a variable coefficient 3D Helmholtz problem is solved using an iterative Krylov subspace linear solver (e.g., BiGCstab, GCR(k), GMRES etc.) [111]. This type of solver is generally more expensive despite grid point models not requiring transformations from spectral to grid point space and vice versa, which offsets some of the extra cost. However, typically up to 80 percent of computations are spent in the solver in grid-point based semi-implicit methods, compared to 10–40% in spectral transforms (depending on the resolution and the number of MPI communications involved) on today’s high performance computing (HPC) architectures. For emerging and future architectures, that may heavily penalize global communication patterns moving to high throughput capabilities, this is a serious concern that needs to be properly investigated and addressed.
3.3 Summary
In this section, we categorized time-integration schemes into two classes Eulerian-based, EBTI, and path-based, PBTI. The former discretizes the original PDE problem in space first, thus obtaining an ODE, and subsequently in time through a suitable time-integration strategy. The PBTI class, instead, solves the PDE problem in a single step, where the advection term is adsorbed into the path (or material) derivative and the right-hand side is formed by forcing terms only. In this case, the system of PDEs can be seen as a physical constraint on the path that can be followed to link two states in the four-dimensional continuum constituted by space and time.
For each of the two categories, EBTI and PBTI, we outlined the most prominent time-integration schemes adopted in the NWP and climate communities. In particular, SE and HEVI for the EBTI class, and the SISL approach for the PBTI class. The latter was the most widely adopted in the past few decades thanks to the hydrostatic approximation ubiquitously used in global weather and climate models. Indeed, PBTI strategies and SISL in particular, were extremely competitive given the large time-steps they allow and their extremely favorable dispersion properties, which yield the correct representation of wave-like solutions—e.g., Rossby and gravity waves. EBTI strategies are instead now emerging as a potential alternative to PBTI in non-hydrostatic models. This is mainly because they can be constructed to ‘filter’ the fast and atmospherically irrelevant acoustic modes that propagate vertically. In fact, SE and HEVI schemes that had been mainly used in the context of limited-area models are now being taken into consideration in global weather models, as they seem to represent an attractive compromise between solution accuracy, time- and energy-to-solution and reliability. In addition, they can address the non-conservation issues typical of PBTI-based approaches, a feature that is particularly relevant for climate simulations. Some additional strategies, beyond HEVI, SE and SISL methods are also under investigation within global NWP and climate simulations, namely IMEX schemes, fully implicit methods and conservative semi-Lagrangian schemes—the latter addressing the conservation issues of traditional SISL schemes. These additional strategies will be briefly discussed in Sect. 5.
Note that while the time-integration approaches described in this section might differ in their implementation details from one model to another, the general concepts and properties, which motivate their use, hold true across all the models adopting each strategy. In the next Sect. 4, we will highlight the time-stepping strategies employed by the main operational global NWP and climate centres and emphasize in more detail why these were selected. In addition, we will outline the implications these choices may have in the context of the changing hardware and weather modeling landscape. Finally, we will introduce some of the (several) projects undertaken within the weather and climate industry to address the computational challenges of the incoming decades.
4 Overview of Operational Time-Integration Strategies Adopted by the NWP and Climate Centers
Time-stepping strategies adopted by the main operational global NWP and climate models as of 2017
Institute | Model | Type | Equation | Time integration | Class | Date | Country |
---|---|---|---|---|---|---|---|
ECMWF | IFS | NWP | H | SISL | PBTI | 1991 | Europe |
MF | ARPEGE | NWP | H | SISL | PBTI | 1991 | France |
UKMO | UM | NWP | NH | SISL | PBTI | 2002 | U.K. |
JMA | GSM | NWP | H | SISL | PBTI | 2005 | Japan |
CMA | GRAPES | NWP | H | SISL | PBTI | 2010 | China |
NRL | NAVGEM | NWP | H | SISL | PBTI | 2013 | U.S. |
EC | GEM | NWP | NH | SISL | PBTI | 2014 | Canada |
NCEP | GFS | NWP | H | SISL | PBTI | 2014 | U.S. |
DWD/MPI | ICON | NWP&Climate | NH | HEVI | EBTI | 2015 | Germany |
MPI | ECHAM | Climate | H | SISL | PBTI | 2003 | Germany |
HC | HADGEM | Climate | NH | SISL | PBTI | 2009 | U.K. |
NOAA/GFDL | FV3 | RD | H/NH | SISL | PBTI | 2004 | U.S. |
CCSR/JAMSTEC | NICAM | RD | NH | SE-HEVI | EBTI | 2008 | Japan |
NCAR | CAM-SEM | RD | H | SE-explicit | EBTI | 2012 | U.S. |
NCAR | MPAS | RD | NH | SE-HEVI | EBTI | 2013 | U.S. |
NCAR | CAM6 | RD | H | HESL | PBTI | 2015 | U.S. |
NPS/NRL | NUMA/ NEPTUNE | RD | NH | IMEX/HEVI | EBTI | 2016 | U.S. |
From this point of view, Table 1 clearly shows that nearly all operational NWP and climate models use PBTI (SISL) approaches and the majority use the hydrostatic approximation—see top part of the Table, where, apart from the German model ICON that uses a non-hydrostatic model and a HEVI scheme, all the others use PBTI schemes, and SISL in particular. These choices are dictated by a desire to maximize time-to-solution performance, that has been (and still is) one of the main objectives in the industry. It is also worth noting that, despite the continuous upgrade of the operational models—see the column “Date” in the Table—all of them keep adopting SISL, even if they transitioned to a non-hydrostatic approximation. This might be an indicator of the inevitable algorithmic inertia within the weather and climate industry due to the strict operational constraints and the relatively complex and large code frameworks developed over several decades.
On the other hand, if we look at research models (bottom part of Table 1, denoted with RD), they are predominantly non-hydrostatic and use EBTI approaches with substantially smaller permitted time-steps. The use of EBTI schemes is justified by their better parallel efficiency (than SISL) that may compensate for the shorter forward-in-time stepping of the dynamical core. Indeed, there seems to be a trend in the weather and climate industry to favour EBTI approaches over PBTI for next-generation non-hydrostatic models, as briefly mentioned in Sect. 3. The associated spatial discretization for the spatial derivatives arising on the right-hand side of each time-stepping strategy can assume various forms that include global spherical harmonics bases, finite-differences, finite-volumes, and spectral element methods (e.g., continuous and discontinuous Galerkin).
Given what we discussed above, it should be clear that the choice of the time-stepping strategy is strongly influenced by the choice of equations, as most non-hydrostatic equations require numerical control of vertical acoustic wave propagation, whereas hydrostatic models filter these modes a priori. The stiffness of the problem arising from the fast propagating acoustic waves needs to be handled carefully. Options include SE methods, HEVI schemes, and, more recently, IMEX methods (discussed in Sect. 5). In essence, many of the approaches originally used in limited-area modeling have been applied in non-hydrostatic global models. It should be also noted that these schemes are favourable in terms of parallel communication patterns, as they require a reduced amount of data to be transferred at each time-step, being mostly nearest-neighbour communication algorithms, and have a higher throughput (flops/communication) than PBTI approaches. This attractive feature is particularly suited to emerging computing architectures that are commonly communication bounded. However, the larger number of time-steps (due to shorter permitted time-steps), can counter balance this advantage in favour of larger-time-step algorithms, including SISL.
Overall, the suitability to emerging hardware architectures of a given time-stepping strategy is of crucial importance for planning a sustainable path to future high-resolution weather and climate prediction systems, as there will be the pressing need to mitigate the exponentially increasing running costs of operational centres. In fact, the use of emerging many-core computing architectures (e.g., GPUs and MICs) help reducing the power consumption of the overall HPC centre where the simulations are run (the power required grows exponentially with the clock-speed of the processor; therefore the use of many-core computing technologies that have a lower clock rate mitigate the growth in energy consumption—see e.g., [9]). With the increased model resolutions envisioned in the next few decades, another critical aspect is the efficient treatment of a massive amount of gridded data (the so-called data tsunami problem). In this case, the adopted time-stepping strategy has important implications, since a larger number of time-steps could imply a larger amount of gridded data produced. This obviously has direct consequences on the costs to maintain and power the servers where the data are stored, and might affect the efficient post-processing and dissemination (to the clients or member states) of the results.
Indeed, these issues are being closely evaluated by several weather and climate agencies through various projects, including the European Exascale Software Initiative (EESI), Energy efficient SCalable Algorithms for weather Prediction at Exascale (ESCAPE) and Excellence in Simulation of Weather And Climate in Europe (ESiWACE). All these initiatives have the general recommendation for research in the areas of development of efficient numerical methods and solvers capable of complying with exabyte data sets and exascale computational efficiency—i.e. next-generation HPC systems.
The efforts being spent to address the computational efficiency issues of weather and climate algorithms run in parallel to the development of the high-resolution medium-range global and nested NWP models, as many government agencies are extensively researching new candidates for their operational services. In fact, these new models must be able to make a quantum leap forward in forecast skill, by being able to efficiently utilize the latest developments in data assimilation (e.g., 4D-En-Var [45]), scale-aware physical parameterizations, and, as just mentioned, modern HPC technologies (e.g., GPU/MIC). From this perspective, several international and interagency activities have been organized to select best candidates for next-generation models. For example, the NOAA High Impact Weather Prediction Project (HIWPP), seeks to improve hydrostatic-scale global modeling systems and demonstrate their skill at the coarsest resolution down to \(\sim \) 10 km. They are also trying to accelerate the development and evaluation of higher-resolution non-hydrostatic, global modeling systems at cloud-resolving (\(\sim \) 3 km) scales. As a part of Research to Operations (R2O) Initiative, the NOAA / National Weather Service (NWS) led the inter-agency effort to develop a unified Next Generation Global Prediction System (NGGPS) for 0–100 days predictions to be used for the next 10–20 years. The new prediction model system designed to upgrade the current operational system, GFS, has to be adaptable to and scalable on evolving HPC architectures. The research and development efforts included the U.S. Navy, NOAA, NCAR and university partners. A similar effort has been undertaken in Japan with the Japan Meteorological Agency (JMA), which has been exploring its own next generation non-hydrostatic global numerical weather prediction model since 2009. The UK MetOffice is also pursuing new modeling efforts through its development of a scalable dynamical core “GungHo” (Globally Uniform, Next Generation, Highly Optimised) [95]. The dynamical cores are also a subject of many intercomparison projects, such as the Dynamical Core Model Intercomparison Project (DCMIP) [101], which in a few editions has hosted more than 20 different models. In addition to the current operational models, several research models may offer good candidates for next generation operational systems replacing the current state-of-the-art solutions.
The overarching objective is to increase the computational efficiency of the models, thereby reducing the operating costs (e.g., energy-to-solution) and the time-to-solution performance, in order to increase the accuracy and resolution of the models at a sustainable economic cost. This aspect is intimately related to the overall co-design of the algorithms underlying weather and climate models where current and emerging hardware and the adopted time-integration method will play a major role. Specifically, the amount of data communicated during each simulation, alongside the length of permitted time-step and the percentage of peak performance achieved on a given computing machine, are a direct consequence of the chosen numerical discretization, and in particular the time-integration. Emerging and future time-integration strategies should take into account all these factors to provide an effective path to sustainable NWP and climate simulations.
5 Emerging Alternatives
For both EBTI and PBTI, we outline some emerging alternatives that are being considered within the community, namely IMEX and fully implicit schemes for the EBTI class, and conservative semi-Lagrangian schemes for the PBTI class. The investigation of these time-stepping schemes for weather and climate is aligned with the general guidelines provided in Sect. 4, where co-design of software and hardware, energy-to- and time-to-solution are key players. In fact, in terms of EBTI strategies, there seems to be a trend to explore (semi-)implicit solutions, that eventually maximize the time-to-solution but would require the development of efficient algorithms to achieve increased percentage of peak performance (of a given machine) and better scalability properties. On the other hand, for PBTI strategies, efforts have been spent on developing conservative SISL schemes, thereby addressing the concerns of lack of accuracy for long-range weather and climate simulations.
5.1 EBTI Strategies
Both the strategies described below aim to improve the time-to-solution requirement of weather and climate simulations. In addition, if coupled with compact-stencil spatial discretizations, including finite volume and spectral element methods, they can be efficiently used on emerging computing architectures, due to the reduced parallel communication costs required. However, the solution procedure for both will require iterative elliptic solvers and iterative Newton-type methods, with all the associated numerical complexities that are to be addressed to make them a suitable alternative for operational NWP and climate.
5.1.1 Implicit–Explicit (IMEX) Methods
5.1.2 Fully-Implicit Methods
Fully-implicit methods are unconditionally stable, like the PBTI methods described previously. However, this flexibility in taking any length time-step size (restricted only by accuracy considerations) is in practice off-set by the prohibitive cost of the iterative solution of both the inner (Krylov solution) and outer (Newton solution) loops of the JFNK method. The condition number of the linear Krylov solution is proportional to the time-step size so taking a large time-step size translates into an increase in the number of Krylov iterations. For this reason it is especially important to choose the proper Krylov method (e.g., the cost of GMRES increases quadratically with the number of iterations/Krylov vectors). Preconditioners become all the more important for this class of time-integration methods if one wishes to build competitive (using the time-to-solution metric) strategies. In fact, in the past two Supercomputing conferences, the Gordon Bell prizes have been awarded to geoscience models using fully-implicit time-integrators–in 2015 for the simulation of the Earth’s mantle [78] and in 2016 for the (dry) simulation of atmospheric flows [113].
5.2 PBTI Strategies
The main drawback of SISL schemes is their inability to conserve, e.g. mass and scalar tracers. Conservative semi-Lagrangian methods exist and are being developed, with the main issue that they usually require computationally demanding re-meshing at each time-step. This is due to the varying-control-volumes in time required to maintain the exact conservative nature of the method. Therefore, their use in the context of weather and climate is limited and currently being investigated.
5.2.1 Conservative Semi-Lagrangian Methods
In Eq. (11), we described the classical semi-Lagrangian methods that are typically used in operational NWP and climate models and we stated that they do not formally conserve, e.g., mass, although in practice they conserve it to within an acceptable level. However, conservative semi-Lagrangian methods do exist and we describe a specific class of them below.
6 Discussion and Concluding Remarks
There are two key aspects that the weather and climate industry need to face in the oncoming years in terms of time-integration strategies: (i) the continuous increment in spatial resolution that is moving the equations of choice towards non-hydrostatic approximations, such that the vertical acoustic modes are not filtered a priori and a separation of horizontal and vertical motions is no longer readily achieved, placing substantially increased constraints on the time-stepping methods; (ii) the emerging computing technologies, that are significantly changing the paradigms traditionally adopted in parallel programming, thus demanding a review of algorithms and numerical solutions to maintain and possibly improve computational efficiency in next-generation HPC systems. In addition to these two points, the overall approach to the time-integration of weather and climate models should consider the operational constraints, time-to-solution (requirement to deliver a 10-days global forecast in 1 h real-time) and cost-to-solution (the latter usually translated into energy-to-solution), accuracy and robustness of the entire simulation framework.
Taking into account all these factors, it is clear that high efficiency time-stepping algorithms are required to allow operational NWP centers to complete extensive simulations with limited computer resources and satisfy the strict operational bounds. For this reason, due to the good efficiency of semi-Lagrangian advection schemes at high advective Courant numbers, the SISL methods have been at the heart of the most successful operational NWP and climate systems, e.g. IFS, UM, GSM, GFS. SISL-based schemes guarantee boundedness of the solution and unconditional stability [96], which have the advantage—compared to explicit schemes—that they permit a relatively large time-step and a very competitive time-to-solution performance [24, 73, 105, 111, 114].
The efficiency and robustness achieved with SISL is not easily replaced by alternative choices [110], although the better scalability that can be exploited on future HPC architectures, due to less reliance on communication exchanges beyond nearest neighborhoods will work in favor of techniques with compact stencils, e.g. finite volume (FV) and spectral element methods (SEM), coupled with EBTI-based approaches. The additional problem of the convergence of meridians at the globe’s poles in classical latitude-longitude grids, which results in very small simulated distances of the grid in the zonal direction (shorter than the physical distance between compute processors in some of today’s very high resolution models) and, thus, extremely small time-steps, has been widely addressed. In particular, the use of reduced quasi-homogeneous grids increases the stability of the solution and icosahedral (cf. ICON, MPAS, NICAM) or cubed sphere (cf. FV3, CAM-SE, NUMA) grids improve the computational efficiency [62, 65, 72]. Despite these improvements, compact-stencil EBTI-based techniques need to overcome additional issues in order to become competitive with PBTI schemes, especially in terms of time-to-solution. From this perspective, the development of efficient parallel preconditioners and their co-design with the underlying hardware is key, especially for IMEX, semi- and fully-implicit methods. The use of compact-stencil EBTI techniques, including FV and SEM, can also allow local grid-refinement, that in conjunction with local sub-time-stepping, can allow for improved resolution in the vicinity of steep topographic slopes at the lower boundary of models (up to 70 degrees in high-resolution global models). Such features can impose severe restrictions on the time-step and subsequently undermine numerical stability [116]. Also, the conservation of important quantities, e.g. mass and scalar tracers, that is of critical importance for climate simulations, is favored in EBTI schemes compared to classical SISL (although note the recent developments in conservative semi-Lagrangian schemes outlined in Sect. 5.2).
- (a)
to overcome the bottlenecks of today’s highly efficient SISL schemes and the associated cost of the solver by overlapping communications and computations [70, 81]; and to overcome accuracy drawbacks related to the large time-step choice while still correctly simulating all relevant wave dispersion relations. Promising approaches for satisfying the latter condition are exponential time integrators [36, 47];
- (b)
to overcome the overly restrictive time-step limitations of EBTI schemes combined with highly scalable horizontal discretizations, either through horizontal/vertical splitting (HEVI) [2, 8, 40] or through combining SISL PBTI methods with discontinuous Galerkin (DG) discretization [99]; and
- (c)
to further the scalability and the adaptation of algorithms to emerging HPC architectures involving SE [32] or fully-implicit time-stepping approaches [113], and further through exploiting additional parallelism with time-parallel algorithms [33].
Footnotes
Notes
Acknowledgements
This work was possible thanks to the ESCAPE project that has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 671627. FXG gratefully acknowledges the support of the U.S. Office of Naval Research through PE-0602435N and the U.S. Air Force Office of Scientific Research, Computational Mathematics program.
References
- 1.Proceedings of the ECMWF Workshop on Non-hydrostatic Modelling, 8–10 November 2010. European Centre for Medium-Range Weather Forecasts, Shinfield Park, Reading, Berkshire, UK (2011). https://books.google.com/books?id=iHE2AwEACAAJ
- 2.Abdi DS, Giraldo FX, Constantinescu EM, Carr III LE, Wilcox LC, Warburton TC (2017) Acceleration of the implicit-explicit non-hydrostatic unified model of the atmosphere (NUMA) on manycore processors. Int J High Perform Comput Appl (in press)Google Scholar
- 3.Alexander R (1977) Diagonally implicit Runge–Kutta methods for stiff ODEs. SIAM J Numer Anal 14(6):1006–1021MathSciNetzbMATHGoogle Scholar
- 4.Arakawa A, Konor CS (2009) Unification of the anelastic and quasi-hydrostatic systems of equations. Mon Weather Rev 137:710–726Google Scholar
- 5.Ascher U, Ruuth S, Wetton B (1995) Implicit-explicit methods for time-dependent partial differential equations. SIAM J Numer Anal 32(3):797–823MathSciNetzbMATHGoogle Scholar
- 6.Baldauf M (2010) Linear stabiliy analysis of Runge–Kutta-based partial time-splitting schemes for the Euler equations. Mon Weather Rev 138:4475–4496Google Scholar
- 7.Baldauf M (2013) A new fast-waves solver for the Runge–Kutta dynamical core. Technical Report 21, Consortium for Small-Scale Modelling. http://www.cosmo-model.org
- 8.Bao L, Klöfkorn R, Nair R (2015) Horizontally explicit and vertically implicit (HEVI) time discretization scheme for a discontinuous Galerkin nonhydrostatic model. Mon Weather Rev 143(3):972–990Google Scholar
- 9.Bauer P, Thorpe A, Brunet G (2015) The quiet revolution of numerical weather prediction. Nature 525(7567):47–55Google Scholar
- 10.Benacchio T, O’Neill W, Klein R (2014) A blended soundproof-to-compressible numerical model for small-to mesoscale atmospheric dynamics. Mon Weather Rev 142(12):4416–4438Google Scholar
- 11.Bénard P (2003) Stability of semi-implicit and iterative centered-implicit time discretizations for various equation systems used in NWP. Mon Weather Rev 131:2479–2491Google Scholar
- 12.Bènard P, Vivoda J, Mašek J, Smolìkovà P, Yessad K, Smith C, Brozkovà R, Geleyn JF (2010) Dynamical kernel of the Aladin-NH spectral limited-area model: Revised formulation and sensitivity experiments. Q J R Meteorol Soc 136:155–169Google Scholar
- 13.Brayton R, Gustavson F, Hachtel G (1972) A new efficient algorithm for solving differential-algebraic systems using implicit backward differentiation formulas. Proc IEEE 60(1):98–108MathSciNetGoogle Scholar
- 14.Bubnovà R, Hello G, Bènard P, Geleyn JF (1995) Integration of the fully elastic equations cast in the hydrostatic pressure terrain-following coordinate in the framework of the ARPEGE/Aladin NWP system. Mon Weather Rev 123:515–535Google Scholar
- 15.Butcher J (1987) The numerical analysis of ordinary differential equations: Runge–Kutta and general linear methods. Wiley-Interscience, New YorkzbMATHGoogle Scholar
- 16.Butcher J (2006) General linear methods. Acta Numer 15:157–256MathSciNetzbMATHGoogle Scholar
- 17.Caluwaerts S, Degrauwe D, Voitus F, Termonia P (2016) Discretization in numerical weather prediction: a modular approach to investigate spectral and local SISL methods. In: Mathematical problems in meteorological modelling, mathematics in industry, Springer, Berlin, vol 24, pp 19–46Google Scholar
- 18.Carpenter M, Kennedy C, Bijl H, Viken S, Vatsa V (2005) Fourth-order Runge-Kutta schemes for fluid mechanics applications. J Sci Comput 25(1–2):157–194MathSciNetzbMATHGoogle Scholar
- 19.Cash J (1979) Diagonally implicit Runge–Kutta formulae with error estimates. IMA J Appl Math 24(3):293–301zbMATHGoogle Scholar
- 20.Cash J (1980) On the integration of stiff systems of ODEs using extended backward differentiation formulae. Numer Math 34(3):235–246MathSciNetzbMATHGoogle Scholar
- 21.Cash J (1983) The integration of stiff initial value problems in ODEs using modified extended backward differentiation formulae. Comput Math Appl 9(5):645–657MathSciNetzbMATHGoogle Scholar
- 22.Ceschino F, Kuntzmann J (1966) Numerical solution of initial value problems. Prentice-Hall, Upper Saddle RiverzbMATHGoogle Scholar
- 23.Colavolpe C, Voitus F, Bénard P (2017) RK-IMEX HEVI schemes for fully compressible atmospheric models with advection: analyses and numerical testing. Q J R Meteorol Soc 143:1336–1350Google Scholar
- 24.Davies T, Cullen M, Malcolm A, Mawson M, Staniforth A, White A, Wood N (2005) A new dynamical core for the Met Office’s global and regional modelling of the atmosphere. Q J R Meteorol Soc 131:1759–1782Google Scholar
- 25.Deconinck W, Bauer P, Diamantakis M, Hamrud M, Kühnlein C, Maciel P, Mengaldo G, Quintino T, Raoult B, Smolarkiewicz P, Wedi N (2017) Atlas: A library for numerical weather prediction and climate modelling. Comput Phys Commun 220:188–204Google Scholar
- 26.Diamantakis M, Davies T, Wood N (2007) An iterative time-stepping scheme for the Met Office’s semi-implicit semi-Lagrangian non-hydrostatic model. Q J R Meteorol Soc 133:997–1011Google Scholar
- 27.Diamantakis M, Magnusson L (2016) Sensitivity of the ECMWF model to semi-Lagrangian departure point iterations. Mon Weather Rev 144:3233–3250Google Scholar
- 28.Durran D (1991) The third-order Adams–Bashforth method: an attractive alternative to leapfrog time differencing. Mon Weather Rev 119(3):702–720Google Scholar
- 29.Durran D (1999) Numerical methods for wave equations in geophysical fluid dynamics. Springer, BerlinzbMATHGoogle Scholar
- 30.Durran D (2008) A physically motivated approach for filtering acoustic waves from the equations governing compressible stratified flow. J Fluid Mech 601:365–379MathSciNetzbMATHGoogle Scholar
- 31.Falcone M, Ferretti R (1998) Convergence analysis for a class of high-order semi-Lagrangian advection schemes. SIAM J Numer Anal 35:909–940MathSciNetzbMATHGoogle Scholar
- 32.Fuhrer O, Chadha T, Hoefler T, Kwasniewski G, Lapillonne X, Leutwyler D, Osuna Schulthess T, Vogt H (2017) Near-global climate simulation at 1 km resolution: establishing a performance baseline on 4888 GPUs with COSMO 5.0. Geoscientific Model Development Discussions. In reviewGoogle Scholar
- 33.Gander M (2013) Multiple shooting and time domain decomposition methods, chap. 50 years of time parallel time integration. MuS-TDD, Heidelberg, pp 69–113Google Scholar
- 34.Gassmann A (2013) A global hexagonal C-grid non-hydrostatic dynamical core (ICON-IAP) designed for energetic consistency. Q J R Meteorol Soc 139(670):152–175Google Scholar
- 35.Gassmann A, Herzog HJ (2007) A consistent time-split numerical scheme applied to the nonhydrostatic compressible equations. Mon Weather Rev 135:20–36Google Scholar
- 36.Gaudreault S, Pudykiewicz J (2016) An efficient exponential time integration method for the numerical solution of the shallow water equations on the sphere. J Comput Phys 322:827–848MathSciNetzbMATHGoogle Scholar
- 37.Giraldo FX (1997) Lagrange-galerkin methods on spherical geodesic grids. J Comput Phys 136(1):197–213MathSciNetzbMATHGoogle Scholar
- 38.Giraldo FX (1998) The Lagrange-Galerkin spectral element method on unstructured quadrilateral grids. J Comput Phys 147(1):114–146MathSciNetzbMATHGoogle Scholar
- 39.Giraldo FX (2000) Lagrange-galerkin methods on spherical geodesic grids: the shallow water equations. J Comput Phys 160(1):336–368MathSciNetzbMATHGoogle Scholar
- 40.Giraldo FX, Kelly JF, Constantinescu EM (2013) Implicit-explicit formulations of a three-dimensional nonhydrostatic unified model of the atmosphere (NUMA). SIAM J Sci Comput 35(5):B1162–B1194MathSciNetzbMATHGoogle Scholar
- 41.Giraldo FX, Restelli M (2008) A study of spectral element and discontinuous Galerkin methods for the Navier–Stokes equations in nonhydrostatic mesoscale atmospheric modeling: Equation sets and test cases. J Comput Phys 227(8):3849–3877MathSciNetzbMATHGoogle Scholar
- 42.Giraldo FX, Restelli M, Läuter M (2010) Semi-implicit formulations of the Navier–Stokes equations: application to nonhydrostatic atmospheric modeling. SIAM J Sci Comput 32(6):3394–3425MathSciNetzbMATHGoogle Scholar
- 43.Godel N, Schomann S, Warburton T, Clemens M (2010) GPU accelerated Adams–Bashforth multirate discontinuous Galerkin FEM simulation of high-frequency electromagnetic fields. IEEE Trans Magn 46(8):2735–2738Google Scholar
- 44.Grabowski W (2016) Towards global large eddy simulation: super-parameterization revisited. J Meteorol Soc Jpn. Ser. II 94(4):327–344Google Scholar
- 45.Gustafsson N, Bojarova J (2014) Four-dimensional ensemble variational (4D-En-Var) data assimilation for the high resolution limited area model (HIRLAM). Nonlinear Process Geophys 21(4):745–762Google Scholar
- 46.Harris L, Lin SJ (2013) A two-way nested global-regional dynamical core on the cubed-sphere grid. Mon Weather Rev 141(1):283–306Google Scholar
- 47.Hochbruck M, Ostermann A (2010) Exponential integrators. Acta Numer 19:209–286MathSciNetzbMATHGoogle Scholar
- 48.Holton J, Haynes P, McIntyre M, Douglass A, Rood R, Pfister L (1995) Stratosphere-troposphere exchange. Rev Geophys 33(4):403–439Google Scholar
- 49.Hortal M (2002) The development and testing of a new two-time-level semi-Lagrangian scheme (SETTLS) in the ECMWF forecast model. Q J R Meteorol Soc 128:1671–1687Google Scholar
- 50.Ineson S, Scaife A (2009) The role of the stratosphere in the European climate response to El Niño. Nat Geosci 2(1):32–36Google Scholar
- 51.Jackiewicz Z (2009) General linear methods for ordinary differential equations. Wiley, New YorkzbMATHGoogle Scholar
- 52.Jameson A, Schmidt W, Turkel E (1981) Numerical solutions of the euler equations by finite volume methods using Runge–Kutta time-stepping schemes. AIAA paper 1259Google Scholar
- 53.Jeevanjee N (2017) Vertical velocity in the gray zone. J Adv Model Earth Syst 9:2304–2316Google Scholar
- 54.Kassam AK, Trefethen L (2005) Fourth-order time-stepping for stiff PDEs. SIAM J Sci Comput 26(4):1214–1233MathSciNetzbMATHGoogle Scholar
- 55.Kennedy C, Carpenter M, Lewis R (2000) Low-storage, explicit Runge–Kutta schemes for the compressible Navier–Stokes equations. Appl Numer Math 35(3):177–219MathSciNetzbMATHGoogle Scholar
- 56.Klemp J, Skamarock W, Dudhia J (2007) Conservative split-explicit time integration methods for the compressible nonhydrostatic equations. Mon Weather Rev 135:2897–2913Google Scholar
- 57.Klemp J, Wilhelmson R (1978) The simulation of three-dimensional convective storm dynamics. J Atmos Sci 35(6):1070–1096Google Scholar
- 58.Knoll D, Keyes D (2004) Jacobian-free Newton-Krylov methods: a survey of approaches and applications. J Comput Phys 193(2):357–397MathSciNetzbMATHGoogle Scholar
- 59.Kværnø A (2004) Singly diagonally implicit Runge-Kutta methods with an explicit first stage. BIT Numer Math 44(3):489–502MathSciNetzbMATHGoogle Scholar
- 60.Laprise R (1992) The Euler equations of motion with hydrostatic pressure as an independent variable. Mon Weather Rev 120(1):197–207Google Scholar
- 61.Lauritzen P, Nair R, Ullrich P (2010) A conservative semi-Lagrangian multi-tracer transport scheme (CSLAM) on the cubed-sphere grid. J Comput Phys 229(5):1401–1424MathSciNetzbMATHGoogle Scholar
- 62.Lin SJ (2004) A “vertically Lagrangian” finite-volume dynamical core for global models. Mon Weather Rev 132(10):2293–2307Google Scholar
- 63.Lock SJ, Wood N, Weller H (2014) Numerical analyses of Runge–Kutta implicit-explicit schemes for horizontally explicit, vertically implicit solutions of atmospheric models. Q J R Meteorol Soc 140(682):1654–1669Google Scholar
- 64.Malardel S, Wedi N (2016) How does subgrid-scale parametrization influence nonlinear spectral energy fluxes in global NWP models? J Geophys Res: Atmos 121:5395–5410Google Scholar
- 65.Mcgregor J, Dix M (2001) The CSIRO conformal-cubic atmospheric GCM. In: IUTAM symposium on advances in mathematical modelling of atmosphere and ocean dynamics, vol 61, pp 197–202. Springer, BerlinGoogle Scholar
- 66.Melvin T, Dubal M, Wood N, Staniforth A, Zerroukat M (2010) An inherently mass-conserving iterative semi-implicit semi-Lagrangian discretization of the non-hydrostatic vertical-slice equations. Q J R Meteorol Soc 136:799–814Google Scholar
- 67.Mengaldo G (2015) Discontinuous spectral/hp element methods: development, analysis and applications to compressible flows. Ph.D. thesis, Imperial College LondonGoogle Scholar
- 68.Mesinger F (1977) Forward-backward scheme, and its use in a limited area model. Contrib Atmos Phys 50:200–210Google Scholar
- 69.Miller M, Smolarkiewicz P (2008) Predicting weather, climate and extreme events. J Comput Phys 7(227):3429–3430Google Scholar
- 70.Mozdzynski G, Hamrud M, Wedi N (2015) A partitioned global address space implementation of the European centre for medium range weather forecasts integrated forecasting system. Int J High Perform Comput Appl 29:261–273Google Scholar
- 71.Polvani L, Kushner P (2002) Tropospheric response to stratospheric perturbations in a relatively simple general circulation model. Geophys Res Lett 29(7):18–21Google Scholar
- 72.Putman W, Suarez M (2011) Cloud-system resolving simulations with the NASA Goddard Earth Observing System global atmospheric model (GEOS-5). Geophys Res Lett 38(16):L16809Google Scholar
- 73.Qian JH, Semazzi F, Scroggs J (1998) A global nonhydrostatic semi-Lagrangian atmospheric model with orography. Mon Weather Rev 126(3):747–771Google Scholar
- 74.Ritchie H (1987) Semi-Lagrangian advection on a gaussian grid. Mon Weather Rev 115:608–619Google Scholar
- 75.Ritchie H, Temperton C, Simmons A, Hortal M, Davies T, Dent D, Hamrud M (1995) Implementation of the semi-Lagrangian method in a high-resolution version of the ECMWF forecast model. Mon Weather Rev 123:489–514Google Scholar
- 76.Rivest C, Staniforth A, Robert A (1994) Spurious resonant response of semi-Lagrangian discretizations to orographic forcing: diagnosis and solution. Mon Weather Rev 122:366–376Google Scholar
- 77.Rudd K, Ferrari S (2015) A constrained integration (CINT) approach to solving partial differential equations using artificial neural networks. Neurocomputing 155:277–285Google Scholar
- 78.Rudi J, Malossi ACI, Isaac T, Stadler G, Gurnis M, Staar P, Ineichen Y, Bekas C, Curioni A, Ghattas O (2015) An extreme-scale implicit solver for complex PDEs: highly heterogeneous flow in Earth’s mantle. In: Proceedings of the international conference for high performance computing, networking, storage and analysis, SC ’15, ACM, pp 5:1–5:12Google Scholar
- 79.Saito K (2003) Time-splitting of advection in the JMA nonhydrostatic model. CAS/JSC WGNE Res Activ Oceanic Modell 33:315–316Google Scholar
- 80.Saito K, Fujita T, Yamada Y, Ishida JI, Kumagai Y, Aranami K, Ohmori S, Nagasawa R, Kumagai S, Muroi C, Kato T, Eito H, Yamazaki Y (2006) The operational JMA nonhydrostatic mesoscale model. Mon Weather Rev 134(4):1266–1298Google Scholar
- 81.Sanan P, Schnepp S, May D (2016) Pipelined, flexible Krylov subspace methods. SIAM J Sci Comput 38(5):C441–C470MathSciNetzbMATHGoogle Scholar
- 82.Satoh M, Matsuno T, Tomita H, Miura H, Nasuno T, Iga S (2008) Nonhydrostatic icosahedral atmospheric model (NICAM) for global cloud resolving simulations. J Comput Phys 227:3486–3514MathSciNetzbMATHGoogle Scholar
- 83.Satoh M, Tomita H, Yashiro H, Miura H, Kodama C, Seiki T, Noda A, Yamada Y, Goto D, Sawada M, Miyoshi T, Niwa Y, Hara M, Ohno T, Iga S, Arakawa T, Inoue T, Kubokawa H (2014) The non-hydrostatic icosahedral atmospheric model: description and development. Progress Earth Planet Sci 1:1–18Google Scholar
- 84.Schiesser W (2012) The numerical method of lines: integration of partial differential equations. Elsevier, AmsterdamzbMATHGoogle Scholar
- 85.Schneider T, Lan S, Stuart A, Teixeira J (2017) Earth system modeling 2.0: a blueprint for models that learn from observations and targeted high-resolution simulations. arXiv preprint arXiv:1709.00037
- 86.Simmons A (1991) Development of a high resolution, semi-Lagrangian version of the ECMWF forecast model. In: Seminar on numerical methods in atmospheric models. ECMWF, ReadingGoogle Scholar
- 87.Skamarock W, Klemp J (1992) The stability of time-split numerical methods for the hydrostatic and the nonhydrostatic elastic equations. Mon Weather Rev 120(9):2109–2127Google Scholar
- 88.Skamarock W, Klemp J (2008) A time-split nonhydrostatic atmospheric model for weather research and forecasting applications. J Comput Phys 227(7):3465–3485MathSciNetzbMATHGoogle Scholar
- 89.Skamarock W, Klemp J, Duda M, Fowler L, Park SH, Ringler T (2012) A multiscale nonhydrostatic atmospheric model using centroidal Voronoi tesselations and C-grid staggering. Mon Weather Rev 140:3090–3105Google Scholar
- 90.Smolarkiewicz P (2011) Modeling atmospheric circulations with soundproof equations. In: Proceedings of the ECMWF workshop on nonhydrostatic modelling, 8–10 November, 2010, Reading, pp. 1–15Google Scholar
- 91.Smolarkiewicz P, Deconinck W, Hamrud M, Kühnlein C, Mozdzynski G, Szmelter J, Wedi N (2016) A finite-volume module for simulating global all-scale atmospheric flows. J Comput Phys 314:287–304MathSciNetzbMATHGoogle Scholar
- 92.Smolarkiewicz P, Kühnlein C, Wedi N (2014) A consistent framework for discrete integrations of soundproof and compressible PDEs of atmospheric dynamics. J Comput Phys 263:185–205MathSciNetzbMATHGoogle Scholar
- 93.Smolarkiewicz P, Pudykiewicz J (1992) A class of semi-Lagrangian approximations for fluids. J Atmos Sci 49(22):2082–2096Google Scholar
- 94.Staniforth A, Côté J (1991) Semi-Lagrangian integration schemes for atmospheric models–a review. Mon Weather Rev 119(9):2206–2223Google Scholar
- 95.Staniforth A, Melvin T, Wood N (2013) Gungho! a new dynamical core for the unified model. In: Proceeding of the ECMWF workshop on recent developments in numerical methods for atmosphere and ocean modellingGoogle Scholar
- 96.Staniforth A, Wood N (2008) Aspects of the dynamical core of a nonhydrostatic, deep-atmosphere, unified weather and climate-prediction model. J Comput Phys 227(7):3445–3464MathSciNetzbMATHGoogle Scholar
- 97.Strang G (1968) On the construction and comparison of difference schemes. SIAM J Numer Anal 5(3):506–517MathSciNetzbMATHGoogle Scholar
- 98.Temperton C, Staniforth A (1987) An efficient two-time-level semi-Lagrangian semi-implicit integration scheme. Q J R Meteorol Soc 113(477):1025–1039Google Scholar
- 99.Tumolo G, Bonaventura L (2015) A semi-implicit, semi-Lagrangian discontinuous Galerkin framework for adaptive numerical weather prediction. Q J R Meteorol Soc 141:2582–2601Google Scholar
- 100.Ullrich P, Jablonowski C (2012) Operator-split Runge–Kutta–Rosenbrock methods for nonhydrostatic atmospheric models. Mon Weather Rev 140(4):1257–1284Google Scholar
- 101.Ullrich P, Jablonowski C, Kent J, Lauritzen P, Nair R, Reed K, Zarzycki C, Hall D, Dazlich D, Heikes R et al (2017) DCMIP2016: a review of non-hydrostatic dynamical core design and intercomparison of participating models. Geosci Model Dev 10(12):4477–4509Google Scholar
- 102.Vos P, Eskilsson C, Bolis A, Chun S, Kirby R, Sherwin S (2011) A generic framework for time-stepping partial differential equations (PDEs): general linear methods, object-oriented implementation and application to fluid problems. Int J Comput Fluid Dyn 25(3):107–125MathSciNetzbMATHGoogle Scholar
- 103.Wedi N, Bauer P, Deconinck W, Diamantakis M, Hamrud M, Kühnlein C, Malardel S, Mogensen K, Mozdzynski G, Smolarkiewicz P (2015) The modelling infrastructure of the integrated forecasting system: recent advances and future challenges. ECMWF Research Department Technical Memorandum 760Google Scholar
- 104.Wedi N, Hamrud M, Mozdzynski G (2013) A fast spherical harmonics transform for global NWP and climate models. Mon Weather Rev 141:3450–3461Google Scholar
- 105.Wedi N, Smolarkiewicz P (2009) A framework for testing global non-hydrostatic models. Q J R Meteorol Soc 135(639):469–484Google Scholar
- 106.Weller H, Lock SJ, Wood N (2013) Runge–Kutta IMEX schemes for the horizontally explicit/vertically implicit (HEVI) solution of wave equations. J Comput Phys 252:365–381MathSciNetzbMATHGoogle Scholar
- 107.Wicker L (2009) A two-step Adams–Bashforth–Moulton split-explicit integrator for compressible atmospheric models. Mon Weather Rev 137(10):3588–3595Google Scholar
- 108.Wicker L, Skamarock W (1998) A time-splitting scheme for the elastic equations incorporating second-order Runge–Kutta time differencing. Mon Weather Rev 126:1992–1999Google Scholar
- 109.Wicker L, Skamarock W (2002) Time-splitting methods for elastic models using forward time schemes. Mon Weather Rev 130(8):2088–2097Google Scholar
- 110.Williamson D (2007) The evolution of dynamical cores for global atmospheric models. J Meteorol Soc Jpn 85B:241–269Google Scholar
- 111.Wood N, Staniforth A, White A, Allen T, Diamantakis M, Gross M, Melvin T, Smith C, Vosper S, Zerroukat M, Thuburn J (2014) An inherently mass-conserving semi-implicit semi-Lagrangian discretization of the deep-atmosphere global non-hydrostatic equations. Q J R Meteorol Soc 140(682):1505–1520Google Scholar
- 112.Xiu D, Karniadakis G (2001) A semi-Lagrangian high-order method for Navier–Stokes equations. J Comput Phys 172:658–684MathSciNetzbMATHGoogle Scholar
- 113.Yang C, Xue W, Fu H, You H, Wang X, Ao Y, Liu F, Gan L, Xu P, Wang L, Yang G, Zheng W (2016) 10m-core scalable fully-implicit solver for nonhydrostatic atmospheric dynamics. In: SC 2016: Proceedings of the international conference for high performance computing, networking, storage and analysis, IEEE, pp 57–68Google Scholar
- 114.Yeh KS, Côté J, Gravel S, Méthot A, Patoine A, Roch M, Staniforth A (2002) The CMC-MRB global environmental multiscale (GEM) model. Part III: nonhydrostatic formulation. Mon Weather Rev 130(2):339–356Google Scholar
- 115.Yessad K, Wedi N (2011) The hydrostatic and non-hydrostatic global model IFS/ARPEGE: deep-layer model formulation and testing. Technical Report 657Google Scholar
- 116.Zängl G (2012) Extending the numerical stability limit of terrain-following coordinate models over steep slopes. Mon Weather Rev 140:3722–3733Google Scholar
- 117.Zerroukat M, Wood N, Staniforth A (2002) SLICE: A semi-Lagrangian inherently conserving and efficient scheme for transport problems. Q J R Meteorol Soc 128(586):2801–2820Google Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.