Abstract
The famous diffusion equation, also known as the heat equation, reads
where \(u(x,t)\) is the unknown function to be solved for, x is a coordinate in space, and t is time. The coefficient α is the diffusion coefficient and determines how fast u changes in time. A quick short form for the diffusion equation is \(u_{t}=\alpha u_{xx}\).
Compared to the wave equation, \(u_{tt}=c^{2}u_{xx}\), which looks very similar, the diffusion equation features solutions that are very different from those of the wave equation. Also, the diffusion equation makes quite different demands to the numerical methods.
You have full access to this open access chapter, Download chapter PDF
Similar content being viewed by others
The famous diffusion equation, also known as the heat equation, reads
where \(u(x,t)\) is the unknown function to be solved for, x is a coordinate in space, and t is time. The coefficient α is the diffusion coefficient and determines how fast u changes in time. A quick short form for the diffusion equation is \(u_{t}=\alpha u_{xx}\).
Compared to the wave equation, \(u_{tt}=c^{2}u_{xx}\), which looks very similar, the diffusion equation features solutions that are very different from those of the wave equation. Also, the diffusion equation makes quite different demands to the numerical methods.
Typical diffusion problems may experience rapid change in the very beginning, but then the evolution of u becomes slower and slower. The solution is usually very smooth, and after some time, one cannot recognize the initial shape of u. This is in sharp contrast to solutions of the wave equation where the initial shape is preserved in homogeneous media – the solution is then basically a moving initial condition. The standard wave equation \(u_{tt}=c^{2}u_{xx}\) has solutions that propagate with speed c forever, without changing shape, while the diffusion equation converges to a stationary solution \(\bar{u}(x)\) as \(t\rightarrow\infty\). In this limit, \(u_{t}=0\), and \(\bar{u}\) is governed by \(\bar{u}^{\prime\prime}(x)=0\). This stationary limit of the diffusion equation is called the Laplace equation and arises in a very wide range of applications throughout the sciences.
It is possible to solve for \(u(x,t)\) using an explicit scheme, as we do in Sect. 3.1, but the time step restrictions soon become much less favorable than for an explicit scheme applied to the wave equation. And of more importance, since the solution u of the diffusion equation is very smooth and changes slowly, small time steps are not convenient and not required by accuracy as the diffusion process converges to a stationary state. Therefore, implicit schemes (as described in Sect. 3.2) are popular, but these require solutions of systems of algebraic equations. We shall use ready-made software for this purpose, but also program some simple iterative methods. The exposition is, as usual in this book, very basic and focuses on the basic ideas and how to implement. More comprehensive mathematical treatments and classical analysis of the methods are found in lots of textbooks. A favorite of ours in this respect is the one by LeVeque [13]. The books by Strikwerda [17] and by Lapidus and Pinder [12] are also highly recommended as additional material on the topic.
3.1 An Explicit Method for the 1D Diffusion Equation
Explicit finite difference methods for the wave equation \(u_{tt}=c^{2}u_{xx}\) can be used, with small modifications, for solving \(u_{t}=\alpha u_{xx}\) as well. The exposition below assumes that the reader is familiar with the basic ideas of discretization and implementation of wave equations from Chapter 2. Readers not familiar with the Forward Euler, Backward Euler, and Crank-Nicolson (or centered or midpoint) discretization methods in time should consult, e.g., Section 1.1 in [9].
3.1.1 The Initial-Boundary Value Problem for 1D Diffusion
To obtain a unique solution of the diffusion equation, or equivalently, to apply numerical methods, we need initial and boundary conditions. The diffusion equation goes with one initial condition \(u(x,0)=I(x)\), where I is a prescribed function. One boundary condition is required at each point on the boundary, which in 1D means that u must be known, u x must be known, or some combination of them.
We shall start with the simplest boundary condition: u = 0. The complete initial-boundary value diffusion problem in one space dimension can then be specified as
With only a first-order derivative in time, only one initial condition is needed, while the second-order derivative in space leads to a demand for two boundary conditions. We have added a source term \(f=f(x,t)\), which is convenient when testing implementations.
Diffusion equations like (3.1) have a wide range of applications throughout physical, biological, and financial sciences. One of the most common applications is propagation of heat, where \(u(x,t)\) represents the temperature of some substance at point x and time t. Other applications are listed in Sect. 3.8.
3.1.2 Forward Euler Scheme
The first step in the discretization procedure is to replace the domain \([0,L]\times[0,T]\) by a set of mesh points. Here we apply equally spaced mesh points
and
Moreover, \(u^{n}_{i}\) denotes the mesh function that approximates \(u(x_{i},t_{n})\) for \(i=0,\ldots,N_{x}\) and \(n=0,\ldots,N_{t}\). Requiring the PDE (3.1) to be fulfilled at a mesh point \((x_{i},t_{n})\) leads to the equation
The next step is to replace the derivatives by finite difference approximations. The computationally simplest method arises from using a forward difference in time and a central difference in space:
Written out,
We have turned the PDE into algebraic equations, also often called discrete equations. The key property of the equations is that they are algebraic, which makes them easy to solve. As usual, we anticipate that \(u^{n}_{i}\) is already computed such that \(u^{n+1}_{i}\) is the only unknown in (3.7). Solving with respect to this unknown is easy:
where we have introduced the mesh Fourier number:
F is the key parameter in the discrete diffusion equation
Note that F is a dimensionless number that lumps the key physical parameter in the problem, α, and the discretization parameters \(\Delta x\) and \(\Delta t\) into a single parameter. Properties of the numerical method are critically dependent upon the value of F (see Sect. 3.3 for details).
The computational algorithm then becomes
-
1.
compute \(u^{0}_{i}=I(x_{i})\) for \(i=0,\ldots,N_{x}\)
-
2.
for \(n=0,1,\ldots,N_{t}\):
-
a)
apply (3.8) for all the internal spatial points \(i=1,\ldots,N_{x}-1\)
-
b)
set the boundary values \(u^{n+1}_{i}=0\) for i = 0 and \(i=N_{x}\)
-
a)
The algorithm is compactly and fully specified in Python:
Note that we use a for α in the code, motivated by easy visual mapping between the variable name and the mathematical symbol in formulas.
We need to state already now that the shown algorithm does not produce meaningful results unless F ≤ 1 ∕ 2. Why is explained in Sect. 3.3.
3.1.3 Implementation
The file diffu1D_u0.py contains a complete function solver_FE_simple for solving the 1D diffusion equation with u = 0 on the boundary as specified in the algorithm above:
A faster alternative is available in the function solver_FE, which adds the possibility of solving the finite difference scheme by vectorization. The vectorized version replaces the explicit loop
by arithmetics on displaced slices of the u array:
For example, the vectorized version runs 70 times faster than the scalar version in a case with 100 time steps and a spatial mesh of 105 cells.
The solver_FE function also features a callback function such that the user can process the solution at each time level. The callback function looks like user_action(u, x, t, n), where u is the array containing the solution at time level n, x holds all the spatial mesh points, while t holds all the temporal mesh points. The solver_FE function is very similar to solver_FE_simple above:
3.1.4 Verification
Exact solution of discrete equations
Before thinking about running the functions in the previous section, we need to construct a suitable test example for verification. It appears that a manufactured solution that is linear in time and at most quadratic in space fulfills the Forward Euler scheme exactly. With the restriction that u = 0 for x = 0,L, we can try the solution
Inserted in the PDE, it requires a source term
With the formulas from Appendix A.4 we can easily check that the manufactured u fulfills the scheme:
which is a 0=0 expression. The computation of the source term, given any u, is easily automated with sympy:
Now we can choose any expression for u and automatically get the suitable source term f. However, the manufactured solution u will in general not be exactly reproduced by the scheme: only constant and linear functions are differentiated correctly by a forward difference, while only constant, linear, and quadratic functions are differentiated exactly by a \([D_{x}D_{x}u]^{n}_{i}\) difference.
The numerical code will need to access the u and f above as Python functions. The exact solution is wanted as a Python function u_exact(x, t), while the source term is wanted as f(x, t). The parameters a and L in u and f above are symbols and must be replaced by float objects in a Python function. This can be done by redefining a and L as float objects and performing substitutions of symbols by numbers in u and f. The appropriate code looks like this:
Here we also make a function I for the initial condition.
The idea now is that our manufactured solution should be exactly reproduced by the code (to machine precision). For this purpose we make a test function for comparing the exact and numerical solutions at the end of the time interval:
The critical value F = 0.5
We emphasize that the value F=0.5 is critical: the tests above will fail if F has a larger value. This is because the Forward Euler scheme is unstable for F > 1 ∕ 2.
The reader may wonder if F = 1 ∕ 2 is safe or if F < 1 ∕ 2 should be required. Experiments show that F = 1 ∕ 2 works fine for \(u_{t}=\alpha u_{xx}\), so there is no accumulation of rounding errors in this case and hence no need to introduce any safety factor to keep F away from the limiting value 0.5.
Checking convergence rates
If our chosen exact solution does not satisfy the discrete equations exactly, we are left with checking the convergence rates, just as we did previously for the wave equation. However, with the Euler scheme here, we have different accuracies in time and space, since we use a second order approximation to the spatial derivative and a first order approximation to the time derivative. Thus, we must expect different convergence rates in time and space. For the numerical error,
we should get convergence rates r = 1 and p = 2 (C t and C x are unknown constants). As previously, in Sect. 2.2.3, we simplify matters by introducing a single discretization parameter h:
where K is any constant. This allows us to factor out only one discretization parameter h from the formula:
The computed rate r should approach 1 with increasing resolution.
It is tempting, for simplicity, to choose K = 1, which gives \(\Delta x=h^{r/p}\), expected to be \(\sqrt{\Delta t}\). However, we have to control the stability requirement: \(F\leq\frac{1}{2}\), which means
implying that \(K=\sqrt{2\alpha}\) is our choice in experiments where we lie on the stability limit F = 1 ∕ 2.
3.1.5 Numerical Experiments
When a test function like the one above runs silently without errors, we have some evidence for a correct implementation of the numerical method. The next step is to do some experiments with more interesting solutions.
We target a scaled diffusion problem where x ∕ L is a new spatial coordinate and \(\alpha t/L^{2}\) is a new time coordinate. The source term f is omitted, and u is scaled by \(\max_{x\in[0,L]}|I(x)|\) (see Section 3.2 in [11] for details). The governing PDE is then
in the spatial domain \([0,L]\), with boundary conditions \(u(0)=u(1)=0\). Two initial conditions will be tested: a discontinuous plug,
and a smooth Gaussian function,
The functions plug and gaussian in diffu1D_u0.py run the two cases, respectively:
These functions make use of the function viz for running the solver and visualizing the solution using a callback function with plotting:
Notice that this viz function stores all the solutions in a list solutions in the callback function. Modern computers have hardly any problem with storing a lot of such solutions for moderate values of N x in 1D problems, but for 2D and 3D problems, this technique cannot be used and solutions must be stored in files.
Our experiments employ a time step \(\Delta t=0.0002\) and simulate for \(t\in[0,0.1]\). First we try the highest value of F: F = 0.5. This resolution corresponds to \(N_{x}=50\). A possible terminal command is
The \(u(x,t)\) curve as a function of x is shown in Fig. 3.1 at four time levels.
Forward Euler scheme for F = 0.5
Movie 1
We see that the curves have saw-tooth waves in the beginning of the simulation. This non-physical noise is smoothed out with time, but solutions of the diffusion equations are known to be smooth, and this numerical solution is definitely not smooth. Lowering F helps: F ≤ 0.25 gives a smooth solution, see Fig. 3.2 (and a movie Footnote 1).
Forward Euler scheme for F = 0.25
Increasing F slightly beyond the limit 0.5, to F = 0.51, gives growing, non-physical instabilities, as seen in Fig. 3.3.
Forward Euler scheme for F = 0.51
Instead of a discontinuous initial condition we now try the smooth Gaussian function for I(x). A simulation for F = 0.5 is shown in Fig. 3.4. Now the numerical solution is smooth for all times, and this is true for any F ≤ 0.5.
Forward Euler scheme for F = 0.5
Experiments with these two choices of I(x) reveal some important observations:
-
The Forward Euler scheme leads to growing solutions if \(F> \frac{1}{2}\).
-
I(x) as a discontinuous plug leads to a saw tooth-like noise for \(F=\frac{1}{2}\), which is absent for \(F\leq\frac{1}{4}\).
-
The smooth Gaussian initial function leads to a smooth solution for all relevant F values (\(F\leq\frac{1}{2}\)).
3.2 Implicit Methods for the 1D Diffusion Equation
Simulations with the Forward Euler scheme show that the time step restriction, \(F\leq\frac{1}{2}\), which means \(\Delta t\leq\Delta x^{2}/(2\alpha)\), may be relevant in the beginning of the diffusion process, when the solution changes quite fast, but as time increases, the process slows down, and a small \(\Delta t\) may be inconvenient. With implicit schemes, which lead to coupled systems of linear equations to be solved at each time level, any size of \(\Delta t\) is possible (but the accuracy decreases with increasing \(\Delta t\)). The Backward Euler scheme, derived and implemented below, is the simplest implicit scheme for the diffusion equation.
3.2.1 Backward Euler Scheme
In (3.5), we now apply a backward difference in time, but the same central difference in space:
which written out reads
Now we assume \(u^{n-1}_{i}\) is already computed, but that all quantities at the ‘‘new’’ time level n are unknown. This time it is not possible to solve with respect to \(u_{i}^{n}\) because this value couples to its neighbors in space, \(u^{n}_{i-1}\) and \(u^{n}_{i+1}\), which are also unknown. Let us examine this fact for the case when \(N_{x}=3\). Equation (3.11) written for \(i=1,\ldots,Nx-1=1,2\) becomes
The boundary values \(u^{n}_{0}\) and \(u^{n}_{3}\) are known as zero. Collecting the unknown new values \(u^{n}_{1}\) and \(u^{n}_{2}\) on the left-hand side and multiplying by \(\Delta t\) gives
This is a coupled 2 × 2 system of algebraic equations for the unknowns \(u^{n}_{1}\) and \(u^{n}_{2}\). The equivalent matrix form is
Terminology: implicit vs. explicit methods
Discretization methods that lead to a coupled system of equations for the unknown function at a new time level are said to be implicit methods. The counterpart, explicit methods, refers to discretization methods where there is a simple explicit formula for the values of the unknown function at each of the spatial mesh points at the new time level. From an implementational point of view, implicit methods are more comprehensive to code since they require the solution of coupled equations, i.e., a matrix system, at each time level. With explicit methods we have a closed-form formula for the value of the unknown at each mesh point.
Very often explicit schemes have a restriction on the size of the time step that can be relaxed by using implicit schemes. In fact, implicit schemes are frequently unconditionally stable, so the size of the time step is governed by accuracy and not by stability. This is the great advantage of implicit schemes.
In the general case, (3.11) gives rise to a coupled \((N_{x}-1)\times(N_{x}-1)\) system of algebraic equations for all the unknown \(u^{n}_{i}\) at the interior spatial points \(i=1,\ldots,N_{x}-1\). Collecting the unknowns on the left-hand side, (3.11) can be written
for \(i=1,\ldots,N_{x}-1\). One can either view these equations as a system where the \(u^{n}_{i}\) values at the internal mesh points, \(i=1,\ldots,N_{x}-1\), are unknown, or we may append the boundary values \(u^{n}_{0}\) and \(u^{n}_{N_{x}}\) to the system. In the latter case, all \(u^{n}_{i}\) for \(i=0,\ldots,N_{x}\) are considered unknown, and we must add the boundary equations to the \(N_{x}-1\) equations in (3.16):
A coupled system of algebraic equations can be written on matrix form, and this is important if we want to call up ready-made software for solving the system. The equations (3.16) and (3.17)–(3.18) correspond to the matrix equation
where \(U=(u^{n}_{0},\ldots,u^{n}_{N_{x}})\), and the matrix A has the following structure:
The nonzero elements are given by
in the equations for internal points, \(i=1,\ldots,N_{x}-1\). The first and last equation correspond to the boundary condition, where we know the solution, and therefore we must have
The right-hand side b is written as
with
We observe that the matrix A contains quantities that do not change in time. Therefore, A can be formed once and for all before we enter the recursive formulas for the time evolution. The right-hand side b, however, must be updated at each time step. This leads to the following computational algorithm, here sketched with Python code:
Regarding verification, the same considerations apply as for the Forward Euler method (Sect. 3.1.4 ).
3.2.2 Sparse Matrix Implementation
We have seen from (3.19) that the matrix A is tridiagonal. The code segment above used a full, dense matrix representation of A, which stores a lot of values we know are zero beforehand, and worse, the solution algorithm computes with all these zeros. With \(N_{x}+1\) unknowns, the work by the solution algorithm is \(\frac{1}{3}(N_{x}+1)^{3}\) and the storage requirements \((N_{x}+1)^{2}\). By utilizing the fact that A is tridiagonal and employing corresponding software tools that work with the three diagonals, the work and storage demands can be proportional to N x only. This leads to a dramatic improvement: with \(N_{x}=200\), which is a realistic resolution, the code runs about 40,000 times faster and reduces the storage to just 1.5 %! It is no doubt that we should take advantage of the fact that A is tridiagonal.
The key idea is to apply a data structure for a tridiagonal or sparse matrix. The scipy.sparse package has relevant utilities. For example, we can store only the nonzero diagonals of a matrix. The package also has linear system solvers that operate on sparse matrix data structures. The code below illustrates how we can store only the main diagonal and the upper and lower diagonals.
The scipy.sparse.linalg.spsolve function utilizes the sparse storage structure of A and performs, in this case, a very efficient Gaussian elimination solve.
The program diffu1D_u0.py contains a function solver_BE, which implements the Backward Euler scheme sketched above. As mentioned in Sect. 3.1.2, the functions plug and gaussian run the case with I(x) as a discontinuous plug or a smooth Gaussian function. All experiments point to two characteristic features of the Backward Euler scheme: 1) it is always stable, and 2) it always gives a smooth, decaying solution.
3.2.3 Crank-Nicolson Scheme
The idea in the Crank-Nicolson scheme is to apply centered differences in space and time, combined with an average in time. We demand the PDE to be fulfilled at the spatial mesh points, but midway between the points in the time mesh:
for \(i=1,\ldots,N_{x}-1\) and \(n=0,\ldots,N_{t}-1\).
With centered differences in space and time, we get
On the right-hand side we get an expression
This expression is problematic since \(u^{n+\frac{1}{2}}_{i}\) is not one of the unknowns we compute. A possibility is to replace \(u^{n+\frac{1}{2}}_{i}\) by an arithmetic average:
In the compact notation, we can use the arithmetic average notation \(\overline{u}^{t}\):
We can also use an average for \(f_{i}^{n+\frac{1}{2}}\):
After writing out the differences and average, multiplying by \(\Delta t\), and collecting all unknown terms on the left-hand side, we get
Also here, as in the Backward Euler scheme, the new unknowns \(u^{n+1}_{i-1}\), \(u^{n+1}_{i}\), and \(u^{n+1}_{i+1}\) are coupled in a linear system AU = b, where A has the same structure as in (3.19), but with slightly different entries:
in the equations for internal points, \(i=1,\ldots,N_{x}-1\). The equations for the boundary points correspond to
The right-hand side b has entries
When verifying some implementation of the Crank-Nicolson scheme by convergence rate testing, one should note that the scheme is second order accurate in both space and time. The numerical error then reads
where r = 2 (C t and C x are unknown constants, as before). When introducing a single discretization parameter, we may now simply choose
which gives
where r should approach 2 as resolution is increased in the convergence rate computations.
3.2.4 The Unifying θ Rule
For the equation
where G(u) is some spatial differential operator, the θ-rule looks like
The important feature of this time discretization scheme is that we can implement one formula and then generate a family of well-known and widely used schemes:
-
θ = 0 gives the Forward Euler scheme in time
-
θ = 1 gives the Backward Euler scheme in time
-
\(\theta=\frac{1}{2}\) gives the Crank-Nicolson scheme in time
In the compact difference notation, we write the θ rule as
We have that \(t_{n+\theta}=\theta t_{n+1}+(1-\theta)t_{n}\).
Applied to the 1D diffusion problem, the θ-rule gives
This scheme also leads to a matrix system with entries
while right-hand side entry b i is
The corresponding entries for the boundary points are as in the Backward Euler and Crank-Nicolson schemes listed earlier.
Note that convergence rate testing with implementations of the theta rule must adjust the error expression according to which of the underlying schemes is actually being run. That is, if θ = 0 (i.e., Forward Euler) or θ = 1 (i.e., Backward Euler), there should be first order convergence, whereas with θ = 0.5 (i.e., Crank-Nicolson), one should get second order convergence (as outlined in previous sections).
3.2.5 Experiments
We can repeat the experiments from Sect. 3.1.5 to see if the Backward Euler or Crank-Nicolson schemes have problems with sawtooth-like noise when starting with a discontinuous initial condition. We can also verify that we can have \(F> \frac{1}{2}\), which allows larger time steps than in the Forward Euler method.
The Backward Euler scheme always produces smooth solutions for any F. Figure 3.5 shows one example. Note that the mathematical discontinuity at t = 0 leads to a linear variation on a mesh, but the approximation to a jump becomes better as N x increases. In our simulation, we specify \(\Delta t\) and F, and set N x to \(L/\sqrt{\alpha\Delta t/F}\). Since \(N_{x}\sim\sqrt{F}\), the discontinuity looks sharper in the Crank-Nicolson simulations with larger F.
Backward Euler scheme for F = 0.5
The Crank-Nicolson method produces smooth solutions for small F, \(F\leq\frac{1}{2}\), but small noise gets more and more evident as F increases. Figures 3.6 and 3.7 demonstrate the effect for F = 3 and F = 10, respectively. Section 3.3 explains why such noise occur.
Crank-Nicolson scheme for F = 3
Crank-Nicolson scheme for F = 10
3.2.6 The Laplace and Poisson Equation
The Laplace equation, \(\nabla^{2}u=0\), and the Poisson equation, \(-\nabla^{2}u=f\), occur in numerous applications throughout science and engineering. In 1D these equations read \(u^{\prime\prime}(x)=0\) and \(-u^{\prime\prime}(x)=f(x)\), respectively. We can solve 1D variants of the Laplace equations with the listed software, because we can interpret \(u_{xx}=0\) as the limiting solution of \(u_{t}=\alpha u_{xx}\) when u reaches a steady state limit where \(u_{t}\rightarrow 0\). Similarly, Poisson’s equation \(-u_{xx}=f\) arises from solving \(u_{t}=u_{xx}+f\) and letting \(t\rightarrow\infty\) so \(u_{t}\rightarrow 0\).
Technically in a program, we can simulate \(t\rightarrow\infty\) by just taking one large time step: \(\Delta t\rightarrow\infty\). In the limit, the Backward Euler scheme gives
which is nothing but the discretization \([-D_{x}D_{x}u=f]^{n+1}_{i}=0\) of \(-u_{xx}=f\).
The result above means that the Backward Euler scheme can solve the limit equation directly and hence produce a solution of the 1D Laplace equation. With the Forward Euler scheme we must do the time stepping since \(\Delta t> \Delta x^{2}/\alpha\) is illegal and leads to instability. We may interpret this time stepping as solving the equation system from \(-u_{xx}=f\) by iterating on a pseudo time variable.
3.3 Analysis of Schemes for the Diffusion Equation
The numerical experiments in Sect. 3.1.5 and 3.2.5 reveal that there are some numerical problems with the Forward Euler and Crank-Nicolson schemes: sawtooth-like noise is sometimes present in solutions that are, from a mathematical point of view, expected to be smooth. This section presents a mathematical analysis that explains the observed behavior and arrives at criteria for obtaining numerical solutions that reproduce the qualitative properties of the exact solutions. In short, we shall explain what is observed in Fig. 3.1–3.7.
3.3.1 Properties of the Solution
A particular characteristic of diffusive processes, governed by an equation like
is that the initial shape \(u(x,0)=I(x)\) spreads out in space with time, along with a decaying amplitude. Three different examples will illustrate the spreading of u in space and the decay in time.
Similarity solution
The diffusion equation (3.42) admits solutions that depend on \(\eta=(x-c)/\sqrt{4\alpha t}\) for a given value of c. One particular solution is
where
is the error function, and a and b are arbitrary constants. The error function lies in \((-1,1)\), is odd around η = 0, and goes relatively quickly to ±1:
As \(t\rightarrow 0\), the error function approaches a step function centered at x = c. For a diffusion problem posed on the unit interval \([0,1]\), we may choose the step at x = 1 ∕ 2 (meaning c = 1 ∕ 2), a = −1 ∕ 2, b = 1 ∕ 2. Then
where we have introduced the complementary error function \(\mbox{erfc}(\eta)=1-\mbox{erf}(\eta)\). The solution (3.45) implies the boundary conditions
For small enough t, \(u(0,t)\approx 1\) and \(u(1,t)\approx 0\), but as \(t\rightarrow\infty\), \(u(x,t)\rightarrow 1/2\) on \([0,1]\).
Solution for a Gaussian pulse
The standard diffusion equation \(u_{t}=\alpha u_{xx}\) admits a Gaussian function as solution:
At t = 0 this is a Dirac delta function, so for computational purposes one must start to view the solution at some time \(t=t_{\epsilon}> 0\). Replacing t by \(t_{\epsilon}+t\) in (3.48) makes it easy to operate with a (new) t that starts at t = 0 with an initial condition with a finite width. The important feature of (3.48) is that the standard deviation σ of a sharp initial Gaussian pulse increases in time according to \(\sigma=\sqrt{2\alpha t}\), making the pulse diffuse and flatten out.
Solution for a sine component
Also, (3.42) admits a solution of the form
The parameters Q and k can be freely chosen, while inserting (3.49) in (3.42) gives the constraint
A very important feature is that the initial shape \(I(x)=Q\sin\left(kx\right)\) undergoes a damping \(\exp{(-\alpha k^{2}t)}\), meaning that rapid oscillations in space, corresponding to large k, are very much faster dampened than slow oscillations in space, corresponding to small k. This feature leads to a smoothing of the initial condition with time. (In fact, one can use a few steps of the diffusion equation as a method for removing noise in signal processing.) To judge how good a numerical method is, we may look at its ability to smoothen or dampen the solution in the same way as the PDE does.
The following example illustrates the damping properties of (3.49). We consider the specific problem
The initial condition has been chosen such that adding two solutions like (3.49) constructs an analytical solution to the problem:
Figure 3.8 illustrates the rapid damping of rapid oscillations \(\sin(100\pi x)\) and the very much slower damping of the slowly varying \(\sin(\pi x)\) term. After about \(t=0.5\cdot 10^{-4}\) the rapid oscillations do not have a visible amplitude, while we have to wait until t ∼ 0.5 before the amplitude of the long wave \(\sin(\pi x)\) becomes very small.
Evolution of the solution of a diffusion problem: initial condition (upper left), 1/100 reduction of the small waves (upper right), 1/10 reduction of the long wave (lower left), and 1/100 reduction of the long wave (lower right)
3.3.2 Analysis of Discrete Equations
A counterpart to (3.49) is the complex representation of the same function:
where \(i=\sqrt{-1}\) is the imaginary unit. We can add such functions, often referred to as wave components, to make a Fourier representation of a general solution of the diffusion equation:
where K is a set of an infinite number of k values needed to construct the solution. In practice, however, the series is truncated and K is a finite set of k values needed to build a good approximate solution. Note that (3.50) is a special case of (3.51) where \(K=\{\pi,100\pi\}\), \(b_{\pi}=1\), and \(b_{100\pi}=0.1\).
The amplitudes b k of the individual Fourier waves must be determined from the initial condition. At t = 0 we have \(u\approx\sum_{k}b_{k}\exp{(ikx)}\) and find K and b k such that
(The relevant formulas for b k come from Fourier analysis, or equivalently, a least-squares method for approximating I(x) in a function space with basis \(\exp{(ikx)}\).)
Much insight about the behavior of numerical methods can be obtained by investigating how a wave component \(\exp{(-\alpha k^{2}t)}\exp{(ikx)}\) is treated by the numerical scheme. It appears that such wave components are also solutions of the schemes, but the damping factor \(\exp{(-\alpha k^{2}t)}\) varies among the schemes. To ease the forthcoming algebra, we write the damping factor as A n. The exact amplification factor corresponding to A is \(A_{\mbox{\footnotesize e}}=\exp{(-\alpha k^{2}\Delta t)}\).
3.3.3 Analysis of the Finite Difference Schemes
We have seen that a general solution of the diffusion equation can be built as a linear combination of basic components
A fundamental question is whether such components are also solutions of the finite difference schemes. This is indeed the case, but the amplitude \(\exp{(-\alpha k^{2}t)}\) might be modified (which also happens when solving the ODE counterpart \(u^{\prime}=-\alpha u\)). We therefore look for numerical solutions of the form
where the amplification factor A must be determined by inserting the component into an actual scheme. Note that A n means A raised to the power of n, n being the index in the time mesh, while the superscript n in \(u^{n}_{q}\) just denotes u at time t n .
Stability
The exact amplification factor is \(A_{\mbox{\footnotesize e}}=\exp{(-\alpha^{2}k^{2}\Delta t)}\). We should therefore require \(|A|<1\) to have a decaying numerical solution as well. If −1 ≤ A < 0, A n will change sign from time level to time level, and we get stable, non-physical oscillations in the numerical solutions that are not present in the exact solution.
Accuracy
To determine how accurately a finite difference scheme treats one wave component (3.53), we see that the basic deviation from the exact solution is reflected in how well A n approximates \(A_{\mbox{\footnotesize e}}^{n}\), or how well A approximates \(A_{\mbox{\footnotesize e}}\). We can plot \(A_{\mbox{\footnotesize e}}\) and the various expressions for A, and we can make Taylor expansions of \(A/A_{\mbox{\footnotesize e}}\) to see the error more analytically.
Truncation error
As an alternative to examining the accuracy of the damping of a wave component, we can perform a general truncation error analysis as explained in Appendix B. Such results are more general, but less detailed than what we get from the wave component analysis. The truncation error can almost always be computed and represents the error in the numerical model when the exact solution is substituted into the equations. In particular, the truncation error analysis tells the order of the scheme, which is of fundamental importance when verifying codes based on empirical estimation of convergence rates.
3.3.4 Analysis of the Forward Euler Scheme
The Forward Euler finite difference scheme for \(u_{t}=\alpha u_{xx}\) can be written as
Inserting a wave component (3.53) in the scheme demands calculating the terms
and
Inserting these terms in the discrete equation and dividing by \(A^{n}e^{ikq\Delta x}\) leads to
and consequently
where
is the numerical Fourier number, and \(p=k\Delta x/2\). The complete numerical solution is then
Stability
We easily see that A ≤ 1. However, the A can be less than −1, which will lead to growth of a numerical wave component. The criterion A ≥ −1 implies
The worst case is when \(\sin^{2}(p/2)=1\), so a sufficient criterion for stability is
or expressed as a condition on \(\Delta t\):
Note that halving the spatial mesh size, \(\Delta x\rightarrow{\frac{1}{2}}\Delta x\), requires \(\Delta t\) to be reduced by a factor of 1 ∕ 4. The method hence becomes very expensive for fine spatial meshes.
Accuracy
Since A is expressed in terms of F and the parameter we now call \(p=k\Delta x/2\), we should also express \(A_{\mbox{\footnotesize e}}\) by F and p. The exponent in \(A_{\mbox{\footnotesize e}}\) is \(-\alpha k^{2}\Delta t\), which equals \(-Fk^{2}\Delta x^{2}=-F4p^{2}\). Consequently,
All our A expressions as well as \(A_{\mbox{\footnotesize e}}\) are now functions of the two dimensionless parameters F and p.
Computing the Taylor series expansion of \(A/A_{\mbox{\footnotesize e}}\) in terms of F can easily be done with aid of sympy:
The result is
Recalling that \(F=\alpha\Delta t/\Delta x^{2}\), \(p=k\Delta x/2\), and that \(\sin^{2}p\leq 1\), we realize that the dominating terms in \(A/A_{\mbox{\footnotesize e}}\) are at most
Truncation error
We follow the theory explained in Appendix B. The recipe is to set up the scheme in operator notation and use formulas from Appendix B.2.4 to derive an expression for the residual. The details are documented in Appendix B.6.1. We end up with a truncation error
Although this is not the true error \(u_{\mbox{\footnotesize e}}(x_{i},t_{n})-u^{n}_{i}\), it indicates that the true error is of the form
for two unknown constants C t and C x .
3.3.5 Analysis of the Backward Euler Scheme
Discretizing \(u_{t}=\alpha u_{xx}\) by a Backward Euler scheme,
and inserting a wave component (3.53), leads to calculations similar to those arising from the Forward Euler scheme, but since
we get
and then
The complete numerical solution can be written
Stability
We see from (3.59) that 0 < A < 1, which means that all numerical wave components are stable and non-oscillatory for any \(\Delta t> 0\).
Truncation error
The derivation of the truncation error for the Backward Euler scheme is almost identical to that for the Forward Euler scheme. We end up with
3.3.6 Analysis of the Crank-Nicolson Scheme
The Crank-Nicolson scheme can be written as
or
Inserting (3.53) in the time derivative approximation leads to
Inserting (3.53) in the other terms and dividing by \(A^{n}e^{ikq\Delta x}\) gives the relation
and after some more algebra,
The exact numerical solution is hence
Stability
The criteria A > −1 and A < 1 are fulfilled for any \(\Delta t> 0\). Therefore, the solution cannot grow, but it will oscillate if \(1-2F\sin^{p}<0\). To avoid such non-physical oscillations, we must demand \(F\leq\frac{1}{2}\).
Truncation error
The truncation error is derived in Appendix B.6.1:
3.3.7 Analysis of the Leapfrog Scheme
An attractive feature of the Forward Euler scheme is the explicit time stepping and no need for solving linear systems. However, the accuracy in time is only \(\mathcal{O}(\Delta t)\). We can get an explicit second-order scheme in time by using the Leapfrog method:
Written out,
We need some formula for the first step, \(u^{1}_{q}\), but for that we can use a Forward Euler step.
Unfortunately, the Leapfrog scheme is always unstable for the diffusion equation. To see this, we insert a wave component \(A^{n}e^{ikx}\) and get
or
which has roots
Both roots have \(|A|> 1\) so the amplitude always grows, which is not in accordance with the physics of the problem. However, for a PDE with a first-order derivative in space, instead of a second-order one, the Leapfrog scheme performs very well. Details are provided in Sect. 4.1.3.
3.3.8 Summary of Accuracy of Amplification Factors
We can plot the various amplification factors against \(p=k\Delta x/2\) for different choices of the F parameter. Figures 3.9, 3.10, and 3.11 show how long and small waves are damped by the various schemes compared to the exact damping. As long as all schemes are stable, the amplification factor is positive, except for Crank-Nicolson when F > 0.5.
Amplification factors for large time steps
Amplification factors for time steps around the Forward Euler stability limit
Amplification factors for small time steps
The effect of negative amplification factors is that A n changes sign from one time level to the next, thereby giving rise to oscillations in time in an animation of the solution. We see from Fig. 3.9 that for F = 20, waves with p ≥ π ∕ 4 undergo a damping close to −1, which means that the amplitude does not decay and that the wave component jumps up and down (flips amplitude) in time. For F = 2 we have a damping of a factor of 0.5 from one time level to the next, which is very much smaller than the exact damping. Short waves will therefore fail to be effectively dampened. These waves will manifest themselves as high frequency oscillatory noise in the solution.
A value p = π ∕ 4 corresponds to four mesh points per wave length of e ikx, while p = π ∕ 2 implies only two points per wave length, which is the smallest number of points we can have to represent the wave on the mesh.
To demonstrate the oscillatory behavior of the Crank-Nicolson scheme, we choose an initial condition that leads to short waves with significant amplitude. A discontinuous I(x) will in particular serve this purpose: Figures 3.6 and 3.7 correspond to F = 3 and F = 10, respectively, and we see how short waves pollute the overall solution.
3.3.9 Analysis of the 2D Diffusion Equation
Diffusion in several dimensions is treated later, but it is appropriate to include the analysis here. We first consider the 2D diffusion equation
which has Fourier component solutions of the form
and the schemes have discrete versions of this Fourier component:
The Forward Euler scheme
For the Forward Euler discretization,
we get
Introducing
we can write the equation for ξ more compactly as
and solve for ξ:
The complete numerical solution for a wave component is
For stability we demand −1 ≤ ξ ≤ 1, and −1 ≤ ξ is the critical limit, since clearly ξ ≤ 1, and the worst case happens when the sines are at their maximum. The stability criterion becomes
For the special, yet common, case \(\Delta x=\Delta y=h\), the stability criterion can be written as
where d is the number of space dimensions: d = 1,2,3.
The Backward Euler scheme
The Backward Euler method,
results in
and
which is always in \((0,1]\). The solution for a wave component becomes
The Crank-Nicolson scheme
With a Crank-Nicolson discretization,
we have, after some algebra,
The fraction on the right-hand side is always less than 1, so stability in the sense of non-growing wave components is guaranteed for all physical and numerical parameters. However, the fraction can become negative and result in non-physical oscillations. This phenomenon happens when
A criterion against non-physical oscillations is therefore
which is the same limit as the stability criterion for the Forward Euler scheme.
The exact discrete solution is
3.3.10 Explanation of Numerical Artifacts
The behavior of the solution generated by Forward Euler discretization in time (and centered differences in space) is summarized at the end of Sect. 3.1.5. Can we, from the analysis above, explain the behavior?
We may start by looking at Fig. 3.3 where F = 0.51. The figure shows that the solution is unstable and grows in time. The stability limit for such growth is F = 0.5 and since the F in this simulation is slightly larger, growth is unavoidable.
Figure 3.1 has unexpected features: we would expect the solution of the diffusion equation to be smooth, but the graphs in Fig. 3.1 contain non-smooth noise. Turning to Fig. 3.4, which has a quite similar initial condition, we see that the curves are indeed smooth. The problem with the results in Fig. 3.1 is that the initial condition is discontinuous. To represent it, we need a significant amplitude on the shortest waves in the mesh. However, for F = 0.5, the shortest wave (p = π ∕ 2) gives the amplitude in the numerical solution as \((1-4F)^{n}\), which oscillates between negative and positive values at subsequent time levels for \(F> \frac{1}{4}\). Since the shortest waves have visible amplitudes in the solution profile, the oscillations becomes visible. The smooth initial condition in Fig. 3.4, on the other hand, leads to very small amplitudes of the shortest waves. That these waves then oscillate in a non-physical way for F = 0.5 is not a visible effect. The oscillations in time in the amplitude \((1-4F)^{n}\) disappear for \(F\leq\frac{1}{4}\), and that is why also the discontinuous initial condition always leads to smooth solutions in Fig. 3.2, where \(F=\frac{1}{4}\).
Turning the attention to the Backward Euler scheme and the experiments in Fig. 3.5, we see that even the discontinuous initial condition gives smooth solutions for F = 0.5 (and in fact all other F values). From the exact expression of the numerical amplitude, \((1+4F\sin^{2}p)^{-1}\), we realize that this factor can never flip between positive and negative values, and no instabilities can occur. The conclusion is that the Backward Euler scheme always produces smooth solutions. Also, the Backward Euler scheme guarantees that the solution cannot grow in time (unless we add a source term to the PDE, but that is meant to represent a physically relevant growth).
Finally, we have some small, strange artifacts when simulating the development of the initial plug profile with the Crank-Nicolson scheme, see Fig. 3.7, where F = 3. The Crank-Nicolson scheme cannot give growing amplitudes, but it may give oscillating amplitudes in time. The critical factor is \(1-2F\sin^{2}p\), which for the shortest waves (p = π ∕ 2) indicates a stability limit F = 0.5. With the discontinuous initial condition, we have enough amplitude on the shortest waves so their wrong behavior is visible, and this is what we see as small instabilities in Fig. 3.7. The only remedy is to lower the F value.
3.4 Exercises
Exercise 3.1 (Explore symmetry in a 1D problem)
This exercise simulates the exact solution (3.48). Suppose for simplicity that c = 0.
-
a)
Formulate an initial-boundary value problem that has (3.48) as solution in the domain \([-L,L]\). Use the exact solution (3.48) as Dirichlet condition at the boundaries. Simulate the diffusion of the Gaussian peak. Observe that the solution is symmetric around x = 0.
-
b)
Show from (3.48) that \(u_{x}(c,t)=0\). Since the solution is symmetric around x = c = 0, we can solve the numerical problem in half of the domain, using a symmetry boundary condition \(u_{x}=0\) at x = 0. Set up the initial-boundary value problem in this case. Simulate the diffusion problem in \([0,L]\) and compare with the solution in a).
Filename: diffu_symmetric_gaussian.
Exercise 3.2 (Investigate approximation errors from a \(u_{x}=0\) boundary condition)
We consider the problem solved in Exercise 3.1 part b). The boundary condition \(u_{x}(0,t)=0\) can be implemented in two ways: 1) by a standard symmetric finite difference \([D_{2x}u]_{i}^{n}=0\), or 2) by a one-sided difference \([D^{+}u=0]^{n}_{i}=0\). Investigate the effect of these two conditions on the convergence rate in space.
Hint
If you use a Forward Euler scheme, choose a discretization parameter \(h=\Delta t=\Delta x^{2}\) and assume the error goes like \(E\sim h^{r}\). The error in the scheme is \(\mathcal{O}(\Delta t,\Delta x^{2})\) so one should expect that the estimated r approaches 1. The question is if a one-sided difference approximation to \(u_{x}(0,t)=0\) destroys this convergence rate.
Filename: diffu_onesided_fd.
Exercise 3.3 (Experiment with open boundary conditions in 1D)
We address diffusion of a Gaussian function as in Exercise 3.1, in the domain \([0,L]\), but now we shall explore different types of boundary conditions on x = L. In real-life problems we do not know the exact solution on x = L and must use something simpler.
-
a)
Imagine that we want to solve the problem numerically on \([0,L]\), with a symmetry boundary condition \(u_{x}=0\) at x = 0, but we do not know the exact solution and cannot of that reason assign a correct Dirichlet condition at x = L. One idea is to simply set \(u(L,t)=0\) since this will be an accurate approximation before the diffused pulse reaches x = L and even thereafter it might be a satisfactory condition if the exact u has a small value. Let \(u_{\mbox{\footnotesize e}}\) be the exact solution and let u be the solution of \(u_{t}=\alpha u_{xx}\) with an initial Gaussian pulse and the boundary conditions \(u_{x}(0,t)=u(L,t)=0\). Derive a diffusion problem for the error \(e=u_{\mbox{\footnotesize e}}-u\). Solve this problem numerically using an exact Dirichlet condition at x = L. Animate the evolution of the error and make a curve plot of the error measure
$$E(t)=\sqrt{\frac{\int_{0}^{L}e^{2}dx}{\int_{0}^{L}udx}}\thinspace.$$Is this a suitable error measure for the present problem?
-
b)
Instead of using \(u(L,t)=0\) as approximate boundary condition for letting the diffused Gaussian pulse move out of our finite domain, one may try \(u_{x}(L,t)=0\) since the solution for large t is quite flat. Argue that this condition gives a completely wrong asymptotic solution as \(t\rightarrow 0\). To do this, integrate the diffusion equation from 0 to L, integrate u xx by parts (or use Gauss’ divergence theorem in 1D) to arrive at the important property
$$\frac{d}{dt}\int_{0}^{L}u(x,t)dx=0,$$implying that \(\int_{0}^{L}udx\) must be constant in time, and therefore
$$\int_{0}^{L}u(x,t)dx=\int_{0}^{L}I(x)dx\thinspace.$$The integral of the initial pulse is 1.
-
c)
Another idea for an artificial boundary condition at x = L is to use a cooling law
$$-\alpha u_{x}=q(u-u_{S}),$$(3.68)where q is an unknown heat transfer coefficient and u S is the surrounding temperature in the medium outside of \([0,L]\). (Note that arguing that u S is approximately \(u(L,t)\) gives the \(u_{x}=0\) condition from the previous subexercise that is qualitatively wrong for large t.) Develop a diffusion problem for the error in the solution using (3.68) as boundary condition. Assume one can take \(u_{S}=0\) ‘‘outside the domain’’ since \(u_{\mbox{\footnotesize e}}\rightarrow 0\) as \(x\rightarrow\infty\). Find a function \(q=q(t)\) such that the exact solution obeys the condition (3.68). Test some constant values of q and animate how the corresponding error function behaves. Also compute E(t) curves as defined above.
Filename: diffu_open_BC.
Exercise 3.4 (Simulate a diffused Gaussian peak in 2D/3D)
-
a)
Generalize (3.48) to multi dimensions by assuming that one-dimensional solutions can be multiplied to solve \(u_{t}=\alpha\nabla^{2}u\). Set c = 0 such that the peak of the Gaussian is at the origin.
-
b)
One can from the exact solution show that \(u_{x}=0\) on x = 0, \(u_{y}=0\) on y = 0, and \(u_{z}=0\) on z = 0. The approximately correct condition u = 0 can be set on the remaining boundaries (say x = L, y = L, z = L), cf. Exercise 3.3. Simulate a 2D case and make an animation of the diffused Gaussian peak.
-
c)
The formulation in b) makes use of symmetry of the solution such that we can solve the problem in the first quadrant (2D) or octant (3D) only. To check that the symmetry assumption is correct, formulate the problem without symmetry in a domain \([-L,L]\times[L,L]\) in 2D. Use u = 0 as approximately correct boundary condition. Simulate the same case as in b), but in a four times as large domain. Make an animation and compare it with the one in b).
Filename: diffu_symmetric_gaussian_2D.
Exercise 3.5 (Examine stability of a diffusion model with a source term)
Consider a diffusion equation with a linear u term:
-
a)
Derive in detail the Forward Euler, Backward Euler, and Crank-Nicolson schemes for this type of diffusion model. Thereafter, formulate a θ-rule to summarize the three schemes.
-
b)
Assume a solution like (3.49) and find the relation between a, k, α, and β.
Hint
Insert (3.49) in the PDE problem.
-
c)
Calculate the stability of the Forward Euler scheme. Design numerical experiments to confirm the results.
Hint
Insert the discrete counterpart to (3.49) in the numerical scheme. Run experiments at the stability limit and slightly above.
-
a)
Repeat c) for the Backward Euler scheme.
-
b)
Repeat c) for the Crank-Nicolson scheme.
-
c)
How does the extra term bu impact the accuracy of the three schemes?
Hint
For analysis of the accuracy, compare the numerical and exact amplification factors, in graphs and/or by Taylor series expansion.
Filename: diffu_stability_uterm.
3.5 Diffusion in Heterogeneous Media
Diffusion in heterogeneous media normally implies a non-constant diffusion coefficient \(\alpha=\alpha(x)\). A 1D diffusion model with such a variable diffusion coefficient reads
A short form of the diffusion equation with variable coefficients is \(u_{t}=(\alpha u_{x})_{x}+f\).
3.5.1 Discretization
We can discretize (3.69) by a θ-rule in time and centered differences in space:
Written out, this becomes
where, e.g., an arithmetic mean can to be used for \(\alpha_{i+\frac{1}{2}}\):
3.5.2 Implementation
Suitable code for solving the discrete equations is very similar to what we created for a constant α. Since the Fourier number has no meaning for varying α, we introduce a related parameter \(D=\Delta t/\Delta x^{2}\).
The code is found in the file diffu1D_vc.py .
3.5.3 Stationary Solution
As \(t\rightarrow\infty\), the solution of the problem (3.69)–(3.72) will approach a stationary limit where \(\partial u/\partial t=0\). The governing equation is then
with boundary conditions \(u(0)=U_{0}\) and \(u(L)=U_{L}\). It is possible to obtain an exact solution of (3.73) for any α. Integrating twice and applying the boundary conditions to determine the integration constants gives
3.5.4 Piecewise Constant Medium
Consider a medium built of M layers. The layer boundaries are denoted \(b_{0},\ldots,b_{M}\), where \(b_{0}=0\) and \(b_{M}=L\). If the layers potentially have different material properties, but these properties are constant within each layer, we can express α as a piecewise constant function according to
The exact solution (3.74) in case of such a piecewise constant α function is easy to derive. Assume that x is in the m-th layer: \(x\in[b_{m},b_{m+1}]\). In the integral \(\int_{0}^{x}(a(\xi))^{-1}d\xi\) we must integrate through the first m − 1 layers and then add the contribution from the remaining part \(x-b_{m}\) into the m-th layer:
Remark
It may sound strange to have a discontinuous α in a differential equation where one is to differentiate, but a discontinuous α is compensated by a discontinuous u x such that \(\alpha u_{x}\) is continuous and therefore can be differentiated as \((\alpha u_{x})_{x}\).
3.5.5 Implementation of Diffusion in a Piecewise Constant Medium
Programming with piecewise function definitions quickly becomes cumbersome as the most naive approach is to test for which interval x lies, and then start evaluating a formula like (3.76). In Python, vectorized expressions may help to speed up the computations. The convenience classes PiecewiseConstant and IntegratedPiecewiseConstant in the Heaviside module were made to simplify programming with functions like (3.75) and expressions like (3.76). These utilities not only represent piecewise constant functions, but also smoothed versions of them where the discontinuities can be smoothed out in a controlled fashion.
The PiecewiseConstant class is created by sending in the domain as a 2-tuple or 2-list and a data object describing the boundaries \(b_{0},\ldots,b_{M}\) and the corresponding function values \(\alpha_{0},\ldots,\alpha_{M-1}\). More precisely, data is a nested list, where data[i][0] holds b i and data[i][1] holds the corresponding value α i , for \(i=0,\ldots,M-1\). Given b i and α i in arrays b and a, it is easy to fill out the nested list data.
In our application, we want to represent α and 1 ∕ α as piecewise constant functions, in addition to the u(x) function which involves the integrals of 1 ∕ α. A class creating the functions we need and a method for evaluating u, can take the form
A visualization method is also convenient to have. Below we plot u(x) along with \(\alpha(x)\) (which works well as long as \(\max\alpha(x)\) is of the same size as \(\max u=\max(U_{0},U_{L})\)).
Figure 3.12 shows the case where
Solution of the stationary diffusion equation corresponding to a piecewise constant diffusion coefficient
By adding the eps parameter to the constructor of the SerialLayers class, we can experiment with smoothed versions of α and see the (small) impact on u. Figure 3.13 shows the result.
Solution of the stationary diffusion equation corresponding to a smoothed piecewise constant diffusion coefficient
3.5.6 Axi-Symmetric Diffusion
Suppose we have a diffusion process taking place in a straight tube with radius R. We assume axi-symmetry such that u is just a function of r and t, with r being the radial distance from the center axis of the tube to a point. With such axi-symmetry it is advantageous to introduce cylindrical coordinates r, θ, and z, where z is in the direction of the tube and \((r,\theta)\) are polar coordinates in a cross section. Axi-symmetry means that all quantities are independent of θ. From the relations \(x=\cos\theta\), \(y=\sin\theta\), and z = z, between Cartesian and cylindrical coordinates, one can (with some effort) derive the diffusion equation in cylindrical coordinates, which with axi-symmetry takes the form
Let us assume that u does not change along the tube axis so it suffices to compute variations in a cross section. Then \(\partial u/\partial z=0\) and we have a 1D diffusion equation in the radial coordinate r and time t. In particular, we shall address the initial-boundary value problem
The condition (3.78) is a necessary symmetry condition at r = 0, while (3.79) could be any Dirichlet or Neumann condition (or Robin condition in case of cooling or heating).
The finite difference approximation will need the discretized version of the PDE for r = 0 (just as we use the PDE at the boundary when implementing Neumann conditions). However, discretizing the PDE at r = 0 poses a problem because of the 1 ∕ r factor. We therefore need to work out the PDE for discretization at r = 0 with care. Let us, for the case of constant α, expand the spatial derivative term to
The last term faces a difficulty at r = 0, since it becomes a 0 ∕ 0 expression caused by the symmetry condition at r = 0. However, L’Hosptial’s rule can be used:
The PDE at r = 0 therefore becomes
For a variable coefficient \(\alpha(r)\) the expanded spatial derivative term reads
We are interested in this expression for r = 0. A necessary condition for u to be axi-symmetric is that all input data, including α, must also be axi-symmetric, implying that \(\alpha^{\prime}(0)=0\) (the second term vanishes anyway because of r = 0). The limit of interest is
The PDE at r = 0 now looks like
so there is no essential difference between the constant coefficient and variable coefficient cases.
The second-order derivative in (3.81) and (3.82) is discretized in the usual way.
The fictitious value \(u^{n}_{-1}\) can be eliminated using the discrete symmetry condition
which then gives the modified approximation to the term with the second-order derivative of u in r at r = 0:
The discretization of the term with the second-order derivative in r at any internal mesh point is straightforward:
To complete the discretization, we need a scheme in time, but that can be done as before and does not interfere with the discretization in space.
3.5.7 Spherically-Symmetric Diffusion
Discretization in spherical coordinates
Let us now pose the problem from Sect. 3.5.6 in spherical coordinates, where u only depends on the radial coordinate r and time t. That is, we have spherical symmetry. For simplicity we restrict the diffusion coefficient α to be a constant. The PDE reads
for \(r\in(0,R)\) and \(t\in(0,T]\). The parameter γ is 2 for spherically-symmetric problems and 1 for axi-symmetric problems. The boundary and initial conditions have the same mathematical form as in (3.77)–(3.80).
Since the PDE in spherical coordinates has the same form as the PDE in Sect. 3.5.6, just with the γ parameter being different, we can use the same discretization approach. At the origin r = 0 we get problems with the term
but L’Hosptial’s rule shows that this term equals \(\gamma\partial^{2}u/\partial r^{2}\), and the PDE at r = 0 becomes
The associated discrete form is then
for a Crank-Nicolson scheme.
Discretization in Cartesian coordinates
The spherically-symmetric spatial derivative can be transformed to the Cartesian counterpart by introducing
Inserting u = v ∕ r in
yields
The two terms in the parenthesis can be combined to
The PDE for v takes the form
For α constant we immediately realize that we can reuse a solver in Cartesian coordinates to compute v. With variable α, a ‘‘reaction’’ term v ∕ r needs to be added to the Cartesian solver. The boundary condition \(\partial u/\partial r=0\) at r = 0, implied by symmetry, forces \(v(0,t)=0\), because
3.6 Diffusion in 2D
We now address diffusion in two space dimensions:
in a domain
with u = 0 on the boundary and \(u(x,y,0)=I(x,y)\) as initial condition.
3.6.1 Discretization
For generality, it is natural to use a θ-rule for the time discretization. Standard, second-order accurate finite differences are used for the spatial derivatives. We sample the PDE at a space-time point \((i,j,n+\frac{1}{2})\) and apply the difference approximations:
Written out,
We collect the unknowns on the left-hand side
where
are the Fourier numbers in x and y direction, respectively.
3.6.2 Numbering of Mesh Points Versus Equations and Unknowns
The equations (3.91) are coupled at the new time level n + 1. That is, we must solve a system of (linear) algebraic equations, which we will write as Ac = b, where A is the coefficient matrix, c is the vector of unknowns, and b is the right-hand side.
Let us examine the equations in Ac = b on a mesh with \(N_{x}=3\) and \(N_{y}=2\) cells in the respective spatial directions. The spatial mesh is depicted in Fig. 3.14. The equations at the boundary just implement the boundary condition u = 0:
We are left with two interior points, with i = 1, j = 1 and i = 2, j = 1. The corresponding equations are
3x2 2D mesh
There are in total 12 unknowns \(u^{n+1}_{i,j}\) for i = 0,1,2,3 and j = 0,1,2. To solve the equations, we need to form a matrix system Ac = b. In that system, the solution vector c can only have one index. Thus, we need a numbering of the unknowns with one index, not two as used in the mesh. We introduce a mapping \(m(i,j)\) from a mesh point with indices \((i,j)\) to the corresponding unknown p in the equation system:
When i and j run through their values, we see the following mapping to p:
That is, we number the points along the x axis, starting with y = 0, and then progress one ‘‘horizontal’’ mesh line at a time. In Fig. 3.14 you can see that the \((i,j)\) and the corresponding single index (p) are listed for each mesh point.
We could equally well have numbered the equations in other ways, e.g., let the j index be the fastest varying index: \(p=m(i,j)=i(N_{y}+1)+j\).
Let us form the coefficient matrix A, or more precisely, insert a matrix element (according Python’s convention with zero as base index) for each of the nonzero elements in A (the indices run through the values of p, i.e., \(p=0,\ldots,11\)):
Here is a more compact visualization of the coefficient matrix where we insert dots for zeros and bullets for non-zero elements:
It is clearly seen that most of the elements are zero. This is a general feature of coefficient matrices arising from discretizing PDEs by finite difference methods. We say that the matrix is sparse.
Let A p,q be the value of element \((p,q)\) in the coefficient matrix A, where p and q now correspond to the numbering of the unknowns in the equation system. We have \(A_{p,q}=1\) for p = q = 0,1,2,3,4,7,8,9,10,11, corresponding to all the known boundary values. Let p be \(m(i,j)\), i.e., the single index corresponding to mesh point \((i,j)\). Then we have
for the equations associated with the two interior mesh points. At these interior points, the single index p takes on the specific values p = 5,6, corresponding to the values \((1,1)\) and \((1,2)\) of the pair \((i,j)\).
The above values for A p,q can be inserted in the matrix:
The corresponding right-hand side vector in the equation system has the entries b p , where p numbers the equations. We have
for the boundary values. For the equations associated with the interior points, we get for p = 5,6, corresponding to i = 1,2 and j = 1:
Recall that \(p=m(i,j)=j(N_{x}+1)+j\) in this expression.
We can, as an alternative, leave the boundary mesh points out of the matrix system. For a mesh with \(N_{x}=3\) and \(N_{y}=2\) there are only two internal mesh points whose unknowns will enter the matrix system. We must now number the unknowns at the interior points:
for \(i=1,\ldots,N_{x}-1\), \(j=1,\ldots,N_{y}-1\).
We can continue with illustrating a bit larger mesh, \(N_{x}=4\) and \(N_{y}=3\), see Fig. 3.15 . The corresponding coefficient matrix with dots for zeros and bullets for non-zeroes looks as follows (values at boundary points are included in the equation system):
4x3 2D mesh
The coefficient matrix is banded
Besides being sparse, we observe that the coefficient matrix is banded: it has five distinct bands. We have the diagonal A i,i, the subdiagonal \(A_{i-1,j}\), the superdiagonal \(A_{i,i+1}\), a lower diagonal \(A_{i,i-(Nx+1)}\), and an upper diagonal \(A_{i,i+(Nx+1)}\). The other matrix entries are known to be zero. With \(N_{x}+1=N_{y}+1=N\), only a fraction 5N−2 of the matrix entries are nonzero, so the matrix is clearly very sparse for relevant N values. The more we can compute with the nonzeros only, the faster the solution methods will potentially be.
3.6.3 Algorithm for Setting Up the Coefficient Matrix
We looked at a specific mesh in the previous section, formulated the equations, and saw what the corresponding coefficient matrix and right-hand side are. Now our aim is to set up a general algorithm, for any choice of N x and N y , that produces the coefficient matrix and the right-hand side vector. We start with a zero matrix and vector, run through each mesh point, and fill in the values depending on whether the mesh point is an interior point or on the boundary.
-
for \(i=0,\ldots,N_{x}\)
-
for \(j=0,\ldots,N_{y}\)
-
\(p=j(N_{x}+1)+i\)
-
if point \((i,j)\) is on the boundary:
-
\(A_{p,p}=1\), \(b_{p}=0\)
-
-
else:
-
fill \(A_{p,m(i-1,j)}\), \(A_{p,m(i+1,j)}\), \(A_{p,m(i,j)}\), \(A_{p,m(i,j-1)}\), \(A_{p,m(i,j+1)}\), and b p
-
-
-
To ease the test on whether \((i,j)\) is on the boundary or not, we can split the loops a bit, starting with the boundary line j = 0, then treat the interior lines \(1\leq j<N_{y}\), and finally treat the boundary line \(j=N_{y}\):
-
for \(i=0,\ldots,N_{x}\)
-
boundary j = 0: \(p=j(N_{x}+1)+i\), \(A_{p,p}=1\)
-
-
for \(j=0,\ldots,N_{y}\)
-
boundary i = 0: \(p=j(N_{x}+1)+i\), \(A_{p,p}=1\)
-
for \(i=1,\ldots,N_{x}-1\)
-
interior point \(p=j(N_{x}+1)+i\)
-
fill \(A_{p,m(i-1,j)}\), \(A_{p,m(i+1,j)}\), \(A_{p,m(i,j)}\), \(A_{p,m(i,j-1)}\), \(A_{p,m(i,j+1)}\), and b p
-
-
boundary \(i=N_{x}\): \(p=j(N_{x}+1)+i\), \(A_{p,p}=1\)
-
-
for \(i=0,\ldots,N_{x}\)
-
boundary \(j=N_{y}\): \(p=j(N_{x}+1)+i\), \(A_{p,p}=1\)
-
The right-hand side is set up as follows.
-
for \(i=0,\ldots,N_{x}\)
-
boundary j = 0: \(p=j(N_{x}+1)+i\), \(b_{p}=0\)
-
-
for \(j=0,\ldots,N_{y}\)
-
boundary i = 0: \(p=j(N_{x}+1)+i\), \(b_{p}=0\)
-
for \(i=1,\ldots,N_{x}-1\)
-
interior point \(p=j(N_{x}+1)+i\)
-
fill b p
-
-
boundary \(i=N_{x}\): \(p=j(N_{x}+1)+i\), \(b_{p}=0\)
-
-
for \(i=0,\ldots,N_{x}\)
-
boundary \(j=N_{y}\): \(p=j(N_{x}+1)+i\), \(b_{p}=0\)
-
3.6.4 Implementation with a Dense Coefficient Matrix
The goal now is to map the algorithms in the previous section to Python code. One should, for computational efficiency reasons, take advantage of the fact that the coefficient matrix is sparse and/or banded, i.e., take advantage of all the zeros. However, we first demonstrate how to fill an N × N dense square matrix, where N is the number of unknowns, here \(N=(N_{x}+1)(N_{y}+1)\). The dense matrix is much easier to understand than the sparse matrix case.
The \(u^{n+1}_{i,j}\) and \(u^{n}_{i,j}\) mesh functions are represented by their spatial values at the mesh points:
It is a good habit (for extensions) to introduce index sets for all mesh points:
The initial condition is easy to fill in:
The memory for the coefficient matrix and right-hand side vector is allocated by
The filling of A goes like this:
Since A is independent of time, it can be filled once and for all before the time loop. The right-hand side vector must be filled at each time level inside the time loop:
We use solve from scipy.linalg and not from numpy.linalg. The difference is stated below.
scipy.linalg versus numpy.linalg
Quote from the SciPy documentation Footnote 2:
scipy.linalg contains all the functions in numpy.linalg plus some other more advanced ones not contained in numpy.linalg.
Another advantage of using scipy.linalg over numpy.linalg is that it is always compiled with BLAS/LAPACK support, while for NumPy this is optional. Therefore, the SciPy version might be faster depending on how NumPy was installed.
Therefore, unless you don’t want to add SciPy as a dependency to your NumPy program, use scipy.linalg instead of numpy.linalg.
The code shown above is available in the solver_dense function in the file diffu2D_u0.py , differing only in the boundary conditions, which in the code can be an arbitrary function along each side of the domain.
We do not bother to look at vectorized versions of filling A since a dense matrix is just used of pedagogical reasons for the very first implementation. Vectorization will be treated when A has a sparse matrix representation, as in Sect. 3.6.7 .
How to debug the computation of A and b
A good starting point for debugging the filling of A and b is to choose a very coarse mesh, say \(N_{x}=N_{y}=2\), where there is just one internal mesh point, compute the equations by hand, and print out A and b for comparison in the code. If wrong elements in A or b occur, print out each assignment to elements in A and b inside the loops and compare with what you expect.
To let the user store, analyze, or visualize the solution at each time level, we include a callback function, named user_action, to be called before the time loop and in each pass in that loop. The function has the signature
where u is a two-dimensional array holding the solution at time level n and time t[n]. The x and y coordinates of the mesh points are given by the arrays x and y, respectively. The arrays xv and yv are vectorized representations of the mesh points such that vectorized function evaluations can be invoked. The xv and yv arrays are defined by
One can then evaluate, e.g., \(f(x,y,t)\) at all internal mesh points at time level n by first evaluating f at all points,
and then use slices to extract a view of the values at the internal mesh points: f_a[1:-1,1:-1]. The next section features an example on writing a user_action callback function.
3.6.5 Verification: Exact Numerical Solution
A good test example to start with is one that preserves the solution u = 0, i.e., f = 0 and \(I(x,y)=0\). This trivial solution can uncover some bugs.
The first real test example is based on having an exact solution of the discrete equations. This solution is linear in time and quadratic in space:
Inserting this manufactured solution in the PDE shows that the source term f must be
We can use the user_action function to compare the numerical solution with the exact solution at each time level. A suitable helper function for checking the solution goes like this:
A true test function for checking the quadratic solution for several different meshes and θ values can take the form
3.6.6 Verification: Convergence Rates
For 2D verification with convergence rate computations, the expressions and computations just build naturally on what we saw for 1D diffusion. Truncation error analysis and other forms of error analysis point to a numerical error formula like
where p, C t , C x , and C y are constants. Often, the analysis of a Crank-Nicolson method can show that p = 2, while the Forward and Backward Euler schemes have p = 1.
When checking the error formula empirically, we need to reduce it to a form \(E=Ch^{r}\) with a single discretization parameter h and some rate r to be estimated. For the Backward Euler method, where p = 1, we can introduce a single discretization parameter according to
where K is a constant. The error formula then becomes
The simplest choice is obviously K = 1. With the Forward Euler method, however, stability requires \(\Delta t=hK\leq h/(4\alpha)\), so \(K\leq 1/(4\alpha)\).
For the Crank-Nicolson method, p = 2, and we can simply choose
since there is no restriction on \(\Delta t\) in terms of \(\Delta x\) and \(\Delta y\).
A frequently used error measure is the \(\ell^{2}\) norm of the error mesh point values. Section 2.2.3 and the formula (2.26) shows the error measure for a 1D time-dependent problem. The extension to the current 2D problem reads
One attractive manufactured solution is
where p can be arbitrary. The required source term is
The function convergence_rates in diffu2D_u0.py implements a convergence rate test. Two potential difficulties are important to be aware of:
-
1.
The error formula is assumed to be correct when \(h\rightarrow 0\), so for coarse meshes the estimated rate r may be somewhat away from the expected value. Fine meshes may lead to prohibitively long execution times.
-
2.
Choosing \(p=\alpha(k_{x}^{2}+k_{y}^{2})\) in the manufactured solution above seems attractive (f = 0), but leads to a slower approach to the asymptotic range where the error formula is valid (i.e., r fluctuates and needs finer meshes to stabilize).
3.6.7 Implementation with a Sparse Coefficient Matrix
We used a sparse matrix implementation in Sect. 3.2.2 for a 1D problem with a tridiagonal matrix. The present matrix, arising from a 2D problem, has five diagonals, but we can use the same sparse matrix data structure scipy.sparse.diags.
Understanding the diagonals
Let us look closer at the diagonals in the example with a 4 × 3 mesh as depicted in Fig. 3.15 and its associated matrix visualized by dots for zeros and bullets for nonzeros. From the example mesh, we may generalize to an \(N_{x}\times N_{y}\) mesh.
The main diagonal has \(N=(N_{x}+1)(N_{y}+1)\) elements, while the sub- and super-diagonals have N − 1 elements. By looking at the matrix above, we realize that the lower diagonal starts in row \(N_{x}+1\) and goes to row N, so its length is \(N-(N_{x}+1)\). Similarly, the upper diagonal starts at row 0 and lasts to row \(N-(N_{x}+1)\), so it has the same length. Based on this information, we declare the diagonals by
Filling the diagonals
We run through all mesh points and fill in elements on the various diagonals. The line of mesh points corresponding to j = 0 are all on the boundary, and only the main diagonal gets a contribution:
Then we run through all interior \(j=\hbox{const}\) lines of mesh points. The first and the last point on each line, i = 0 and \(i=N_{x}\), correspond to boundary points:
For the interior mesh points \(i=1,\ldots,N_{x}-1\) on a mesh line \(y=\hbox{const}\) we can start with the main diagonal. The entries to be filled go from i = 1 to \(i=N_{x}-1\) so the relevant slice in the main vector is m(1,j):m(Nx,j):
The upper array for the superdiagonal has its index 0 corresponding to row 0 in the matrix, and the array entries to be set go from \(m(1,j)\) to \(m(N_{x}-1,j)\):
The subdiagonal (lower array), however, has its index 0 corresponding to row 1, so there is an offset of 1 in indices compared to the matrix. The first nonzero occurs (interior point) at a mesh line \(j=\hbox{const}\) corresponding to matrix row \(m(1,j)\), and the corresponding array index in lower is then \(m(1,j)\). To fill the entries from \(m(1,j)\) to \(m(N_{x}-1,j)\) we set the following slice in lower:
For the upper diagonal, its index 0 corresponds to matrix row 0, so there is no offset and we can set the entries correspondingly to upper:
The lower2 diagonal, however, has its first index 0 corresponding to row \(N_{x}+1\), so here we need to subtract the offset \(N_{x}+1\):
We can now summarize the above code lines for setting the entries in the sparse matrix representation of the coefficient matrix:
The next task is to create the sparse matrix from these diagonals:
Filling the right-hand side; scalar version
Setting the entries in the right-hand side is easier, since there are no offsets in the array to take into account. The right-hand side is in fact similar to the one previously shown, when we used a dense matrix representation (the right-hand side vector is, of course, independent of what type of representation we use for the coefficient matrix). The complete time loop goes as follows.
Filling the right-hand side; vectorized version.
Since we use a sparse matrix and try to speed up the computations, we should examine the loops and see if some can be easily removed by vectorization. In the filling of A we have already used vectorized expressions at each \(j=\hbox{const}\) line of mesh points. We can very easily do the same in the code above and remove the need for loops over the i index:
The most tricky part of this code snippet is the loading of values from the one-dimensional array c into the two-dimensional array u. With our numbering of unknowns from left to right along ‘‘horizontal’’ mesh lines, the correct reordering of the one-dimensional array c as a two-dimensional array requires first a reshaping to an (Ny+1,Nx+1) two-dimensional array and then taking the transpose. The result is an (Nx+1,Ny+1) array compatible with u both in size and appearance of the function values.
The spsolve function in scipy.sparse.linalg is an efficient version of Gaussian elimination suited for matrices described by diagonals. The algorithm is known as sparse Gaussian elimination, and spsolve calls up a well-tested C code called SuperLU Footnote 3.
The complete code utilizing spsolve is found in the solver_sparse function in the file diffu2D_u0.py .
Verification
We can easily extend the function quadratic from Sect. 3.6.5 to include a test of the solver_sparse function as well.
3.6.8 The Jacobi Iterative Method
So far we have created a matrix and right-hand side of a linear system Ac = b and solved the system for c by calling an exact algorithm based on Gaussian elimination. A much simpler implementation, which requires no memory for the coefficient matrix A, arises if we solve the system by iterative methods. These methods are only approximate, and the core algorithm is repeated many times until the solution is considered to be converged.
Numerical scheme and linear system
To illustrate the idea of the Jacobi method, we simplify the numerical scheme to the Backward Euler case, θ = 1, so there are fewer terms to write:
The idea of the Jacobi iterative method is to introduce an iteration, here with index r, where we in each iteration treat \(u^{n+1}_{i,j}\) as unknown, but use values from the previous iteration for the other unknowns \(u^{n+1}_{i\pm 1,j\pm 1}\).
Iterations
Let \(u^{n+1,r}_{i,j}\) be the approximation to \(u^{n+1}_{i,j}\) in iteration r, for all relevant i and j indices. We first solve with respect to \(u^{n+1}_{i,j}\) to get the equation to solve:
The iteration is introduced by using iteration index r, for computed values, on the right-hand side and r + 1 (unknown in this iteration) on the left-hand side:
Initial guess
We start the iteration with the computed values at the previous time level:
Relaxation
A common technique in iterative methods is to introduce a relaxation, which means that the new approximation is a weighted mean of the approximation as suggested by the algorithm and the previous approximation. Naming the quantity on the left-hand side of (3.99) as \(u^{n+1,*}_{i,j}\), a new approximation based on relaxation reads
Under-relaxation means ω < 1, while over-relaxation has ω > 1.
Stopping criteria
The iteration can be stopped when the change from one iteration to the next is sufficiently small (≤ϵ), using either an infinity norm,
or an L 2 norm,
Another widely used criterion measures how well the equations are solved by looking at the residual (essentially \(b-Ac^{r+1}\) if \(c^{r+1}\) is the approximation to the solution in iteration r + 1). The residual, defined in terms of the finite difference stencil, is
One can then iterate until the norm of the mesh function R i,j is less than some tolerance:
Code-friendly notation
To make the mathematics as close as possible to what we will write in a computer program, we may introduce some new notation: u i,j is a short notation for \(u^{n+1,r+1}_{i,j}\), \(u^{-}_{i,j}\) is a short notation for \(u^{n+1,r}_{i,j}\), and \(u^{(s)}_{i,j}\) denotes \(u^{n+1-s}_{i,j}\). That is, u i,j is the unknown, \(u^{-}_{i,j}\) is its most recently computed approximation, and s counts time levels backwards in time. The Jacobi method (3.99) takes the following form with the new notation:
Generalization of the scheme
We can also quite easily introduce the θ rule for discretization in time and write up the Jacobi iteration in that case as well:
The final update of u applies relaxation:
3.6.9 Implementation of the Jacobi Method
The Jacobi method needs no coefficient matrix and right-hand side vector, but it needs an array for u in the previous iteration. We call this array u_, using the notation at the end of the previous section (at the same time level). The unknown itself is called u, while u_n is the computed solution one time level back in time. With a θ rule in time, the time loop can be coded like this:
The vectorized version should be quite straightforward to understand once one has an understanding of how a standard 2D finite stencil is vectorized.
The first natural verification is to use the test problem in the function quadratic from Sect. 3.6.5. This problem is known to have no approximation error, but any iterative method will produce an approximate solution with unknown error. For a tolerance 10−k in the iterative method, we can, e.g., use a slightly larger tolerance \(10^{-(k-1)}\) for the difference between the exact and the computed solution.
Even for a very coarse 4 × 4 mesh, the Jacobi method requires 26 iterations to reach a tolerance of 10−5, which is quite many iterations, given that there are only 25 unknowns.
3.6.10 Test Problem: Diffusion of a Sine Hill
It can be shown that
is a solution of the 2D homogeneous diffusion equation \(u_{t}=\alpha(u_{xx}+u_{yy})\) in a rectangle \([0,L_{x}]\times[0,L_{y}]\), for any value of the amplitude A. This solution vanishes at the boundaries, and the initial condition is the product of two sines. We may choose A = 1 for simplicity.
It is difficult to know if our solver based on the Jacobi method works properly since we are faced with two sources of errors: one from the discretization, E Δ, and one from the iterative Jacobi method, E i . The total error in the computed u can be represented as
One error measure is to look at the maximum value, which is obtained for the midpoint \(x=L_{x}/2\) and \(y=L_{x}/2\). This midpoint is represented in the discrete u if N x and N y are even numbers. We can then compute E u as \(E_{u}=|\max u_{\mbox{\footnotesize e}}-\max u|\), when we know an exact solution \(u_{\mbox{\footnotesize e}}\) of the problem.
What about E Δ? If we use the maximum value as a measure of the error, we have in fact analytical insight into the approximation error in this particular problem. According to Sect. 3.3.9, the exact solution (3.108) of the PDE problem is also an exact solution of the discrete equations, except that the damping factor in time is different. More precisely, (3.66) and (3.67) are solutions of the discrete problem for θ = 1 (Backward Euler) and \(\theta=\frac{1}{2}\) (Crank-Nicolson), respectively. The factors raised to the power n is the numerical amplitude, and the errors in these factors become
We are now in a position to compute E i numerically. That is, we can compute the error due to iterative solution of the linear system and see if it corresponds to the convergence tolerance used in the method. Note that the convergence is based on measuring the difference in two consecutive approximations, which is not exactly the error due to the iteration, but it is a kind of measure, and it should have about the same size as E i .
The function demo_classic_iterative in diffu2D_u0.py implements the idea above (also for the methods in Sect. 3.6.12). The value of E i is in particular printed at each time level. By changing the tolerance in the convergence criterion of the Jacobi method, we can see that E i is of the same order of magnitude as the prescribed tolerance in the Jacobi method. For example: \(E_{\Delta}\sim 10^{-2}\) with \(N_{x}=N_{y}=10\) and \(\theta=\frac{1}{2}\), as long as \(\max u\) has some significant size (\(\max u> 0.02\)). An appropriate value of the tolerance is then 10−3, such that the error in the Jacobi method does not become bigger than the discretization error. In that case, E i is around \(5\cdot 10^{-3}\). The corresponding number of Jacobi iterations (with ω = 1) varies from 31 to 12 during the time simulation (for \(\max u> 0.02\)). Changing the tolerance to 10−5 causes many more iterations (61 to 42) without giving any contribution to the overall accuracy, because the total error is dominated by E Δ.
Also, with an \(N_{x}=N_{y}=20\), the spatial accuracy increases and many more iterations are needed (143 to 45), but the dominating error is from the time discretization. However, with such a finer spatial mesh, a higher tolerance in the convergence criterion 10−4 is needed to keep \(E_{i}\sim 10^{-3}\). More experiments show the disadvantage of the very simple Jacobi iteration method: the number of iterations increases with the number of unknowns, keeping the tolerance fixed, but the tolerance should also be lowered to avoid the iteration error to dominate the total error. A small adjustment of the Jacobi method, as described in Sect. 3.6.12, provides a better method.
3.6.11 The Relaxed Jacobi Method and Its Relation to the Forward Euler Method
We shall now show that solving the Poisson equation \(-\alpha\nabla^{2}u=f\) by the Jacobi iterative method is in fact equivalent to using a Forward Euler scheme on \(u_{t}=\alpha\nabla^{2}u+f\) and letting \(t\rightarrow\infty\).
A Forward Euler discretization of the 2D diffusion equation,
can be written out as
where \(h=\Delta x=\Delta y\) has been introduced for simplicity. The scheme can be reordered as
with
but this latter form is nothing but the relaxed Jacobi method applied to
From the equivalence above we know a couple of things about the Jacobi method for solving \(-\nabla^{2}u=f\):
-
1.
The method is unstable if ω > 1 (since the Forward Euler method is then unstable).
-
2.
The convergence is really slow as the iteration index increases (coming from the fact that the Forward Euler scheme requires many small time steps to reach the stationary solution).
These observations are quite disappointing: if we already have a time-dependent diffusion problem and want to take larger time steps by an implicit time discretization method, we will with the Jacobi method end up with something close to a slow Forward Euler simulation of the original problem at each time level. Nevertheless, the are two reasons for why the Jacobi method remains a fundamental building block for solving linear systems arising from PDEs: 1) a couple of iterations remove large parts of the error and this is effectively used in the very efficient class of multigrid methods; and 2) the idea of the Jacobi method can be developed into more efficient methods, especially the SOR method, which is treated next.
3.6.12 The Gauss-Seidel and SOR Methods
If we update the mesh points according to the Jacobi method (3.98) for a Backward Euler discretization with a loop over \(i=1,\ldots,N_{x}-1\) and \(j=1,\ldots,N_{y}-1\), we realize that when \(u^{n+1,r+1}_{i,j}\) is computed, \(u^{n+1,r+1}_{i-1,j}\) and \(u^{n+1,r+1}_{i,j-1}\) are already computed, so these new values can be used rather than \(u^{n+1,r}_{i-1,j}\) and \(u^{n+1,r}_{i,j-1}\) (respectively) in the formula for \(u^{n+1,r+1}_{i,j}\). This idea gives rise to the Gauss-Seidel iteration method, which mathematically is just a small adjustment of (3.98):
Observe that the way we access the mesh points in the formula (3.109 ) is important: points with i − 1 must be computed before points with i, and points with j − 1 must be computed before points with j. Any sequence of mesh points can be used in the Gauss-Seidel method, but the particular math formula must distinguish between already visited points in the current iteration and the points not yet visited.
The idea of relaxation (3.101) can equally well be applied to the Gauss-Seidel method. Actually, the Gauss-Seidel method with an arbitrary 0 < ω ≤ 2 has its own name: the Successive Over-Relaxation method, abbreviated as SOR.
The SOR method for a θ rule discretization, with the shortened u and \(u^{-}\) notation, can be written
The sequence of mesh points in (3.110) is \(i=1,\ldots,N_{x}-1\), \(j=1,\ldots,N_{y}-1\) (but whether i runs faster or slower than j does not matter).
3.6.13 Scalar Implementation of the SOR Method
Since the Jacobi and Gauss-Seidel methods with relaxation are so similar, we can easily make a common code for the two:
The idea here is to introduce u__ to be used for already computed values (u) in the Gauss-Seidel/SOR version of the implementation, or just values from the previous iteration (u_) in case of the Jacobi method.
3.6.14 Vectorized Implementation of the SOR Method
Vectorizing the Gauss-Seidel iteration step turns out to be non-trivial. The problem is that vectorized operations typically imply operations on arrays where the sequence in which we visit the elements does not matter. In particular, this principle makes vectorized code trivial to parallelize. However, in the Gauss-Seidel algorithm, the sequence in which we visit the elements in the arrays does matter, and it is well known that the basic method as explained above cannot be parallelized. Therefore, also vectorization will require new thinking.
The strategy for vectorizing (and parallelizing) the Gauss-Seidel method is to use a special numbering of the mesh points called red-black numbering: every other point is red or black as in a checkerboard pattern. This numbering requires N x and N y to be even numbers. Here is an example of a 6 × 6 mesh:
The idea now is to first update all the red points. Each formula for updating a red point involves only the black neighbors. Thereafter, we update all the black points, and at each black point, only the recently computed red points are involved.
The scalar implementation of the red-black numbered Gauss-Seidel method is really compact, since we can update values directly in u (this guarantees that we use the most recently computed values). Here is the relevant code for the Backward Euler scheme in time and without a source term:
The vectorized version must be based on slices. Looking at a typical red-black pattern, e.g.,
we want to update the internal points (marking boundary points with x):
It is impossible to make one slice that picks out all the internal red points. Instead, we need two slices. The first involves points marked with R:
This slice is specified as 1::2 for i and 1::2 for j, or with slice objects:
The second slice involves the red points with R:
The slices are
For the black points, the first slice involves the B points:
with slice objects
The second set of black points is shown here:
with slice objects
That is, we need four sets of slices. The simplest way of implementing the algorithm is to make a function with variables for the slices representing i, i − 1, i + 1, j, j − 1, and j + 1, here called ic (‘‘i center’’), im1 (‘‘i minus 1’’, ip1 (‘‘i plus 1’’), jc, jm1, and jp1, respectively.
The formula returned from update is to be compared with (3.110).
The relaxed Jacobi iteration can be implemented by
The Gauss-Seidel (or SOR) updates need four different steps. The ic and jc slices are specified above. For each of these, we must specify the corresponding im1, ip1, jm1, and jp1 slices. The code below contains the details.
The function solver_classic_iterative in diffu2D_u0.py contains a unified implementation of the relaxed Jacobi and SOR methods in scalar and vectorized versions using the techniques explained above.
3.6.15 Direct Versus Iterative Methods
Direct methods
There are two classes of methods for solving linear systems: direct methods and iterative methods. Direct methods are based on variants of the Gaussian elimination procedure and will produce an exact solution (in exact arithmetics) in an a priori known number of steps. Iterative methods, on the other hand, produce an approximate solution, and the amount of work for reaching a given accuracy is usually not known.
The most common direct method today is to use the LU factorization procedure to factor the coefficient matrix A as the product of a lower-triangular matrix L (with unit diagonal terms) and an upper-triangular matrix U: A = LU. As soon as we have L and U, a system of equations LUc = b is easy to solve because of the triangular nature of L and U. We first solve Ly = b for y (forward substitution), and thereafter we find c from solving Uc = y (backward substitution). When A is a dense N × N matrix, the LU factorization costs \(\frac{1}{3}N^{3}\) arithmetic operations, while the forward and backward substitution steps each require of the order N 2 arithmetic operations. That is, factorization dominates the costs, while the substitution steps are cheap.
Symmetric, positive definite coefficient matrices often arise when discretizing PDEs. In this case, the LU factorization becomes \(A=LL^{T}\), and the associated algorithm is known as Cholesky factorization. Most linear algebra software offers highly optimized implementations of LU and Cholesky factorization as well as forward and backward substitution (scipy.linalg is the relevant Python package).
Finite difference discretizations lead to sparse coefficient matrices. An extreme case arose in Sect. 3.2.1 where A was tridiagonal. For a tridiagonal matrix, the amount of arithmetic operations in the LU and Cholesky factorization algorithms is just of the order N, not N 3. Tridiagonal matrices are special cases of banded matrices, where the matrices contain just a set of diagonal bands. Finite difference methods on regularly numbered rectangular and box-shaped meshes give rise to such banded matrices, with 5 bands in 2D and 7 in 3D for diffusion problems. Gaussian elimination only needs to work within the bands, leading to much more efficient algorithms.
If \(A_{i,j}=0\) for j > i + p and j < i − p, p is the half-bandwidth of the matrix. We have in our 2D problem \(p=N_{x}+2\), while in 3D, \(p=(N_{x}+1)(N_{y}+1)+2\). The cost of Gaussian elimination is then \(\mathcal{O}(Np^{2})\), so with \(p\ll N\), we see that banded matrices are much more efficient to compute with. By reordering the unknowns in clever ways, one can reduce the work of Gaussian elimination further. Fortunately, the Python programmer has access to such algorithms through the scipy.sparse.linalg package.
Although a direct method is an exact algorithm, rounding errors may in practice accumulate and pollute the solution. The effect grows with the size of the linear system, so both for accuracy and efficiency, iterative methods are better suited than direct methods for solving really large linear systems.
Iterative methods
The Jacobi and SOR iterative methods belong to a class of iterative methods where the idea is to solve Au = b by splitting A into two parts, A = M − N, such that solving systems Mu = c is easy and efficient. With the splitting, we get a system
which suggests an iterative method
where \(u^{r+1}\) is a new approximation to u in the r + 1-th iteration. To initiate the iteration, we need a start vector u 0.
The Jacobi and SOR methods are based on splitting A into a lower tridiagonal part L, the diagonal D, and an upper tridiagonal part U, such that \(A=L+D+U\). The Jacobi method corresponds to M = D and N = −L − U. The Gauss-Seidel method employs M = L + D and N = −U, while the SOR method corresponds to
The relaxed Jacobi method has similar expressions:
With the matrix forms of the Jacobi and SOR methods as written above, we could in an implementation alternatively fill the matrix A with entries and call general implementations of the Jacobi or SOR methods that work on a system Au = b. However, this is almost never done since forming the matrix A requires quite some code and storing A in the computer’s memory is unnecessary. It is much easier to just apply the Jacobi and SOR ideas to the finite difference stencils directly in an implementation, as we have shown in detail.
Nevertheless, the matrix formulation of the Jacobi and SOR methods have been important for analyzing their convergence behavior. One can show that the error \(u^{r}-u\) fulfills \(u^{r}-u=G^{r}(u^{0}-u)\), where \(G=M^{-1}N\) and G k is a matrix exponential. For the method to converge, \(\lim_{r\rightarrow\infty}||G^{r}||=0\) is a necessary and sufficient condition. This implies that the spectral radius of G must be less than one. Since G is directly related to the finite difference scheme for the underlying PDE problem, one can in principle compute the spectral radius. For a given PDE problem, however, this is not a practical strategy, since it is very difficult to develop useful formulas. Analysis of model problems, usually related to the Poisson equation, reveals some trends of interest: the convergence rate of the Jacobi method goes like h 2, while that of SOR with an optimal ω goes like h, where h is the spatial spacing: \(h=\Delta x=\Delta y\). That is, the efficiency of the Jacobi method quickly deteriorates with the increasing mesh resolution, and SOR is much to be preferred (even if the optimal ω remains an open question). We refer to Chapter 4 of [16] for more information on the convergence theory. One important result is that if A is symmetric and positive definite, then SOR will converge for any 0 < ω < 2.
The optimal ω parameter can be theoretically established for a Poisson problem as
This formula can be used as a guide also in other problems.
The Jacobi and the SOR methods have their great advantage of being trivial to implement, so they are obviously popular of this reason. However, the slow convergence of these methods limits the popularity to fairly small linear systems (i.e., coarse meshes). As soon as the matrix size grows, one is better off with more sophisticated iterative methods like the preconditioned Conjugate gradient method, which we now turn to.
Finally, we mention that there is a variant of the SOR method, called the Symmetric Successive Over-relaxation method, known as SSOR, where one runs a standard SOR sweep through the mesh points and then a new sweep while visiting the points in reverse order.
3.6.16 The Conjugate Gradient Method
There is no simple intuitive derivation of the Conjugate gradient method, so we refer to the many excellent expositions in the literature for the idea of the method and how the algorithm is derived. In particular, we recommend the books [1, 16, 2, 5]. A brief overview is provided in the Wikipedia article Footnote 4. Here, we just state the pros and cons of the method from a user’s perspective and how we utilize it in code.
The original Conjugate gradient method is limited to linear systems Au = b, where A is a symmetric and positive definite matrix. There are, however, extensions of the method to non-symmetric matrices.
A major advantage of all conjugate gradient methods is that the matrix A is only used in matrix-vector products, so we do not need form and store A if we can provide code for computing a matrix-vector product Au. Another important feature is that the algorithm is very easy to vectorize and parallelize. The primary downside of the method is that it converges slowly unless one has an effective preconditioner for the system. That is, instead of solving Au = b, we try to solve \(M^{-1}Au=M^{-1}b\) in the hope that the method works better for this preconditioned system. The matrix M is the preconditioner or preconditioning matrix. Now we need to perform matrix-vector products \(y=M^{-1}Au\), which is done in two steps: first the matrix-vector product v = Au is carried out and then the system My = v must be solved. Therefore, M must be cheap to compute and systems My = v must be cheap to solve.
A perfect preconditioner is M = A, but in each iteration in the Conjugate gradient method one then has so solve a system with A as coefficient matrix! A key idea is to let M be some kind of cheap approximation to A. The simplest preconditioner is to set M = D, where D is the diagonal of A. This choice means running one Jacobi iteration as preconditioner. Exercise 3.8 shows that the Jacobi and SOR methods can also be viewed as preconditioners.
Constructing good preconditioners is a scientific field on its own. Here we shall treat the topic just very briefly. For a user having access to the scipy.sparse. linalg library, there are Conjugate gradient methods and preconditioners readily available:
-
For positive definite, symmetric systems: cg (the Conjugate gradient method)
-
For symmetric systems: minres (Minimum residual method)
-
For non-symmetric systems:
-
gmres (GMRES: Generalized minimum residual method)
-
bicg (BiConjugate gradient method)
-
bicgstab (Stabilized BiConjugate gradient method)
-
cgs (Conjugate gradient squared method)
-
qmr (Quasi-minimal residual iteration)
-
-
Preconditioner: spilu (Sparse, incomplete LU factorization)
The ILU preconditioner is an attractive all-round type of preconditioner that is suitable for most problems on serial computers. A more efficient preconditioner is the multigrid method, and algebraic multigrid is also an all-round choice as preconditioner. The Python package PyAMG Footnote 5 offers efficient implementations of the algebraic multigrid method, to be used both as a preconditioner and as a stand-alone iterative method.
The matrix arising from implicit time discretization methods applied to the diffusion equation is symmetric and positive definite. Thus, we can use the Conjugate gradient method (cg), typically in combination with an ILU preconditioner. The code is very similar to the one we created when solving the linear system by sparse Gaussian elimination, the main difference is that we now allow for calling up the Conjugate gradient function as an alternative solver.
The number of iterations in the Conjugate gradient method is of interest, but is unfortunately not available from the cg function. Therefore, we perform a trick: in each iteration a user function CG_callback is called where we accumulate the number of iterations in a list CG_iter.
3.6.17 What Is the Recommended Method for Solving Linear Systems?
There is no clear answer to this question. If you have enough memory and computing time available, direct methods such as spsolve are to be preferred since they are easy to use and finds almost an exact solution. However, in larger 2D and in 3D problems, direct methods usually run too slowly or require too much memory, so one is forced to use iterative methods. The fastest and most reliable methods are in the Conjugate Gradient family, but these require suitable preconditioners. ILU is an all-round preconditioner, but it is not suited for parallel computing. The Jacobi and SOR iterative methods are easy to implement, and popular for that reason, but run slowly. Jacobi iteration is not an option in real problems, but SOR may be.
3.7 Random Walk
Models leading to diffusion equations, see Sect. 3.8, are usually based on reasoning with averaged physical quantities such as concentration, temperature, and velocity. The underlying physical processes involve complicated microscopic movement of atoms and molecules, but an average of a large number of molecules is performed in a small volume before the modeling starts, and the averaged quantity inside this volume is assigned as a point value at the centroid of the volume. This means that concentration, temperature, and velocity at a space-time point represent averages around the point in a small time interval and small spatial volume.
Random walk is a principally different kind of modeling procedure compared to the reasoning behind partial differential equations. The idea in random walk is to have a large number of ‘‘particles’’ that undergo random movements. Averaging can then be used afterwards to compute macroscopic quantities like concentration. The ‘‘particles’’ and their random movement represent a very simplified microscopic behavior of molecules, much simpler and computationally much more efficient than direct molecular simulation Footnote 6, yet the random walk model has been very powerful to describe a wide range of phenomena, including heat conduction, quantum mechanics, polymer chains, population genetics, neuroscience, hazard games, and pricing of financial instruments.
It can be shown that random walk, when averaged, produces models that are mathematically equivalent to diffusion equations. This is the primary reason why we treat random walk in this chapter: two very different algorithms (finite difference stencils and random walk) solve the same type of problems. The simplicity of the random walk algorithm makes it particularly attractive for solving diffusion equations on massively parallel computers. The exposition here is as simple as possible, and good thorough derivation of the models is provided by Hjorth-Jensen [7].
3.7.1 Random Walk in 1D
Imagine that we have some particles that perform random moves, either to the right or to the left. We may flip a coin to decide the movement of each particle, say head implies movement to the right and tail means movement to the left. Each move is one unit length. Physicists use the term random walk for this type of movement. The movement is also known as drunkard’s walk Footnote 7. You may try this yourself: flip the coin and make one step to the left or right, and repeat the process.
We introduce the symbol N for the number of steps in a random walk. Figure 3.16 shows four different random walks with N = 200.
Ensemble of 4 random walks, each with 200 steps
3.7.2 Statistical Considerations
Let S k be the stochastic variable representing a step to the left or to the right in step number k. We have that \(S_{k}=-1\) with probability p and \(S_{k}=1\) with probability q = 1 − p. The variable S k is known as a Bernoulli variable Footnote 8. The expectation of S k is
and the variance is
The position after k steps is another stochastic variable
The expected position is
All the S k variables are independent. The variance therefore becomes
We see that \(\hbox{Var}(\bar{X}_{k})\) is proportional with the number of steps k. For the very important case \(p=q=\frac{1}{2}\), \(\hbox{E}[\bar{X}_{k}]=0\) and \(\hbox{Var}(\bar{X}_{k})=k\).
How can we estimate \(\hbox{E}[\bar{X}_{k}]=0\) and \(\hbox{Var}(\bar{X}_{k})=N\)? We must have many random walks of the type in Fig. 3.16. For a given k, say k = 100, we find all the values of \(\bar{X}_{k}\), name them \(\bar{x}_{0,k}\), \(\bar{x}_{1,k}\), \(\bar{x}_{2,k}\), and so on. The empirical estimate of \(\hbox{E}[\bar{X}_{k}]\) is the average,
while an empirical estimate of \(\hbox{Var}(\bar{X}_{k})\) is
That is, we take the statistics for a given K across the ensemble of random walks (‘‘vertically’’ in Fig. 3.16). The key quantities to record are \(\sum_{i}\bar{x}_{i,k}\) and \(\sum_{i}\bar{x}_{i,k}^{2}\).
3.7.3 Playing Around with Some Code
Scalar code
Python has a random module for drawing random numbers, and this module has a function uniform(a, b) for drawing a uniformly distributed random number in the interval \([a,b)\). If an event happens with probability p, we can simulate this on the computer by drawing a random number r in \([0,1)\), because then r ≤ p with probability p and r > p with probability 1 − p:
A random walk with N steps, starting at x 0, where we move to the left with probability p and to the right with probability 1 − p can now be implemented by
Vectorized code
Since N is supposed to be large and we want to repeat the process for many particles, we should speed up the code as much as possible. Vectorization is the obvious technique here: we draw all the random numbers at once with aid of numpy, and then we formulate vector operations to get rid of the loop over the steps (k). The numpy.random module has vectorized versions of the functions in Python’s built-in random module. For example, numpy.random.uniform(a, b, N) returns N random numbers uniformly distributed between a (included) and b (not included).
We can then make an array of all the steps in a random walk: if the random number is less than or equal to p, the step is −1, otherwise the step is 1:
The value of position[k] is the sum of all steps up to step k. Such sums are often needed in vectorized algorithms and therefore available by the numpy.cumsum function:
The resulting array in this demo has elements 1, 1 + 3 = 4, \(1+3+4=8\), and \(1+3+4+6=14\).
We can now vectorize the random_walk1D function:
This code runs about 10 times faster than the scalar version. With a parallel numpy library, the code can also automatically take advantage of hardware for parallel computing because each of the four array operations can be trivially parallelized.
Fixing the random sequence
During software development with random numbers it is advantageous to always generate the same sequence of random numbers, as this may help debugging processes. To fix the sequence, we set a seed of the random number generator to some chosen integer, e.g.,
Calls to random_walk1D_vec give positions of the particle as depicted in Fig. 3.17. The particle starts at the origin and moves with \(p=\frac{1}{2}\). Since the seed is the same, the plot to the left is just a magnification of the first 1000 steps in the plot to the right.
1000 (left) and 50,000 (right) steps of a random walk
Verification
When we have a scalar and a vectorized code, it is always a good idea to develop a unit test for checking that they produce the same result. A problem in the present context is that the two versions apply two different random number generators. For a test to be meaningful, we need to fix the seed and use the same generator. This means that the scalar version must either use np.random or have this as an option. An option is the most flexible choice:
Using random=np.random, the r variable gets computed by np.random.uniform, and the sequence of random numbers will be the same as in the vectorized version that employs the same generator (given that the seed is also the same). A proper test function may be to check that the positions in the walk are the same in the scalar and vectorized implementations:
Note that we employ == for arrays with real numbers, which is normally an inadequate test due to rounding errors, but in the present case, all arithmetics consists of adding or subtracting one, so these operations are expected to have no rounding errors. Comparing two numpy arrays with == results in a boolean array, so we need to call the all() method to ensure that all elements are True, i.e., that all elements in the two arrays match each other pairwise.
3.7.4 Equivalence with Diffusion
The original random walk algorithm can be said to work with dimensionless coordinates \(\bar{x}_{i}=-N+i\), \(i=0,1,\ldots,2N+1\) (\(i\in[-N,N]\)), and \(\bar{t}_{n}=n\), \(n=0,1,\ldots,N\). A mesh with spacings \(\Delta x\) and \(\Delta t\) with dimensions can be introduced by
If we implement the algorithm with dimensionless coordinates, we can just use this rescaling to obtain the movement in a coordinate system without unit spacings.
Let \(P^{n+1}_{i}\) be the probability of finding the particle at mesh point \(\bar{x}_{i}\) at time \(\bar{t}_{n+1}\). We can reach mesh point \((i,n+1)\) in two ways: either coming in from the left from \((i-1,n)\) or from the right (\(i+1,n)\). Each has probability \(\frac{1}{2}\) (if we assume \(p=q=\frac{1}{2}\)). The fundamental equation for \(P^{n+1}_{i}\) is
(This equation is easiest to understand if one looks at the random walk as a Markov process and applies the transition probabilities, but this is beyond scope of the present text.)
Subtracting \(P^{n}_{i}\) from (3.113) results in
Readers who have seen the Forward Euler discretization of a 1D diffusion equation recognize this scheme as very close to such a discretization. We have
or in dimensionless coordinates
Similarly, we have
Equation (3.113) is therefore equivalent with the dimensionless diffusion equation
or the diffusion equation
with diffusion coefficient
This derivation shows the tight link between random walk and diffusion. If we keep track of where the particle is, and repeat the process many times, or run the algorithms for lots of particles, the histogram of the positions will approximate the solution of the diffusion equation for the local probability \(P^{n}_{i}\).
Suppose all the random walks start at the origin. Then the initial condition for the probability distribution is the Dirac delta function \(\delta(x)\). The solution of (3.114) can be shown to be
where \(\alpha=\frac{1}{2}\).
3.7.5 Implementation of Multiple Walks
Our next task is to implement an ensemble of walks (for statistics, see Sect. 3.7.2) and also provide data from the walks such that we can compute the probabilities of the positions as introduced in the previous section. An appropriate representation of probabilities \(P^{n}_{i}\) are histograms (with i along the x axis) for a few selected values of n.
To estimate the expectation and variance of the random walks, Sect. 3.7.2 points to recording \(\sum_{j}x_{j,k}\) and \(\sum_{j}x_{j,k}^{2}\), where x j,k is the position at time/step level k in random walk number j. The histogram of positions needs the individual values x i,k for all i values and some selected k values.
We introduce position[k] to hold \(\sum_{j}x_{j,k}\), position2[k] to hold \(\sum_{j}(x_{j,k})^{2}\), and pos_hist[i,k] to hold x i,k. A selection of k values can be specified by saying how many, num_times, and let them be equally spaced through time:
This is one of the few situations where we want integer division (//) or real division rounded to an integer.
Scalar version
Our scalar implementation of running num_walks random walks may go like this:
Vectorized version
We have already vectorized a single random walk. The additional challenge here is to vectorize the computation of the data for the histogram, pos_hist, but given the selected steps in pos_hist_times, we can find the corresponding positions by indexing with the list pos_hist_times: position[post_hist_times], which are to be inserted in pos_hist[n,:].
Improved vectorized version
Looking at the vectorized version above, we still have one potentially long Python loop over n. Normally, num_walks will be much larger than N. The vectorization of the loop over N certainly speeds up the program, but if we think of vectorization as also a way to parallelize the code, all the independent walks (the n loop) can be executed in parallel. Therefore, we should include this loop as well in the vectorized expressions, at the expense of using more memory.
We introduce the array walks to hold the N + 1 steps of all the walks: each row represents the steps in one walk.
Since all the steps are independent, we can just make one long vector of enough random numbers (N*num_walks), translate these numbers to ±1, then we reshape the array such that the steps of each walk are stored in the rows.
The next step is to sum up the steps in each walk. We need the np.cumsum function for this, with the argument axis=1 for indicating a sum across the columns:
Now walks can be computed by
The position vector is the sum of all the walks. That is, we want to sum all the rows, obtained by
A corresponding expression computes the squares of the positions. Finally, we need to compute pos_hist, but that is a matter of grabbing some of the walks (according to pos_hist_times):
The complete vectorized algorithm without any loop can now be summarized:
What is the gain of the vectorized implementations? One important gain is that each vectorized operation can be automatically parallelized if one applies a parallel numpy library like Numba Footnote 9. On a single CPU, however, the speed up of the vectorized operations is also significant. With N = 1000 and 50,000 repeated walks, the two vectorized versions run about 25 and 18 times faster than the scalar version, with random_walks1D_vec1 being fastest.
Remark on vectorized code and parallelization
Our first attempt on vectorization removed the loop over the N steps in a single walk. However, the number of walks is usually much larger than N, because of the need for accurate statistics. Therefore, we should rather remove the loop over all walks. It turns out, from our efficiency experiments, that the function random_walks1D_vec2 (with no loops) is slower than random_walks1D_vec1. This is a bit surprising and may be explained by less efficiency in the statements involving very large arrays, containing all steps for all walks at once.
From a parallelization and improved vectorization point of view, it would be more natural to switch the sequence of the loops in the serial code such that the shortest loop is the outer loop:
The vectorized version of this code, where we just vectorize the loop over n, becomes
This function runs significantly faster than the random_walks1D_vec1 function above, typically 1.7 times faster. The code is also more appropriate in a parallel computing context since each vectorized statement can work with data of size num_walks over the compute units, repeated N times (compared with data of size N, repeated num_walks times, in random_walks1D_vec1).
The scalar code with switched loops, random_walks1D2 runs a bit slower than the original code in random_walks1D, so with the longest loop as the inner loop, the vectorized function random_walks1D2_vec1 is almost 60 times faster than the scalar counterpart, while the code random_walks1D_vec2 without loops is only around 18 times faster. Taking into account the very large arrays required by the latter function, we end up with random_walks1D2_vec1 as the preferred implementation.
Test function
During program development, it is highly recommended to carry out computations by hand for, e.g., N=4 and num_walks=3. Normally, this is done by executing the program with these parameters and checking with pen and paper that the computations make sense. The next step is to use this test for correctness in a formal test function.
First, we need to check that the simulation of multiple random walks reproduces the results of random_walk1D, random_walk1D_vec1, and random_walk1D_vec2 for the first walk, if the seed is the same. Second, we run three random walks (N=4) with the scalar and the two vectorized versions and check that the returned arrays are identical.
For this type of test to be successful, we must be sure that exactly the same set of random numbers are used in the three versions, a fact that requires the same random number generator and the same seed, of course, but also the same sequence of computations. This is not obviously the case with the three random_walk1D* functions we have presented. The critical issue in random_walk1D_vec1 is that the first random numbers are used for the first walk, the second set of random numbers is used for the second walk and so on, to be compatible with how the random numbers are used in the function random_walk1D. For the function random_walk1D_vec2 the situation is a bit more complicated since we generate all the random numbers at once. However, the critical step now is the reshaping of the array returned from np.where: we must reshape as (num_walks, N) to ensure that the first N random numbers are used for the first walk, the next N numbers are used for the second walk, and so on.
We arrive at the test function below.
Such test functions are indispensable for further development of the code as we can at any time test whether the basic computations remain correct or not. This is particularly important in stochastic simulations since without test functions and fixed seeds, we always experience variations from run to run, and it can be very difficult to spot bugs through averaged statistical quantities.
3.7.6 Demonstration of Multiple Walks
Assuming now that the code works, we can just scale up the number of steps in each walk and the number of walks. The latter influences the accuracy of the statistical estimates. Figure 3.18 shows the impact of the number of walks on the expectation, which should approach zero. Figure 3.19 displays the corresponding estimate of the variance of the position, which should grow linearly with the number of steps. It does, seemingly very accurately, but notice that the scale on the y axis is so much larger than for the expectation, so irregularities due to the stochastic nature of the process become so much less visible in the variance plots. The probability of finding a particle at a certain position at time (or step) 800 is shown in Fig. 3.20. The dashed red line is the theoretical distribution (3.116) arising from solving the diffusion equation (3.114) instead. As always, we realize that one needs significantly more statistical samples to estimate a histogram accurately than the expectation or variance.
Estimated expected value for 1000 steps, using 100 walks (upper left), 10,000 (upper right), 100,000 (lower left), and 1,000,000 (lower right)
Estimated variance over 1000 steps, using 100 walks (upper left), 10,000 (upper right), 100,000 (lower left), and 1,000,000 (lower right)
Estimated probability distribution at step 800, using 100 walks (upper left), 10,000 (upper right), 100,000 (lower left), and 1,000,000 (lower right)
3.7.7 Ascii Visualization of 1D Random Walk
If we want to study (very) long time series of random walks, it can be convenient to plot the position in a terminal window with the time axis pointing downwards. The module avplotter in SciTools has a class Plotter for plotting functions in the terminal window with the aid of ascii symbols only. Below is the code required to visualize a simple random walk, starting at the origin, and considered over when the point x = −1 is reached. We use a spacing \(\Delta x=0.05\) (so x = −1 corresponds to i = −20).
Observe that we implement an infinite loop, but allow a smooth interrupt of the program by Ctrl+c through Python’s KeyboardInterrupt exception. This is a useful recipe that can be used in many occasions!
The output looks typically like
Positions beyond the limits of the x axis appear with a value. A long file Footnote 10 contains the complete ascii plot corresponding to the function run_random_walk above.
3.7.8 Random Walk as a Stochastic Equation
The (dimensionless) position in a random walk, \(\bar{X}_{k}\), can be expressed as a stochastic difference equation:
where s is a Bernoulli variable Footnote 11, taking on the two values s = −1 and s = 1 with equal probability:
The s variable in a step is independent of the s variable in other steps.
The difference equation expresses essentially the sum of independent Bernoulli variables. Because of the central limit theorem, X k , will then be normally distributed with expectation \(k\hbox{E}[s]\) and \(k\hbox{Var}(s)\). The expectation and variance of a Bernoulli variable with values r = 0 and r = 1 are p and \(p(1-p)\), respectively. The variable s = 2r − 1 then has expectation \(2\hbox{E}[r]-1=2p-1=0\) and variance \(2^{2}\hbox{Var}(r)=4p(1-p)=1\). The position X k is normally distributed with zero expectation and variance k, as we found in Sect. 3.7.2.
The central limit theorem tells that as long as k is not small, the distribution of X k remains the same if we replace the Bernoulli variable s by any other stochastic variable with the same expectation and variance. In particular, we may let s be a standardized Gaussian variable (zero mean, unit variance).
Dividing (3.117 ) by \(\Delta t\) gives
In the limit \(\Delta t\rightarrow 0\), \(s/\Delta t\) approaches a white noise stochastic process. With \(\bar{X}(t)\) as the continuous process in the limit \(\Delta t\rightarrow 0\) (\(X_{k}\rightarrow X(t_{k})\)), we formally get the stochastic differential equation
where W(t) is a Wiener process Footnote 12. Then X is also a Wiener process. It follows from the stochastic ODE dX = dW that the probability distribution of X is given by the Fokker-Planck equation Footnote 13 (3.114). In other words, the key results for random walk we found earlier can alternatively be derived via a stochastic ordinary differential equation and its related Fokker-Planck equation.
3.7.9 Random Walk in 2D
The most obvious generalization of 1D random walk to two spatial dimensions is to allow movements to the north, east, south, and west, with equal probability \(\frac{1}{4}\).
The left plot in Fig. 3.21 provides an example on 200 steps with this kind of walk. We may refer to this walk as a walk on a rectangular mesh as we move from any spatial mesh point \((i,j)\) to one of its four neighbors in the rectangular directions: \((i+1,j)\), \((i-1,j)\), \((i,j+1)\), or \((i,j-1)\).
Random walks in 2D with 200 steps: rectangular mesh (left) and diagonal mesh (right)
3.7.10 Random Walk in Any Number of Space Dimensions
From a programming point of view, especially when implementing a random walk in any number of dimensions, it is more natural to consider a walk in the diagonal directions NW, NE, SW, and SE. On a two-dimensional spatial mesh it means that we go from \((i,j)\) to either \((i+1,j+1)\), \((i-1,j+1)\), \((i+1,j-1)\), or \((i-1,j-1)\). We can with such a diagonal mesh (see right plot in Fig. 3.21) draw a Bernoulli variable for the step in each spatial direction and trivially write code that works in any number of spatial directions:
A vectorized version is desired. We follow the ideas from Sect. 3.7.3, but each step is now a vector in d spatial dimensions. We therefore need to draw Nd random numbers in r, compute steps in the various directions through np.where(r <=p, -1, 1) (each step being −1 or 1), and then we can reshape this array to an N × d array of step vectors. Doing an np.cumsum summation along axis 0 will add the vectors, as this demo shows:
With such summation of step vectors, we get all the positions to be filled in the position array:
3.7.11 Multiple Random Walks in Any Number of Space Dimensions
As we did in 1D, we extend one single walk to a number of walks (num_walks in the code).
Scalar code
As always, we start with implementing the scalar case:
Vectorized code
Significant speed-ups can be obtained by vectorization. We get rid of the loops in the previous function and arrive at the following vectorized code.
3.8 Applications
3.8.1 Diffusion of a Substance
The first process to be considered is a substance that gets transported through a fluid at rest by pure diffusion. We consider an arbitrary volume V of this fluid, containing the substance with concentration function \(c(\boldsymbol{x},t)\). Physically, we can think of a very small volume with centroid x at time t and assign the ratio of the volume of the substance and the total volume to \(c(\boldsymbol{x},t)\). This means that the mass of the substance in a small volume \(\Delta V\) is approximately \(\varrho c\Delta V\), where \(\varrho\) is the density of the substance. Consequently, the total mass of the substance inside the volume V is the sum of all \(\varrho c\Delta V\), which becomes the volume integral \(\int_{V}\varrho cdV\).
Let us reason how the mass of the substance changes and thereby derive a PDE governing the concentration c. Suppose the substance flows out of V with a flux q. If \(\Delta S\) is a small part of the boundary \(\partial V\) of V, the volume of the substance flowing out through dS in a small time interval \(\Delta t\) is \(\varrho\boldsymbol{q}\cdot\boldsymbol{n}\Delta t\Delta S\), where n is an outward unit normal to the boundary \(\partial V\), see Fig. 3.23. We realize that only the normal component of q is able to transport mass in and out of V. The total outflow of the mass of the substance in a small time interval \(\Delta t\) becomes the surface integral
Assuming conservation of mass, this outflow of mass must be balanced by a loss of mass inside the volume. The increase of mass inside the volume, during a small time interval \(\Delta t\), is
assuming \(\varrho\) is constant, which is reasonable. The outflow of mass balances the loss of mass in V, which is the increase with a minus sign. Setting the two contributions equal to each other ensures balance of mass inside V. Dividing by \(\Delta t\) gives
Note the minus sign on the right-hand side: the left-hand side expresses loss of mass, while the integral on the right-hand side is the gain of mass.
Four random walks with 5000 steps in 2D
An arbitrary volume of a fluid
Now, letting \(\Delta t\rightarrow 0\), we have
so
To arrive at a PDE, we express the surface integral as a volume integral using Gauss’ divergence theorem:
Since \(\varrho\) is constant, we can divide by this quantity. If the integral is to vanish for an arbitrary volume V, the integrand must vanish too, and we get the mass conservation PDE for the substance:
A fundamental problem is that this is a scalar PDE for four unknowns: c and the three components of q. We therefore need additional equations. Here, Fick’s law comes at rescue: it models how the flux q of the substance is related to the concentration c. Diffusion is recognized by mass flowing from regions with high concentration to regions of low concentration. This principle suggests that q is proportional to the negative gradient of c:
where α is an empirically determined constant. The relation (3.121) is known as Fick’s law. Inserting (3.121) in (3.120) gives a scalar PDE for the concentration c:
3.8.2 Heat Conduction
Heat conduction is a well-known diffusion process. The governing PDE is in this case based on the first law of thermodynamics: the increase in energy of a system is equal to the work done on the system, plus the supplied heat. Here, we shall consider media at rest and neglect work done on the system. The principle then reduces to a balance between increase in internal energy and supplied heat flow by conduction.
Let \(e(x,t)\) be the internal energy per unit mass. The increase of the internal energy in a small volume \(\Delta V\) in a small time interval \(\Delta t\) is then
where \(\varrho\) is the density of the material subject to heat conduction. In an arbitrary volume V, as depicted in Fig. 3.23, the corresponding increase in internal energy becomes the volume integral
This increase in internal energy is balanced by heat supplied by conduction. Let q be the heat flow per time unit. Through the surface \(\partial V\) of V the following amount of heat flows out of V during a time interval \(\Delta t\):
The simplified version of the first law of thermodynamics then states that
The minus sign on the right-hand side ensures that the integral there models net inflow of heat (since n is an outward unit normal, q ⋅ n models outflow). Dividing by \(\Delta t\) and notifying that
we get (in the limit \(\Delta t\rightarrow 0\))
This is the integral equation for heat conduction, but we aim at a PDE. The next step is therefore to transform the surface integral to a volume integral via Gauss’ divergence theorem. The result is
If this equality is to hold for all volumes V, the integrand must vanish, and we have the PDE
Sometimes the supplied heat can come from the medium itself. This is the case, for instance, when radioactive rock generates heat. Let us add this effect. If \(f(\boldsymbol{x},t)\) is the supplied heat per unit volume per unit time, the heat supplied in a small volume is \(f\Delta t\Delta V\), and inside an arbitrary volume V the supplied generated heat becomes
Adding this to the integral statement of the (simplified) first law of thermodynamics, and continuing the derivation, leads to the PDE
There are four unknown scalar fields: e and q. Moreover, the temperature T, which is our primary quantity to compute, does not enter the model yet. We need an additional equation, called the equation of state, relating e, \(V=1/\varrho=\), and T: \(e=e(V,T)\). By the chain rule we have
The first coefficient \(\partial e/\partial T\) is called specific heat capacity at constant volume, denoted by c v :
The specific heat capacity will in general vary with T, but taking it as a constant is a good approximation in many applications.
The term \(\partial e/\partial V\) models effects due to compressibility and volume expansion. These effects are often small and can be neglected. We shall do so here. Using \(\partial e/\partial t=c_{v}\partial T/\partial t\) in the PDE gives
We still have four unknown scalar fields (T and q). To close the system, we need a relation between the heat flux q and the temperature T called Fourier’s law:
which simply states that heat flows from hot to cold areas, along the path of greatest variation. In a solid medium, k depends on the material of the medium, and in multi-material media one must regard k as spatially dependent. In a fluid, it is common to assume that k is constant. The value of k reflects how easy heat is conducted through the medium, and k is named the coefficient of heat conduction.
We now have one scalar PDE for the unknown temperature field \(T(\boldsymbol{x},t)\):
3.8.3 Porous Media Flow
The requirement of mass balance for flow of a single, incompressible fluid through a deformable (elastic) porous medium leads to the equation
where p is the fluid pressure, q is the fluid velocity, u is the displacement (deformation) of the medium, S is the storage coefficient of the medium (related to the compressibility of the fluid and the material in the medium), and α is another coefficient. In many circumstances, the last term with u can be neglected, an assumption that decouples the equation above from a model for the deformation of the medium. The famous Darcy’s law relates q to p:
where K is the permeability of the medium, μ is the dynamic viscosity of the fluid, \(\varrho\) is the density of the fluid, and g is the acceleration of gravity, here taken as \(\boldsymbol{g}=-g\boldsymbol{k}\). Combining the two equations results in the diffusion model
Boundary conditions consist of specifying p or q ⋅ n (i.e., normal velocity) at each point of the boundary.
3.8.4 Potential Fluid Flow
Let v be the velocity of a fluid. The condition \(\nabla\times\boldsymbol{v}=0\) is relevant for many flows, especially in geophysics when viscous effects are negligible. From vector calculus it is known that \(\nabla\times\boldsymbol{v}=0\) implies that v can be derived from a scalar potential field ϕ: \(\boldsymbol{v}=\nabla\phi\). If the fluid is incompressible, \(\nabla\cdot\boldsymbol{v}=0\), it follows that \(\nabla\cdot\nabla\phi=0\), or
This Laplace equation is sufficient for determining ϕ and thereby describe the fluid motion. This type of flow is known as potential flow Footnote 14. One very important application where potential flow is a good model is water waves. As boundary condition we must prescribe \(\boldsymbol{v}\cdot\boldsymbol{n}=\partial\phi/\partial n\). This gives rise to what is known as a pure Neumann problem and will cause numerical difficulties because ϕ and ϕ plus any constant are two solutions of the problem. The simplest remedy is to fix the value of ϕ at a point.
3.8.5 Streamlines for 2D Fluid Flow
The streamlines in a two-dimensional stationary fluid flow are lines tangential to the flow. The stream function Footnote 15 ψ is often introduced in two-dimensional flow such that its contour lines, \(\psi=\hbox{const}\), gives the streamlines. The relation between ψ and the velocity field \(\boldsymbol{v}=(u,v)\) is
It follows that \(\nabla\boldsymbol{v}=\psi_{yx}-\psi_{xy}=0\), so the stream function can only be used for incompressible flows. Since
we can derive the relation
which is a governing equation for the stream function \(\psi(x,y)\) if the vorticity ω is known.
3.8.6 The Potential of an Electric Field
Under the assumption of time independence, Maxwell’s equations for the electric field E become
where ρ is the electric charge density and ϵ0 is the electric permittivity of free space (i.e., vacuum). Since \(\nabla\times\boldsymbol{E}=0\), E can be derived from a potential \(\varphi\), \(\boldsymbol{E}=-\nabla\varphi\). The electric field potential is therefore governed by the Poisson equation
If the medium is heterogeneous, ρ will depend on the spatial location r. Also, ϵ0 must be exchanged with an electric permittivity function \(\epsilon(\boldsymbol{r})\).
Each point of the boundary must be accompanied by, either a Dirichlet condition \(\varphi(\boldsymbol{r})=\varphi_{D}(\boldsymbol{r})\), or a Neumann condition \(\frac{\partial\varphi(\boldsymbol{r})}{\partial n}=\varphi_{N}(\boldsymbol{r})\).
3.8.7 Development of Flow Between Two Flat Plates
Diffusion equations may also arise as simplified versions of other mathematical models, especially in fluid flow. Consider a fluid flowing between two flat, parallel plates. The velocity is uni-directional, say along the z axis, and depends only on the distance x from the plates; \(\boldsymbol{u}=u(x,t)\boldsymbol{k}\). The flow is governed by the Navier-Stokes equations,
where p is the pressure field, unknown along with the velocity u, \(\varrho\) is the fluid density, μ the dynamic viscosity, and f is some external body force. The geometric restrictions of flow between two flat plates puts restrictions on the velocity, \(\boldsymbol{u}=u(x,t)\boldsymbol{i}\), and the z component of the Navier-Stokes equations collapses to a diffusion equation:
if f z is the component of f in the z direction.
The boundary conditions are derived from the fact that the fluid sticks to the plates, which means u = 0 at the plates. Say the location of the plates are z = 0 and z = L. We then have
One can easily show that \(\partial p/\partial z\) must be a constant or just a function of time t. We set \(\partial p/\partial z=-\beta(t)\). The body force could be a component of gravity, if desired, set as \(f_{z}=\gamma g\). Switching from z to x as independent variable gives a very standard one-dimensional diffusion equation:
The boundary conditions are
while some initial condition
must also be prescribed.
The flow is driven by either the pressure gradient β or gravity, or a combination of both. One may also consider one moving plate that drives the fluid. If the plate at x = L moves with velocity \(U_{L}(t)\), we have the adjusted boundary condition
3.8.8 Flow in a Straight Tube
Now we consider viscous fluid flow in a straight tube with radius R and rigid walls. The governing equations are the Navier-Stokes equations, but as in Sect. 3.8.7, it is natural to assume that the velocity is directed along the tube, and that it is axi-symmetric. These assumptions reduced the velocity field to \(\boldsymbol{u}=u(r,x,t)\boldsymbol{i}\), if the x axis is directed along the tube. From the equation of continuity, \(\nabla\cdot\boldsymbol{u}=0\), we see that u must be independent of x. Inserting \(\boldsymbol{u}=u(r,t)\boldsymbol{i}\) in the Navier-Stokes equations, expressed in axi-symmetric cylindrical coordinates, results in
Here, \(\beta(t)=-\partial p/\partial x\) is the pressure gradient along the tube. The associated boundary condition is \(u(R,t)=0\).
3.8.9 Tribology: Thin Film Fluid Flow
Thin fluid films are extremely important inside machinery to reduce friction between gliding surfaces. The mathematical model for the fluid motion takes the form of a diffusion problem and is quickly derived here. We consider two solid surfaces whose distance is described by a gap function \(h(x,y)\). The space between these surfaces is filled with a fluid with dynamic viscosity μ. The fluid may move partially because of pressure gradients and partially because the surfaces move. Let \(U\boldsymbol{i}+V\boldsymbol{j}\) be the relative velocity of the two surfaces and p the pressure in the fluid. The mathematical model builds on two principles: 1) conservation of mass, 2) assumption of locally quasi-static flow between flat plates.
The conservation of mass equation reads \(\nabla\cdot\boldsymbol{u}\), where u is the local fluid velocity. For thin films the detailed variation between the surfaces is not of interest, so \(\nabla\cdot\boldsymbol{u}=0\) is integrated (average) in the direction perpendicular to the surfaces. This gives rise to the alternative mass conservation equation
where z is the coordinate perpendicular to the surfaces, and q is then the volume flux in the fluid gap.
Locally, we may assume that we have steady flow between two flat surfaces, with a pressure gradient and where the lower surface is at rest and the upper moves with velocity \(U\boldsymbol{i}+V\boldsymbol{j}\). The corresponding mathematical problem is actually the limit problem in Sect. 3.8.7 as \(t\rightarrow\infty\). The limit problem can be solved analytically, and the local volume flux becomes
The idea is to use this expression locally also when the surfaces are not flat, but slowly varying, and if U, V, or p varies in time, provided the time variation is sufficiently slow. This is a common quasi-static approximation, much used in mathematical modeling.
Inserting the expression for q via p, U, and V in the equation \(\nabla\boldsymbol{q}=0\) gives a diffusion PDE for p:
The boundary conditions must involve p or q at the boundary.
3.8.10 Propagation of Electrical Signals in the Brain
One can make a model of how electrical signals are propagated along the neuronal fibers that receive synaptic inputs in the brain. The signal propagation is one-dimensional and can, in the simplest cases, be governed by the Cable equation Footnote 16:
where \(V(x,t)\) is the voltage to be determined, c m is capacitance of the neuronal fiber, while r l and r m are measures of the resistance. The boundary conditions are often taken as V = 0 at a short circuit or open end, \(\partial V/\partial x=0\) at a sealed end, or \(\partial V/\partial x\propto V\) where there is an injection of current.
3.9 Exercises
Exercise 3.6 (Stabilizing the Crank-Nicolson method by Rannacher time stepping)
It is well known that the Crank-Nicolson method may give rise to non-physical oscillations in the solution of diffusion equations if the initial data exhibit jumps (see Sect. 3.3.6). Rannacher [15] suggested a stabilizing technique consisting of using the Backward Euler scheme for the first two time steps with step length \(\frac{1}{2}\Delta t\). One can generalize this idea to taking 2m time steps of size \(\frac{1}{2}\Delta t\) with the Backward Euler method and then continuing with the Crank-Nicolson method, which is of second-order in time. The idea is that the high frequencies of the initial solution are quickly damped out, and the Backward Euler scheme treats these high frequencies correctly. Thereafter, the high frequency content of the solution is gone and the Crank-Nicolson method will do well.
Test this idea for m = 1,2,3 on a diffusion problem with a discontinuous initial condition. Measure the convergence rate using the solution (3.45) with the boundary conditions (3.46)–(3.47) for t values such that the conditions are in the vicinity of ±1. For example, \(t<5a1.6\cdot 10^{-2}\) makes the solution diffusion from a step to almost a straight line. The program diffu_erf_sol.py shows how to compute the analytical solution.
Project 3.7 (Energy estimates for diffusion problems)
This project concerns so-called energy estimates for diffusion problems that can be used for qualitative analytical insight and for verification of implementations.
-
a)
We start with a 1D homogeneous diffusion equation with zero Dirichlet conditions:
$$u_{t} =\alpha u_{x}x,\quad x\in\Omega=(0,L),\ t\in(0,T],$$(3.133)$$u(0,t) =u(L,t)=0,\quad t\in(0,T],$$(3.134)$$u(x,0) =I(x),\quad x\in[0,L]\thinspace.$$(3.135)The energy estimate for this problem reads
$$||u||_{L^{2}}\leq||I||_{L^{2}},$$(3.136)where the \(||\cdot||_{L^{2}}\) norm is defined by
$$||g||_{L^{2}}=\sqrt{\int_{0}^{L}g^{2}dx}\thinspace.$$(3.137)The quantify \(||u||_{L^{2}}\) or \(\frac{1}{2}||u||_{L^{2}}\) is known as the energy of the solution, although it is not the physical energy of the system. A mathematical tradition has introduced the notion energy in this context.
The estimate (3.136) says that the ‘‘size of u’’ never exceeds that of the initial condition, or more precisely, it says that the area under the u curve decreases with time.
To show (3.136), multiply the PDE by u and integrate from 0 to L. Use that uu t can be expressed as the time derivative of u 2 and that \(u_{x}xu\) can integrated by parts to form an integrand \(u_{x}^{2}\). Show that the time derivative of \(||u||_{L^{2}}^{2}\) must be less than or equal to zero. Integrate this expression and derive (3.136).
-
b)
Now we address a slightly different problem,
$$u_{t} =\alpha u_{x}x+f(x,t),\quad x\in\Omega=(0,L),\ t\in(0,T],$$(3.138)$$u(0,t) =u(L,t)=0,\quad t\in(0,T],$$(3.139)$$u(x,0) =0,\quad x\in[0,L]\thinspace.$$(3.140)The associated energy estimate is
$$||u||_{L^{2}}\leq||f||_{L^{2}}\thinspace.$$(3.141)(This result is more difficult to derive.)
Now consider the compound problem with an initial condition I(x) and a right-hand side \(f(x,t)\):
$$u_{t} =\alpha u_{x}x+f(x,t),\quad x\in\Omega=(0,L),\ t\in(0,T],$$(3.142)$$u(0,t) =u(L,t)=0,\quad t\in(0,T],$$(3.143)$$u(x,0) =I(x),\quad x\in[0,L]\thinspace.$$(3.144)Show that if w 1 fulfills (3.133)–(3.135) and w 2 fulfills (3.138)–(3.140), then \(u=w_{1}+w_{2}\) is the solution of (3.142)–(3.144). Using the triangle inequality for norms,
$$||a+b||\leq||a||+||b||,$$show that the energy estimate for (3.142)–(3.144) becomes
$$||u||_{L^{2}}\leq||I||_{L^{2}}+||f||_{L^{2}}\thinspace.$$(3.145) -
c)
One application of (3.145) is to prove uniqueness of the solution. Suppose u 1 and u 2 both fulfill (3.142)–(3.144). Show that \(u=u_{1}-u_{2}\) then fulfills (3.142)–(3.144) with f = 0 and I = 0. Use (3.145) to deduce that the energy must be zero for all times and therefore that \(u_{1}=u_{2}\), which proves that the solution is unique.
-
d)
Generalize (3.145) to a 2D/3D diffusion equation \(u_{t}=\nabla\cdot(\alpha\nabla u)\) for \(x\in\Omega\).
Hint
Use integration by parts in multi dimensions:
where \(\frac{\partial u}{\partial n}=\boldsymbol{n}\cdot\nabla u\), n being the outward unit normal to the boundary \(\partial\Omega\) of the domain Ω.
-
a)
Now we also consider the multi-dimensional PDE \(u_{t}=\nabla\cdot(\alpha\nabla u)\). Integrate both sides over Ω and use Gauss’ divergence theorem, \(\int_{\Omega}\nabla\cdot\boldsymbol{q}\,\mathrm{d}x=\int_{\partial\Omega}\boldsymbol{q}\cdot\boldsymbol{n}\,\mathrm{d}s\) for a vector field q. Show that if we have homogeneous Neumann conditions on the boundary, \(\partial u/\partial n=0\), area under the u surface remains constant in time and
$$\int_{\Omega}u\,\mathrm{d}x=\int_{\Omega}I\,\mathrm{d}x\thinspace.$$(3.146) -
b)
Establish a code in 1D, 2D, or 3D that can solve a diffusion equation with a source term f, initial condition I, and zero Dirichlet or Neumann conditions on the whole boundary.
We can use (3.145) and (3.146) as a partial verification of the code. Choose some functions f and I and check that (3.145) is obeyed at any time when zero Dirichlet conditions are used. Iterate over the same I functions and check that (3.146) is fulfilled when using zero Neumann conditions.
-
c)
Make a list of some possible bugs in the code, such as indexing errors in arrays, failure to set the correct boundary conditions, evaluation of a term at a wrong time level, and similar. For each of the bugs, see if the verification tests from the previous subexercise pass or fail. This investigation shows how strong the energy estimates and the estimate (3.146) are for pointing out errors in the implementation.
Filename: diffu_energy.
Exercise 3.8 (Splitting methods and preconditioning)
In Sect. 3.6.15, we outlined a class of iterative methods for Au = b based on splitting A into A = M − N and introducing the iteration
The very simplest splitting is M = I, where I is the identity matrix. Show that this choice corresponds to the iteration
where \(r^{k-1}\) is the residual in the linear system in iteration k − 1. The formula (3.147) is known as Richardson’s iteration. Show that if we apply the simple iteration method (3.147) to the preconditioned system \(M^{-1}Au=M^{-1}b\), we arrive at the Jacobi method by choosing M = D (the diagonal of A) as preconditioner and the SOR method by choosing \(M=\omega^{-1}D+L\) (L being the lower triangular part of A). This equivalence shows that we can apply one iteration of the Jacobi or SOR method as preconditioner.
Problem 3.9 (Oscillating surface temperature of the earth)
Consider a day-and-night or seasonal variation in temperature at the surface of the earth. How deep down in the ground will the surface oscillations reach? For simplicity, we model only the vertical variation along a coordinate x, where x = 0 at the surface, and x increases as we go down in the ground. The temperature is governed by the heat equation
in some spatial domain \(x\in[0,L]\), where L is chosen large enough such that we can assume that T is approximately constant, independent of the surface oscillations, for x > L. The parameters \(\varrho\), c v , and k are the density, the specific heat capacity at constant volume, and the heat conduction coefficient, respectively.
-
a)
Derive the mathematical model for computing \(T(x,t)\). Assume the surface oscillations to be sinusoidal around some mean temperature T m . Let \(T=T_{m}\) initially. At x = L, assume \(T\approx T_{m}\).
-
b)
Scale the model in a) assuming k is constant. Use a time scale \(t_{c}=\omega^{-1}\) and a length scale \(x_{c}=\sqrt{2\alpha/\omega}\), where \(\alpha=k/(\varrho c_{v})\). The primary unknown can be scaled as \(\frac{T-T_{m}}{2A}\).
Show that the scaled PDE is
$$\frac{\partial u}{\partial\bar{t}}=\frac{1}{2}\frac{\partial^{2}u}{\partial x^{2}},$$with initial condition \(u(\bar{x},0)=0\), left boundary condition \(u(0,\bar{t})=\sin(\bar{t})\), and right boundary condition \(u(\bar{L},\bar{t})=0\). The bar indicates a dimensionless quantity.
Show that \(u(\bar{x},\bar{t})=e^{-\bar{x}}\sin(\bar{x}-\bar{t})\) is a solution that fulfills the PDE and the boundary condition at \(\bar{x}=0\) (this is the solution we will experience as \(\bar{t}\rightarrow\infty\) and \(L\rightarrow\infty\)). Conclude that an appropriate domain for x is \([0,4]\) if a damping \(e^{-4}\approx 0.18\) is appropriate for implementing \(\bar{u}\approx\hbox{const}\); increasing to \([0,6]\) damps \(\bar{u}\) to 0.0025.
-
c)
Compute the scaled temperature and make animations comparing two solutions with \(\bar{L}=4\) and \(\bar{L}=8\), respectively (keep \(\Delta x\) the same).
Problem 3.10 (Oscillating and pulsating flow in tubes)
We consider flow in a straight tube with radius R and straight walls. The flow is driven by a pressure gradient \(\beta(t)\). The effect of gravity can be neglected. The mathematical problem reads
We consider two models for \(\beta(t)\). One plain, sinusoidal oscillation:
and one with periodic pulses,
Note that both models can be written as \(\beta=A\sin^{m}(\omega t)\), with m = 1 and m = 16, respectively.
-
a)
Scale the mathematical model, using the viscous time scale \(\varrho R^{2}/\mu\).
-
b)
Implement the scaled model from a), using the unifying θ scheme in time and centered differences in space.
-
c)
Verify the implementation in b) using a manufactured solution that is quadratic in r and linear in t. Make a corresponding test function.
Hint
You need to include an extra source term in the equation to allow for such tests. Let the spatial variation be \(1-r^{2}\) such that the boundary condition is fulfilled.
-
a)
Make animations for m = 1,16 and α = 1,0.1. Choose T such that the motion has reached a steady state (non-visible changes from period to period in u).
-
b)
For \(\alpha\gg 1\), the scaling in a) is not good, because the characteristic time for changes (due to the pressure) is much smaller than the viscous diffusion time scale (α becomes large). We should in this case base the short time scale on 1 ∕ ω. Scale the model again, and make an animation for m = 1,16 and α = 10.
Filename: axisymm_flow.
Problem 3.11 (Scaling a welding problem)
Welding equipment makes a very localized heat source that moves in time. We shall investigate the heating due to welding and choose, for maximum simplicity, a one-dimensional heat equation with a fixed temperature at the ends, and we neglect melting. We shall scale the problem, and besides solving such a problem numerically, the aim is to investigate the appropriateness of alternative scalings.
The governing PDE problem reads
Here, u is the temperature, \(\varrho\) the density of the material, c a heat capacity, k the heat conduction coefficient, f is the heat source from the welding equipment, and U s is the initial constant (room) temperature in the material.
A possible model for the heat source is a moving Gaussian function:
where A is the strength, σ is a parameter governing how peak-shaped (or localized in space) the heat source is, and v is the velocity (in positive x direction) of the source.
-
a)
Let x c , t c , u c , and f c be scales, i.e., characteristic sizes, of x, t, u, and f, respectively. The natural choice of x c and f c is L and A, since these make the scaled x and f in the interval \([0,1]\). If each of the three terms in the PDE are equally important, we can find t c and u c by demanding that the coefficients in the scaled PDE are all equal to unity. Perform this scaling. Use scaled quantities in the arguments for the exponential function in f too and show that
$$\bar{f}=e^{-\frac{1}{2}\beta^{2}(\bar{x}-\gamma\bar{t})^{2}},$$where β and γ are dimensionless numbers. Give an interpretation of β and γ.
-
b)
Argue that for large γ we should base the time scale on the movement of the heat source. Show that this gives rise to the scaled PDE
$$\frac{\partial\bar{u}}{\partial\bar{t}}=\gamma^{-1}\frac{\partial^{2}\bar{u}}{\partial\bar{x}^{2}}+\bar{f},$$and
$$\bar{f}=\exp{\left(-\frac{1}{2}\beta^{2}(\bar{x}-\bar{t})^{2}\right)}\thinspace.$$Discuss when the scalings in a) and b) are appropriate.
-
c)
One aim with scaling is to get a solution that lies in the interval \([-1,1]\). This is not always the case when u c is based on a scale involving a source term, as we do in a) and b). However, from the scaled PDE we realize that if we replace \(\bar{f}\) with \(\delta\bar{f}\), where δ is a dimensionless factor, this corresponds to replacing u c by \(u_{c}/\delta\). So, if we observe that \(\bar{u}\sim 1/\delta\) in simulations, we can just replace \(\bar{f}\) by \(\delta\bar{f}\) in the scaled PDE.
Use this trick and implement the two scaled models. Reuse software for the diffusion equation (e.g., the solver function in diffu1D_vc.py). Make a function run(gamma, beta=10, delta=40, scaling=1, animate=False) that runs the model with the given γ, β, and δ parameters as well as an indicator scaling that is 1 for the scaling in a) and 2 for the scaling in b). The last argument can be used to turn screen animations on or off.
Experiments show that with γ = 1 and β = 10, δ = 20 is appropriate. Then \(\max|\bar{u}|\) will be larger than 4 for γ = 40, but that is acceptable.
Equip the run function with visualization, both animation of \(\bar{u}\) and \(\bar{f}\), and plots with \(\bar{u}\) and \(\bar{f}\) for t = 0.2 and t = 0.5.
Hint
Since the amplitudes of \(\bar{u}\) and \(\bar{f}\) differs by a factor δ, it is attractive to plot \(\bar{f}/\delta\) together with \(\bar{u}\).
-
d)
Use the software in c) to investigate γ = 0.2,1,5,40 for the two scalings. Discuss the results.
Filename: welding.
Exercise 3.12 (Implement a Forward Euler scheme for axi-symmetric diffusion)
Based on the discussion in Sect. 3.5.6, derive in detail the discrete equations for a Forward Euler in time, centered in space, finite difference method for axi-symmetric diffusion. The diffusion coefficient may be a function of the radial coordinate. At the outer boundary r = R, we may have either a Dirichlet or Robin condition. Implement this scheme. Construct appropriate test problems.
Filename: FE_axisym.
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
Author information
Authors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. The images or other third party material in this chapter are included in the chapter’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2017 The Author(s)
About this chapter
Cite this chapter
Linge, S., Langtangen, H.P. (2017). Diffusion Equations. In: Finite Difference Computing with PDEs. Texts in Computational Science and Engineering, vol 16. Springer, Cham. https://doi.org/10.1007/978-3-319-55456-3_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-55456-3_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-55455-6
Online ISBN: 978-3-319-55456-3
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)