Acceleration of Classical Molecular Dynamics Simulations

Abstract

In this chapter, we describe the acceleration and parallelization in classical molecular dynamics simulations. As electrostatic interactions are computationally intensive, the importance of the particle mesh Ewald (PME) method and the fast multipole method (FMM) will increase. These methods will be described here. In addition, general techniques for hierarchical parallelization on the latest general-purpose supercomputers (especially connected by a three-dimensional torus network), together with the critical importance of the data array structure, are explained. We show the optimization and benchmark results in the parallel environments of the molecular dynamics calculation programs, MODYLAS and GENESIS.

Appendix: FMM

The principle of the fast multipole method (FMM) is briefly explained in this Appendix. See Ref. [15] for exact mathematical treatment of the transformation of multipole moment and local expansion coefficients, such as M2M, M2L, and L2L.

5.1.1 Fundamentals of Multipole Expansion

As shown in Fig. 5.15, the atom i of the charge $q_i$ in a certain area forms the potential at a distant point Q. The reciprocal $1/r_i$ of the distance between two points is calculated using the distance $\Delta { r_i}$ from the origin O to the atom i in the region, the distance r from the origin O to the point Q, and the angle $\gamma _i$ formed by the points $\Delta {r_i}$ and r with the point O. This can be expressed as

$$\begin{aligned} \frac{1}{r_i} = \frac{1}{\sqrt{r^2 - 2r\Delta r_i \cos \gamma _i + \Delta r_i^2}} = \frac{1}{r \sqrt{1 - 2 \frac{\Delta r_i}{r} \cos \gamma _i + \left( \frac{\Delta r_i}{r} \right) ^2 }} \end{aligned}$$

(5.14)

from the cosine theorem. When this equation is rewritten by Taylor expansion, it becomes

$$\begin{aligned} \frac{1}{r_i}= & {} \frac{1}{r} + \frac{\Delta r_i}{r^2} \cos \gamma _i + \frac{\Delta r_i^2}{2 r^3} \left( 3 \cos ^2 \gamma _i - 1 \right) + \frac{\Delta r_i^3}{2 r^4} \left( 5 \cos ^3 \gamma _i - 3 \cos \gamma _i \right) + \cdots \nonumber \\= & {} \frac{1}{r} \sum _n P_n(\cos \gamma _i) \left( \frac{\Delta r_i}{r} \right) ^n, \end{aligned}$$

(5.15)

where $P_n(\cos \gamma _i)$ is a Legendre polynomial. This Eq. (5.15) is called multipole expansion. $P_n(\cos \gamma _i)$ is written as

$$\begin{aligned} P_n(\cos \gamma _i) = \sum _{m=-m}^{n} Y_n^{-m} (\theta _i, \phi _i) Y_n^m (\theta ,\phi ) \end{aligned}$$

(5.16)

using the spherical harmonics, $Y_n^m(\theta ,\phi )$ for spherical coordinates $(\Delta r_i, \theta _i,\phi _i)$ and $(r, \theta , \phi )$ of the two vectors $\Delta r_i$ and r. In Eq. (5.16), the coordinate variables of atom i and point Q are completely separated into two factors. Therefore, the sum with respect to the charges in a region can be represented independently of the position of the point Q. The sum

$$\begin{aligned} M_n^m = \sum _i q_i Y_n^{-m}(\theta _i,\phi _i) {\Delta r_i}^n \end{aligned}$$

(5.17)

is the multipole moment formed by the charge in the region. The electrostatic field formed by all charges in the region can be expressed as

$$\begin{aligned} \sum _i \frac{q_i}{r_i} = \frac{1}{r^{n+1}} \sum _{n=0} \sum _{m=-n}^{n} M_n^m Y_n^{m} (\theta ,\phi ). \end{aligned}$$

(5.18)

Fig. 5.15

Multipole expansion of electrostatic interaction

5.1.2 Division of Unit Cells

In FMM, unit cells are divided into small spaces (subcells). As described in the next section, the interaction calculation is performed using these subcells. There are several ways to divide unit cells into subcells; however, an octree structure is commonly used, in which each side of the unit cell is divided into two equal parts and the whole is divided into eight subcells (Fig. 5.16). This division is repeated until the subcell reaches the desired size of small subcells, i.e., level 0 for the unit cell and level 1 after dividing once. The number of subcells of level n is $8^n$. In practical use, tens of atoms are assigned to the smallest subcell. This smallest subcell is a unit of regional division. We assign subcells to compute nodes and cores to perform parallel calculations.

Fig. 5.16

Division of unit cell

5.1.3 Interaction Calculation by FMM

We assume that the unit cell is divided to level 4, as shown in Fig. 5.17. We now calculate the potential of an atom in region A. First, we evaluate the electrostatic interaction directly up to the second nearest neighbor (B) of region A of level 4. For subcells beyond region B, the interaction calculation is performed using multipole moments.

Fig. 5.17

Interactions between atoms and subcell

Region C is within the second nearest neighbor of the subcell of level 3, but excluding region B. The interaction calculation with region C is performed using the multipole moment of level 4. Region D is within the second nearest neighbor of the subcell of level 2, but excluding region C. The interaction calculation with region B is performed using the multipole moment of level 3. Hereinafter, regions E and F are determined in the same manner. Subcells beyond region F (image cell of third or further nearest neighbor of the unit cell) are treated as those of level 0. Thus, the interactions can be calculated for each region without excess or deficiency. By using small subcells for interactions with neighboring regions and large subcells for interactions with distant regions, it is possible to efficiently perform interaction calculations without decreasing accuracy.

5.1.4 Multipole Expansion and Local Expansion

The procedure for determining the potential acting on the i atom in region A is shown (Fig. 5.18). First, the contributions from regions A and B is obtained by direct calculation. This procedure is called particle to particle (P2P) in FMM. For region C, the multipole moment is calculated in level 4 subcells according to Eq. (5.17). This operation is called particle to multipole (P2M). Next, the center of the multipole moment is moved to the center of subcell A. The expression obtained by this transformation is the form of Taylor expansion at the center of subcell A, which is called local expansion. This shift operation of the expansion center is called multipole to local (M2L) . By using the local expansion coefficient and the relative position from the center subcell of atom i, its potential, force, and virial can be calculated, which is called local to particle (L2P). For the further subcell D, the multipole moment of the level 4 subcell is obtained by P2M similarly to C. The expansion center of the multipole moment is shifted to the center position of the level 3 subcell (i.e., multipole to multipole (M2M)). Similarly, with respect to the other seven subcells, the center of expansion is shifted by M2M. The obtained multipole moments are summed to obtain a multipole moment of level 3, which is transformed to the local expansion coefficient of the level 3 subcell, including subcell A, by M2L operation. The local expansion coefficient is shifted to the center of the subcell of level A (i.e., local to local (L2L)). The contributions from the previously obtained subcell C and the subcells D are summed and then the interaction is evaluated by L2P. The contribution from image cells more distant than region F in Fig. 5.17 can be included by applying Ewald’s method to the multipole moment of image cells [46].

Fig. 5.18

Interaction between atoms and subcell

5.1.5 Accuracy of FMM

FMM is a strictly formulated analytical method that includes error evaluation. The error depends on the expansion order of the multipole moment; i.e., as the order increases, the calculation accuracy monotonically increases. Table 5.1 shows the benchmark results of potential and force in a system consisting of 10,125,000 atoms using MODYLAS software. The accuracy of FMM was evaluated by comparison with the result of the PME method with parameters giving a sufficiently high accuracy. The values obtained at the 4th to 8th expansion order coincide with the exact value in the range of 4–9 digits. There is also an accuracy of 4–7 digits for force; i.e., enough accuracy can be obtained for MD calculations  even in expansions up to the 4thorder.

Table 5.1 Potential and force obtained by FMM