Cluster Computing

, Volume 18, Issue 4, pp 1363–1377 | Cite as

A domain decomposition strategy for hybrid parallelization of moving particle semi-implicit (MPS) method for computer cluster

  • Davi Teodoro Fernandes
  • Liang-Yee Cheng
  • Eric Henrique Favero
  • Kazuo Nishimoto


The MPS is a particle method developed to simulate incompressible flows with free surfaces that has several applications in nonlinear hydrodynamics. Much effort has been done to handle the large amount of particles required for simulating practical problems with desired refinement. However, the efficient use of the currently available computational resources, such as computer cluster, remains as a challenge. The present paper proposes a new strategy to parallelize the MPS method for fully distributed computing in cluster, which enables to simulate models with hundreds of millions of particles and keeps the required runtime within reasonable limits, as shown by the analysis of scalability and performance. The proposed strategy uses a non-geometric dynamic domain decomposition method that provides homogeneous load balancing and for very large models the scalability is supra-linear. Also, the domain decomposition (DD) is carried out only in the initial setup. As a result, the DD method is based on renumbering of particles using an original fully distributed sorting algorithm. Moreover, unlike the usual strategies, none of the processors require access to global data of the particles on any time step. Therefore, the limit for the maximum size of the model depends more on the total memory of the allocated nodes than the quantity of the local memory of each node. Thus, by extending the application of MPS method to very large models, this study contributes to consolidating the method as a practical tool to investigate complex engineering problems.


Cluster computing Domain decomposition Particle method Computational fluid dynamics 



The authors would like to express their gratitude to PETROBRAS S.A for the financial support to present research.


  1. 1.
    Koshizuka, S., Tamako, H., Oka, Y.: A particle method for incompressible viscous flow with fluid fragmentation. CFD J. 4(1), 29–46 (1995)Google Scholar
  2. 2.
    Iwashita, T., Shimasaki, M.: Parallel processing of 3-D eddy current analysis with moving conductor using parallelized ICCG solver with renumbering process. IEEE Trans. Magn. 36(4), 1504–1509 (2000)CrossRefGoogle Scholar
  3. 3.
    Motezuki, F.K., Cheng, L.Y., Tsukamoto, M.M.: A study on parallelized solvers for moving particle semi-implicit method (MPS). In: Proceedings of the 20th International Congress of Mechanical Engineering (COBEM2009). Gramado, Brazil (2009)Google Scholar
  4. 4.
    Yao, Z., Wang, J.S., Cheng, M.: Improved o(n) neighbor list method using domain decomposition and data sorting. High Perform. Comput. Eng. Syst. (HPCES) 161(1–2), 27–35 (2004)Google Scholar
  5. 5.
    Nyland, L., Prins, J., Yun, R.H., Hermans, J., Kum, H., Wang, L.: Achieving scalable parallel molecular dynamics using dynamic spatial domain decomposition techniques. J. Parallel Distrib. Comput. 47, 125–138 (1997)CrossRefGoogle Scholar
  6. 6.
    Gotoh, H., Khayyer, A., Ikari, H., Chiemi, H.: 3D-CMPS method for improvement of water surface tracking in breaking waves. In: 4th SPHERIC Workshop, pp. 265–272. Nantes, France (2009)Google Scholar
  7. 7.
    Iribe, T., Fujisawa, T., Koshizuka, S.: Reduction of communication in parallel computing of particle method for flow simulation of seaside areas. Cost Eng. 52(4), 287–304 (2010)CrossRefGoogle Scholar
  8. 8.
    Gotoh, H., Sakai, T.: Key issues in the particle method for computation of wave breaking. Cost Eng. 53(2–3), 171–179 (2006)CrossRefGoogle Scholar
  9. 9.
    Ikeda, H., Koshizuka, S., Oka, Y.: Numerical analysis of jet injection behavior for fuel-coolant interaction using particle method. J. Nucl. Sci. Tech. 38(3), 174–182 (2001)CrossRefGoogle Scholar
  10. 10.
    Formaggia, L., Sala, M., Saleri, F.: Domain decomposition techniques. Tech. rep, Sandia National Laboratories, Albuquerque, USA (2005)Google Scholar
  11. 11.
    Knuth, D.E.: The Art of Computer Programming. Sorting and Searching, vol. 3. Addison Wesley Longman Publishing Co., Inc., Redwood City (1998)Google Scholar
  12. 12.
    Meloni, S., Rosati, M., Colombo, L.: Efficiently particle labeling in atomistic simulations. J. Chem. Phys. 126, 121102 (2007)CrossRefGoogle Scholar
  13. 13.
    The OpenMP API specification for parallel programming.
  14. 14.
    Message passing interface (MPI).
  15. 15.
    VTK, Visualization toolkit.
  16. 16.
    Issa, R., Violeau, D.: Test case 2: 3D-dambreaking. Tech. rep, SPH European Research Interest Community—SPHERIC ERCOFTAC, Rome, Italy (2006)Google Scholar
  17. 17.
    Issa, R., Violeau, D.: SPHERIC Test2, 3D schematic dam break and evolution of the free surface.
  18. 18.
    Ganglia monitoring system—scalable distributed monitoring system for high-performance computing systems.
  19. 19.
    Marr, D.T., Binns, F., Hill, D.L., Hinton, G., Koufaty, D.A., Miller, J.A., Upton, M.: Hyper-threading technology architecture and microarchitecture. Intel Technol. J. 6–1, 4–15 (2002)Google Scholar
  20. 20.
    Grama, A., Gupta, A., Karypis, G., Kumar, V.: Introduction to Parallel Computing, 2nd edn. Addison Wesley, Boston (2003)Google Scholar
  21. 21.
    Ogino, M., Shioya, R., Kawai, H., Yoshimura, S.: Seismic response analysis of nuclear pressure vessel model with ADVENTRUE system on the Earth Simulator. J. Earth Simul. 2, 41–54 (2005)Google Scholar
  22. 22.
    Ihmsen, M., Akinci, N., Becker, M., Teschner, M.: A parallel SPH implementation on multi-core CPUs. Comput. Graph. Forum. 30(1), 99–112 (2011)CrossRefGoogle Scholar
  23. 23.
    Hori, C., Gotoh, H., Ikari, H., Khayyer, A.: GPU-acceleration for Moving Particle Semi-implicit Method. Comput. Fluids 51, 174 (2011)Google Scholar
  24. 24.
    Ovaysi, S., Piri, M.: Multi-GPU acceleration of direct pore-scale modeling of fluid flow in natural porous media. Comput. Phys. Commun. 183, 1890–1898 (2012)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Bellezi, C.A., Cheng, L.Y.: Particle based numerical analysis of green water on FSOP deck. In: 32nd International Conference on Ocean, Offshore and Arctic Engineering, OMAE2013-11553, Nantes, France (2013)Google Scholar
  26. 26.
    Murotani, K., Koshizuka, S., Tamai, T., Shibata, K., Mitsume, N., Yoshimura, S., Tanaka, S., Hasegawa, K., Nagai, E., Fujisawa, T.: Development of hierarchical domain decomposition explicit MPS method and application to large-scale tsunami analysis with floating objects. J. Adv. Simul. Sci. Eng. (JASSE) 1(1), 16–35 (2014)CrossRefGoogle Scholar
  27. 27.
    Duan, G., Chen, B.: Comparison of parallel solvers for Moving Particle Semi-Implicit method. Eng. Comput. 32(3), 834–862 (2015)CrossRefGoogle Scholar
  28. 28.
    Taniguchi, D., Sato, L.M., Cheng, L.Y.: Explicit moving particle simulation method on GPU clusters. Blucher Mech. Eng. Proc. 1(1), 1155 (2014)Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Numerical Offshore Tank of Escola PolitécnicaUniversity of São PauloSão PauloBrazil
  2. 2.Cidade UniversitáriaSão PauloBrazil
  3. 3.Department of Construction Engineering, Escola PolitécnicaUniversity of São PauloSão PauloBrazil

Personalised recommendations