The European Physical Journal Special Topics

, Volume 227, Issue 14, pp 1757–1778 | Cite as

Adaptive grid implementation for parallel continuum mechanics methods in particle simulations

  • Miriam MehlEmail author
  • Michael Lahnert
Part of the following topical collections:
  1. Particle Methods in Natural Science and Engineering


In this tutorial review paper, we present our minimally invasive approach for integrating dynamically adaptive tree-structured grids into existing simulation software that has been developed for regular Cartesian grids. We introduce different physical models that we target and that span a wide range of typical simulation characteristics – from grid-based Lattice-Boltzmann, finite volume and finite difference discretized models to particle-based molecular dynamics models. We derive the respective typical data access requirements and extensions of the algorithms to adaptively refined grids along with possible grid adaptivity criteria. In addition, after introducing basics of tree-structured adaptively refined grids, we present the adaptive grid framework p4est and our enhancement of p4est in order to provide a grid and partitioning infrastructure that can easily be used in existing simulation codes. Finally, we explain how such a grid infrastructure can be integrated into regular grid codes in general in three major steps and how we integrated p4est in the soft matter simulation package ESPResSo in particular. A summary of results from previously published performance and scalability studies together with new results for more realistic coupled simulation scenarios shows the efficiency and validity of the resulting new version of ESPResSo.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    T. Weinzierl, M. Mehl, SIAM J. Sci. Comput. 33, 2732 (2011)MathSciNetGoogle Scholar
  2. 2.
    R.S. Sampath, S.S. Adavani, H. Sundar, I. Lashuk, G. Biros, Dendro: parallel algorithms for multigrid and AMR methods on 2:1 balanced octrees, in SC ‘08: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing (2008), pp. 1–12Google Scholar
  3. 3.
    C. Burstedde, L.C. Wilcox, O. Ghattas, SIAM J. Sci. Comput. 33, 1103 (2011)MathSciNetGoogle Scholar
  4. 4.
    H. Klimach, K. Jain, S.P. Roller, End-to-end parallel simulations with APES, in Advances in parallel computing, (IOS Press, 2014), Vol. 25, pp. 703–711Google Scholar
  5. 5.
    A. Lintermann, S. Schlimpert, J.H. Grimmen, C. Günther, M. Meinke, W. Schrüder, Comput. Methods Appl. Mech. Eng. (2014) 277, 131ADSGoogle Scholar
  6. 6.
    T. Tu, D.R. O’Hallaron, O. Ghattas, Scalable parallel octree meshing for teraScale applications, in SC ‘05: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (ACM/IEEE, 2005)Google Scholar
  7. 7.
    F. Schornbaum, U. Rüde, SIAM J. Sci. Comput. 38, C96 (2016)Google Scholar
  8. 8.
    F. Schornbaum, U. Rüde, SIAM J. Sci. Comput. 40, C358 (2018)Google Scholar
  9. 9.
    S.G. Parker, Future Gener. Comput. Syst. 22, 204 (2006)Google Scholar
  10. 10.
    M. Wahib, N. Maruyama, T. Aoki, Daino: A high-level framework for parallel and effcient AMR on GPUs, in SC16: International Conference for High Performance Computing, Networking, Storage and Analysis (IEEE, 2016), pp. 621–632Google Scholar
  11. 11.
    A. Dubey, A. Almgren, J. Bell, M. Berzins, S. Brandt, G. Bryan, P. Colella, D. Graves, M. Lijewski, F. Lüffler, B. O’Shea, J. Parallel Distrib. Comput. 74, 3217 (2014)Google Scholar
  12. 12.
    H.J. Limbach, A. Arnold, B.A. Mann, C. Holm, Comput. Phys. Commun. 174, 704 (2006)ADSGoogle Scholar
  13. 13.
    A. Arnold, O. Lenz, S. Kesselheim, R. Weeber, F. Fahrenberger, D. Roehm, P. Košovan, C. Holm, ESPResSo 3.1: Molecular dynamics software for coarse-grained models, in Meshfree methods for partial differential equations VI, Lecture Notes in Computational Science and Engineering, edited by M. Griebel, M.A. Schweitzer (Springer, Berlin, Heidelberg, 2012), Vol. 89, pp. 1–23Google Scholar
  14. 14.
    M. Bader, C. Böck, J. Schwaiger, C. Vigh, SIAM J. Sci. Comput. 32, 212 (2010)MathSciNetGoogle Scholar
  15. 15.
    M. Bader, Space-filling curves: an introduction with applications in scientific computing (Springer, Heidelberg, New York, 2013)Google Scholar
  16. 16.
    G. Inci, A. Arnold, A. Kronenburg, R. Weeber, Aerosol Sci. Technol. 48, 842 (2014)ADSGoogle Scholar
  17. 17.
    J. Hüpfner, T. Richter, P. Košovan, C. Holm, M. Wilhelm, Progr. Colloid. Polym. Sci. 140, 140 (2013)Google Scholar
  18. 18.
    B.J. Reynwar, G. Illya, V.A. Harmandaris, M.M. Müller, K. Kremer, M. Deserno, Nature 447, 461 (2007)ADSGoogle Scholar
  19. 19.
    M. Kuron, A. Arnold, Eur. Phys. J. E 38, 20 (2015)Google Scholar
  20. 20.
    S. Kesselheim, M. Sega, C. Holm, Soft Matter 8, 9480 (2012)ADSGoogle Scholar
  21. 21.
    K. Breitsprecher, P. Košovan, C. Holm, J. Phys.: Condens. Matter 26, 284108 (2014)Google Scholar
  22. 22.
    K. Breitsprecher, P. Košovan, C. Holm, J. Phys.: Condens. Matter 26, 284114 (2014)Google Scholar
  23. 23.
    S. Succi, The Lattice Boltzmann equation for fluid dynamics and beyond (Clarendon Press, 2001)Google Scholar
  24. 24.
    D.A. Wolf-Gladrow, Lattice-gas cellular automata and Lattice Boltzmann models – an introduction (Springer, 2000)Google Scholar
  25. 25.
    S. Succi, The Lattice Boltzmann equation: for complex states of flowing matter (Oxford University Press, 2018)Google Scholar
  26. 26.
    P.L. Bhatnagar, E.P. Gross, M. Krook, Phys. Rev. 94, 511 (1954)ADSGoogle Scholar
  27. 27.
    D. d’Humières, S. Succi, D. d’Humières, I. Ginzburg, M. Krafczyk, P. Lallemand, L.-S. Luo, Philos. Trans. R. Soc. A: Math. Phys. Eng. Sci. 360, 437 (2002)ADSGoogle Scholar
  28. 28.
    L.-S. Luo, W. Liao, X. Chen, Y. Peng, W. Zhang, Phys. Rev. E 83, 056710 (2011)ADSGoogle Scholar
  29. 29.
    I. Ginzburg, F. Verhaeghe, D. d’Humières, Commun. Comput. Phys. 3, 427 (2008)MathSciNetGoogle Scholar
  30. 30.
    I. Ginzburg, F. Verhaeghe, D. d’Humieres. Commun. Comput. Phys. 3, 519 (2008)Google Scholar
  31. 31.
    S. Hou, J. Sterling, S. Chen, G.D. Doolen, A Lattice Boltzmann subgrid model for high Reynolds number flow, in Pattern formation and lattice gas automata (American Mathematical Society, 1996), Vol. 6, pp. 151–166Google Scholar
  32. 32.
    R. Brownlee, A. Gorban, J. Levesley, Physica A 387, 385 (2008)ADSMathSciNetGoogle Scholar
  33. 33.
    X. Shan, X.-F. Yuan, H. Chen, J. Fuid Mech. 550, 413 (2006)ADSGoogle Scholar
  34. 34.
    R. Deiterding, S.L. Wood, J. Phys.: Conf. Ser. 753, 082005 (2016)Google Scholar
  35. 35.
    M. Geier, A. Greiner, J.G. Korvink, Phys. Rev. E 73, 066705 (2006)ADSGoogle Scholar
  36. 36.
    P. Asinari, Phys. Rev. E 78, 016701 (2008)ADSGoogle Scholar
  37. 37.
    S. Seeger, H. Hoffmann, Continuum Mech. Thermodyn. 12, 403 (2000)ADSMathSciNetGoogle Scholar
  38. 38.
    M. Geier, M. Schönherr, A. Pasquali, M. Krafczyk, Comput. Math. Appl. 70, 507 (2015)MathSciNetGoogle Scholar
  39. 39.
    E.K. Far, M. Geier, K. Kutscher, M. Krafczyk, Comput. Fluids 140, 222 (2016)MathSciNetGoogle Scholar
  40. 40.
    K. Kutscher, M. Geier, M. Krafczyk, Comput. Fluids, in pressGoogle Scholar
  41. 41.
    M. Geier, M. Schönherr, Computation 5, 15 (2017)Google Scholar
  42. 42.
    M. Wittmann, T. Zeiser, G. Hager, G. Wellein, Modeling and analyzing performance for highly optimized propagation steps of the Lattice Boltzmann method on sparse lattices, arXiv:1410.0412 (2014)Google Scholar
  43. 43.
    U.D. Schiller, Thermal uctuations and boundary conditions in the Lattice Boltzmann method, Ph.D. thesis, Johannes Gutenberg-Universität, Mainz, 2008Google Scholar
  44. 44.
    L. Li, R. Mei, J.F. Klausner, J. Comput. Phys. 237, 366 (2013)ADSMathSciNetGoogle Scholar
  45. 45.
    A. Pasquali, M. Geier, M. Krafczyk, Comput. Math. Appl., in pressGoogle Scholar
  46. 46.
    A. Fakhari, T. Lee, Phys. Rev. E 89, 033310 (2014)ADSGoogle Scholar
  47. 47.
    A. Fakhari, T. Lee, Comput. Fluids 107, 205 (2015)MathSciNetGoogle Scholar
  48. 48.
    O. Filippova, D. Hänel, J. Comput. Phys. 147, 219 (1998)ADSGoogle Scholar
  49. 49.
    D. Yu, R. Mei, W. Shyy, Int. J. Numer. Methods Fluids 39, 99 (2002)ADSGoogle Scholar
  50. 50.
    J. Tölke, S. Freudiger, M. Krafczyk, Comput. Fluids 35, 820 (2006)Google Scholar
  51. 51.
    M. Geier, A. Greiner, J.G. Korvink, Eur. Phys. J. Special Topics 171, 173 (2009)ADSGoogle Scholar
  52. 52.
    H. Chen, Phys. Rev. E 58, 3955 (1998)ADSGoogle Scholar
  53. 53.
    H. Chen, O. Filippova, J. Hoch, K. Molvig, R. Shock, C. Teixeira, R. Zhang, Physica A 362, 158 (2006)ADSGoogle Scholar
  54. 54.
    M. Rohde, D. Kandhai, J.J. Derksen, H. Van den Akker, Int. J. Numer. Methods Fluids 51, 439 (2006)ADSGoogle Scholar
  55. 55.
    P. Neumann, Hybrid multiscale simulation approaches for micro- and nano ows, Ph.D. thesis, Technische Universität München, 2013Google Scholar
  56. 56.
    M. Lahnert, C. Burstedde, F. Weik, Towards Lattice-Boltzmann on dynamically adaptive grids – minimally-invasive grid exchange in ESPResSo. Englisch, in ECCOMAS Congress 2016, VII European Congress on Computational Methods in Applied Sciences and Engineering, edited by M. Papadrakakis, et al.(ECCOMAS, 2016)Google Scholar
  57. 57.
    M. Mehl, T. Neckel, P. Neumann, Int. J. Numer. Methods Fluids 65, 67 (2010)Google Scholar
  58. 58.
    M. Lahnert, C. Burstedde, M. Mehl, Scalable Lattice-Boltzmann Simulation on Dynamically Adaptive Grids, submittedGoogle Scholar
  59. 59.
    F. Capuani, I. Pagonabarraga, D. Frenkel, J. Chem. Phys. 121, 973 (2004)ADSGoogle Scholar
  60. 60.
    G. Rempfer, Electrokinetic transport phenomena in soft-matter systems, Ph.D. thesis, University of Stuttgart, 2018Google Scholar
  61. 61.
    G. Rempfer, G.B. Davies, C. Holm, J. de Graaf, J. Chem. Phys. 145, 044901 (2016)ADSGoogle Scholar
  62. 62.
    P.P. Ewald, Ann. Phys. 369, 253 (1921)Google Scholar
  63. 63.
    S.W. de Leeuw, J.W. Perram, E.R. Smith, Proc. R. Soc. London A: Math. Phys. Eng. Sci. 373, 27 (1980)ADSGoogle Scholar
  64. 64.
    R.W. Hockney, J.W. Eastwood, Computer simulation using particles (Taylor & Francis, Inc, Bristol, PA, USA, 1988)Google Scholar
  65. 65.
    M. Deserno, C. Holm, J. Chem. Phys. 109, 7678 (1998)ADSGoogle Scholar
  66. 66.
    M. Deserno, C. Holm, J. Chem. Phys. 109, 7694 (1998)ADSGoogle Scholar
  67. 67.
    A. Arnold, K. Breitsprecher, F. Fahrenberger, S. Kesselheim, O. Lenz, C. Holm, Entropy 15, 4569 (2013)ADSGoogle Scholar
  68. 68.
    J.E. Jones, Proc. Roy. Soc. London A: Math. Phys. Eng. Sci. 106, 463 (1924)ADSGoogle Scholar
  69. 69.
    J.E. Lennard-Jones, Proc. Phys. Soc. 43, 461 (1931)ADSGoogle Scholar
  70. 70.
    M. Buchholz, Framework zur Parallelisierung vonMolekulardynamiksimulationen in verfahrenstechnischen Anwendungen, Dissertation, München, Institut für Informatik, Technische Universität MünchenGoogle Scholar
  71. 71.
    T. Isaac, C. Burstedde, L.C. Wilcox, O. Ghattas, SIAM J. Sci. Comput. 37, C497 (2015)Google Scholar
  72. 72.
    M. Lahnert, T. Aoki, C. Burstedde, M. Mehl, Minimally-invasive integration of p4est in ESPResSo for adaptive Lattice-Boltzmann, in The 30th Computational Fluid Dynamics Symposium (Japan Society of Fluid Mechanics, 2016)Google Scholar
  73. 73.
    C. Burstedde, O. Ghattas, L.C. Wilcox, Towards adaptive mesh PDE simulations on petascale computers, in Proceedings of Teragrid (2008), Vol. 8Google Scholar
  74. 74.
    I. Tischler, Implementing adaptive electrokinetics in ESPResSo, MA thesis, University of Stuttgart, 2018Google Scholar
  75. 75.
    M. Brunn, Coupling of particle simulation and Lattice Boltzmann background flow on adaptive grids, MA thesis, Universität Stuttgart, 2017Google Scholar
  76. 76.
    S. Hirschmann, M. Brunn, M. Lahnert, C.W. Glass, D. Pflüger, Load Balancing with p4est for Short-Range Molecular Dynamics with ESPResSo, in Advances in parallel computing, edited by S. Bassini, et al. (IOS Press, 2017), Vol. 32, pp. 455–464Google Scholar
  77. 77.
    P. Ahlrichs, B. Dünweg, J. Chem. Phys. 111, 8225 (1999)ADSGoogle Scholar
  78. 78.
    B. Dünweg, A.J.C. Ladd, Adv. Polym. Sci. 221, 89 (2009)Google Scholar
  79. 79.
    S. Hirschmann, et al., Load-balancing and spatial adaptivity for coarse-grained molecular dynamics applications, in High Performance Computing in Science and Engineering ‘18 (Springer, 2018), forthcomingGoogle Scholar
  80. 80.
    F. Schornbaum, Block-structured adaptive mesh refinement for simulations on extreme-scale supercomputers, Doctoral thesis, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), 2018, p. 152Google Scholar
  81. 81.
    G. Wellein, T. Zeiser, G. Hager, S. Donath, Comput. Fluids 35, 910 (2006)Google Scholar
  82. 82.
    R.M.M. Smeets, U.F. Keyser, D. Krapf, M.Y. Wu, N.H. Dekker, C. Dekker, Nano Lett. 6, 89 (2006)ADSGoogle Scholar
  83. 83.
    G. Rempfer, S. Ehrhardt, C. Holm, J. de Graaf, Macromol. Theor. Simul. 26, 160051 (2016)Google Scholar
  84. 84.
    B. Kurz, Lattice-Boltzmann simulationen auf mehreren GPUs, Bachelor’s thesis, University of Stuttgart, 2018Google Scholar

Copyright information

© EDP Sciences, Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Institute for Parallel and Distributed SystemsStuttgartGermany

Personalised recommendations