Abstract
Obtaining a good load balance is a significant challenge in scaling up lattice-Boltzmann simulations of realistic sparse problems to the exascale. Here we analyze the effect of weighted decomposition on the performance of the HemeLB lattice-Boltzmann simulation environment, when applied to sparse domains. Prior to domain decomposition, we assign wall and in/outlet sites with increased weights which reflect their increased computational cost. We combine our weighted decomposition with a second optimization, which is to sort the lattice sites according to a space filling curve. We tested these strategies on a sparse bifurcation and very sparse aneurysm geometry, and find that using weights reduces calculation load imbalance by up to 85 %, although the overall communication overhead is higher than some of our runs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Palabos LBM Wiki (2011). http://wiki.palabos.org/
Cresta case study: application soars above petascale after tools collaboration (2014). http://www.cresta-project.eu/images/cresta_casestudy1_2014.pdf
ParMETIS (2014). http://glaros.dtc.umn.edu/gkhome/metis/parmetis/overview
Axner, L., Bernsdorf, J., Zeiser, T., Lammers, P., Linxweiler, J., Hoekstra, A.G.: Performance evaluation of a parallel sparse lattice Boltzmann solver. J. Computat. Phys. 227(10), 4895–4911 (2008)
Barad, M.F., Colella, P., Schladow, S.G.: An adaptive cut-cell method for environmental fluid mechanics. Int. J. Numer. Methods Fluids 60(5), 473–514 (2009)
Bhatnagar, P.L., Gross, E.P., Krook, M.: A model for collision processes in gases. i. small amplitude processes in charged and neutral one-component systems. Phys. Rev. 94, 511–525 (1954)
Bouzidi, M., Firdaouss, M., Lallemand, P.: Momentum transfer of a Boltzmann-lattice fluid with boundaries. Phys. Fluids 13(11), 3452–3459 (2001)
Carver, H.B., Groen, D., Hetherington, J., Nash, R.W., Bernabeu, M.O., Coveney, P.V.: Coalesced communication: a design pattern for complex parallel scientific software. Advances in Engineering Software (2015, in press)
Catalyurek, U.V., Boman, E.G., Devine, K.D., Bozdag, D., Heaphy, R., Riesen, L.A.: Hypergraph-based dynamic load balancing for adaptive scientific computations. In: IEEE International Parallel and Distributed Processing Symposium, 2007, IPDPS 2007, pp. 1–11. IEEE (2007)
Godenschwager, C., Schornbaum, F., Bauer, M., Köstler, H., Rüde, U.: A framework for hybrid parallel flow simulations with a trillion cells in complex geometries. In: Proceedings of SC13: International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2013, pp. 35:1–35:12. ACM, New York (2013)
Groen, D., Borgdorff, J., Bona-Casas, C., Hetherington, J., Nash, R.W., Zasada, S.J., Saverchenko, I., Mamonski, M., Kurowski, K., Bernabeu, M.O., Hoekstra, A.G., Coveney, P.V.: Flexible composition and execution of high performance, high fidelity multiscale biomedical simulations. Interface Focus 3(2), 20120087 (2013)
Groen, D., Henrich, O., Janoschek, F., Coveney, P.V., Harting, J.: Lattice-boltzmann methods in fluid dynamics: turbulence and complex colloidal fluids. In: Jülich Blue Gene/P Extreme Scaling Workshop (2011)
Groen, D., Hetherington, J., Carver, H.B., Nash, R.W., Bernabeu, M.O., Coveney, P.V.: Analyzing and modeling the performance of the HemeLB lattice-Boltzmann simulation environment. J. Comput. Sci. 4(5), 412–422 (2013)
Hasert, M., Masilamani, K., Zimny, S., Klimach, H., Qi, J., Bernsdorf, J., Roller, S.: Complex fluid simulations with the parallel tree-based lattice boltzmannsolver musubi. J. Comput. Sci. 5, 784–794 (2013)
Hendrickson, B., Kolda, T.G.: Graph partitioning models for parallel computing. Parallel Comput. 26(12), 1519–1534 (2000)
Mazzeo, M.D., Coveney, P.V.: HemeLB: a high performance parallel lattice-Boltzmann code for large scale fluid flow in complex geometries. Comput. Phys. Commun. 178(12), 894–914 (2008)
Mazzeo, M.D., Manos, S., Coveney, P.V.: In situ ray tracing and computational steering for interactive blood flow simulation. Comput. Phys. Commun. 181, 355–370 (2010)
Nash, R.W., Carver, H.B., Bernabeu, M.O., Hetherington, J., Groen, D., Krüger, T., Coveney, P.V.: Choice of boundary condition for lattice-Boltzmann simulation of moderate Reynolds number flow in complex domains. Phys. Rev. E 89, 023303 (2014)
Peters, A., Melchionna, S., Kaxiras, E., Lätt, J., Sircar, J., Bernaschi, M., Bison, M., Succi, S.: Multiscale simulation of cardiovascular flows on the ibm bluegene/p: full heart-circulation system at red-blood cell resolution. In: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2010, Washington, DC, USA, pp. 1–10. IEEE Computer Society (2010)
Acknowledgements
We thank Timm Krueger for his valuable input. This work has received funding from the CRESTA and MAPPER projects within the EC-FP7 (ICT-2011.9.13) under Grant Agreements nos. 287703 and 261507, and from EPSRC Grants EP/I017909/1 (www.2020science.net) and EP/I034602/1. This work made use of the HECToR supercomputer at EPCC in Edinburgh, funded by the Office of Science and Technology through EPSRC’s High End Computing Programme.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Groen, D., Chacra, D.A., Nash, R.W., Jaros, J., Bernabeu, M.O., Coveney, P.V. (2015). Weighted Decomposition in High-Performance Lattice-Boltzmann Simulations: Are Some Lattice Sites More Equal than Others?. In: Markidis, S., Laure, E. (eds) Solving Software Challenges for Exascale. EASC 2014. Lecture Notes in Computer Science(), vol 8759. Springer, Cham. https://doi.org/10.1007/978-3-319-15976-8_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-15976-8_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-15975-1
Online ISBN: 978-3-319-15976-8
eBook Packages: Computer ScienceComputer Science (R0)