Abstract
Considering that simulations of turbulent combustion are computationally expensive, this chapter takes a decidedly different perspective, that of high-performance computing (HPC). The cost scaling arguments of non-reacting turbulence simulations are revisited and it is shown that the cost scaling for reacting flows is much more stringent for comparable conditions, making parallel computing and HPC indispensable. Hardware abstractions of typical parallel supercomputers are presented which show that for design of an efficient and optimal program, it is essential to exploit both distributed memory parallelism and shared-memory parallelism, i.e. hierarchical parallelism. Principles of efficient programming at various levels of parallelism are illustrated using archetypal code examples. The vast array of numerical methods, particularly schemes for spatial and temporal discretization, are examined in terms of tradeoffs they present from an HPC perspective. Aspects of data analytics that invariably result from large feature-rich data sets generated by combustion simulations are covered briefly.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
This distinction is also referred to sometimes as internode versus intra-node or node-level parallelism.
- 2.
Early computer architectures had just a single level of cache between the processor and main memory. However, with the number of processors and their speed, increasing dramatically the current architecture has multiple (upto 3) cache levels. Furthermore, in multicore architectures, some levels of caches are not shared amongst all cores on a node, but subsets of them.
- 3.
This is a bit of an illusion. What a program addresses is not directly the physical location of memory but what is known as virtual memory. All arrays are stored contiguously in virtual memory, i.e. the memory addresses of successive elements of an array are contiguous in the virtual address space. The translation of the virtual address space to physical memory addresses is handled by memory management layer of an operating system.
- 4.
The Intel compiler suite has a very useful compiler option that generates a detailed vectorization report. When turned on, it generates output to a file that lists every portion of the code that was attempted to be vectorized by the compiler, whether the vectorization was successful and what, if anything, prevented a loop from being vectorized.
- 5.
The term multiprocessing is itself very general, simply referring to a system with multiple processors. The multicore nodes commonly found today may be thought to belong a subset known as symmetric multiprocessors (SMP) which strictly means that all the processors share all the memory and I/O resources equally and are orchestrated by one instance of an operating system kernel. The reality may be somewhere in between. Most modern node architectures have multiple sets of multicore CPUs, and they are not all exactly equal since at least one or more layers of memory hierarchy are not equally shared by all the cores. They are better described by a category known as non-uniform memory access (NUMA) nodes.
- 6.
NVIDIA, which has pioneered use of GPUs for scientific computing, has developed a full-fledged programming model—CUDA—that provides constructs for the thread hierarchy. The smallest group of threads that execute a common instruction is called a warp and from a performance perspective having all threads in a warp do the same computation without diverging is critical.
- 7.
The difference between spectral and pseudo-spectral methods lie in how the nonlinear convective term was handled. In the spectral method, all computations were in the wavenumber space, but pseudo-spectral methods use an intermediate step to transform the velocity Fourier modes to physical space, compute the convective term in the physical space and transform it back to the Fourier space.
References
GRI-Mech 3.0. http://www.me.berkeley.edu/gri_mech
Legion: a data-centric parallel programming system. http://legion.stanford.edu
MPI forum. http://mpi-forum.org
Open multi-processing. http://www.openmp.org
PETSc web page. http://www.mcs.anl.gov/petsc
Programming guidelines for vectorization. https://software.intel.com/en-us/node/695829
Trilinos home page. https://trilinos.org
Austin W, Ballard G, Kolda T (2016) Parallel tensor compression for large-scale scientific data. In: IPDPS’16: Proceedings of the 30th IEEE international parallel and distributed processing symposium, pp 912–922. https://doi.org/10.1109/IPDPS.2016.67
Bennett J, Abbasi H, Bremer PT, Grout R, Gyulassy A, Jin T, Klasky S, Kolla H, Parashar M, Pascucci V, Pebay P, Thompson D, Yu H, Zhang F, Chen J (2012) Combining in-situ and in-transit processing to enable extreme-scale scientific analysis. In: International conference on high performance computing, networking, storage and analysis (SC)
Bennett J, Clay R, Baker G, Gamell M, Hollman D, Knight S, Kolla H, Sjaardema G, Slattengren N, Teranishi K, Wilke J, Bettencourt M, Bova S, Franko K, Lin P, Grant R, Hammond S, Olivier S, Kale L, Jain N, Mikida E, Aiken A, Bauer M, Lee W, Slaughter E, Treichler S, Berzins M, Harman T, Humphrey A, Schmidt J, Sunderland D, McCormick P, Gutierrez S, Schulz M, Bhatele A, Boehme D, Bremer PT, Gamblin T (2015) ASC ATDM Level 2 Milestone #5325: asynchronous many-task runtime system analysis and assessment for next generation platforms. Technical report SAND2015-8312, Sandia National Laboratories, Albuquerque, NM
Bray K, Libby PA (1976) Interaction effects in turbulent premixed flames. Phys Fluids 19(11):1687–1701
Bruno C, Sankaran V, Kolla H, Chen J (2015) Impact of multi-component diffusion in turbulent combustion using direct numerical simulations. Combust Flame 162:4313–4330
Burke M, Chaos M, Ju Y, Dryer F, Klippenstein S (2012) Comprehensive \({\rm H}_2/{\rm O}_2\) kinetic model for high-pressure combustion. Int J Chem Kinet 44:444–474
Chaudhuri S, Kolla H, Dave H, Hawkes E, Chen J, Law C (2017) Flame thickness and conditional scalar dissipation rate in a premixed temporal turbulent reacting jet. Combust Flame 184:273–285
Chen JH, Choudhary A, de Supinski B, DeVries M, Hawkes ER, Klasky S, Liao WK, Ma KL, Mellor-Crummey J, Podhorszki N, Sankaran R, Shende S, Yoo CS (2009) Terascale direct numerical simulations of turbulent combustion using S3D. Comput Sci Discov 2:015001
Curran H, Gaffuri P, Pitz W, Westbrook C (2002) A comprehensive modeling study of iso-octane oxidation. Combust Flame 129:253–280
Denning P (2005) The locality principle. Commun ACM 48(7):19–24
Descombes S, Duarte M, Dumont T, Laurent F, Louvet V, Massot M (2014) Analysis of operator splitting in the non-asymptotic regime for nonlinear reaction-diffusion equations. Application to the dynamics of premixed flames. SIAM J Numer Anal 52:1311–1334
Echekki T, Chen J (1996) Unsteady strain rate and curvature effects in turbulent premixed methane-air flames. Combust Flame 106:184–202
Echekki T, Chen J (2002) High-temperature combustion in autoigniting non-homogeneous hydrogen/air mixtures. Proc Combust Inst 29:2061–2068
Franzelli B, Riber E, Sanjose M, Poinsot T (2010) A two-step chemical scheme for kerosene-air premixed flames. Combust Flame 157:1364–1373
Gamell M, Teranishi K, Mayo J, Kolla H, Heroux M, Chen J, Parashar M (2017) Modeling and simulating multiple failure masking enabled by local recovery for stencil-based applications at extreme scales. IEEE Trans Parallel Distrib Syst
Gropp W, Lusk E, Skjellum A (1994) Using MPI: portable parallel programming with the message-passing interface. Scientific and engineering computation series. MIT Press
Jenkins K, Cant R (1999) Direct numerical simulation of turbulent flame kernels. In: Knight D, Sakell L (eds) Proceedings of the second AFOSR conference on DNS and LES. Kluwer
Kee R, Rupley F, Miller J (1990) The CHEMKIN thermodynamic data base. Technical report SAND-87-8215B, Sandia National Laboratories
Kee RJ, Dixon-Lewis G, Warnatz J, Coltrin ME, Miller JA (1986) A Fortran computer code package for the evaluation of gas-phase multicomponent transport properties. Technical report SAND86-8246, Sandia National Laboratories
Kennedy C, Carpenter M (1994) Several new numerical methods for compressible shear-layer simulations. Appl Numer Math 14:397–433
Kolla H, Hawkes E, Kerstein A, Chen J (2014) On velocity and reactive scalar spectra in turbulent premixed flames. J Fluid Mech 754:456–487
Kolla H, Zhao XY, Chen J, Swaminathan N (2016) Velocity and reactive scalar dissipation spectra in turbulent premixed flames. Combust Sci Technol 188:1424–1439
Lele S (1992) Compact finite-difference schemes with spectral like resolution. J Comput Phys 103:16–42
Luong M, Luo Z, Lu T, Chung S, Yoo C (2013) Direct numerical simulations of the ignition of lean primary reference fuel/air mixtures under HCCI condition. Combust Flame 160:2038–2047
Najm H, Knio O (2005) Modeling low Mach number reacting flow with detailed chemistry and transport. J Sci Comput 25:263–287
Nonaka A, Bell J, Day M, Gilet C, Almgren A, Minion M (2012) A deferred correction coupling strategy for low Mach number flow with complex chemistry. Combust Theory Model
Pei Y, Mehl M, Liu W, Lu T, Pitz W, Som S (2015) A multi-component blend as a diesel fuel surrogate for compression ignition engine applications. J Eng Gas Turbines Power (GTP-15-1057)
Poinsot T, Veynante D (2012) Theoretical and numerical combustion, chapter 1. Edwards
Raman V, Fox RO (2016) Modeling of fine-particle formation in turbulent flames. Annu Rev Fluid Mech 48:159–190
Sankaran R, Hawkes E, Chen J, Lu T, Law C (2007) Structure of a spatially developing turbulent lean methane-air Bunsen flame. Proc Combust Inst 31:1291–1298
Schendel ER, Pendse SV, Jenkins J, Boyuka II DA, Gong Z, Lakshminarasimhan S, Liu Q, Kolla H, Chen J, Klasky S, Ross R, Samatova NF (2012) ISOBAR Hybrid compression-I/O interleaving for large-scale parallel I/O optimization. In: Proceedings of the 21st international symposium on high-performance parallel and distributed computing, HPDC ’12. ACM, New York, NY, USA, pp 61–72. https://doi.org/10.1145/2287076.2287086. http://doi.acm.org/10.1145/2287076.2287086
Strang G (1968) On the construction and comparison of difference schemes. SIAM J Numer Anal 5:503–517
Tomboulides A, Lee J, Orszag S (1997) Numerical simulation of Low-Mach number reactive flows. J Sci Comput 12:139–167
Treichler S, Bauer M, Bhagatwala A, Borghesi G, Sankaran R, Kolla H, McCormick P, Slaughter E, Lee W, Aiken A, Chen J (2018) S3D-Legion: an exascale software for direct numerical simulation (DNS) of turbulent combustion with complex multicomponent chemistry. In: Straatsma T, Williams KAT (eds) Exascale scientific applications scalability and performance portability, chapter 12. CRC Press
Ye Y, Neuroth T, Sauer F, Ma KL, Borghesi G, Konduri A, Kolla H, Chen J (2016) In situ generated probability distribution functions for interactive post hoc visualization and analysis. In: IEEE sixth symposium on large data analysis and visualization (LDAV)
Acknowledgements
This work was supported by Sandia National Laboratories, a multi-mission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC., a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energys National Nuclear Security Administration under contract DE-NA-0003525. The authors acknowledge the support of the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations—the Office of Science and the National Nuclear Security Administration—responsible for the planning and preparation of a capable exascale ecosystem including software, applications, hardware, advanced system engineering and early testbed platforms to support the nations exascale computing imperative. Helpful comments and inputs from Dr. Xinyu Zhao (University of Connecticut), Dr. Giulio Borghesi (Sandia National Labs) and Dr. Aditya Konduri (Sandia National Labs) are gratefully acknowledged.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Kolla, H., Chen, J.H. (2018). Turbulent Combustion Simulations with High-Performance Computing. In: De, S., Agarwal, A., Chaudhuri, S., Sen, S. (eds) Modeling and Simulation of Turbulent Combustion. Energy, Environment, and Sustainability. Springer, Singapore. https://doi.org/10.1007/978-981-10-7410-3_3
Download citation
DOI: https://doi.org/10.1007/978-981-10-7410-3_3
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-7409-7
Online ISBN: 978-981-10-7410-3
eBook Packages: EngineeringEngineering (R0)