Difference between revisions of "Codes"

From CFD Benchmark
Jump to: navigation, search
(YALES2)
(General comments on the codes)
 
(36 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
= Code Description =
 
= Code Description =
  
This page is dedicated to a short presentation of the three low-Mach number codes used for the rest of this benchmarl: [https://www.coria-cfd.fr/index.php/YALES2 YALES2], [http://www.lss.ovgu.de/lss/en/Research/Computational+Fluid+Dynamics.html DINO] and [http://nek5000.mcs.anl.gov Nek5000].  
+
This page is dedicated to a short presentation of the three '''low-Mach number''' codes used for the rest of this benchmarl: [https://www.coria-cfd.fr/index.php/YALES2 YALES2], [http://www.lss.ovgu.de/lss/en/Research/Computational+Fluid+Dynamics.html DINO] and [http://nek5000.mcs.anl.gov Nek5000].  
  
Since these codes have already been the subject of many publications and are not completely new, only the most relevant features are discussed in what follows, with suitable references for those readers needing more details.
+
Since these codes have already been the subject of many publications and are not completely new, '''only the most relevant features are discussed in what follows''', with suitable references for those readers needing more details.
  
 
== YALES2 ==
 
== YALES2 ==
  
YALES2 is a massively parallel multiphysics platform<ref>V. Moureau. Yales2 public website [https://www.coria-cfd.fr/index.php/YALES2].</ref> developed since 2009 by Moureau, Lartigue and co-workers at CORIA (Rouen, France).
+
YALES2 is a '''massively parallel multiphysics platform'''<ref name="moureau2011"/> developed since 2009 by Moureau, Lartigue and co-workers at CORIA (Rouen, France).
 
+
It is dedicated to the high-fidelity simulation of '''low-Mach number flows in complex geometries'''.
YALES2-website,moureau2011large developed since 2009 by Moureau, Lartigue and co-workers at CORIA (Rouen, France).
+
It is based on the '''Finite Volumes formulation''' of the '''Navier-Stokes equations''' and it can solve both '''non-reacting''' and '''reacting flows'''.
It is dedicated to the high-fidelity simulation of low-Mach number flows in complex geometries.
+
It is based on the Finite Volumes formulation of the Navier-Stokes equations and it can solve both non-reacting and reacting flows.
+
 
It can actually solve various physical problems thanks to its original structure which is composed of a main numerical library accompanied with tens of dedicated solvers (for acoustics, multiphase flows, heat transfer, radiation\ldots) which can be coupled with one another.
 
It can actually solve various physical problems thanks to its original structure which is composed of a main numerical library accompanied with tens of dedicated solvers (for acoustics, multiphase flows, heat transfer, radiation\ldots) which can be coupled with one another.
YALES2 relies on unstructured meshes and a fully parallel dynamic mesh adaptation technique to improve the resolution in physically-relevant zones to mitigate the computational cost<ref>~\cite{benard2015mesh}.</ref>.  
+
YALES2 relies on '''unstructured meshes''' and a '''fully parallel dynamic mesh adaptation technique''' to improve the resolution in physically-relevant zones to mitigate the computational cost<ref name="benard2015"/>.  
As a result it can easily handle meshes composed of billions of tetrahedra, thus enabling the Direct Numerical Simulation of laboratory and semi-industrial configurations.
+
As a result it can easily handle meshes composed of billions of tetrahedra, thus enabling the '''Direct Numerical Simulation''' of laboratory and semi-industrial configurations.
It is now composed of nearly 500,000 lines of object-oriented Fortran and the parallelism is currently ensured by a pure MPI paradigm, although a hybrid OpenMP/MPI as well as a GPU version are under development.
+
It is now composed of nearly 500,000 lines of object-oriented Fortran and the parallelism is currently ensured by a pure '''MPI''' paradigm, although a hybrid '''OpenMP/MPI''' as well as a GPU version are under development.
  
As most low-Mach number codes, the time-advancement is based on a projection-correction method following the pioneering work of~\cite{chorin1968}.
+
As most low-Mach number codes, the time-advancement is based on a '''projection-correction method''' following the pioneering work of<ref name="chorin1968"/>.
The prediction step uses a method which is a blend between a 4th-order Runge-Kutta method and a 4th-order Lax-Wendroff-like method~\cite{KraushaarPhD}, combined with a 4th-order node-based centered finite-volume discretization of the convective and diffusive terms.
+
The prediction step uses a method which is a blend between a '''4th-order Runge-Kutta method''' and a '''4th-order Lax-Wendroff-like method'''<ref name="kraushaar2011"/>, combined with a '''4th-order node-based centered finite-volume discretization''' of the convective and diffusive terms.
Moreover, to improve the performance of the correction step, the pressure of the previous iteration is included in the prediction step to limit the splitting errors.
+
Moreover, to improve the performance of the correction step, the pressure of the previous iteration is included in the prediction step to limit the '''splitting errors'''.
The correction step is required to ensure mass conservation, using a pressure that arises from a Poisson equation.
+
The correction step is required to ensure '''mass conservation''', using a pressure that arises from a '''Poisson equation'''.
This is performed numerically by solving a linear system on the pressure at each node of the mesh thanks to a dedicated in-house version of the deflated conjugate gradient algorithm, which has been optimized for solving elliptic equations on massively parallel machines~\cite{malandain2013optimization}.
+
This is performed numerically by solving a linear system on the pressure at each node of the mesh thanks to a dedicated in-house version of the deflated conjugate gradient algorithm, which has been optimized for solving elliptic equations on massively parallel machines<ref name="malandain2013"/>.
  
When considering multispecies and non-isothermal flows, the extension of the classical projection method proposed by Pierce et al.~\cite{pierce2004progress} is used to account for variable-density flows.
+
When considering '''multispecies''' and '''non-isothermal flows''', the extension of the classical projection method proposed by Pierce et al.<ref name="pierce2004"/> is used to account for '''variable-density flows'''.
All the thermodynamic and transport properties are provided by the Cantera software~\cite{goodwin2003open}, which has been fully re-implemented in Fortran to avoid any performance issues.
+
All the thermodynamic and transport properties are provided by the '''Cantera software'''<ref name="goodwin2003"/>, which has been fully re-implemented in '''Fortran''' to avoid any performance issues.
Each species thermodynamic property is specified by 5th-order polynomials on two temperature ranges (below and above 1000 K).
+
Each species thermodynamic property is specified by '''5th-order polynomials''' on two temperature ranges (below and above 1000 K).
 
The mixture is supposed to be both thermally and mechanically perfect.
 
The mixture is supposed to be both thermally and mechanically perfect.
Regarding transport properties, several type of models are implemented in YALES2: 1) the default approach is based on the computation of transport properties for each species (using tabulated molecular potentials), then combining those to obtain mixture-averaged coefficients for viscosity, conductivity and diffusion velocities (Hirschfelder and Curtiss approximation~\cite{hirschfelder1964molecular}); 2) alternatively, simplified laws (for example the Sutherland law~\cite{Sutherland}) can be used for the viscosity, while fixed values for Prandtl and Schmidt numbers allow the computation of conductivity and diffusion coefficients; and 3) a mix of both approaches, for example computing viscosity and conductivity with a mixture-averaged approximation and then imposing a Lewis number for each species.
+
 
 +
Regarding transport properties, several type of models are implemented in YALES2:  
 +
 
 +
1) the default approach is based on the computation of transport properties for each species (using tabulated molecular potentials), then combining those to obtain mixture-averaged coefficients for '''viscosity''', '''conductivity''' and '''diffusion velocities''' (Hirschfelder and Curtiss approximation<ref name="hirschfelder1964"/>;  
 +
 
 +
2) alternatively, simplified laws (for example the Sutherland law<ref name="sutherland1893"/> can be used for the '''viscosity''', while fixed values for '''Prandtl''' and '''Schmidt numbers''' allow the computation of '''conductivity''' and '''diffusion coefficients''';  
 +
 
 +
3) a mix of both approaches, for example computing '''viscosity''' and '''conductivity''' with a mixture-averaged approximation and then imposing a '''Lewis number''' for each species.
 
This last approach has been retained in the present benchmark.
 
This last approach has been retained in the present benchmark.
  
Finally, the source terms used in the reacting simulations are modeled with an Arrhenius law with the necessary modifications needed to take into account three-body or pressure-dependent reactions.
+
Finally, the source terms used in the '''reacting simulations''' are modeled with an '''Arrhenius law''' with the necessary modifications needed to take into account three-body or pressure-dependent reactions.
  
From a numerical point of view, it must be noticed that both the diffusion and reaction processes occur at time scales which can be orders of magnitude smaller than the convective time scale.
+
From a numerical point of view, it must be noticed that both the '''diffusion''' and '''reaction processes''' occur at time scales which can be orders of magnitude smaller than the convective time scale.
Solving these phenomena with explicit methods would thus drastically limit the global timestep of the whole simulation and induce an overwhelming CPU cost.
+
Solving these phenomena with explicit methods would thus drastically limit the '''global timestep''' of the whole simulation and induce an overwhelming CPU cost.
To mitigate these effects when dealing with multi-species reacting flows, the classical operator splitting technique is used.
+
To mitigate these effects when dealing with '''multi-species reacting flows''', the classical operator '''splitting technique''' is used.
The diffusion process is solved with a fractional timestep method inside each convective iteration, each substep being limited by a Fourier condition to ensure stability.  
+
The '''diffusion process''' is solved with a fractional timestep method inside each convective iteration, each substep being limited by a '''Fourier condition''' to ensure stability.  
This method gets activated only when the mesh is very fine (typically when performing DNS with very diffusive species like H$_2$ or H, as in the present project); otherwise an explicit treatment is sufficient.
+
This method gets activated only when the mesh is very fine (typically when performing DNS with very diffusive species like <math>H_2</math> or <math>H</math>, as in the present project); otherwise an explicit treatment is sufficient.
The chemical source terms are then integrated with a dedicated stiff solver, namely the CVODE library from SUNDIALS~\cite{cohen1996cvode,cvodeUG,sundials}.  
+
The chemical source terms are then integrated with a dedicated stiff solver, namely the '''CVODE''' library from SUNDIALS<ref name="cohen1996cvode"/><ref name="cvodeUG">A.C. Hindmarsh and R. Serban. User documentation for '''CVODE'''. [https://computing.llnl.gov/sites/default/files/public/cv_guide.pdf].</ref><ref name="sundials"/>.  
 
To that purpose, each control volume is considered as an isolated reactor with both constant pressure and enthalpy.
 
To that purpose, each control volume is considered as an isolated reactor with both constant pressure and enthalpy.
  
 
Using the stiff integration technique results in a very strong load imbalance between the various regions of the flow: the fresh gases are solved in a very small number of integration steps while the inner flame region may require tens or even hundreds of integration steps.  
 
Using the stiff integration technique results in a very strong load imbalance between the various regions of the flow: the fresh gases are solved in a very small number of integration steps while the inner flame region may require tens or even hundreds of integration steps.  
To overcome this difficulty, a dedicated MPI dynamic scheduler based on a work-sharing algorithm ensures global load-balancing.
+
To overcome this difficulty, a dedicated '''MPI''' dynamic scheduler based on a work-sharing algorithm ensures global load-balancing.
  
 
== DINO ==
 
== DINO ==
  
DINO is a 3-D DNS code used for incompressible or low-Mach number flows, the latter approach being used in this project.  
+
DINO is a '''3-D DNS code''' used for '''incompressible''' or '''low-Mach number flows''', the latter approach being used in this project.  
The development of DINO started in the group of D. Thev\'enin (Univ. of Magdeburg) in 2013.  
+
The development of DINO started in the group of D. Thévenin (Univ. of Magdeburg) in 2013.  
DINO is a Fortran-90 code, written on top of a 2-D pencil decomposition to enable efficient large-scale parallel simulations on distributed-memory supercomputers by coupling with the open-source library 2-DECOMP\&FFT~\cite{2decompfft}.  
+
DINO is a '''Fortran-90 code''', written on top of a 2-D pencil decomposition to enable efficient large-scale parallel simulations on distributed-memory supercomputers by coupling with the open-source library 2-DECOMP&FFT<ref name="li2010"/>.  
 
The code offers different features and algorithms in order to investigate different physicochemical processes.  
 
The code offers different features and algorithms in order to investigate different physicochemical processes.  
Spatial derivative are computed by default using sixth-order central finite differences.  
+
Spatial derivative are computed by default using '''sixth-order central finite differences'''.  
Time integration relies on several Runge-Kutta solvers.  
+
Time integration relies on '''several Runge-Kutta solvers'''.  
In what follows, an explicit 4th-order Runge-Kutta approach has been used.  
+
In what follows, an explicit '''4th-order Runge-Kutta approach''' has been used.  
A 3rd-order semi-implicit Runge-Kutta integration can be activated as needed, when considering stiff chemistry.  
+
A '''3rd-order semi-implicit Runge-Kutta integration''' can be activated as needed, when considering stiff chemistry.  
In this case, non-stiff terms are still computed with explicit Runge-Kutta, while the PyJac package~\cite{pyjac} is used to integrate in an implicit manner all chemistry terms with an analytical Jacobian computation.  
+
In this case, non-stiff terms are still computed with explicit Runge-Kutta, while the '''PyJac package'''<ref>Create analytical jacobian matrix source code for chemical kinetics.</ref> is used to integrate in an implicit manner all chemistry terms with an analytical Jacobian computation.  
All thermodynamic, chemical and transport properties are computed using the open-source library Cantera 2.4.0~\cite{cantera}.
+
All thermodynamic, chemical and transport properties are computed using the open-source library '''Cantera 2.4.0'''<ref>Cantera.</ref>.  
The transport properties can be computed based on three different models: 1) Constant Lewis numbers, 2) mixture-averaged, 3) full multicomponent diffusion, by coupling either again with Cantera or with the open-source library EGlib \cite{Ern}.  
+
The Poisson equation is solved using Fast Fourier Transform (FFT) for periodic as well as for non-periodic boundary conditions, relying in the latter case on an in-house pre- and post-processing technique.
+
The I/O operations are implemented using two different approaches: (1) binary MPI-I/O using 2-DECOMP\&FFT for check-points and restart files; (2) HDF5 files used for analysis and visualization.
+
  
Multi-phase flows can be simulated in DINO using resolved or non-resolved (point) particles and droplets using a Lagrangian approach~\cite{Abdelsamie2019,Abdelsamie2020_dles}.
+
The transport properties can be computed based on three different models:
Complex boundaries are represented by a novel second-order immersed boundary method implementation (IBM) based on a directional extrapolation scheme~\cite{Chi2020}.
+
More details about the implemented algorithms can be found in particular in~\cite{Abdelsamie2016,Chi2019}.
+
Since DINO has been developed as a multi-purpose code for analyzing many different reacting and non-reacting flows~\cite{Abdelsamie2015_dles,Oster2018,Abdelsamie2019_dles}, a detailed verification and validation is obviously essential.
+
  
 +
1) Constant Lewis numbers,
 +
 +
2) mixture-averaged,
 +
 +
3) full multicomponent diffusion, by coupling either again with '''Cantera''' or with the open-source library EGlib<ref name="ern1995"/>.
 +
 +
The '''Poisson equation''' is solved using Fast Fourier Transform ('''FFT''') for '''periodic''' as well as for '''non-periodic boundary conditions''', relying in the latter case on an in-house pre- and post-processing technique.
 +
The I/O operations are implemented using two different approaches: (1) binary '''MPI-I/O''' using 2-DECOMP&FFT for check-points and restart files; (2) HDF5 files used for analysis and visualization.
 +
 +
'''Multi-phase flows''' can be simulated in DINO using resolved or non-resolved (point) particles and droplets using a '''Lagrangian approach'''<ref name="abdelsamie2019"/><ref name="abdelsamie2020"/>. Complex boundaries are represented by a novel second-order immersed boundary method implementation ('''IBM''') based on a directional extrapolation scheme<ref name="chi2020"/>.
 +
More details about the implemented algorithms can be found in particular in<ref name="abdelsamie2016"/>.
 +
Since DINO has been developed as a multi-purpose code for analyzing many different '''reacting''' and '''non-reacting flows'''<ref name="chi2018"/><ref name="oster2018"/><ref name="abdelsamie2019-2"/>, a detailed verification and validation is obviously essential.
  
 
== Nek5000 ==
 
== Nek5000 ==
  
This spectral element low-Mach number reacting flow solver is based on the highly-efficient open-source solver Nek5000~\cite{Nek5000} extended by a plugin developed at ETH implementing a high-order splitting scheme for low-Mach number reacting flows~\cite{tomboulides1997}.  
+
This '''spectral element low-Mach number reacting flow solver''' is based on the highly-efficient open-source solver Nek5000<ref>Nek5000 version v17.0, Argonne National Laboratory, IL, U.S.A.</ref> extended by a plugin developed at ETH implementing a '''high-order splitting scheme''' for '''low-Mach number reacting flows'''<ref name="tomboulides1997"/>.  
The spectral element method (SEM) is a high-order weighted residual technique for spatial discretization that combines the accuracy of spectral methods with the geometric flexibility of the finite element method allowing for accurately representation of complex geometries~\cite{deville2002}.
+
The spectral element method ('''SEM''') is a '''high-order weighted residual technique''' for spatial discretization that combines the accuracy of spectral methods with the geometric flexibility of the finite element method allowing for accurately representation of complex geometries<ref name="deville2002"/>.
The computational domain is decomposed into $E$ conforming elements, which are quadrilaterals (in 2-D) or hexahedra (in 3-D) that conform to the domain boundaries.  
+
The computational domain is decomposed into <math>E</math> conforming elements, which are '''quadrilaterals''' ('''in 2-D''') or '''hexahedra''' (in '''3-D''') that conform to the domain boundaries.  
Within each element, functions are expanded as $N$th-order polynomials so that resolution can be increased either by decreasing the element size (\emph{h}-type refinement) or by increasing the polynomial order (\emph{p}-type refinement; typically N = 7 - 15).  
+
Within each element, functions are expanded as <math>N</math>th-order polynomials so that resolution can be increased either by decreasing the element size (<math>h</math>-type refinement) or by increasing the polynomial order (<math>p</math>-type refinement; typically <math>N = 7 - 15</math>).  
The grids can be unstructured and allow for static local refinement, while adaptive mesh refinement has been recently developed~\cite{Tanarro2020}.  
+
The grids can be unstructured and allow for '''static local refinement''', while '''adaptive mesh refinement''' has been recently developed<ref name="tanarro2020"/>.  
By casting the polynomial approximation in tensor-product form, the differential operators on $N^3$ gridpoints per element can be evaluated with only $O(N^4)$ work and $O(N^3)$ storage.
+
By casting the '''polynomial approximation''' in tensor-product form, the differential operators on <math>N^3</math> gridpoints per element can be evaluated with only <math>O(N^4)</math> work and <math>O(N^3)</math> storage.
  
The principal advantage of the SEM is that convergence is exponential in $N$, yielding minimal numerical dispersion and dissipation, so that significantly fewer grid points per wavelength are required in order to accurately propagate a turbulent structure  
+
The principal advantage of the '''SEM''' is that convergence is exponential in <math>N</math>, yielding minimal '''numerical dispersion''' and '''dissipation''', so that significantly fewer grid points per wavelength are required in order to accurately propagate a turbulent structure over the extended time required in high Reynolds number simulations.  
over the extended time required in high Reynolds number simulations.  
+
Nek5000 uses locally '''structured basis coefficients''' (<math>N\times N \times N</math> arrays), which allow direct addressing and tensor-product-based derivative evaluation that can be cast as efficient matrix-matrix products involving <math>N^2</math> operators applied to <math>N^3</math> data values for each element.  
Nek5000 uses locally structured basis coefficients ($N\times N \times N$ arrays), which allow direct addressing and tensor-product-based derivative evaluation that can be cast as efficient matrix-matrix products involving $N^2$ operators applied to $N^3$ data values for each element.  
+
As a result, data movement per grid point is the same as for '''low-order methods'''.  
As a result, data movement per grid point is the same as for low-order methods.  
+
 
It uses scalable domain-decomposition-based iterative solvers with efficient preconditioners.  
 
It uses scalable domain-decomposition-based iterative solvers with efficient preconditioners.  
Communication is based on the Message Passing Interface (MPI) standard, and the code has proven scalability to over one million ranks.  
+
Communication is based on the Message Passing Interface ('''MPI''') standard, and the code has proven scalability to over one million ranks.  
 
Nek5000 provides balanced I/O latency among all processors and reduces the overhead or even completely hides the I/O latency by using dedicated I/O communicators in the optimal case.
 
Nek5000 provides balanced I/O latency among all processors and reduces the overhead or even completely hides the I/O latency by using dedicated I/O communicators in the optimal case.
  
Time advancement is performed using the splitting scheme proposed in~\cite{tomboulides1997} to decouple the highly non-linear and stiff thermochemistry (species and energy governing equations) from the hydrodynamic system (continuity and momentum).
+
'''Time advancement''' is performed using the '''splitting scheme''' proposed in<ref name="tomboulides1997" /> to decouple the highly non-linear and stiff thermochemistry (species and energy governing equations) from the hydrodynamic system (continuity and momentum).
Species and energy equations are integrated without further splitting using the implicit stiff integrator solver CVODE from the SUNDIALS package~\cite{cohen1996cvode,cvodeUG,sundials} that uses backward differentiation formulas (BDF).
+
Species and energy equations are integrated without further '''splitting using the implicit stiff integrator solver CVODE from the SUNDIALS package'''<ref name="cohen1996cvode" /><ref name="cvodeUG" /><ref name="sundials" /> that uses backward differentiation formulas (BDF).
The continuity and momentum equations are integrated using a second- or third-order semi-implicit formulation (EXT/BDF) treating the non-linear advection term explicitly~\cite{deville2002}.
+
The '''continuity''' and '''momentum equations''' are integrated using a '''second-''' or '''third-order semi-implicit formulation''' (EXT/BDF) treating the non-linear advection term explicitly<ref name="deville2002" />.
The thermodynamic properties, detailed chemistry, and transport properties are provided by optimized subroutines compatible with Chemkin~\cite{chemkin}.
+
The thermodynamic properties, detailed chemistry, and transport properties are provided by optimized subroutines compatible with '''Chemkin'''<ref name="chemkin"/>.
 
+
The reacting flow solver can handle complex time-varying geometries and has been used for instance to simulate laboratory-scale internal combustion engines~\cite{MS, TCC}.
+
It can account for conjugate fluid-solid heat transfer and detailed gas phase as well as surface kinetics~\cite{catalytic}.
+
  
 +
The reacting flow solver can handle complex time-varying geometries and has been used for instance to simulate laboratory-scale internal combustion engines<ref name="MS"/><ref name="TCC"/>.
 +
It can account for conjugate fluid-solid heat transfer and detailed gas phase as well as surface kinetics<ref name="catalytic"/>.
  
 
== General comments on the codes ==
 
== General comments on the codes ==
  
The three aforementioned solvers are all unsteady, high-fidelity codes based on the low-Mach number formulation of the Navier-Stokes equations.
+
The three '''aforementioned solvers''' are all '''unsteady''', '''high-fidelity codes''' based on the '''low-Mach number''' formulation of the '''Navier-Stokes equations'''.
They can perform both Direct Numerical Simulations or Large Eddy Simulations of reacting flows, only DNS being considered here.
+
They can perform both '''Direct Numerical Simulations''' or '''Large Eddy Simulations''' of reacting flows, only '''DNS''' being considered here.
 
However, they differ in a certain number of points, mainly from the numerical point of view.
 
However, they differ in a certain number of points, mainly from the numerical point of view.
 
The aim of this section is to emphasize those major differences.
 
The aim of this section is to emphasize those major differences.
First, YALES2 is an unstructured code, designed to handle any type of elements; its main application field pertains to LES of industrially relevant flows, though DNS is possible as well.  
+
First, '''YALES2''' is an '''unstructured code''', designed to handle any type of elements; its main application field pertains to '''LES''' of industrially relevant flows, though '''DNS''' is possible as well.  
On the other hand, both DINO and Nek5000 are mostly dedicated to DNS of configurations found in fundamental research.
+
On the other hand, both '''DINO''' and '''Nek5000''' are mostly dedicated to '''DNS''' of configurations found in fundamental research.
Both DINO and Nek5000 can only deal with quads or hexas, with the major difference that DINO is based on a structured connectivity while Nek5000 can use unstructured meshes (pavings).
+
Both '''DINO''' and '''Nek5000''' can only deal with '''quads''' or '''hexas''', with the major difference that '''DINO''' is based on a '''structured connectivity''' while '''Nek5000''' can use '''unstructured meshes''' (pavings).
As a consequence, both DINO and Nek5000 employ higher-order numerical schemes compared to YALES2, limited at best to a 4th-order scheme.
+
As a consequence, both '''DINO''' and '''Nek5000''' employ '''higher-order numerical schemes''' compared to '''YALES2''', limited at best to a '''4th-order scheme'''.
All codes rely on dedicated libraries to compute the thermo-chemical properties of the flow, either Chemkin, Cantera, or in-house versions of those.
+
All codes rely on dedicated libraries to compute the thermo-chemical properties of the flow, either '''Chemkin''', '''Cantera''', or in-house versions of those.
 
Moreover, they also rely at least to some extent on external software to perform the temporal integration of the stiff chemical source terms.
 
Moreover, they also rely at least to some extent on external software to perform the temporal integration of the stiff chemical source terms.
Regarding the Poisson equation for pressure which must be solved by all codes, DINO relies on a spectral formulation by performing direct and inverse Fourier transforms, which is possible thanks to its structured mesh.
+
Regarding the Poisson equation for pressure which must be solved by all codes, '''DINO''' relies on a spectral formulation by performing direct and inverse Fourier transforms, which is possible thanks to its structured mesh.
On the other hand, both YALES2 and Nek5000 use an iterative solver with an efficient preconditioning technique; this method is more versatile and should be computationally more efficient for large and complex geometrical configurations.
+
On the other hand, both '''YALES2''' and '''Nek5000''' use an iterative solver with an efficient preconditioning technique; this method is more versatile and should be computationally more efficient for large and complex geometrical configurations.
Nek5000 employs CVODE to integrate the thermochemical equations without further splitting of the different terms accounting for convection, diffusion and chemistry.
+
'''Nek5000''' employs CVODE to integrate the thermochemical equations without further '''splitting''' of the different terms accounting for convection, diffusion and chemistry.
The main differences between the three codes are summarized in Table~\ref{Tab:codes}.  
+
The main differences between the three codes are summarized in the table below.  
 
Please note that the presented values are those used for the benchmark, even though some other options are available in each codes.
 
Please note that the presented values are those used for the benchmark, even though some other options are available in each codes.
  
<nowiki>
+
{| class="wikitable alternance center"
\begin{table}[H]
+
|+ Table 1: Major numerical properties of the three high-fidelity codes as used in this benchmark
\resizebox{\textwidth}{!}{\begin{tabular}{|| l || c || c || c ||}
+
|-
\hline
+
! scope="col" | Code
\hline
+
! scope="col" | YALES2
Code & YALES2 & DINO & Nek5000 \\
+
! scope="col" | DINO
\hline
+
! scope="col" | Nek5000
\hline
+
|-
Connectivity & Unstructured & Structured & Unstructured \\
+
| Connectivity
\hline
+
| Unstructured
Discretization Type & Finite Volumes & Finite Differences & Spectral Elements \\
+
| Structured
\hline
+
| Unstructured
Grid point distribution & Regular hexahedra. & Regular hexahedra & Regular hexahedra with GLL points \\
+
|-
\hline
+
| Discretization Type
Spatial order & 4th & 6th & 7th - 15th (typically) \\
+
| Finite Volumes
\hline
+
| Finite Differences
Temporal method & expl. RK4 & expl. RK4 / semi-impl. RK3 & semi-impl. BDF3 \\
+
| Spectral Elements
\hline
+
|-
Pressure solver & CG with Deflation Prec. & FFT-based & CG/GMRES with Jacobi/Schwartz Prec. \\
+
| Grid point distribution
\hline
+
| Regular hexahedra
Thermo-chemistry & Cantera (re-coded) & Cantera & Chemkin interface \\
+
| Regular hexahedra
\hline
+
| Regular hexahedra with GLL points
Chemistry integration & CVODE & PyJac & CVODE \\
+
|-
\hline
+
| Spatial order
Operator splitting & Yes & ??? & No \\
+
| 4th
\hline
+
| 6th
Parallel paradigm & MPI & MPI & MPI \\
+
| 7th - 15th (typically)
        \hline
+
|-
\hline
+
| Temporal method
\end{tabular}}
+
| expl. RK4
\caption{Major numerical properties of the three high-fidelity codes as used in this benchmark}
+
| expl. RK4 / semi-impl. RK3
\label{Tab:codes}
+
| semi-impl. BDF3
\end{table}
+
|-
</nowiki>
+
| Pressure solver
 +
| CG with Deflation MPrec.
 +
| FFT-based
 +
| CG/GMRES with Jacobi/Schwartz Prec.
 +
|-
 +
| Thermo-chemistry
 +
| Cantera (re-coded)
 +
| Cantera
 +
| Chemkin interface
 +
|-
 +
| Chemistry integration
 +
| CVODE
 +
| expl. RK4
 +
| CVODE
 +
|-
 +
| Operator splitting
 +
| Yes
 +
| No
 +
| No
 +
|-
 +
| Parallel paradigm
 +
| MPI
 +
| MPI
 +
| MPI
 +
|}
 +
 
 +
= References =
 +
 
 +
<references>
 +
<ref name="moureau2011">
 +
<bibtex>
 +
@article{Moureau2011,
 +
author= {V. Moureau, P. Domingo, and L. Vervisch},
 +
title= {From large-eddy simulation to direct numerical simulation of a lean premixed swirl flame: Filtered laminar flame-pdf modeling},
 +
journal={Combust. Flame},
 +
year= {2011},
 +
volume={158},
 +
number={7},
 +
pages={1340--1357},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="benard2015">
 +
<bibtex>
 +
@article{Benard2015,
 +
author= {P. Benard, G. Balarac, V. Moureau, C. Dobrzynski, G. Lartigue, and Y. D’Angelo},
 +
title= {Mesh adaptation for largeeddy simulations in complex geometries},
 +
journal={Int. J. Numer. Methods Fluids},
 +
year= {2015},
 +
volume={81},
 +
pages={719--740},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="chorin1968">
 +
<bibtex>
 +
@article{Chorin1968,
 +
author= {A.J. Chorin},
 +
title= {Numerical solution of the Navier-Stokes equations},
 +
journal={Math. Computation},
 +
year= {1968},
 +
volume={22},
 +
number={104},
 +
pages={745--762},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="kraushaar2011">
 +
<bibtex>
 +
@article{Kraushaar2011,
 +
author= {M. Kraushaar},
 +
title= {Application of the compressible and low-Mach number approaches to Large-Eddy Simulation of turbulent flows in aero-engines},
 +
journal={ PhD thesis, PhD, Institut National Polytechnique de Toulouse-INPT},
 +
year= {2011},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="malandain2013">
 +
<bibtex>
 +
@article{Malandain2013,
 +
author= {M. Malandain, N. Maheu, and V. Moureau},
 +
title= {Optimization of the deflated conjugate gradient algorithm for the solving of elliptic equations on massively parallel machines},
 +
journal={J. Comput. Phys.},
 +
year= {2013},
 +
volume={238},
 +
pages={32--47},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="pierce2004">
 +
<bibtex>
 +
@article{Pierce2004,
 +
author= {C.D. Pierce and P. Moin},
 +
title= {Progress-variable approach for large-eddy simulation of non-premixed turbulent combustion},
 +
journal={J. Fluid Mech.},
 +
year= {2004},
 +
volume={504},
 +
pages={73--97},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="goodwin2003">
 +
<bibtex>
 +
@article{Goodwin2003,
 +
author= {A.J. Chorin},
 +
title= {An open-source, extensible software suite for CVD process simulation},
 +
journal={Chemical Vapor Deposition XVI and EUROCVD},
 +
year= {2003},
 +
volume={14},
 +
pages={2003--2008},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="hirschfelder1964">
 +
<bibtex>
 +
@article{Hirschfelder1964,
 +
author= {J. Hirschfelder, R.B. Bird, and C.F. Curtiss},
 +
title= {Molecular Theory of Gases and Liquids},
 +
journal={Wiley},
 +
year= {1964},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="sutherland1893">
 +
<bibtex>
 +
@article{Sutherland1893,
 +
author= {W. Sutherland. LII},
 +
title= {The viscosity of gases and molecular force},
 +
journal={The London, Edinburgh, and Dublin Philosophical
 +
Magazine and Journal of Science},
 +
year= {1893},
 +
volume={36},
 +
number={223},
 +
pages={507--531},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="cohen1996cvode">
 +
<bibtex>
 +
@article{Cohen1996cvode,
 +
author= {S.D. Cohen, A.C. Hindmarsh, and P.F. Dubois},
 +
title= {CVODE, a stiff/nonstiff ODE solver in C},
 +
journal={Computers Phys.},
 +
year= {1996},
 +
volume={10},
 +
number={2},
 +
pages={138--143},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="sundials">
 +
<bibtex>
 +
@article{Sundials,
 +
author= {A.C. Hindmarsh, P.N. Brown, K.E. Grant, S.L. Lee, R. Serban, D.E. Shumaker, and C.S. Woodward},
 +
title= {SUNDIALS: Suite of nonlinear and differential/algebraic equation solvers},
 +
journal={ACM Trans. Math. Soft. (TOMS)},
 +
year= {2005},
 +
volume={31},
 +
number={3},
 +
pages={363--396},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="li2010">
 +
<bibtex>
 +
@article{Li2010,
 +
author= {N. Li and S. Laizet},
 +
title= {2DECOMP&FFT - a highly scalable 2D decomposition library and FFT interface},
 +
journal={Cray User’s Group 2010 conference, Edinburgh},
 +
year= {2010},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="ern1995">
 +
<bibtex>
 +
@article{Ern1995,
 +
author= {A. Ern and V. Giovangigli},
 +
title= {Fast and accurate multicomponent transport property evaluation},
 +
journal={J. Comput. Phys.},
 +
year= {1995},
 +
volume={120},
 +
pages={105--116},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="abdelsamie2019">
 +
<bibtex>
 +
@article{Abdelsamie2019,
 +
author= {A. Abdelsamie and D. Thévenin.},
 +
title= { On the behavior of spray combustion in a turbulent spatially-evolving jet investigated by direct numerical simulation},
 +
journal={Proc. Combust. Inst.},
 +
year= {2019},
 +
volume={37},
 +
number={3},
 +
pages={2493--2502},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="abdelsamie2020">
 +
<bibtex>
 +
@article{Abdelsamie2020,
 +
author= {A. Abdelsamie and D. Thévenin},
 +
title= {Nanoparticle behavior and formation in turbulent spray flames investigated by DNS},
 +
journal={Direct and Large Eddy Simulation XII, ERCOFTAC Series},
 +
year= {2020},
 +
volume={27},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="chi2020">
 +
<bibtex>
 +
@article{Chi2020,
 +
author= {C. Chi, A. Abdelsamie, and D. Thévenin},
 +
title= {A directional ghost-cell immersed boundary method for incompressible flows},
 +
journal={J. Comput. Phys.},
 +
year= {2020},
 +
volume={404},
 +
pages={109122--109142},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="chi2018">
 +
<bibtex>
 +
@article{Chi2018,
 +
author= {C. Chi, A. Abdelsamie, and D. Thévenin},
 +
title= {Direct numerical simulations of hotspot-induced ignition in homogeneous hydrogen-air pre-mixtures and ignition spot tracking},
 +
journal={Flow Turbul. Combust.},
 +
year= {2018},
 +
volume={101},
 +
number={1},
 +
pages={103--121},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="oster2018">
 +
<bibtex>
 +
@article{Oster2018,
 +
author= {T. Oster, A. Abdelsamie, M. Motejat, T. Gerrits, C. Rössl, D. Thévenin, and H. Theisel},
 +
title= {On-the-fly tracking of flame surfaces for the visual analysis of combustion processes},
 +
journal={Comput. Graph. Forum},
 +
year= {2018},
 +
volume={37},
 +
number={6},
 +
pages={358--369},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="abdelsamie2019-2">
 +
<bibtex>
 +
@article{Abdelsamie2019-2,
 +
author= {A. Abdelsamie and D. Thévenin},
 +
title= {Impact of scalar dissipation rate on turbulent spray combustion investigated by DNS},
 +
journal={Direct and Large-Eddy Simulation XI, ERCOFTAC Series},
 +
year= {2019},
 +
volume={25},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="tomboulides1997">
 +
<bibtex>
 +
@article{Tomboulides1997,
 +
author= {A.G. Tomboulides, J.C.Y. Lee, and S.A. Orszag},
 +
title= {Numerical simulation of low Mach number reactive flows},
 +
journal={J. Sci. Comput.},
 +
year= {1997},
 +
volume={12},
 +
pages={139--167},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="deville2002">
 +
<bibtex>
 +
@article{Deville2002,
 +
author= {M.O. Deville, P.F. Fischer, and E.H. Mund},
 +
title= {High-order Methods for Incompressible Fluid Flows},
 +
journal={Cambridge University Press},
 +
year= {2002},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="tanarro2020">
 +
<bibtex>
 +
@article{Tanarro2020,
 +
author= {A. Tanarro, F. Mallor, N. Offermans, A. Peplinski, R. Vinuesa, and P. Schlatter},
 +
title= {Enabling adaptive mesh refinement for spectral-element simulations of turbulence around wing sections},
 +
journal={Flow Turbul. Combust},
 +
year= {2020},
 +
volume={105},
 +
number={2},
 +
pages={415--436},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="chemkin">
 +
<bibtex>
 +
@article{Chemkin,
 +
author= {R.J. Kee, F.M. Rupley, and J.A. Miller},
 +
title= {Chemkin-II: A Fortran chemical kinetics package for the analysis of gas-phase chemical kinetics},
 +
journal={SAND-89-8009},
 +
year= {1989},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="MS">
 +
<bibtex>
 +
@article{MS,
 +
author= {M. Schmitt, C.E. Frouzakis, A.G. Tomboulides, Y.M. Wright, and K. Boulouchos},
 +
title= {Direct numerical simulation of the effect of compression on the flow, temperature and composition under engine-like conditions},
 +
journal={Proc. Combust. Inst.},
 +
year= {2015},
 +
volume={35},
 +
pages={3069--3077},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="TCC">
 +
<bibtex>
 +
@article{TCC,
 +
author= {G.K. Giannakopoulos, C.E. Frouzakis, P.F. Fischer, A.G. Tomboulides, and K. Boulouchos},
 +
title= {LES of the gasexchange process inside an internal combustion engine using a high-order method},
 +
journal={Flow Turbul. Combust.},
 +
year= {2020},
 +
volume={104},
 +
pages={673--692},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="catalytic">
 +
<bibtex>
 +
@article{Catalytic,
 +
author= {B.O. Arani, C.E. Frouzakis, J. Mantzaras, and K. Boulouchos},
 +
title= {Three-dimensional direct numerical simulations of turbulent fuel-lean H2/air hetero-/homogeneous combustion over Pt with detailed chemistry},
 +
journal={Proc. Combust. Inst.},
 +
year= {2017},
 +
volume={36},
 +
number={3},
 +
pages={4355--4363},
 +
}
 +
</bibtex>
 +
</ref>
 +
<ref name="abdelsamie2016">
 +
<bibtex>
 +
@article{Abdelsamie2016,
 +
author= {B.O. Arani, C.E. Frouzakis, J. Mantzaras, and K. BoulouchosA. Abdelsamie, G. Fru, F. Dietzsch, G. Janiga, and D. Thévenin},
 +
title= {Towards direct numerical simulations of lowMach number turbulent reacting and two-phase flows using immersed boundaries},
 +
journal={Comput. Fluids},
 +
year= {2016},
 +
volume={131},
 +
number={5},
 +
pages={123--141},
 +
}
 +
</bibtex>
 +
</ref>
 +
</references>

Latest revision as of 16:26, 16 September 2020

Code Description

This page is dedicated to a short presentation of the three low-Mach number codes used for the rest of this benchmarl: YALES2, DINO and Nek5000.

Since these codes have already been the subject of many publications and are not completely new, only the most relevant features are discussed in what follows, with suitable references for those readers needing more details.

YALES2

YALES2 is a massively parallel multiphysics platform[1] developed since 2009 by Moureau, Lartigue and co-workers at CORIA (Rouen, France). It is dedicated to the high-fidelity simulation of low-Mach number flows in complex geometries. It is based on the Finite Volumes formulation of the Navier-Stokes equations and it can solve both non-reacting and reacting flows. It can actually solve various physical problems thanks to its original structure which is composed of a main numerical library accompanied with tens of dedicated solvers (for acoustics, multiphase flows, heat transfer, radiation\ldots) which can be coupled with one another. YALES2 relies on unstructured meshes and a fully parallel dynamic mesh adaptation technique to improve the resolution in physically-relevant zones to mitigate the computational cost[2]. As a result it can easily handle meshes composed of billions of tetrahedra, thus enabling the Direct Numerical Simulation of laboratory and semi-industrial configurations. It is now composed of nearly 500,000 lines of object-oriented Fortran and the parallelism is currently ensured by a pure MPI paradigm, although a hybrid OpenMP/MPI as well as a GPU version are under development.

As most low-Mach number codes, the time-advancement is based on a projection-correction method following the pioneering work of[3]. The prediction step uses a method which is a blend between a 4th-order Runge-Kutta method and a 4th-order Lax-Wendroff-like method[4], combined with a 4th-order node-based centered finite-volume discretization of the convective and diffusive terms. Moreover, to improve the performance of the correction step, the pressure of the previous iteration is included in the prediction step to limit the splitting errors. The correction step is required to ensure mass conservation, using a pressure that arises from a Poisson equation. This is performed numerically by solving a linear system on the pressure at each node of the mesh thanks to a dedicated in-house version of the deflated conjugate gradient algorithm, which has been optimized for solving elliptic equations on massively parallel machines[5].

When considering multispecies and non-isothermal flows, the extension of the classical projection method proposed by Pierce et al.[6] is used to account for variable-density flows. All the thermodynamic and transport properties are provided by the Cantera software[7], which has been fully re-implemented in Fortran to avoid any performance issues. Each species thermodynamic property is specified by 5th-order polynomials on two temperature ranges (below and above 1000 K). The mixture is supposed to be both thermally and mechanically perfect.

Regarding transport properties, several type of models are implemented in YALES2:

1) the default approach is based on the computation of transport properties for each species (using tabulated molecular potentials), then combining those to obtain mixture-averaged coefficients for viscosity, conductivity and diffusion velocities (Hirschfelder and Curtiss approximation[8];

2) alternatively, simplified laws (for example the Sutherland law[9] can be used for the viscosity, while fixed values for Prandtl and Schmidt numbers allow the computation of conductivity and diffusion coefficients;

3) a mix of both approaches, for example computing viscosity and conductivity with a mixture-averaged approximation and then imposing a Lewis number for each species. This last approach has been retained in the present benchmark.

Finally, the source terms used in the reacting simulations are modeled with an Arrhenius law with the necessary modifications needed to take into account three-body or pressure-dependent reactions.

From a numerical point of view, it must be noticed that both the diffusion and reaction processes occur at time scales which can be orders of magnitude smaller than the convective time scale. Solving these phenomena with explicit methods would thus drastically limit the global timestep of the whole simulation and induce an overwhelming CPU cost. To mitigate these effects when dealing with multi-species reacting flows, the classical operator splitting technique is used. The diffusion process is solved with a fractional timestep method inside each convective iteration, each substep being limited by a Fourier condition to ensure stability. This method gets activated only when the mesh is very fine (typically when performing DNS with very diffusive species like or , as in the present project); otherwise an explicit treatment is sufficient. The chemical source terms are then integrated with a dedicated stiff solver, namely the CVODE library from SUNDIALS[10][11][12]. To that purpose, each control volume is considered as an isolated reactor with both constant pressure and enthalpy.

Using the stiff integration technique results in a very strong load imbalance between the various regions of the flow: the fresh gases are solved in a very small number of integration steps while the inner flame region may require tens or even hundreds of integration steps. To overcome this difficulty, a dedicated MPI dynamic scheduler based on a work-sharing algorithm ensures global load-balancing.

DINO

DINO is a 3-D DNS code used for incompressible or low-Mach number flows, the latter approach being used in this project. The development of DINO started in the group of D. Thévenin (Univ. of Magdeburg) in 2013. DINO is a Fortran-90 code, written on top of a 2-D pencil decomposition to enable efficient large-scale parallel simulations on distributed-memory supercomputers by coupling with the open-source library 2-DECOMP&FFT[13]. The code offers different features and algorithms in order to investigate different physicochemical processes. Spatial derivative are computed by default using sixth-order central finite differences. Time integration relies on several Runge-Kutta solvers. In what follows, an explicit 4th-order Runge-Kutta approach has been used. A 3rd-order semi-implicit Runge-Kutta integration can be activated as needed, when considering stiff chemistry. In this case, non-stiff terms are still computed with explicit Runge-Kutta, while the PyJac package[14] is used to integrate in an implicit manner all chemistry terms with an analytical Jacobian computation. All thermodynamic, chemical and transport properties are computed using the open-source library Cantera 2.4.0[15].

The transport properties can be computed based on three different models:

1) Constant Lewis numbers,

2) mixture-averaged,

3) full multicomponent diffusion, by coupling either again with Cantera or with the open-source library EGlib[16].

The Poisson equation is solved using Fast Fourier Transform (FFT) for periodic as well as for non-periodic boundary conditions, relying in the latter case on an in-house pre- and post-processing technique. The I/O operations are implemented using two different approaches: (1) binary MPI-I/O using 2-DECOMP&FFT for check-points and restart files; (2) HDF5 files used for analysis and visualization.

Multi-phase flows can be simulated in DINO using resolved or non-resolved (point) particles and droplets using a Lagrangian approach[17][18]. Complex boundaries are represented by a novel second-order immersed boundary method implementation (IBM) based on a directional extrapolation scheme[19]. More details about the implemented algorithms can be found in particular in[20]. Since DINO has been developed as a multi-purpose code for analyzing many different reacting and non-reacting flows[21][22][23], a detailed verification and validation is obviously essential.

Nek5000

This spectral element low-Mach number reacting flow solver is based on the highly-efficient open-source solver Nek5000[24] extended by a plugin developed at ETH implementing a high-order splitting scheme for low-Mach number reacting flows[25]. The spectral element method (SEM) is a high-order weighted residual technique for spatial discretization that combines the accuracy of spectral methods with the geometric flexibility of the finite element method allowing for accurately representation of complex geometries[26]. The computational domain is decomposed into conforming elements, which are quadrilaterals (in 2-D) or hexahedra (in 3-D) that conform to the domain boundaries. Within each element, functions are expanded as th-order polynomials so that resolution can be increased either by decreasing the element size (-type refinement) or by increasing the polynomial order (-type refinement; typically ). The grids can be unstructured and allow for static local refinement, while adaptive mesh refinement has been recently developed[27]. By casting the polynomial approximation in tensor-product form, the differential operators on gridpoints per element can be evaluated with only work and storage.

The principal advantage of the SEM is that convergence is exponential in , yielding minimal numerical dispersion and dissipation, so that significantly fewer grid points per wavelength are required in order to accurately propagate a turbulent structure over the extended time required in high Reynolds number simulations. Nek5000 uses locally structured basis coefficients ( arrays), which allow direct addressing and tensor-product-based derivative evaluation that can be cast as efficient matrix-matrix products involving operators applied to data values for each element. As a result, data movement per grid point is the same as for low-order methods. It uses scalable domain-decomposition-based iterative solvers with efficient preconditioners. Communication is based on the Message Passing Interface (MPI) standard, and the code has proven scalability to over one million ranks. Nek5000 provides balanced I/O latency among all processors and reduces the overhead or even completely hides the I/O latency by using dedicated I/O communicators in the optimal case.

Time advancement is performed using the splitting scheme proposed in[25] to decouple the highly non-linear and stiff thermochemistry (species and energy governing equations) from the hydrodynamic system (continuity and momentum). Species and energy equations are integrated without further splitting using the implicit stiff integrator solver CVODE from the SUNDIALS package[10][11][12] that uses backward differentiation formulas (BDF). The continuity and momentum equations are integrated using a second- or third-order semi-implicit formulation (EXT/BDF) treating the non-linear advection term explicitly[26]. The thermodynamic properties, detailed chemistry, and transport properties are provided by optimized subroutines compatible with Chemkin[28].

The reacting flow solver can handle complex time-varying geometries and has been used for instance to simulate laboratory-scale internal combustion engines[29][30]. It can account for conjugate fluid-solid heat transfer and detailed gas phase as well as surface kinetics[31].

General comments on the codes

The three aforementioned solvers are all unsteady, high-fidelity codes based on the low-Mach number formulation of the Navier-Stokes equations. They can perform both Direct Numerical Simulations or Large Eddy Simulations of reacting flows, only DNS being considered here. However, they differ in a certain number of points, mainly from the numerical point of view. The aim of this section is to emphasize those major differences. First, YALES2 is an unstructured code, designed to handle any type of elements; its main application field pertains to LES of industrially relevant flows, though DNS is possible as well. On the other hand, both DINO and Nek5000 are mostly dedicated to DNS of configurations found in fundamental research. Both DINO and Nek5000 can only deal with quads or hexas, with the major difference that DINO is based on a structured connectivity while Nek5000 can use unstructured meshes (pavings). As a consequence, both DINO and Nek5000 employ higher-order numerical schemes compared to YALES2, limited at best to a 4th-order scheme. All codes rely on dedicated libraries to compute the thermo-chemical properties of the flow, either Chemkin, Cantera, or in-house versions of those. Moreover, they also rely at least to some extent on external software to perform the temporal integration of the stiff chemical source terms. Regarding the Poisson equation for pressure which must be solved by all codes, DINO relies on a spectral formulation by performing direct and inverse Fourier transforms, which is possible thanks to its structured mesh. On the other hand, both YALES2 and Nek5000 use an iterative solver with an efficient preconditioning technique; this method is more versatile and should be computationally more efficient for large and complex geometrical configurations. Nek5000 employs CVODE to integrate the thermochemical equations without further splitting of the different terms accounting for convection, diffusion and chemistry. The main differences between the three codes are summarized in the table below. Please note that the presented values are those used for the benchmark, even though some other options are available in each codes.

Table 1: Major numerical properties of the three high-fidelity codes as used in this benchmark
Code YALES2 DINO Nek5000
Connectivity Unstructured Structured Unstructured
Discretization Type Finite Volumes Finite Differences Spectral Elements
Grid point distribution Regular hexahedra Regular hexahedra Regular hexahedra with GLL points
Spatial order 4th 6th 7th - 15th (typically)
Temporal method expl. RK4 expl. RK4 / semi-impl. RK3 semi-impl. BDF3
Pressure solver CG with Deflation MPrec. FFT-based CG/GMRES with Jacobi/Schwartz Prec.
Thermo-chemistry Cantera (re-coded) Cantera Chemkin interface
Chemistry integration CVODE expl. RK4 CVODE
Operator splitting Yes No No
Parallel paradigm MPI MPI MPI

References

  1. V. Moureau, P. Domingo,, L. Vervisch, From large-eddy simulation to direct numerical simulation of a lean premixed swirl flame: Filtered laminar flame-pdf modeling, Combust. Flame 158(7):1340--1357, 2011, Bibtex
    Author : V. Moureau, P. Domingo,, L. Vervisch
    Title : From large-eddy simulation to direct numerical simulation of a lean premixed swirl flame: Filtered laminar flame-pdf modeling
    In : Combust. Flame -
    Address :
    Date : 2011
  2. P. Benard, G. Balarac, V. Moureau, C. Dobrzynski, G. Lartigue,, Y. D’Angelo, Mesh adaptation for largeeddy simulations in complex geometries, Int. J. Numer. Methods Fluids 81:719--740, 2015, Bibtex
    Author : P. Benard, G. Balarac, V. Moureau, C. Dobrzynski, G. Lartigue,, Y. D’Angelo
    Title : Mesh adaptation for largeeddy simulations in complex geometries
    In : Int. J. Numer. Methods Fluids -
    Address :
    Date : 2015
  3. A.J. Chorin, Numerical solution of the Navier-Stokes equations, Math. Computation 22(104):745--762, 1968, Bibtex
    Author : A.J. Chorin
    Title : Numerical solution of the Navier-Stokes equations
    In : Math. Computation -
    Address :
    Date : 1968
  4. M. Kraushaar, Application of the compressible and low-Mach number approaches to Large-Eddy Simulation of turbulent flows in aero-engines, PhD thesis, PhD, Institut National Polytechnique de Toulouse-INPT , 2011, Bibtex
    Author : M. Kraushaar
    Title : Application of the compressible and low-Mach number approaches to Large-Eddy Simulation of turbulent flows in aero-engines
    In : PhD thesis, PhD, Institut National Polytechnique de Toulouse-INPT -
    Address :
    Date : 2011
  5. M. Malandain, N. Maheu,, V. Moureau, Optimization of the deflated conjugate gradient algorithm for the solving of elliptic equations on massively parallel machines, J. Comput. Phys. 238:32--47, 2013, Bibtex
    Author : M. Malandain, N. Maheu,, V. Moureau
    Title : Optimization of the deflated conjugate gradient algorithm for the solving of elliptic equations on massively parallel machines
    In : J. Comput. Phys. -
    Address :
    Date : 2013
  6. C.D. Pierce, P. Moin, Progress-variable approach for large-eddy simulation of non-premixed turbulent combustion, J. Fluid Mech. 504:73--97, 2004, Bibtex
    Author : C.D. Pierce, P. Moin
    Title : Progress-variable approach for large-eddy simulation of non-premixed turbulent combustion
    In : J. Fluid Mech. -
    Address :
    Date : 2004
  7. A.J. Chorin, An open-source, extensible software suite for CVD process simulation, Chemical Vapor Deposition XVI and EUROCVD 14:2003--2008, 2003, Bibtex
    Author : A.J. Chorin
    Title : An open-source, extensible software suite for CVD process simulation
    In : Chemical Vapor Deposition XVI and EUROCVD -
    Address :
    Date : 2003
  8. J. Hirschfelder, R.B. Bird,, C.F. Curtiss, Molecular Theory of Gases and Liquids, Wiley , 1964, Bibtex
    Author : J. Hirschfelder, R.B. Bird,, C.F. Curtiss
    Title : Molecular Theory of Gases and Liquids
    In : Wiley -
    Address :
    Date : 1964
  9. W. Sutherland. LII, The viscosity of gases and molecular force, The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 36(223):507--531, 1893, Bibtex
    Author : W. Sutherland. LII
    Title : The viscosity of gases and molecular force
    In : The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science -
    Address :
    Date : 1893
  10. 10.0 10.1 S.D. Cohen, A.C. Hindmarsh,, P.F. Dubois, CVODE, a stiff/nonstiff ODE solver in C, Computers Phys. 10(2):138--143, 1996, Bibtex
    Author : S.D. Cohen, A.C. Hindmarsh,, P.F. Dubois
    Title : CVODE, a stiff/nonstiff ODE solver in C
    In : Computers Phys. -
    Address :
    Date : 1996
  11. 11.0 11.1 A.C. Hindmarsh and R. Serban. User documentation for CVODE. [1].
  12. 12.0 12.1 A.C. Hindmarsh, P.N. Brown, K.E. Grant, S.L. Lee, R. Serban, D.E. Shumaker,, C.S. Woodward, SUNDIALS: Suite of nonlinear and differential/algebraic equation solvers, ACM Trans. Math. Soft. (TOMS) 31(3):363--396, 2005, Bibtex
    Author : A.C. Hindmarsh, P.N. Brown, K.E. Grant, S.L. Lee, R. Serban, D.E. Shumaker,, C.S. Woodward
    Title : SUNDIALS: Suite of nonlinear and differential/algebraic equation solvers
    In : ACM Trans. Math. Soft. (TOMS) -
    Address :
    Date : 2005
  13. N. Li, S. Laizet, 2DECOMP&FFT - a highly scalable 2D decomposition library and FFT interface, Cray User’s Group 2010 conference, Edinburgh , 2010, Bibtex
    Author : N. Li, S. Laizet
    Title : 2DECOMP&FFT - a highly scalable 2D decomposition library and FFT interface
    In : Cray User’s Group 2010 conference, Edinburgh -
    Address :
    Date : 2010
  14. Create analytical jacobian matrix source code for chemical kinetics.
  15. Cantera.
  16. A. Ern, V. Giovangigli, Fast and accurate multicomponent transport property evaluation, J. Comput. Phys. 120:105--116, 1995, Bibtex
    Author : A. Ern, V. Giovangigli
    Title : Fast and accurate multicomponent transport property evaluation
    In : J. Comput. Phys. -
    Address :
    Date : 1995
  17. A. Abdelsamie, D. Thévenin., On the behavior of spray combustion in a turbulent spatially-evolving jet investigated by direct numerical simulation, Proc. Combust. Inst. 37(3):2493--2502, 2019, Bibtex
    Author : A. Abdelsamie, D. Thévenin.
    Title : On the behavior of spray combustion in a turbulent spatially-evolving jet investigated by direct numerical simulation
    In : Proc. Combust. Inst. -
    Address :
    Date : 2019
  18. A. Abdelsamie, D. Thévenin, Nanoparticle behavior and formation in turbulent spray flames investigated by DNS, Direct and Large Eddy Simulation XII, ERCOFTAC Series 27, 2020, Bibtex
    Author : A. Abdelsamie, D. Thévenin
    Title : Nanoparticle behavior and formation in turbulent spray flames investigated by DNS
    In : Direct and Large Eddy Simulation XII, ERCOFTAC Series -
    Address :
    Date : 2020
  19. C. Chi, A. Abdelsamie,, D. Thévenin, A directional ghost-cell immersed boundary method for incompressible flows, J. Comput. Phys. 404:109122--109142, 2020, Bibtex
    Author : C. Chi, A. Abdelsamie,, D. Thévenin
    Title : A directional ghost-cell immersed boundary method for incompressible flows
    In : J. Comput. Phys. -
    Address :
    Date : 2020
  20. B.O. Arani, C.E. Frouzakis, J. Mantzaras,, K. BoulouchosA. Abdelsamie, G. Fru, F. Dietzsch, G. Janiga,, D. Thévenin, Towards direct numerical simulations of lowMach number turbulent reacting and two-phase flows using immersed boundaries, Comput. Fluids 131(5):123--141, 2016, Bibtex
    Author : B.O. Arani, C.E. Frouzakis, J. Mantzaras,, K. BoulouchosA. Abdelsamie, G. Fru, F. Dietzsch, G. Janiga,, D. Thévenin
    Title : Towards direct numerical simulations of lowMach number turbulent reacting and two-phase flows using immersed boundaries
    In : Comput. Fluids -
    Address :
    Date : 2016
  21. C. Chi, A. Abdelsamie,, D. Thévenin, Direct numerical simulations of hotspot-induced ignition in homogeneous hydrogen-air pre-mixtures and ignition spot tracking, Flow Turbul. Combust. 101(1):103--121, 2018, Bibtex
    Author : C. Chi, A. Abdelsamie,, D. Thévenin
    Title : Direct numerical simulations of hotspot-induced ignition in homogeneous hydrogen-air pre-mixtures and ignition spot tracking
    In : Flow Turbul. Combust. -
    Address :
    Date : 2018
  22. T. Oster, A. Abdelsamie, M. Motejat, T. Gerrits, C. Rössl, D. Thévenin,, H. Theisel, On-the-fly tracking of flame surfaces for the visual analysis of combustion processes, Comput. Graph. Forum 37(6):358--369, 2018, Bibtex
    Author : T. Oster, A. Abdelsamie, M. Motejat, T. Gerrits, C. Rössl, D. Thévenin,, H. Theisel
    Title : On-the-fly tracking of flame surfaces for the visual analysis of combustion processes
    In : Comput. Graph. Forum -
    Address :
    Date : 2018
  23. A. Abdelsamie, D. Thévenin, Impact of scalar dissipation rate on turbulent spray combustion investigated by DNS, Direct and Large-Eddy Simulation XI, ERCOFTAC Series 25, 2019, Bibtex
    Author : A. Abdelsamie, D. Thévenin
    Title : Impact of scalar dissipation rate on turbulent spray combustion investigated by DNS
    In : Direct and Large-Eddy Simulation XI, ERCOFTAC Series -
    Address :
    Date : 2019
  24. Nek5000 version v17.0, Argonne National Laboratory, IL, U.S.A.
  25. 25.0 25.1 A.G. Tomboulides, J.C.Y. Lee,, S.A. Orszag, Numerical simulation of low Mach number reactive flows, J. Sci. Comput. 12:139--167, 1997, Bibtex
    Author : A.G. Tomboulides, J.C.Y. Lee,, S.A. Orszag
    Title : Numerical simulation of low Mach number reactive flows
    In : J. Sci. Comput. -
    Address :
    Date : 1997
  26. 26.0 26.1 M.O. Deville, P.F. Fischer,, E.H. Mund, High-order Methods for Incompressible Fluid Flows, Cambridge University Press , 2002, Bibtex
    Author : M.O. Deville, P.F. Fischer,, E.H. Mund
    Title : High-order Methods for Incompressible Fluid Flows
    In : Cambridge University Press -
    Address :
    Date : 2002
  27. A. Tanarro, F. Mallor, N. Offermans, A. Peplinski, R. Vinuesa,, P. Schlatter, Enabling adaptive mesh refinement for spectral-element simulations of turbulence around wing sections, Flow Turbul. Combust 105(2):415--436, 2020, Bibtex
    Author : A. Tanarro, F. Mallor, N. Offermans, A. Peplinski, R. Vinuesa,, P. Schlatter
    Title : Enabling adaptive mesh refinement for spectral-element simulations of turbulence around wing sections
    In : Flow Turbul. Combust -
    Address :
    Date : 2020
  28. R.J. Kee, F.M. Rupley,, J.A. Miller, Chemkin-II: A Fortran chemical kinetics package for the analysis of gas-phase chemical kinetics, SAND-89-8009 , 1989, Bibtex
    Author : R.J. Kee, F.M. Rupley,, J.A. Miller
    Title : Chemkin-II: A Fortran chemical kinetics package for the analysis of gas-phase chemical kinetics
    In : SAND-89-8009 -
    Address :
    Date : 1989
  29. M. Schmitt, C.E. Frouzakis, A.G. Tomboulides, Y.M. Wright,, K. Boulouchos, Direct numerical simulation of the effect of compression on the flow, temperature and composition under engine-like conditions, Proc. Combust. Inst. 35:3069--3077, 2015, Bibtex
    Author : M. Schmitt, C.E. Frouzakis, A.G. Tomboulides, Y.M. Wright,, K. Boulouchos
    Title : Direct numerical simulation of the effect of compression on the flow, temperature and composition under engine-like conditions
    In : Proc. Combust. Inst. -
    Address :
    Date : 2015
  30. G.K. Giannakopoulos, C.E. Frouzakis, P.F. Fischer, A.G. Tomboulides,, K. Boulouchos, LES of the gasexchange process inside an internal combustion engine using a high-order method, Flow Turbul. Combust. 104:673--692, 2020, Bibtex
    Author : G.K. Giannakopoulos, C.E. Frouzakis, P.F. Fischer, A.G. Tomboulides,, K. Boulouchos
    Title : LES of the gasexchange process inside an internal combustion engine using a high-order method
    In : Flow Turbul. Combust. -
    Address :
    Date : 2020
  31. B.O. Arani, C.E. Frouzakis, J. Mantzaras,, K. Boulouchos, Three-dimensional direct numerical simulations of turbulent fuel-lean H2/air hetero-/homogeneous combustion over Pt with detailed chemistry, Proc. Combust. Inst. 36(3):4355--4363, 2017, Bibtex
    Author : B.O. Arani, C.E. Frouzakis, J. Mantzaras,, K. Boulouchos
    Title : Three-dimensional direct numerical simulations of turbulent fuel-lean H2/air hetero-/homogeneous combustion over Pt with detailed chemistry
    In : Proc. Combust. Inst. -
    Address :
    Date : 2017