luo-qin / streams

Supersonic TuRbulEnt Accelerated navier stokes Solver

!=============================================================
!
! ███████╗████████╗██████╗ ███████╗ █████╗ ███╗   ███╗███████╗
! ██╔════╝╚══██╔══╝██╔══██╗██╔════╝██╔══██╗████╗ ████║██╔════╝
! ███████╗   ██║   ██████╔╝█████╗  ███████║██╔████╔██║███████╗
! ╚════██║   ██║   ██╔══██╗██╔══╝  ██╔══██║██║╚██╔╝██║╚════██║
! ███████║   ██║   ██║  ██║███████╗██║  ██║██║ ╚═╝ ██║███████║
! ╚══════╝   ╚═╝   ╚═╝  ╚═╝╚══════╝╚═╝  ╚═╝╚═╝     ╚═╝╚══════╝
!
! Supersonic TuRbulEnt Accelerated navier stokes Solver
!
! input file description
!
!=============================================================

STREAmS performs Direct Numerical Simulations of compressible turbulent flows in Cartesian geometry solving the fully compressible Navier-Stokes equations. Currently, three canonical wall-bounded flows can be simulated:

• compressible turbulent channel flow
• compressible zero-pressure-gradient turbulent boundary layer
• supersonic oblique shock wave/turbulent boundary-layer interaction

STREAmS can be used both on local cluster and massively parallel HPC architectures.

Compiling

STREAmS requires (1) a Fortran compiler and (2) an MPI library. For the GPU CUDA version, the PGI compiler is required (tested using PGI 19.4 or more recent compilers). A Makefile with predefined settings is available to facilitate compiling. Different compiling modes can be selected by changing the three variables:

COMPILE 
MODE    
PREC    

COMPILE currently supports these choices:

• pgi-cuda: PGI compiler Cuda Fortran asynchronous version with MPI library (tested with OpenMPI provided by PGI)
• pgi-cuda-sync: PGI compiler Cuda Fortran synchronous version with MPI library (tested with OpenMPI provided by PGI)
• pgi: PGI compiler CPU version with OpenMPI library (tested with OpenMPI provided by PGI)
• intel: Intel compiler with MPI library (tested with IntelMPI library)
• gnu: gnu compiler with MPI library (tested with OpenMPI)
• ibmxl: XL IBM compiler with MPI library (tested with IBM MPI library)
• cray-cuda: PGI compiler with Cray mpich library without support of CUDA-Aware MPI (currently oriented to Pitz-Daint cluster)

MODE can be one of:

• opt: optimized compilation, for standard production runs
• debug: debugging compilation, with run-time checks enabled and backtracing when available

PREC defines the precision of the float numbers:

• double: double precision floating point numbers
• single: single precision floating point numbers

Running

STREAmS can be usually executed using a standard MPI launcher, e.g. mpirun. In addition to the executable, in the running folder you need:

• input.dat: file defining the physical and numerical setup of the simulation, to be customized according to the desired needs. Detailed description input.dat file is given below. Some examples of input.dat files are available in the examples folder.
• database_bl.dat: only required for the boundary layer and shock-boundary layer interaction flows (this file does not have to be modified by the user). The file is available in the examples folder.

To run a simulation, type, e.g.:

mpirun -np 8 ./streams

or (for SLURM jobs)

srun ./streams

For CUDA versions in cluster environments, you must distribute MPI processes according to the number of GPUs available for each node. For CINECA Marconi-100 cluster -- 4 GPUS per node -- a possible submission script using 8 GPUs is:

#!/bin/bash
#SBATCH -N 2 
#SBATCH --tasks-per-node 4
#SBATCH --cpus-per-task 32
#SBATCH --ntasks-per-socket 2
#SBATCH --mem=128G 
#SBATCH --partition=m100_usr_prod
#SBATCH --time=00:30:00 
#SBATCH --gres=gpu:4

module load profile/global hpc-sdk/20.5--binary

mpirun -np 8 ./streams

For CSCS Pitz-Daint cluster -- 1 GPU per node -- a possible submission script using 8 GPUs is:

#!/bin/bash
#SBATCH -N 8
#SBATCH --tasks-per-node 1 
#SBATCH --mem=15G 
#SBATCH --partition=debug 
#SBATCH --time=00:30:00 
#SBATCH --gres=gpu:1
#SBATCH --constraint=gpu 
module swap PrgEnv-cray PrgEnv-pgi
srun ./streams

Preparing input.dat file

iflow defines the type of flow. 0 = channel, 1 = boundary layer, 2 = shock/boundary-layer interaction

rlx rly rlz real numbers defining the size of the Cartesian domain along the three directions (x=streamwise, y=wall-normal, z=spanwise)

nxmax nymax nzmax integer numbers defining the number of grid nodes along each direction

nymaxwr rlywr dyp_target specify the wall-normal mesh features. dyp_target is the desired spacing at the wall in inner units, . nymaxwr denotes the number of grid points in the wall-resolved region, ranging from y=0 (wall) up to y=rlywr, where a sinh mapping is applied. Both nymaxwr and rlywr can be specified when iflow = 1 (turbulent boundary layer), then a geometric progression is applied from y=rlywr up to y = rly. For iflow = 2 (shock/boundary-layer interaction), nymaxwr must be specified but rlywr is automatically computed.

nblocks(1) nblocks(3) define the MPI decomposition along x (streamwise) and z (spanwise). These numbers must be consistent to nxmax, nymax, and nzmax. In particular the following divisions must have zero remainder: nxmax/nblocks(1), nzmax/nzmax. Moreover, for iflow>0 cases also nxmax/nblocks(3) and nymax/nblocks(1) must have zero remainder. The product nblocks(1)*nblocks(3) must equal the total number of launched MPI processes.

tresduc ximp deflec is the WENO threshold (WENO is active if the shock sensor value exceeds tresduc), ximp is the shock abscissa and deflec is the shock angle. ximp and deflec are used only if iflow > 0.

idiski ncyc cfl nstep nprint io_type idiski selects the start type (0=init, 1=restart, 2=collect statistics at runtime), ncyc the total number of iterations, cfl is the CFL number. The time step is re-evaluated every nstep iterations, nprint the residual file is printed every nprint iterations and io_type is selects the type of I/O (io_type=0 no I/O, io_type=1, serial I/O, io_type=2, MPI I/O).

rm re (friction) twall/taw visc_type s2tinf Mach number, number, ratio between wall temperature and adiabatic wall temperature, type of viscosity law (visc_type=1 power law, visc_type=2, Sutherland law) and ratio between S(110.4K) and reference temperature. If s2tinf=0 s2tinf is automatically computed by the solver.

istat nstat cumulative flow statistics are evaluate every istat iterations. nstat is the number of streamwise stations at which flow statistics are extracted. nstat is meaningful for boundary layer flow (iflow > 0)

xstat streamwise locations of boundary layer flow statistics (iflow > 0) 20. 30. 40. 50. 60. 70. 80. 87.

dtsave dtsave_restart plot3d vtk dt_save is the time interval between field output, dtsave_restart is the time interval between output of restart files, enable_plot3d>0 activates the plot3d format output, vtk>0 activates vtk format output.

irand if < 0 produce not reproducible random sequences based on current time, if >=0 produces random sequences which are reproducible across different runs with the same configuration

Typical input files for canonical flow cases are available.

Understanding the output files

Plot3d binary output files -- e.g. plot3dgrid.xyz and field_0001.q -- can be read using Tecplot or Paraview

VTK Rectilinear grid files -- e.g. field_0001.vtr -- can be read using Paraview.

Other statistics files are automatically produced according to the input configuration.

Channel flow output files

Residual file output_streams.dat

When running channel flow cases (iflow=0) the residuals file (output_streams.dat) contains seven columns:

1. number of cycles,
2. elapsed time,
3. streamwise momentum residual,
4. pressure gradient,
5. bulk density (conserved to machine accuracy),
6. bulk flow velocity (conserved to machine accuracy),
7. bulk temperature. For instance, plotting the fourth column vs. the second allows the user to check the time history of the pressure gradient.

Mean flow statistics channstat.prof

The file `channstat.prof` contains the mean channel flow statistics.
This file is printed at the end of each run and it contains the mean flow statistics averaged
in the homogeneous spatial directions and in time (statistics are progressively updated in time at each restart if idiski=2, or collected from scratch if idisk=1).
The file `channstat.prof` contains 15 columns:

1. is the wall-normal coordinate, normalized with the channel half width
2. the wall-normal coordinate in viscous units
3. the wall-normal coordinate transformed according to Trettel & Larsson in viscous units
4. the mean streamwise velocity averaged according to Favre, normalized with the bulk flow velocity
5. the mean streamwise velocity averaged according to Favre, normalized with the friction velocity
6. the mean streamwise velocity transformed according to van Driest, normalized with the friction velocity
7. the mean streamwise velocity transformed according to Trettel & Larsson in viscous units
8. the mean density profile, normalized with the mean wall density
9. the Favre streamwise Reynolds stress, normalized with the wall-shear stress
10. the Favre wall-normal Reynolds stress, normalized with the wall-shear stress
11. the Favre spanwise Reynolds stress, normalized with the wall-shear stress
12. the Favre shear Reynolds stress, normalized with the wall-shear stress
13. the mean temperature profile, normalized with the wall temperature
14. the density fluctuations normalized with the mean wall density
15. The temperature fluctuations, normalized with the wall temperature

Boundary layer output files

Residual file output_streams.dat

When running boundary layer cases (iflow=1 or 2) the residuals file (output_streams.dat) contains three columns:

1. number of cycles,
2. elapsed time,
3. streamwise momentum residual,

cf files

The files cf_xxx.dat are ASCII files and the number xxx refers to the Cartesian streamwise MPI block to which the file belong. These files is printed at the end of each run. Statistics are progressively updated in time at each restart if idiski=2, or collected from scratch if idisk=1. The file contains 13 columns:

1. streamwise coordinate normalized with the inlet boundary layer thickness
2. skin-friction coefficient
3. friction Reynolds number
4. Compressible shape factor
5. Incompressible shape factor
6. boundary layer thickness, normalized with the inlet boundary layer thickness
7. Compressible displacement thickness
8. Compressible momentum thickness
9. friction velocity
10. Reynolds number based on the incompressible momentum thickness
11. Incompressible friction coefficient, according to van Driest II transofrmation
12. Reynolds number based on the compressible momentum thickness
13. Mean pressure rms at the wall normalized with the wall-shear stress

stat files

The files stat_nnn.dat are ASCII containing the mean boundary layer statistics. The files are printed at the end of each run. Statistics are progressively updated in time at each restart if idiski=2, or collected from scratch if idisk=1. The number nnn indicates the global mesh index in the streamwise direction at which statistics are printed. The files contains 10 columns:

1. , wall-normal coordinate, normalized with the local boundary layer thickness
2. wall-normal coordinate, normalized with the viscous lenght scale
3. the Favre averaged streamwise velocity, normalized with the friction velocity
4. the streamwise velocity transformed according to van Driest
5. the density scaled streamwise velocity rms
6. the density scaled wall-normal velocity rms, in viscous units
7. the density scaled spanwise velocity rms, in viscous units
8. the density scaled Reynolds shear stress, in viscous units
9. the square root of the mean density, normalized with the wall density
10. The pressure rms, normalized with the square root of the wall-shear stress.