cfd-go / tclb

TCLB - Templated MPI+CUDA/CPU Lattice Boltzmann code

TCLB Solver

TCLB is a MPI+CUDA, MPI+CPU or MPI+HIP high-performance Computational Fluid Dynamics simulation code, based on the Lattice Boltzmann Method. It provides a clear interface for calculation of complex physics, and the implementation of new models.

Stable release (master branch):

Current release (develop branch):

How to use it

Install

git clone https://github.com/CFD-GO/TCLB.git
cd TCLB

Configure

make configure
./configure

Compile

make d2q9

Run

CLB/d2q9/main example/flow/2d/karman.xml

More information

Documentation

The documentation (including tutorials) is published at docs.tclb.io.

For the develop version, the most recent documentation can be found at develop.docs.tclb.io.

You can contribute to the documentation at CFD-GO/TCLB_docs.

Supported architectures

This code is designed to run on Linux with CUDA. We strongly recommend using Linux for compilation, computation and postprocessing.

Nevertheless, TCLB can be compiled on Windows using the Windows Subsystem for Linux, with CUDA supported on some system configurations (see nVidia's website for more info). It also can be compiled on MacOS (CPU only). Both Debian and Red Hat based Linux distributions are supported by the install.sh script described below, as is MacOS (with brew package manager).

Dependencies

For the code to compile and work you'll need a few things:

• R and some R packages (optparse, rtemplate, gvector, polyAlgebra)
• MPI. We recommend OpenMPI
• To use your GPU, you'll need nVidia CUDA or AMD HIP/ROCm

Optionally, you may need:

• To integrate TCLB with R, you'll need R package rinside
• To integrate TCLB with Python, you'll need python, numpy with libraries and headers
• To develop a model using Python, you'll need python, sympy and R package reticulate

You can install many of these with the provided tools/install.sh script (note that this requires sudo):

sudo tools/install.sh essentials   # Installs essential system packages needed by TCLB
sudo tools/install.sh r            # Installs R
sudo tools/install.sh openmpi      # Installs OpenMPI
     tools/install.sh rdep         # Installs needed R packages
sudo tools/install.sh cuda         # Installs CUDA (we recommend to do it on your own)
sudo tools/install.sh python-dev   # Installs Python libraries with headers

You can run the tools/install.sh script with the --dry option, which will print the commands to run, so you can run them on your own. We do not recommend running anything with sudo without checking

develop Branch:

If you want a more recent version, you could try the development branch with git checkout develop

CPU

To compile the code for CPU, you can use the --disable-cuda option for ./configure:

./configure --disable-cuda

HIP

To compile the code for AMD GPUs (ROCm), you can use the --enable-hip option for ./configure:

./configure --enable-hip

Parallel run

To run TCLB in parallel (both on multiple CPU and multiple GPU), you can use the standard syntax of MPI parallel run:

mpirun -np 8 CLB/d2q9/main example/flow/2d/karman.xml

Running on clusters

To assist with using TCLB on HPC clusters (SLURM/PBS), there are scripts provided in the TCLB_cluster repository.

LBM-DEM computation

TCLB code can be coupled with Discrete Element Method (DEM) codes, to enable computation of flow with particles.

The DEM codes that TCLB can be integrated with are:

• LIGGGHTS
• LAMMPS
• ESYS-Particle

Refer to the documentation for instructions on compilation and coupling.

About

Authors

TCLB began development in 2012 with the aim at providing a framework for efficient CFD computations with LBM, mainly for research.

Author: Łukasz Łaniewski-Wołłk

Major contributors:

• Michał Dzikowski
• Travis Mitchell

Contributors:

• Nathan Di Vaira
• Grzegorz Gruszczyński
• Bryce Hill
• Jon McCullough
• Paweł Obrępalski
• Wojciech Regulski
• Mariusz Rutkowski
• Dmytro Sashko

Developed at:

• Zakład Aerodynamiki at Politechnika Warszawska (Warsaw University of Technology)
• School of Mechanical & Mining Engineering at University of Queensland
• Interdisciplinary Centre for Mathematical and Computational Modelling at University of Warsaw

Citation

Please use appropriate citations if using this software in any research publication. The publication should cite the original paper about TCLB and papers which describe the used LBM models. You can find the list of TCLB publications at docs.tclb.io/general-info/publications/. You can also find the information about published articles in the source code of the models. The code can be cited additionally, by its Zenodo DOI.

License

This software is distributed under the GPL v3 License.

If you need this software under a different license, please contact the main author.

Contact: lukasz.laniewski(monkey)pw.edu.pl

tclb's Issues

[internal] New execution interface and development plan

As things are getting hard to maintain I propose a change of overall approach of configuration and execution of TCLB.

These are notes for a possible direction of development

• Integrate TCLB_cluster into the main repo
• Make a common wrapper script for running all calculations
  • The model selection would be based on something line <CLBConfig model="...">
  • It would run both native and slurm
  • It would run the code coupling (eg. ESYS/TCLB) based on best practices.
  • Could generate batch script (without submitting it to the queue).
• Make machine specific alternation to configuration in a mechanism similar to the one done in TCLB_cluster
• Allow for easier execution of tests on different machines/architectures
• Make configuration more persistent (now you have to remember the ./configure options that you used on specific machine) (cmake has a similar mechanism, and can be considered as a solution)

The plan would be to:

• Close V6.3 including DEM and all nice proposed features (#186 #198)
• Begin V6.4 or V7.0 with #157 and maybe #198 + #125
• make the wrappers for configuration and execution and do #124 at the same time.

<Params> multiple arguments

The xml file looks like this:

    <Params thing="A"  thing-zone="B"/>

Where it would work different to

    <Params thing-zone="A"  thing="B"/>

because order of attributes matters. BUT it's non conforming with XML standard* . I know that it's a bug in xml case file, but it's all over examples etc. and we do support that.
Therefore only supported version should be

    <Params thing="B"/>
    <Params thing-zone="A" />

because order of child nodes is meaningful in XML.

We could add special name/zone argument/node:

    <Params    thing="B"       other-thing="B"  />
    <Params   zone="zone"  thing="B"  other-thing="B" />

or

       <Params   thing="B"   other-thing="B" >
       <Zonal   name="zone"     thing="B"      other-thing="B" />
       <Zonal   name="zone2"  thing="B"   other-thing="B" />
       </Params>
      <Params yetanother="1"/>

or maybe another.

In general this is bug IMHO, and need to be clarified before our miracle DOM-based handling ;)

@llaniewski , @shkodm I could work on that, but in some cases it will break cases. We could make "converter" but still. And which option is most usefull/most expandable.

Other thing is that XML guys prefer

<param>
<name>name</name>
<value>value</value>
</param>

or (I would vote for this, still short but allows <[[CDATA and binary/base64 input easly)

<param name="name">value</param>

over

<param name="value" />

*) section 3.1 of the XML recommendation. It says, "Note that the order of attribute specifications in a start-tag or empty-element tag is not significant."', http://stackoverflow.com/questions/726395/order-of-xml-attributes-after-dom-processing

Travis-CI support for CUDA

There is a persisting problem in support for CUDA on updated Travis-CI virtual machines. The update occurred on May 4. Our compilations encountered error first here:

https://travis-ci.org/llaniewski/TCLB/jobs/229029926

Suggested remedies:

• Force improvement of the nvidia apt source - no idea how to do it
• Move to container-based infrastructure, as cuda packages are included in their whitelist - in progress #145
• Keep the virtual machine based setup, but use apt: addon for cuda packages - seems to work #142

New repo

...for tools/scripts that could be used to work witch CLB: preprocessing/post-processing/gui ;) etc
I was thinking about pyCLB, but it suggests python API.
Nice acronym, or we go with TCLB-tools :]

Illegal Instruction - CSV output prevented for all(?) models

relevant: http://stackoverflow.com/questions/29899021/cuda-an-illegal-instruction-was-encountered-when-accessing-structure ???

on newton-01

module load cuda/5.5.0 openmpi R
make clean
./configure [--enable-debug]
./CLB/d2q9/main ./expamples/karman.xml

cat karman.xml

<CLBConfig version="2.0" output="output/">
        <Geometry nx="1024" ny="100">
        <MRT><Box/></MRT>
        <WVelocity name="Inlet"><Inlet/></WVelocity>
        <EPressure name="Outlet"><Outlet/></EPressure>
        <Inlet nx='1' dx='5'><Box/></Inlet>
        <Outlet nx='1' dx='-5'><Box/></Outlet>
        <Wall mask="ALL">
            <Channel/>
            <Wedge dx="120" nx="20" dy="50" ny="20" direction="LowerRight"/>
            <Wedge dx="120" nx="20" dy="30" ny="20" direction="UpperRight"/>
            <Wedge dx="140" nx="20" dy="50" ny="20" direction="LowerLeft"/>
            <Wedge dx="140" nx="20" dy="30" ny="20" direction="UpperLeft"/>
        </Wall>
    </Geometry>
    <Model>
        <Params Velocity="0.01"/>
        <Params nu="0.02"/>
        <Params InletTemperature="1"
            InitTemperature="0"
            FluidAlfa="0.01"
            SolidAlfa="0.166"
            Coriolis="0.001"
            Tension="0.01"
            EOSScale="0.05"
            PressDiffInObj="1"
            Smag="0.16"
        />
    </Model>
    <VTK Iterations="1000"/>
    <Log Iterations="100"/>
    <Solve Iterations="10000"/>
</CLBConfig>

plain d2q9 model

[ 0] FATAL ERROR: unknown error in LatticeContainer.inc.cpp at line 618

        container->RunInterior< Primal, NoGlobals, BaseIteration > (kernelStream); break;
    case ITER_GLOBS:
        container->RunInterior< Primal, IntegrateGlobals, BaseIteration >(kernelStream); break;
    case ITER_OBJ:
#ifdef ADJOINT
        container->RunInterior< Primal, OnlyObjective, BaseIteration >(kernelStream);
#endif
                break;
    }

    MPIStream_B();

>>>>>>>   CudaDeviceSynchronize();


    Snap = tab1;
    MarkIteration();
};

cuda-memcheck:

========= Illegal Instruction
=========     at 0x00001de0 in void RunKernel<Node_Run<eOperationType=0, eCalculateGlobals=1, eStage=0>>(void)
=========     by thread (0,1,0) in block (6,0,0)
=========     Saved host backtrace up to driver entry point at kernel launch time
=========     Host Frame:/tools/CUDA/libcuda/346.46/libcuda.so (cuLaunchKernel + 0x2cd) [0x150cad]
=========     Host Frame:/tools/CUDA/5.5.0/lib64/libcudart.so.5.5 [0xe108]
=========     Host Frame:/tools/CUDA/5.5.0/lib64/libcudart.so.5.5 (cudaLaunch + 0x143) [0x2cb53]
=========     Host Frame:./CLB/d2q9/main (_ZN60_GLOBAL__N__39_tmpxft_0000281e_00000000_6_cuda_cpp1_ii_omega10cudaLaunchIcEE9cudaErrorPT_ + 0x18) [0x44f5d]
=========     Host Frame:./CLB/d2q9/main [0x42365]
=========     Host Frame:./CLB/d2q9/main [0x42370]
=========     Host Frame:./CLB/d2q9/main (_Z9RunKernelI8Node_RunIL14eOperationType0EL17eCalculateGlobals1EL6eStage0EEEvv + 0x9) [0x44f94]
=========     Host Frame:./CLB/d2q9/main (_ZN16LatticeContainer12RunInteriorTI8Node_RunIL14eOperationType0EL17eCalculateGlobals1EL6eStage0EEEEvP11CUstream_st + 0x96) [0x484de]
=========     Host Frame:./CLB/d2q9/main (_ZN16LatticeContainer11RunInteriorIL14eOperationType0EL17eCalculateGlobals1EL6eStage0EEEvP11CUstream_st + 0x23) [0x4499f]
=========     Host Frame:./CLB/d2q9/main (_ZN7Lattice9IterationEiii + 0xd82) [0x529ec]
=========     Host Frame:./CLB/d2q9/main (_ZN7Lattice7IterateEii + 0x385) [0x4e51f]
=========     Host Frame:./CLB/d2q9/main (_ZN7acSolve4InitEv + 0x178) [0x8ddb2]
=========     Host Frame:./CLB/d2q9/main (_ZN7HandlerC1EN4pugi8xml_nodeEP6Solver + 0x62) [0x3eee2]
=========     Host Frame:./CLB/d2q9/main (_ZN13GenericAction15ExecuteInternalEv + 0x4e) [0x8d46e]
=========     Host Frame:./CLB/d2q9/main (_ZN13MainContainer4InitEv + 0x157) [0x8dc11]
=========     Host Frame:./CLB/d2q9/main (_ZN7HandlerC1EN4pugi8xml_nodeEP6Solver + 0x62) [0x3eee2]
=========     Host Frame:./CLB/d2q9/main (main + 0xba7) [0x3a9e4]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xfd) [0x1eead]
=========     Host Frame:./CLB/d2q9/main [0x392b9]
=========
========= Program hit error 30 on CUDA API call to cudaThreadSynchronize 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/tools/CUDA/libcuda/346.46/libcuda.so [0x2e40d3]
=========     Host Frame:/tools/CUDA/5.5.0/lib64/libcudart.so.5.5 (cudaThreadSynchronize + 0x166) [0x2b616]
=========     Host Frame:./CLB/d2q9/main (_ZN16LatticeContainer12RunInteriorTI8Node_RunIL14eOperationType0EL17eCalculateGlobals1EL6eStage0EEEEvP11CUstream_st + 0x9b) [0x484e3]
=========     Host Frame:./CLB/d2q9/main (_ZN16LatticeContainer11RunInteriorIL14eOperationType0EL17eCalculateGlobals1EL6eStage0EEEvP11CUstream_st + 0x23) [0x4499f]
=========     Host Frame:./CLB/d2q9/main (_ZN7Lattice9IterationEiii + 0xd82) [0x529ec]
=========     Host Frame:./CLB/d2q9/main (_ZN7Lattice7IterateEii + 0x385) [0x4e51f]
=========     Host Frame:./CLB/d2q9/main (_ZN7acSolve4InitEv + 0x178) [0x8ddb2]
[ 0] FATAL ERROR: unknown error in LatticeContainer.inc.cpp at line 618
=========     Host Frame:./CLB/d2q9/main (_ZN7HandlerC1EN4pugi8xml_nodeEP6Solver + 0x62) [0x3eee2]
=========     Host Frame:./CLB/d2q9/main (_ZN13GenericAction15ExecuteInternalEv + 0x4e) [0x8d46e]
=========     Host Frame:./CLB/d2q9/main (_ZN13MainContainer4InitEv + 0x157) [0x8dc11]
=========     Host Frame:./CLB/d2q9/main (_ZN7HandlerC1EN4pugi8xml_nodeEP6Solver + 0x62) [0x3eee2]
=========     Host Frame:./CLB/d2q9/main (main + 0xba7) [0x3a9e4]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xfd) [0x1eead]
=========     Host Frame:./CLB/d2q9/main [0x392b9]
=========
========= ERROR SUMMARY: 2 errors

outfile marking to prevent overwrite/mixup

@mdzik :
Models (with variants) would be similar, maybe some sort of vti outfiles suffix so results wont mix-up?

@llaniewski :
As for outfiles - for now, as earlier, the information about the model that was executed is only in the output/...config....xml file. If you think the model name should be added to the output files names, maybe at it as an enchancement issue.

options ? :

• tag keyword in xml, that would add tag to ALL possible output files
• do-not-overwrite-a-thing - if outfile exist use date-base prefix/tag

Output on CPU version

(to be fully confirmed)
Output on plain CPU version seems to be seriously messed up, some "\n" may be missing
I need to confirm that on plain new develop clone.
Any ideas why it could be? For GPU works OK. Less than month old-bug I think

Coveralls

I don't think Coveralls integration is working, it got status "pending", but there is nothing to do.
It is nice thing to look at that, but I don't think it's serving it's purpose. If there is indeed some missconfig, I vote to disable it for good.
We could go back to that when we split models, and stable ones would have more complete test's

Move copy and template to model makefile

Background

Because of the old policy of separating the code generation (done with R) and compilation, the two things are done in separate makefiles (makefile.main and CLB/[model]/makefile). The reason for that was to be able to generate the code on a machine with R, copy it to a cluster and then compile with gcc/nvcc.

Problem

This makes a mess in makefile.main, and makes compilation wait for all the code generation to finish (which is slowing things down on parallel compilation).

Possible solution

The code generation could be moved to the model makefile (CLB/[model]/makefile) to resolve this problem, as either way the code is always now compiled on architectures that have R.

Coveralls not working - no source to look at

The file "CLB/d2q9/Geometry.cpp" isn't available on github. Either it's been removed, or the repo root directory needs to be updated.

Command order in Dynamics.R

I found that Dynamics.R compilation is not working

this works fine:

AddDensity( name="f[0]", dx= 0, dy= 0, group="f")
#....

AddDensity( name="h[0]", dx= 0, dy= 0, group="h")
#....

AddDensity( name="h_Z", dx=0, dy=0, group="HZ")

AddField( name="nw_x", stencil2d=1, group="nw")
AddField( name="nw_y", stencil2d=1, group="nw")
AddField("phi"      ,stencil2d=1 );

AddStage("BaseIteration", "Run", 
         load=DensityAll$group == "f" | DensityAll$group == "h"| DensityAll$group == "HZ",  
         save=Fields$group=="f" | Fields$group=="h" | Fields$group=="nw" | Fields$group == "HZ"
         ) 
AddStage("CalcPhi", 
         save=Fields$name=="phi" ,  
         load=DensityAll$group == "h"
         )
AddStage("BaseInit", "Init",  save=Fields$group=="f" | Fields$group=="h"| Fields$group == "HZ") 
AddStage("CalcWallNormall", "CalcNormal",   
         save=Fields$group=="nw",
         fixedPoint=TRUE
         ) 

AddAction("Iteration", c("BaseIteration","CalcPhi"))
AddAction("Init", c("BaseInit","CalcPhi", "CalcWallNormall"))

but if you change order, eq:

AddDensity( name="f[0]", dx= 0, dy= 0, group="f")
#....

AddField( name="nw_x", stencil2d=1, group="nw") # <<<<<<<<

AddDensity( name="h[0]", dx= 0, dy= 0, group="h")
#....
AddDensity( name="h_Z", dx=0, dy=0, group="HZ")

solver brakes, and h_Z does not gets "advected" with zero-zero velocity through iterations.

Possible fixes:

1. fix the problem itself, but for AddStage it might require some sort of delayed compilation ;)
2. AddDensity called AFTER AddField should cause compilation error. This must be implemented for all Add* functions where order matters.

Obsolete сhunk of code

Surprisingly, I found out that src/inter.cpp is not actually used anywhere. Grepping it gives, that its only occurrence is the makefile. Last (and second..) commit to it dates back to 2014. Besides it includes Rcpp header, which is is not even TCLB dependency.

@llaniewski , what was that? Should we just delete it?

VTK for subregion

Add capability to specify a region in <VTK/> export.

Commandline options

Make commandline key-value options, which would overwrite attributes of main xml element.

Most useful:

• output - for changing output prefix
• model - for future model-based dispatch

PythonHandler cleanup

• Clean up naming and tests in python handler, XML tag and PythonHandler class from tools.
• Include and features requested during 6.1 preparation

Residuals/(convergence criterium)

As far, as I remember we don't have any support for monitoring quantities
I am pretty sure, it is possible to implement both "Stop when certain value is reached" and also saving history of residuals, aka ANSYS-like functionality. All this applied to quantities("U", "T", whatever)

Anyone interested? I can try to implement it.

python with config.h not config.mk

The EMBEDED_PYTHON should be in config.h, not as a "-D" flag to the compiler:

TCLB/src/config.mk.in

Line 71 in f5e0bdc

OPT+=-D EMBEDED_PYTHON

otherwise it won't recompile everything after enabling it in configure.

[V6.2 RC] Automatic symmetries

Add implicit Option "AutoSym" which will add symmetry node types and take care of symmetries in directly in LatticeAccess.

Needed:
Good definition on how to reflect each Field. Need different treatment of scalar fields, vector fields, and LBM densities.

FailCheck

I found that FailCheck is not working

The xml file looks like this:

<Solve Iterations="100000">
<VTK Iterations="100" />
<Failcheck Iterations="50">
<VTK />
</Failcheck>
</Solve>

I get this error:

There IS complete NaN Rho field, but: on BC (left/right) it's not, so

                if (!cond) {
                        real_t tmp;
                        solver->lattice->GetRho(reg,&tmp);
                        cond = (std::isnan(tmp));

is OK, because temp is first element of Rho array, which is BC, and not NaN

@shkodm I will fix that, but do we want 2 separate versions? Is this a bug or a future? :)

Cleanup Geometry

in the same manner as Handlers

Other ideas/suggestions?

I think we don't need geometry objects, but it might be usable in future for DOM handling.
I would convert present long if-else into calls for static methods of proper classes:

    } else if (strcmp(n.name(), "OffgridSphere") == 0) {
/// a lot of code

to

    } else if (strcmp(n.name(), "OffgridSphere") == 0) {
      OffgridSphere::Render(args)

or even

    } else if ( OffgridSphere::IsThisMe(n) ) {
      OffgridSphere::Render(args)

Profiling of LatticeAccess

I ran profiling for the TCLB. Special interest is LatticeAccess.cpp.Rt parsing time.
Full report for d2q9 and d3q27 models: https://gist.github.com/shkodm/607d98a0dfe3c244a1197016f4b125d7
Top list of compilation-time eaters for d3q27. Top of d2q9 is pretty much similar.

function name	total.time	total.pct	self.time	self.pct
"<Anonymous>"	145.46	100.00	2.42	1.66
"eval"	145.46	100.00	2.00	1.37
"lapply"	145.20	99.82	1.60	1.10
"FUN"	145.18	99.81	1.42	0.98
"txtlapply"	142.20	97.76	0.00	0.00
"field.access"	140.80	96.80	0.00	0.00
"Ops.gvector.apply"	114.84	78.95	0.12	0.08
"fun"	110.98	76.30	0.60	0.41
"offset.p"	95.22	65.46	0.06	0.04
"load.field"	91.98	63.23	0.00	0.00
"aggregate"	70.38	48.38	0.56	0.38
"aggregate.pAlg"	70.06	48.16	1.20	0.82
"*"	67.22	46.21	0.04	0.03
"*.pAlg"	65.74	45.19	1.80	1.24
"save.field"	48.82	33.56	0.00	0.00
"aggregate.data.frame"	48.46	33.32	1.90	1.31
"+"	40.68	27.97	0.16	0.11
"["	33.28	22.88	1.22	0.84
"+.pAlg"	32.32	22.22	0.10	0.07
"C"	30.54	21.00	0.02	0.01
"ToC"	30.42	20.91	0.04	0.03

Any ideas about what can be done with it?

Symmetry in d3q27 is inconsistent

I found that Symmetry in d3q27 is not working

Please fill:

My configuration:
run.txt

Same config work fine with d3q27_cummulant, faile with d3q27
Relevant symmetry code:

CudaDeviceFunction void NSymmetry()
{
<?R Symmetry(2, -1) ?>
}

CudaDeviceFunction void SSymmetry()
{
<?R Symmetry(2, 1) ?>
}

is the same for both models :/

BUT to have it work in d3q27 one need to change N with S in XML. Ideas?

PythonInline doesn't work on Travis-CI container

Something is wrong when I try to run TCLB tests on Travis-CI container infrastructure with PythonInline

https://travis-ci.org/llaniewski/TCLB/jobs/232514466#L1268

Everything else seems to work.

Branch: https://github.com/llaniewski/TCLB/tree/travis-container

@mdzik any ideas?

AddSetting

• should have obligatory default
• most test should use that default (without explicit setting in XML)
• we should provide AddStandardSetting('Viscosity') routine, so we have common defaults and common name around the solver

I could do that, but do we agree?

Using SHA checksum for testing results is insustainable

As we go into testing on multiple platforms (cuda/cpu, different compilers etc), the comparison based on comparing checksums of vti files cannot work. The result files have to be compared as numerical fields, up to a certain precision.

Currently running the tools/tests.sh fails on most architectures, as the sha sums are from Travis-CI VMs.

Possible solutions:

• base comparison on absolute difference between fields below some epsilon, done with:
  • vtk modules in python (lot of potential compatibility issues there)
  • some quick write up of a simple vti loader for R/python (not based of vtk, and no support for other formats)
  • pvpython?
• base comparison only on other types of output (TXT, Log, etc)

Commented out SaveMemoryDump

@mdzik @shkodm Why Saving is commented out in SaveMemoryDump?

https://github.com/CFD-GO/TCLB/blame/develop/src/Handlers.cpp.Rt#L2280

(Blame shows @mdzik)

Persisting problem with travis-ci tests

The previews issue #144 was solved by #142 and modified in #151
But problem in compilation still occures and random builds don't finish (end with errored, not failed)
The move to travis-ci container infrastructure was blocked by an issue with pythongeom in #143 , but I think that the idea should be revisited.

Introduction of NUMBA/PyCUDA integration

I found that https://devblogs.nvidia.com/numba-python-cuda-acceleration/

Question is:
This might be nice for custom inprocessing (on gpu) or preprocessing. BUT: how "easy" is access to "f[]" on GPU? Is it documented?
I was thinking about something like

<PythonGpuCall module="XX" method="XX" densities="XXX">

This could be in C/CUDA, but then we would need to compile it on the fly and use dynamic loading (also duable)

Use better checkpointing strategy

The current log-n strategy is suboptimal, especially for large memory capacity.
The plan is tu use some dynamic optimal checkpointing.

Is the TCLB can be used to simulate the fluid with free surface?

Hello everyone,
Is the Code can be used to simulate the fluid with free surface? For example, I want to simulate the flowing process of water in a channel, the water flow in from one side and flow out in other side.
Thanks a lot!
Best regard,
Henry

Clean up configure

Long overdue cleanup of the configure script:

• Set/Discover path to R
• Set/Discover path to nvcc
• Set/Discover path to g++ used
• Acquire options for future RCallback
• Delegate Catalyst, Python, R discovery to external script maybe

Make half-way bounce back easy to use

Improve on the usability of the cut-calculation for half-way bounce back.
For instance by making a R template function for generating such BB condition.

Example refactoring

@TravisMitchell Pointed out, that examples need to be restructured in the same manner as models.
I'm working on that

Extra xml added after config in export is wrong

I found that XML saving in XXX_config_P00_000000.xml is not working right

The xml file looks like this:

<?xml version="1.0"?>
<CLBConfig version="2.0" output="output/">
    <!-- this is OK -->
</CLBConfig>
<Run model="d2q9_kuper">
    <Code version="v6.0-beta-529-g31f6a5f" precision="double" cross="GPU" />
</Run>

I get this error during post-processing parsing:

xml.sax._exceptions.SAXParseException: /XXXXX/run_config_P00_00000000.xml:47:0: junk after document element

Which refers to part. this is wrong: XML states that document should have single root element

% xmllint NX1024_prom/id1/output/run_config_P00_00000000.xml                                                                                                                               ~/praca/bench_MP_Laplace
NX1024_prom/id1/output/run_config_P00_00000000.xml:47: parser error : Extra content at the end of the document
<Run model="d2q9_kuper">
^

Bugs in TAPENADE

This is an issue to keep track of bugs in TAPENADE.

1. Bug in initialization of dual variable

double W;

double A(double x) {
        x = x + 3; // <-- This line is crusial
        return x*x;
}

double B(double x) {
        double ret = 1;
        ret = A(W-x);
        return ret;
}

d3q19 on single precision

I've noticed, that d3q19 model fails to compile for single precision on CPU.
Presumably, due to type conflict somewhere in the model...

./configure --disable-cuda --disable-double

I get this error:

In file included from cuda.cpp:645:0:
Dynamics_sp.c: In member function 'void Node_Run<(eOperationType)0u, (eCalculateGlobals)1u, (eStage)2u>::CollisionMRT()':
Dynamics_sp.c:403:5: error: no matching function for call to 'max(real_t&, double)'
Dynamics_sp.c:403:5: note: candidate is:
In file included from cuda.cpp:21:0:
cross.h:239:40: note: template<class T> const T& max(const T&, const T&)
cross.h:239:40: note:   template argument deduction/substitution failed:
In file included from cuda.cpp:645:0:
Dynamics_sp.c:403:5: note:   deduced conflicting types for parameter 'const T' ('float' and 'double')
In file included from cuda.cpp:3043:0:
Dynamics_sp.c: In member function 'void Node_Run<(eOperationType)0u, (eCalculateGlobals)1u, (eStage)0u>::CollisionMRT()':
Dynamics_sp.c:403:5: error: no matching function for call to 'max(real_t&, double)'
Dynamics_sp.c:403:5: note: candidate is:
In file included from cuda.cpp:21:0:
cross.h:239:40: note: template<class T> const T& max(const T&, const T&)
cross.h:239:40: note:   template argument deduction/substitution failed:
In file included from cuda.cpp:3043:0:
Dynamics_sp.c:403:5: note:   deduced conflicting types for parameter 'const T' ('float' and 'double')
In file included from cuda.cpp:5551:0:
Dynamics_sp.c: In member function 'void Node_Run<(eOperationType)0u, (eCalculateGlobals)1u, (eStage)1u>::CollisionMRT()':
Dynamics_sp.c:403:5: error: no matching function for call to 'max(real_t&, double)'
Dynamics_sp.c:403:5: note: candidate is:
In file included from cuda.cpp:21:0:
cross.h:239:40: note: template<class T> const T& max(const T&, const T&)
cross.h:239:40: note:   template argument deduction/substitution failed:
In file included from cuda.cpp:5551:0:
Dynamics_sp.c:403:5: note:   deduced conflicting types for parameter 'const T' ('float' and 'double')

Compilation with:
./configure --enable-cuda --disable-double
is fine.

Merge policy

New rule proposal:
ANY new, model-related future/helper (auto BC, feq/collision helper etc.) will not be merged without reference to wiki (in code) OR at minimum meaningful API doc-string. Things like lib/boundary.R are unusable without that

Also, maybe we should come out with some merging guide, especially if we want someone to use it ;)

R profiling doens't work on Ubuntu for Windows

Requested reminder:
When using Ubuntu on Windows, calling commands such as make d2q9 results in an error related to timers, caused by a wrong default setting in the makefile if I recall correctly.

d3q27 MRT?

I found that something might be wrong with our "MRT"??

https://github.com/CFD-GO/TCLB/blob/v6.1_candidate/models/flow/d3q27/Dynamics.c.Rt#L182

	real_t gamma  = 1-omega;
//...
<?R
	S = PV("S",1:nrow(U)-1);
	S[EQ$order <= 2] = PV("gamma")
	S[EQ$order >  2] = PV("gamma2")
//...
?>
//...
real_t gamma2 = gamma;

Doesn't is mean, that it's not even TRT? it is (in fact) SRT-in-moments?

This might be serious issue for any external usage.

Execution after Failcheck finds NAN

It would be nice if we could execute something still after the failcheck finds NANs.

I propose this syntax:

<Failcheck>
  <VTK name="withnan" what="ALL"/>
</Failcheck>

This would write the VTK file after finding the nan. This could help identify that is happening better. I thinks it can be done with something like ExecuteInternal().

TCLB License

This issue is for contributors only.

If you agree for licensing your contribution to TCLB code under GPL v3 license, please add a comment in this Issue.

It is sufficient to write I agree to license my contributions to TCLB under GPL v3.

Note: Please don't start a discussion here. If you want to discuss the issue, please write an email.

Refactoring of all models

This should include:

• Change of legacy Bounce-Back to collision type (better logic and no more need for mask="ALL")
• Make MRT/BGK as options, and have one main Collision type (better performance)
• Separating all symmetries to another NodeType group, and execute before all
• Adding all NodeTypes explicitly in Dynamics.R (standard can be done as in #114)

Correct progress bar for logs

The current updating progress bar clogs the output log on clusters (lot of \r).

Additionally: print estimated time to finish

A separate ESYS-Particle xml element is needed

Problem

Because of my lazy naming, we ended up with a <RemoteForceInterface> element which both runs the Remote Force Interface (RFI) and executes (or connects to) ESYS-Particle code. This naming doesn't make sens and has to be changed.

Proposed solution

Separate elements should be available. The <RemoteForceInterface integrator="..."> should run RFI and connect to a selected integrator. A separate <ESYSParticle> element should run the ESYS-Particle code, and there should be different elements for other DEM codes (LAMMPS in near future).

Comments

This change is not backwards compatible. Old xml files involving RemoteForceInterface would have to be changed.

CONTRIBUTING.md typos

General spelling mistakes

Tracker

During some talk at UQ, a tracker was proposed, which would log usage of the code. Generally the idea is to send simple REST (http) requests logging in the usage of the code for the developers to have an general idea of the usage of the code. The information could potentially include:

• domain (for identification of universities etc)
• CPU/GPU information
• Total mesh size
• Total number of iterations
• commit hash
• model
• average MLBups
  A notice could be displayed in ./configure script and it could be deactivated on the level of ./configure --without-tracker or something similar.

It's a proposal for now. But I'm strongly considering it.

Practical implementation would need an efficient host receiving the information. The information would have to be send both at the beginning of the simulation and at end. I could be done in a wrapper script proposed in #214 or as part of the main executable.

create "all futures enabled d2q9" model

to test all, not physics related stuff. simple d2q9, but any new base solver future should be enabled there, to prevent conflicts etc. Good model to unittest etc
also, base for any new model

Add support for different Topology Parametrization

Add different topology parametrization modes like 1/2D extrusion, periodic repetition, etc.

Change of branch structure

The current approach to development structure of branches on git proved to be very problematic. Additionally the master branch is rarely updated. That's why I decided that the repository will change the approach, to a one more resembling gitflow:

• master branch will be the default branch to clone and will have the last (mostly stable) version of TCLB, with fixes (what develop is now)
• develop will be the next release candidate version, with all big changes well defined in Issues and Pull Requests
• feature/... branch for big mods that do not work yet to be published (like chimera)

This approach will take effect in 2 weeks on 17.01.2019

Please do not respond in this issue.

Notice for all people who have forked TCLB: @hbcbh1999, @zcy9311, @vonHousen, @ndivaira, @edk261, @JunhuaPAN, @younglch, @dlightbody, @bhill23, @linhanGE, @Leixushu, @mrutkowski-aero, @uqdwan11, @yuelipku, @ggruszczynski, @Zitzeronion, @PeiyongWang, @JonMcCullough, @taraspiotr, @Pabloob, @myousefi2016, @Raflen, @okpatil4u, @niyaa, @franjesus, @wojtasMEiL, @hietwll, @TravisMitchell, @mdzik, @TGajek, @llaniewski, @shkodm

This is preliminary issue: "clean up" milestone

Things to be cleared BEFORE moving forward with TCLB:

1. Put hold on any changes in TCLB codebase
2. Separate models into stable/experimental/obsolete
3. All current models are flagged experimental (or obsolete)
4. Move d2q9 (_new?) and d3q19 (?) to stable but:
   1. Unit test for values
   2. Improve comments and references. Remove/upgrade ALL obsolete BC code
   3. Rename to match conventions (nu->KinematicViscosity etc)
   4. Track nonneeded BC code/helper functions
   5. Rename things like CollisionMRT if name is misleading
5. Check which of models left in experimental have actual support, assign maintainer and push him to clean model to match stable standards
6. Clear out experimental models from abandoned ones, move them to obsolete

Despite how it looks, I think it could be done in few hours. Big part of it is already done.