otbrown / lambrex

Lattice Boltzmann code built on AMReX

LAMBReX 🐑👑

Lattice Boltzmann code built on AMReX

Build Instructions

LAMBReX

From the home directory create a build directory and cd to it -- for example mkdir build followed by cd build. Then simply run cmake .., and make. liblambrex.a can be found in build/src, the binaries for the example calculation and tests can be found in build/example and build/tests. You can also just run make test to run all tests. If using conan for Catch2, please run conan install .. before cmake.

Dependencies

AMReX

AMReX is a C++ adaptive mesh refinement library. The source can be downloaded from the AMReX GitHub.

Important: It is recommended to build AMReX with cmake for compatibility with LAMBReX. AMReX is updated regularly. LAMBReX is attempting to move with it, so minimum version 19.08 is REQUIRED, but it is recommended to use the current release.

To build the static AMReX library with cmake in way that is compatible with the current version of LAMBReX, one can follow the commands given in (or run) amrex_cmake.sh.

Catch2

The C++ Automated Tests in a Header (Catch2) test framework is used for testing LAMBReX. It can be found at Catch2. System-wide installation using cmake is recommended, as described here. Alternatively, it can be installed with Conan (see below), a conanfile is provided.

Conan

Conan is a C/C++ package manager. The recommended way to install it is with pip, simply using pip install conan. If you don't have and don't want pip installed, alternatives can be found here.

Examples

At present there is only one example code, a simple simulation of a "pulse" of enhanced density on a single plane, with fully periodic boundary conditions. This calculation was chosen for its simplicity, and the output was compared against the original code on which the Lattice Boltzmann calculation part of LAMBReX is based. This example is run as part of the tests run by make test.

lambrex's Issues

Tests SIGSEGV

$ ctest
Test project /Users/rupe/working/hilemms/lambrex-test/build/lambrex
    Start 1: Single Value Initialisation
1/2 Test #1: Single Value Initialisation ......***Failed    0.33 sec
    Start 2: Single Value Initialisation
2/2 Test #2: Single Value Initialisation ......***Failed    0.13 sec

0% tests passed, 2 tests failed out of 2

Total Test time (real) =   0.48 sec

The following tests FAILED:
	  1 - Single Value Initialisation (Failed)
	  2 - Single Value Initialisation (Failed)
Errors while running CTest

And trying the catch exe directly:

$ bin/c2InitTests
MPI initialized with 1 MPI processes
AMReX (19.02-2-g7d428cb32120) initialized
AMReX (19.02-2-g7d428cb32120) finalized
*** The MPI_Comm_dup() function was called after MPI_FINALIZE was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[mbp13-rnash2-1.epcc.ed.ac.uk:17459] Local abort after MPI_FINALIZE started completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

We can try to debug next week. Cheers!

Use function to initialise ParmParse

Second form of amrex::Initialize allows the user to specify a function to initialise the ParmParse database. Investigate what this means, and possibly make use of to alleviate difficulties with AmrSim construction. Drawback is possible requirement for MPI communicator.

void Initialize (int& argc, char**& argv, bool build_parm_parse=true,
                 MPI_Comm mpi_comm = MPI_COMM_WORLD,
                 const std::function<void()>& func_parm_parse = {},
                 std::ostream& a_osout = std::cout,
                 std::ostream& a_oserr = std::cerr);

Ensure consistency between refinement levels

Related to #8.

Adjust dt and dm with dx in order to ensure density and speed of sound are consistent between levels. Adjustments must also be made to the relaxation rates. Follow the approach used in Rohde at al. doi:10.1002/fld.1140.

how to install it

Performance benchmark

Hi, do you have any primary benchmark data about the performance on GPUs, especially overhead of AMRex.

Cases with periodic boundary conditions would work.

Update build documentation

• Test new build (including Conan and improved CMake) in Linux environment
• Update documentation accordingly

Implement Multi-level AmrSim

Implement pulse example simulation with one (or more!) refined levels. Requires implementation of remaining AmrCore override methods, and function for calculating dt at refined levels.

Initialise/Finalise no longer necessary

This used to be a necessary workaround due to reliance on parmparse, and static const declaration of geometry, but can probably be removed as of an amrex update from some time ago. Investigate and do better if possible.

AMReX 19.06 no longer requires blitz++

Hi -- I saw your comment at https://github.com/otbrown/LAMBReX/tree/AMR about blitz++ being required -- just an FYI that as of version 19.06, AMReX no longer has any dependence on blitz.

Rewrite Simulation class as AMR class

In order to actually make use of AMReX's power, the Simulation class should probably be converted to an AMR class (or inherit from it). This is the class AMReX uses to manipulate multiple levels of refinement.

Most likely a major rewrite.

Run with MPI

Test example calculation when AMReX is compiled with MPI. Maybe even use a parallel function!

Some questions about LBM in AMReX

hi, Recently, I have been referring to LAMBReX to realize the LBM code running on AMReX, but I have encountered some problems. How can I contact you?

CMake for running pulse regression test

I'm not happy about make running the pulse example! This won't work in a cross-compiling env (eg ARCHER).

Have a think about the best way to do this.

add_custom_command(
  TARGET pulse
  POST_BUILD
  COMMAND ${CMAKE_COMMAND} -E echo "Running $<TARGET_FILE_NAME:pulse> ..."
  COMMAND pulse > pulse.out
  )

add_test(
  NAME "pulse_diff"
  COMMAND ${CMAKE_COMMAND} -E compare_files pulse.out blessed_pulse.out
  WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/bins
)

Boundary Conditions

Using global sizes to limit loops:

for (k = lo[2]; k <= hi[2] && k < NZ; ++k)

means only data we expect is used, but likely means that built in functions for handling boundary conditions will look at uninitialised data.

Possible solution: Use global limit on data initialisation, then use FillPatch to interpolate data to fill out grid. Then use validbox sizes only for all loops.

Additionally, consider meaning of boundary conidtions at refined levels. Boxes which cover whole domain are periodic, but smaller boxes on refined grids may not be.

• Data initialisation corrected (see d310aa6)
• Boundary conditions at refined levels understood