lanl / clamr

Cell-Based Adaptive Mesh Refinement

The CLAMR code is a cell-based adaptive mesh refinement (AMR) mini-app developed as a testbed for hybrid algorithm development using MPI and OpenCL GPU code.

The CLAMR code is open-sourced under its LANL copyright /*

• Copyright (c) 2011-2019, Triad National Security, LLC.
• All rights Reserved.
• CLAMR -- LA-CC-11-094
• Copyright 2011-2019. Triad National Security, LLC. This software was produced
• under U.S. Government contract 89233218CNA000001 for Los Alamos National
• Laboratory (LANL), which is operated by Triad National Security, LLC
• for the U.S. Department of Energy. The U.S. Government has rights to use,
• reproduce, and distribute this software. NEITHER THE GOVERNMENT NOR
• TRIAD NATIONAL SECURITY, LLC MAKES ANY WARRANTY, EXPRESS OR IMPLIED, OR
• ASSUMES ANY LIABILITY FOR THE USE OF THIS SOFTWARE. If software is modified
• to produce derivative works, such modified software should be clearly marked,
• so as not to confuse it with the version available from LANL.
• See LICENSE file for full copyright

*/

Contributions to CLAMR are welcomed as long as they do not substantially change the nature of the code.

To build the CLAMR executables. CLAMR now used cmake for its builds

// In-tree build cmake . // out-of-tree build cmake

// Optimized build (part optimized RelWithDebInfo is default) cmake -DCMAKE_BUILD_TYPE=release . // Graphics options (OpenGL is the default) cmake -DGRAPHICS_TYPE=[None|OpenGL|MPE]

There are two real-time graphics packages. The default is OpenGL. An alternative real-time graphics package uses MPE. To use this package, configure with "cmake -DGRAPHICS_TYPE=MPE". The OpenGL option is automatically turned off when selecting MPE. MPE is part of the MPICH package from Argonne National Laboratory. It usually is not on a system and will need to be installed. A setup script and version 1.9.1 of the MPE package are available in the download directory.

make

Seven executables are currently built. The first four are "standalone" versions which run one implementation of the routines

clamr: Calls the MPI/GPU versions of each call. Option to check the results against other verisons

clamr_gpuonly: Calls the GPU versions of each call. Option to check the results against the cpu calls

clamr_cpuonly: Calls the CPU versions of each call. Option to check the results against the gpu calls

clamr_mpionly: Calls the MPI versions of each call. Option to check the results against the cpu calls

Check versions. These versions run multiple implementations and check correctness of the implementations

clamr_gpucheck: Calls the GPU and CPU versions of each call and checks the results against each other

clamr_mpicheck: Calls the CPU and MPI/CPU versions of each call and checks the results against each other

clamr_checkall: Calls the GPU, CPU, MPI and GPU/MPI versions of each call and checks the results against each other.

More executables are planned

Currently the executables run only on NVIDIA GPUs. Fixing the kernels to run on ATI GPUs and MIC is of great interest

The numerical algorithm still does not handle "dry" conditions properly and will crash

Many other limitations exist -- coarsening has not been implemented and boundary conditions need some more work

Current performance shows about a 30x speedup on the GPU versus the CPU using NVIDIA Tesla 2090s and Intel CPUs

See the PAPERS file for a list of publications related to the CLAMR code (Papers.bib for bibtex format)

clamr's Issues

Crash - Invalid Reads

Hi Bob,

I'm trying to trace down a crash when I start including some j7 stuff into MallocPlus. When I run clamr_quo (you can pull by fork of CLAMR to see this), I get the following crash (attached is some Valgrind output). It may be my code, so I'm still looking. That being said, j7 appears to be Valgrind clean and I haven't yet integrated any real code into MallocPlus. So, it may be that I've changed the memory layout enough to tickle this error. I can't say for sure. If you like, take a look. I'll continue looking on my end.

It crashes almost immediately:

$ mpirun -n 2 ./clamr_quo
System Info ###
Nodes: 1
NUMA Nodes: 0
Sockets: 1
Cores: 1
PUs: 1
MPI Procs on Node: 2
MPI_COMM_WORLD ranks in subComm: 0
rank: 0 is subComm rank: 0
Size comm world 2 subComm 1
--- num openmp threads: 1
[ubuntu-12:48907] *** Process received signal ***
[ubuntu-12:48907] Signal: Segmentation fault (11)

Thanks!

--- num openmp threads: 1
==48416== Invalid read of size 4
==48415== Invalid read of size 4
==48416== at 0x4553C2: Mesh::is_left_boundary(int) (mesh.h:302)
==48416== by 0x474E47: Mesh::calc_celltype(unsigned long) (mesh.cpp:7304)
==48416== by 0x463629: Mesh::init(int, int, double, partition_method, int) (mesh.cpp:1901)
==48416== by 0x44499C: main (clamr_quo.cpp:180)
==48416== Address 0x0 is not stack'd, malloc'd or (recently) free'd
==48416==
[ubuntu-12:48416] *** Process received signal ***
==48415== at 0x4553C2: Mesh::is_left_boundary(int) (mesh.h:302)
==48415== by 0x474E47: Mesh::calc_celltype(unsigned long) (mesh.cpp:7304)
==48415== by 0x463629: Mesh::init(int, int, double, partition_method, int) (mesh.cpp:1901)
==48415== by 0x44499C: main (clamr_quo.cpp:180)
==48415== Address 0x0 is not stack'd, malloc'd or (recently) free'd
==48415==
[ubuntu-12:48415] *** Process received signal ***
[ubuntu-12:48416] Signal: Segmentation fault (11)
[ubuntu-12:48416] Signal code: Address not mapped (1)
[ubuntu-12:48416] Failing at address: (nil)
[ubuntu-12:48415] Signal: Segmentation fault (11)
[ubuntu-12:48415] Signal code: Address not mapped (1)
[ubuntu-12:48415] Failing at address: (nil)
[ubuntu-12:48415] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x6e2fcb0]
[ubuntu-12:48416] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x6e2fcb0]
[ubuntu-12:48415] [ 1] ./clamr_quo(_ZN4Mesh16is_left_boundaryEi+0x42) [0x4553c2]
[ubuntu-12:48415] [ 2] ./clamr_quo(_ZN4Mesh13calc_celltypeEm+0xb6) [0x474e48]
[ubuntu-12:48415] [ 3] ./clamr_quo(_ZN4Mesh4initEiid16partition_methodi+0x8cc) [0x46362a]
[ubuntu-12:48415] [ 4] ./clamr_quo(main+0x189) [0x44499d]
[ubuntu-12:48415] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x705e76d]
[ubuntu-12:48415] [ 6] ./clamr_quo() [0x444759]
[ubuntu-12:48415] *** End of error message ***
[ubuntu-12:48416] [ 1] ./clamr_quo(_ZN4Mesh16is_left_boundaryEi+0x42) [0x4553c2]
==48415== Invalid read of size 4
[ubuntu-12:48416] [ 2] ./clamr_quo(_ZN4Mesh13calc_celltypeEm+0xb6) [0x474e48]
[ubuntu-12:48416] [ 3] ./clamr_quo(_ZN4Mesh4initEiid16partition_methodi+0x8cc) [0x46362a]
[ubuntu-12:48416] [ 4] ./clamr_quo(main+0x189) [0x44499d]
[ubuntu-12:48416] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x705e76d]
[ubuntu-12:48416] [ 6] ./clamr_quo() [0x444759]
[ubuntu-12:48416] ** End of error message ***
==48416== Invalid read of size 4
==48415== at 0x4553DC: Mesh::is_left_boundary(int) (mesh.h:302)
==48415== by 0x474E47: Mesh::calc_celltype(unsigned long) (mesh.cpp:7304)
==48415== by 0x463629: Mesh::init(int, int, double, partition_method, int) (mesh.cpp:1901)
==48415== by 0x44499C: main (clamr_quo.cpp:180)
==48415== Address 0x8b58540 is 8 bytes after a block of size 8 alloc'd
==48415== at 0x4C2B1C7: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==48415== by 0x44A457: _gnu_cxx::new_allocator::allocate(unsigned long, void const) (new_allocator.h:92)
==48415== by 0x44A0AC: std::_Vector_base<int, std::allocator >::_M_allocate(unsigned long) (in /home/samuel/devel/CLAMR/clamr_quo)
==48415== by 0x449DDE: std::vector<int, std::allocator >::_M_fill_insert(__gnu_cxx::_normal_iterator<int, std::vector<int, std::allocator > >, unsigned long, int const&) (vector.tcc:419)
==48415== by 0x44982F: std::vector<int, std::allocator >::insert(__gnu_cxx::__normal_iterator<int, std::vector<int, std::allocator > >, unsigned long, int const&) (stl_vector.h:944)
==48415== by 0x4494B9: std::vector<int, std::allocator >::resize(unsigned long, int) (stl_vector.h:632)
==48415== by 0x4625AB: Mesh::Mesh(int, int, int, int, int, int, int, int) (mesh.cpp:1699)
==48415== by 0x444963: main (clamr_quo.cpp:170)
==48415==
==48415== Invalid read of size 4
==48415== at 0x455430: Mesh::is_right_boundary(int) (mesh.h:303)
==48415== by 0x474E7C: Mesh::calc_celltype(unsigned long) (mesh.cpp:7305)
==48415== by 0x463629: Mesh::init(int, int, double, partition_method, int) (mesh.cpp:1901)
==48416== at 0x4553DC: Mesh::is_left_boundary(int) (mesh.h:302)
==48415== by 0x44499C: main (clamr_quo.cpp:180)
==48415== Address 0x0 is not stack'd, malloc'd or (recently) free'd
==48416== by 0x474E47: Mesh::calc_celltype(unsigned long) (mesh.cpp:7304)

clamr fails to build in Spack

Found in https://travis-ci.org/proxyapps/proxy-ci/jobs/409647992:

[ 87%] Building CXX object CMakeFiles/clamr_openmponly.dir/state.cpp.o
/home/ci/spack/lib/spack/env/gcc/g++  -DHAVE_CONFIG_H -DMIXED_PRECISION -I/home/ci/spack/var/spack/stage/clamr-master-md6rutwflpy7csefnnxs4kzccutxvkly/CLAMR -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include -I/usr/lib/x86_64-linux-gnu/openmpi/include -I/tmp/ci/spack-stage/spack-stage-fxG4Bg/CLAMR/spack-build  -ftree-vectorize -O2 -g -DNDEBUG -fPIE   -fopenmp -std=gnu++11 -o CMakeFiles/clamr_openmponly.dir/state.cpp.o -c /home/ci/spack/var/spack/stage/clamr-master-md6rutwflpy7csefnnxs4kzccutxvkly/CLAMR/state.cpp
/home/ci/spack/var/spack/stage/clamr-master-md6rutwflpy7csefnnxs4kzccutxvkly/CLAMR/state.cpp: In member function 'void State::calc_finite_difference_regular_cells(double)':
/home/ci/spack/var/spack/stage/clamr-master-md6rutwflpy7csefnnxs4kzccutxvkly/CLAMR/state.cpp:3349:46: error: cannot convert 'double***' to 'state_t*** {aka float***}' in initialization
        state_t ***pstate = mesh->meshes[lev].pstate;
                                              ^~~~~~

facen128lev4 test fails in min precision

Failure is seg fault after 0th iteration of :
"mpirun -n 2 ./clamr_mpiopenmponly -n 128 -i 100 -t 3000"
(where OMP_NUM_THREADS set to 2)
Oddly, the test passes with verbose output "ctest -R facen128lev4 -V"
Running from terminal mpirun -n 2 ./clamr_mpiopenmponly -n 128 -i 100 -t 3000 with 2 OMP threads also works with no issue.

Tag a release

The last release is from 2014, please tag a new one.

CLAMR fails to build on GCC<8

The -mprefer-vector-width=x option seems to have been added only recently in GCC, so building with 7.4 is currently failing.

OpenMP SIMD question

We are working on a static tool that flagged a potential problem with an OpenMP simd loop in CLAMR, however we are unsure if the problem we detected is real.

CLAMR/l7/l7_update.c

Lines 554 to 560 in a5f7a67

	#ifdef _OPENMP_SIMD
	#pragma omp simd
	#endif
	for (int j=0; j<send_count; j++){
	local_indices[offset] = l7_id_db->indices_local_to_send[offset];
	offset++;
	}

Does the offset++ line in the above loop introduce a dependency between iterations that prevents this loop from from using SIMD instructions?

I have read through the OpenMP specification for simd and tried to look for some other examples, but I am still not sure how this loop would be handled.

cpuonly test fails for minimum precision

The test for cpuonly: "./clamr_cpuonly -n 128 -i 100 -t 3000" fails for minimum precision. At iteration 3000, a NAN is encountered.

Got a NAN on cycle 3000

This was tested on a Broadwell architecture with Intel 18.0.2 and openmpi 2.1.3

Should the test be changed to work for minimum precision?

The compiler could not find the sys/sysctl.h header file

Hi,I encountered the following error when compiling. I looked up some information and thought that the glibc version of my system was higher (2.34). Glic did not have this file（sys/sysctl.h） after 2.32. Is there any plan to adapt to the high version of glibc?

fatal error: sys/sysctl.h: No such file or directory
     358       62 | #include <sys/sysctl.h>
     359          |          ^~~~~~~~~~~~~~
     360    compilation terminated.
  >> 361    make[2]: *** [memstats/CMakeFiles/memstats.dir/build.make:85: memstats/CMakeFiles/memstats.dir/memstats.c.o] Error 1
     362    make[2]: Leaving directory '/home/stage/root/spack-stage-clamr-master-fenfgrhpxophuuyquc7iu2k4mnn2xdk7/spack-build-fenfgrh'
  >> 363    make[1]: *** [CMakeFiles/Makefile2:2164: memstats/CMakeFiles/memstats.dir/all] Error 2
     364    make[1]: *** Waiting for unfinished jobs....
     365    Scanning dependencies of target mpmesh
     366    Scanning dependencies of target pmpmesh
     367    Scanning dependencies of target pmesh
     368    Scanning dependencies of target mesh

openmponly test fails for min precision

NAN on iteration 2900 in test 1

Not sure if this is a bug or just limitation of precision?

Fix L7_ASSERT in L7_Update

Hi Bob,

Are you seeing these?

$ mpirun -n 3 ./clamr_quo
System Info ###
Nodes: 1
NUMA Nodes: 0
Sockets: 1
Cores: 1
PUs: 1
MPI Procs on Node: 3
MPI_COMM_WORLD ranks in subComm: 0
rank: 0 is subComm rank: 0
Size comm world 3 subComm 1
--- num openmp threads: 1
Mass of initialized cells equal to 92652.9978195
CPU: setup time time was 0.1072 0.1077 0.1070 s
Memory used in startup 7832 7812 7808 kB
Memory peak in startup 7832 7812 7808 kB
Memory free at startup 1521316 1521316 1521316 kB
Memory available at startup 2043828 2043828 2043828 kB
Iteration 0 timestep n/a Sim Time 0.0 cells 17028 Mass Sum 92652.9978195
[pe:1] L7 Error L7_UPDATE (/home/samuel/devel/CLAMR/l7/l7_update.c:142[!(l7_id > 0)]) (l7_id <= 0) (-1)
[pe:2] L7 Error L7_UPDATE (/home/samuel/devel/CLAMR/l7/l7_update.c:142[!(l7_id > 0)]) (l7_id <= 0) (-1)
[pe:0] L7 Error L7_UPDATE (/home/samuel/devel/CLAMR/l7/l7_update.c:142[!(l7_id > 0)]) (l7_id <= 0) (-1)
[pe:1] L7 Error L7_UPDATE (/home/samuel/devel/CLAMR/l7/l7_update.c:142[!(l7_id > 0)]) (l7_id <= 0) (-1)
[pe:2] L7 Error L7_UPDATE (/home/samuel/devel/CLAMR/l7/l7_update.c:142[!(l7_id > 0)]) (l7_id <= 0) (-1)
[pe:0] L7 Error L7_UPDATE (/home/samuel/devel/CLAMR/l7/l7_update.c:142[!(l7_id > 0)]) (l7_id <= 0) (-1)
[pe:1] L7 Error L7_UPDATE (/home/samuel/devel/CLAMR/l7/l7_update.c:142[!(l7_id > 0)]) (l7_id <= 0) (-1)
[pe:2] L7 Error L7_UPDATE (/home/samuel/devel/CLAMR/l7/l7_update.c:142[!(l7_id > 0)]) (l7_id <= 0) (-1)
[pe:0] L7 Error L7_UPDATE (/home/samuel/devel/CLAMR/l7/l7_update.c:142[!(l7_id > 0)]) (l7_id <= 0) (-1)
DEBUG -- cell 6117 lev 2

clamr fails to build with gcc-8.0.1

[ 29%] Building CXX object crux/CMakeFiles/pcrux.dir/crux.cpp.o
cd /tmp/ci/spack-stage/spack-stage-PXVdjp/CLAMR/spack-build/crux && /home/ci/spack/lib/spack/env/gcc/g++  -DHAVE_CONFIG_H -DHAVE_MPI -DMIXED_PRECISION -I/home/ci/spack/var/spack/stage/clamr-master-tqfbejmjaw4527dau4gzfq5zr6ea3hvj/CLAMR -I/usr/include/openmpi-x86_64 -I/usr/lib64/openmpi/include -I/tmp/ci/spack-stage/spack-stage-PXVdjp/CLAMR/spack-build  -ftree-vectorize -ftree-vectorize -O2 -g -DNDEBUG -fPIC   -o CMakeFiles/pcrux.dir/crux.cpp.o -c /home/ci/spack/var/spack/stage/clamr-master-tqfbejmjaw4527dau4gzfq5zr6ea3hvj/CLAMR/crux/crux.cpp
In file included from /usr/include/c++/8/map:60,
                 from /home/ci/spack/var/spack/stage/clamr-master-tqfbejmjaw4527dau4gzfq5zr6ea3hvj/CLAMR/MallocPlus/MallocPlus.h:56,
                 from /home/ci/spack/var/spack/stage/clamr-master-tqfbejmjaw4527dau4gzfq5zr6ea3hvj/CLAMR/MallocPlus/MallocPlus.cpp:59:
/usr/include/c++/8/bits/stl_tree.h: In instantiation of 'class std::_Rb_tree<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, malloc_plus_memory_entry*>, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, malloc_plus_memory_entry*> >, cmp_str, std::allocator<std::pair<const std::__cxx11::basic_string<char>, malloc_plus_memory_entry*> > >':
/usr/include/c++/8/bits/stl_map.h:151:17:   required from 'class std::map<std::__cxx11::basic_string<char>, malloc_plus_memory_entry*, cmp_str>'
/home/ci/spack/var/spack/stage/clamr-master-tqfbejmjaw4527dau4gzfq5zr6ea3hvj/CLAMR/MallocPlus/MallocPlus.cpp:95:48:   required from here
/usr/include/c++/8/bits/stl_tree.h:452:21: error: static assertion failed: comparison object must be invocable with two arguments of key type
       static_assert(__is_invocable<_Compare&, const _Key&, const _Key&>{},
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [MallocPlus/CMakeFiles/pMallocPlus.dir/build.make:66: MallocPlus/CMakeFiles/pMallocPlus.dir/MallocPlus.cpp.o] Error 1
make[2]: Leaving directory '/tmp/ci/spack-stage/spack-stage-PXVdjp/CLAMR/spack-build'
make[1]: *** [CMakeFiles/Makefile2:2558: MallocPlus/CMakeFiles/pMallocPlus.dir/all] Error 2

Details here: https://travis-ci.org/proxyapps/proxy-ci/jobs/375671286 and https://api.travis-ci.org/v3/job/375671286/log.txt

Broken Build

Hi,

Just pulled the master head and I get the following when trying to build clamr_quo.

/home/samuel/scratch/CLAMR/hash/hash.c:6:28: fatal error: hashlib_kern.inc: No such file or directory
compilation terminated.
make[3]: *** [hash/CMakeFiles/hash.dir/hash.c.o] Error 1
make[2]: *** [hash/CMakeFiles/hash.dir/all] Error 2
make[1]: *** [CMakeFiles/clamr_quo.dir/rule] Error 2
make: *** [clamr_quo] Error 2

Updating from Python 2 to Python 3

Our codebase will soon no longer support Python 2. We have CLAMR within our tests and have noticed that there is some code that is explicitly Python 2, including the generate_image.py script and a call in clamr_plot.py:

CLAMR/tests/hlopenmp_paper/clamr_plot.py

Line 15 in 07ae359

print "Syntax: python2.7 self_plot.py <FILES>"; exit(1)

We were curious if there is any plan to update CLAMR, or if the python2->python3 updates from our side may be desirable in a PR to this repo.

cannot convert 'state_t*** {aka float***}' to 'double***' in assignment ( gentrimatrix_p(knum, jnum, inum, elsize, __FILE__, __LINE_ 2956 _) )

Spack CI found:

/tmp/spack-stage/ci/clamr-master-gpvlo4l5oyt7p3hnhphclvkd5g4aq4wd/spack-src/genmalloc/genmalloc.h:72:70: error: cannot convert 'state_t*** {aka float***}' to 'double***' in assignment ( gentrimatrix_p(knum, jnum, inum, elsize, __FILE__, __LINE__) )

Details see here: https://travis-ci.org/proxyapps/proxy-ci/jobs/580050392

We only need to build against master as there is still no release (#55), please tag one.