trixi-framework / libtrixi Goto Github PK

Addendum to #56 : Do we need and can we make sure that the versions in file VERSION and LibTrixi.jl/Project.toml agree?

At the very least, we should check that

• everything compiles
• the example programs run without errors

In the future, we should of course also add numeric tests that involve the results from running a Trixi.jl simulation through one of our example programs, but that's something for another time...

I think it would be extremely helpful during development (and for debugging production) to be able to simply activate certain debug statements inside the code. I am thinking of something like an environment variable LIBTRIXI_DEBUG, which can be either set to 1 or 0 and which enables additional messages. One could even consider to make this more configurable, e.g., to have different debug levels such as all, c, julia to only activate a subset of the debug statements.

Something I do not get straight: When we follow the "Julia-REPL-like" approach to call everything else, we need a Julia runtime environment. Currently (#1) the system environment is used, but in the end, when we need to switch MPI and so on, we need a dedicated one.

1. Where should this Julia project reside? In the source tree? At a user specified location? Auto-generated by cmake install?
2. A user will typically execute his code from somewhere else. How does libtrixi know the location of the project to activate. It cannot be hard-coded. Should it be passed to trixi_initialize()?

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

@bgeihe had some issues with coupling MESSy to Trixi.jl that seem to be due to mismatching MPI libraries/p4est libraries. We should figure out how to do this properly and then document it for the users (and us!).

Right now, trixi_initialize in LibTrixi.jl sets up a concrete simulation. However, it might be useful/necessary to separate this into two functions:

• One function is only ever called once and does all "global" initialization of libtrixi/Trixi.jl (e.g., this could continue to be named trixi_initialize)
• One function is called to create a concrete new simulation setup (e.g., this could be named trixi_create_simulation or something similar)

In our CMake build, we should extract the version from the VERSION file and provide it as macro definitions to libtrixi, e.g., LIBTRIXI_VERSION_MAJOR, LIBTRIXI_VERSION_MINOR, LIBTRIXI_VERSION_PATCH, and LIBTRIXI_VERSION (for a full version string).

Then, we should provide API functions that allow one to query the version information from the library, e.g.,

int libtrixi_version_major();
int libtrixi_version_minor();
int libtrixi_version_patch();
const char* libtrixi_version();

plus of course the corresponding Fortran bindings. The first three should always just return exactly the corresponding part as integers, while the last function can include details after the patch version such as -dev or +build0.

IMHO, such an ability of a library to identify itself is crucial for proper usability and debugging of downstream programs.

When doing something like

trixi_initialize( project_path, NULL );
[...]
trixi_finalize();
trixi_initialize( project_path, NULL );

we get

libtrixi: finalize
ERROR in /home/bene/libtrixi/src/api.c:95 (trixi_initialize):
The following Julia code could not be evaluated: println("Module LibTrixi.jl loaded")

i.e. after trixi_finalize(), and thereby jl_atexit_hook(0), and the following trixi_initialize(), and thereby jl_init(), the subsequent calls to jl_eval_string fail. To be precise, the first call with using Pkg does not fail (but maybe does not work correctly either?), but the following call with Pkg.activate(...) fails. (Normally it is not visible in console output, so the first visible failure is the call with println(...).

When looking at the current coverage builds, it seems like neither Coveralls nor Codecov are trackingn our C header files. Is this intentional or an oversight @bgeihe?

To be sure, as long as we do not have inline implementations of functions that exist only in headers, everything of relevant functionality should be tested. However, I am wondering if we should include the header files nonetheless.

This is not complete yet, e.g. switching to system MPI and p4est.

Originally posted by @bgeihe in #10 (comment)

Should we somewhere add an example CMakeLists.txt which compiles the fortran_hello_world as a standalone cmake project? Or maybe just add the cmake code to README.md?

For reference:

#
# Fortran Hello Trixi
#



# Specify minimum cmake version
cmake_minimum_required ( VERSION 3.0 )

# Specify project info
project ( fortran_hello_world VERSION 0.1.0 DESCRIPTION "Example for using libtrixi Fortran" )

# Enable Fortran
enable_language(Fortran)



# Find libtrixi
find_library ( TRIXI_LIBRARY trixi REQUIRED)

# Find libtrixi.mod
find_path ( TRIXI_MOD_DIR libtrixi.mod PATH_SUFFIXES lib REQUIRED)

# Find MPI
find_package( MPI REQUIRED )

# Find Julia
list ( APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}" )
find_package( Julia REQUIRED )



# Define the executable
add_executable(${PROJECT_NAME} fortran_hello_world.f90)

# Add libraries for linking
target_link_libraries(${PROJECT_NAME} PRIVATE ${MPI_Fortran_LIBRARIES} ${TRIXI_LIBRARY} ${JULIA_LIBRARY})

# Add include directories
target_include_directories(${PROJECT_NAME} PRIVATE ${MPI_Fortran_INCLUDE_DIRS} ${TRIXI_MOD_DIR})

Requires CMAKE_PREFIX_PATH=<libtrixi_install_directory> when running cmake.

Originally posted by @bgeihe in #1 (comment)

Mother issue for testing.

Goals:

• #73
• #82
• #67
• #85
• #49

Files / folders:

• trixi.h is the public interface definition, it should not change
• Organize C-API implementation according to purpose (setup.c, version.c, control.c, data.c)
• C-API (api.c), Fortran-API (api.f90), Julia-API (api_c.jl, api_jl.jl) remain in single files
• function pointers must be made available to multiple c-files. Move them to function_pointers.h/c?
• General purpose routines like print_and_die are moved to auxiliary.h/c

Code:

• all interface routines should start with trixi_
• trixi_version_* is replaced by trixi_version_library_* to be consistent with trixi_version_julia*
• julia_eval_string is renamed to trixi_eval_julia and used in examples

• Create developer documentation on how to create a new release
• Verify that JuliaRegistrator is properly set up
• Verify that TagBot is properly set up
• Refer from LibTrixi.jl README to main README and main docs
• Follow steps in developer docs to create a new release
• Add reference to stable docs
• Update repo link to point to stable docs

Create a nice Fortran example for using LibTrixi.jl.

Since last week the setup of the Julia project fails, also on main.

The error message is attached but not very instructive.

*** An error occurred in MPI_Init_thread
ci_fail.txt

Comparing to the last successful CI run for main, the differences in Julia packages are:

35c34
< ⌅ [fa961155] + CEnum v0.4.2
---
>   [fa961155] + CEnum v0.4.2
56c55
<   [6b5a15aa] + P4est_jll v2.8.1+3
---
>   [6b5a15aa] + P4est_jll v2.8.1+2
159c152
<   [b4f34e82] + Distances v0.10.10
---
>   [b4f34e82] + Distances v0.10.9
194c187
<   [7ed4a6bd] + LinearSolve v2.9.2
---
>   [7ed4a6bd] + LinearSolve v2.8.1
206c199
<   [8913a72c] + NonlinearSolve v2.1.0
---
>   [8913a72c] + NonlinearSolve v2.0.1
210c203
<   [1dea7af3] + OrdinaryDiffEq v6.58.0
---
>   [1dea7af3] + OrdinaryDiffEq v6.57.0
233c226
<   [47a9eef4] + SparseDiffTools v2.7.0
---
>   [47a9eef4] + SparseDiffTools v2.6.0
292c286
<   [1dea7af3] + OrdinaryDiffEq v6.58.0
---
>   [1dea7af3] + OrdinaryDiffEq v6.57.0

So, this could be related to
https://github.com/trixi-framework/P4est.jl/pull/88/files
see the discussion
trixi-framework/P4est.jl#88

The issue can be reproduced on rocinante. (precompilation step of the libtrixi-julia project)

It works on my local machine.

Is the runtime of the shim library variant and the PackageCompiler variant comparable to running Trixi.jl in a pure Julia setting?

As mentioned in #116 (comment), with #116 we had to introduce
https://github.com/trixi-framework/libtrixi/blob/43098506d3b25d558afbcacc7abb390ae5557324/docs/make.jl#L41C5-L41C35
in order to make the cross reference to a parent directory not fail. However, this is highly dangerous, since it effectively disables hard errors for any cross reference failures, potentially hiding issues when adding new documentation. This should thus be fixed soon.

xref: JuliaDocs/Documenter.jl#2271

With this little code written so far, it should be possible to get coverage to at least >90%. I'd say we should strive to hit 95% once and then stay above it...

We might need to introduce a more sophisticated way of testing things, e.g., using a proper (3rd party or hand-written) testing framework.

Here are some initial thoughts for an API that would allow one to do more than just control the main time integration loop from outside. The idea is to keep this initial post updated (by anyone) with the results of the subsequent discussion.

Basic querying

Functions to get basic information from Trixi.jl.

int trixi_ndims(int handle);         // Return number of spatial dimensions
int trixi_nelements(int handle);     // Return number of elements (cells)
int trixi_polydeg(int handle);       // Return polynomial degree of DGSEM approximation
int trixi_nvariables(int handle);    // Return number of (conservative) variables
int trixi_ndofs(int handle);         // Return total number of degrees of freedom
int trixi_ndofs_element(int handle); // Return number of degrees of freedom for one element

Raw data exchange

Direct access to solution data u.

void trixi_load_u(double* u, int handle);        // Load data from Trixi.jl into `u`
void trixi_store_u(const double* u, int handle); // Store data from `u` in Trixi.jl

Block-structured data exchange

DGSEM has >> 1 degrees of freedom (DOF) per element. However, other programs might not care for our Gauss node distribution and just want equidistant data (a.k.a. finite-volume-type block-structured data), e.g., 8x8x8 cell-centered values per element. Thus we provide convenience functions that allow loading and storing in such block-structured format and do the conversion to Gauss nodes internally.

void trixi_set_blocksize(int blocksize, int handle);   // Set 1D block size
int  trixi_get_blocksize(int handle);                  // Get 1D block size
int  trixi_ndofs_block(int handle);                    // Return total #DOFs for block-structured data
void trixi_load_u_block(double* u, int handle);        // Load block-structured data from Trixi.jl
void trixi_store_u_block(const double* u, int handle); // Store block structured data in Trixi.jl

Now that we have releases, we should add a Zenodo DOI.

For Julia-based repos we can use JuliaRegistrator together with TagBot to automatically create new releases. For libtrixi, we ideally want something similarly easy to use, maybe with the ability to create a release including a nice summary of merged PRs/closed issues since the last release upon pushing a tag with a pattern of v#.#.#.

Thus, we might want to have a workflow that uses one action to create a release summary/changelog (e.g., https://github.com/mikepenz/release-changelog-builder-action), and then passes it own to another action that creates the actual release (e.g., https://github.com/softprops/action-gh-release).

Right now we only show how to use hand-crafted Makefiles to build an executable that links to libtrixi. It would IMHO be nice to provide a small CMake module that allows one to properly link one's code against libtrixi if building a CMake project.

In preparation for that, #76 will already provide a LIBTRIXI_VERSION file in share/julia that can be parsed by CMake to provide version information.

Implement a canonical fix for #54 once JuliaParallel/MPI.jl#746 is resolved.

In the meantime think about using using the internal variable that is set in run_init_hooks,

https://github.com/JuliaParallel/MPI.jl/blob/2a2bd68a6dc53660108bbc16690e4972de6318d5/src/MPI.jl#L55

to check if we need to run it again. That is, something like

      "if MPI.Initialized() && !MPI._finished_loading[];\n"

Originally posted by @sloede in #54 (review)

Unfortunately Fortran is not natively supported by google test. A way around would be to write a Fortran program testing everything that is currently covered by the C-tests, write an interface to this function, and call it from a C++ google test.
It feels a little strange though, because we would call an interface function to Fortran, which would then call interface functions to C.

Alternatively, provided we keep running our examples, we could make sure that the examples use all of the provided Fortran functions.

The latter however seems not straightforward as apparently the Fortran interface calls are not considered in the coverage reports.

Originally posted by @bgeihe in #73 (comment)

Right now we put them in the lib/ dir, because I guess this is how we conceptually think about them. However, they really belong into the include directory, since that's where gfortran/ifort will look for them.

I overlooked that the filter expressions

One could rewrite this (non-critical) piece of code or bump the the compat entry.

Command line:

LIBTRIXI_DEBUG=all mpirun -n 2 ./bin/simple_trixi_controller_c ../libtrixi-julia ./share/libtrixi/LibTrixi.jl/examples/libelixir_p4est2d_dgsem_euler_sedov.jl

After initialization and

Simulation running 'CompressibleEulerEquations2D' with DGSEM(polydeg=4)
[...]

there is no progress anymore. Both processes use 100% CPU.

libelixir_tree2d_dgsem_euler.jl still works as expected.

We should make a conscious decision about at least a few very basic data types we want to use in the beginning.

For floating-point data, I think there is not much to discuss: double precision is IMHO the only viable option.

For integer data, there are several options, since C defaults to 32-bit int while Julia defaults to 64-bit Int64 (which we use in Trixi.jl):

1. We could just use int for everything. That's by far the simplest option, but might put a hard limit on the maximum problem size (e.g., with N = 3 and 64 DOFs/element, this restricts to at most ~34 mio. elements).
2. We could use long for everything. This is similarly simple in terms of the API, and there is no chance for accidentally using a wrong type. It might put a (memory, usability) burden on everyone not working with large problems though.
3. We could use long for everything that might conceivably grow beyond 2.1 billion, such as number of elements, number of DOFs etc., and int for everything else. This is conservative with memory, but also easy to get wrong ("which integer was used for this API call again?").
4. We could provide two separate APIs: a default one foo(int) and one with large-integer support foo(long). This gives users flexibility, but causes overhead for the libtrixi maintainers (everything has to be set up to work for both sizes, additional testing etc.).
5. We could eschew using hard-coded integer values and make it configurable during build time. That is, have the ability to compile with 32-bit integers or 64-bit integers. This gives users flexibility, but creates overhead for libtrixi maintainers.

I don't think we need to make a decision now and stick to it for eternity. However, we should make some decision now, while being aware of the immediate implications/limitations. A later switch is always possible, but might be a breaking change (and, of course, cause more overhead).

We should explicitly set the C/Fortran versions we want to compile against such that there are no surprises about features being supported differently on different compilers, or ambiguity about which features we use or need.

Since requiring an arbitrarily recent language standard limits the choice of compilers, I suggest to try to play it safe for now and require C11 and Fortran 2008 2018 (if those actually work).

EDIT: We actually need Fortran 2018 😱

• #7
• #3
• #8
• #9
• #17
• #15
• #10
• #31
• #11

For the future (neither time nor mission critical at the moment):

• #22

The current cmake setup is able to make install the libtrixi shared object and include files.

Should we keep providing this?

Missing once #20 and #21 are deployed:

• LibTrixi.jl (just copy it?)

No, I’d say that it’s either PackageCompiler or our C library. I would treat them as mutually exclusive, not a composition. So you either use our C library or the PC approach during installation. For that, we also need a new CMakeLists.txt (or better yet, a selector whether you want to use the C shim or PC) and then fix the installation part accordingly. What’s also missing right now is the ability to use the Fortran examples.

Originally posted by @sloede in #76 (comment)

EDIT (by @sloede): In this course, one should also introduce the ability to do full out-of-source builds. That is, no compilation artifacts etc. should end up in LibTrixi.jl/lib.

The goal is to be able to control the currently implemented dummy simulation in LibTrixi.jl from our example Fortran program.

Once we have a fully functional libtrixi/LibTrixi.jl based mockup, we should go ahead and add Trixi.jl and OrdinaryDiffEq.jl to LibTrixi.jl. Then, we can finally set up a proper simulation.

If this really makes a performance difference, we might have to compile libtrixi as a static library, i.e., libtrixi.a.

Originally posted by @sloede in #20 (comment)

Right now, only the C Interface is really tested. If we want LibTrixi.jl to be considered also as a standalone package, we should add regular Julia tests as well.

Right now, the various examples serve different purposes, and it not clear (not documented at all?) what their respective purposes is. For example, I always thought that simple_trixi_controller was the simplest of examples, but it turns out you need to compile it with MPI enabled.

Maybe we could add a small overview section to the docs that explains which "features" are tested in each of the examples?

When using julia version 1.8 the following error occurs:

[...]
UndefVarError(var=:invokelatest)
[...]

Do we need to replace invokelatest by Base.invokelatest?

Right now our CI runs two examples (each in C and Fortran) and checks whether they finish without errors.

Do we want more?

• Unit tests?
• Tests with random or perturbed data?
• Third party testing framework?

If you do not have it installed, you might see something like:

~/libtrixi_main/examples (main *) $ ./trixi_controller_simple_c ~/install/libtrixi-julia ../LibTrixi.jl/examples/libelixir_p4est2d_dgsem_euler_sedov.jl

*** Trixi controller ***   Initialize Trixi
  Downloaded artifact: MKL
  Downloaded artifact: MKL
fatal: error thrown and no exception handler available.
InitError(mod=:MKL_jll, error=ErrorException("Unable to automatically download/install artifact 'MKL' from sources listed in '/home/bene/.julia/packages/MKL_jll/8Hu7G/Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/d670351e2fcdac07d02cf73bda5f53e9bea796a6
    Error: IOError: could not spawn setenv(`7z x /tmp/jl_KNK5I7o7of-download.gz -so`,...

T8code has made its way into Trixi:
https://github.com/trixi-framework/Trixi.jl/pull/1426/files#diff-6058d487915aa254cfe1a5f8da6315b8c1fef4da04df38b0089a599803e28e5a

We therefore need to adapt libtrixi:

• For the CI: Install t8code (C version) in place of p4est (t8code includes p4est)
• Adapt libtrixi-init-julia to make julia use the system (C version) of t8code

In case an external application initializes MPI and then initializes libtrixi, which internally initializes MPI.jl, I see the following crash:

[14244] signal (11.1): Segmentation fault
in expression starting at none:1
MPI_Comm_size at /usr/lib/libmpi.so.40 (unknown line)
MPI_Comm_size at /home/bene/.julia/dev/MPI/src/api/generated_api.jl:999 [inlined]
Comm_size at /home/bene/.julia/dev/MPI/src/comm.jl:79
unknown function (ip: 0x7f86c3d06442)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_call at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/interpreter.c:126
eval_value at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/interpreter.c:226
eval_stmt_value at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/interpreter.c:177 [inlined]
eval_body at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/interpreter.c:624
jl_interpret_toplevel_thunk at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/interpreter.c:762
jl_toplevel_eval_flex at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/toplevel.c:912
jl_toplevel_eval_flex at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/toplevel.c:856
ijl_toplevel_eval_in at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/toplevel.c:971
ijl_eval_string at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/jlapi.c:113
trixi_initialize at /home/bene/UniKoeln/libtrixi/src/trixi.c:58
main at /home/bene/UniKoeln/libtrixi/examples/messy_mockup.c:45
unknown function (ip: 0x7f86e6dd784f)
__libc_start_main at /usr/lib/libc.so.6 (unknown line)
_start at ./messy_mockup (unknown line)
Allocations: 618896 (Pool: 617990; Big: 906); GC: 1
Segmentation fault (core dumped)

Indeed, the comm object on the julia side has lots of 0x0 pointers in this situation.

In case MPI is initialized via the C-API from within libtrixi, everything works as expected.

A more or less minimal example can be found in branch
https://github.com/trixi-framework/libtrixi/tree/embedding_julia_with_mpi

Interestingly enough, something similar works with Python:
https://github.com/trixi-framework/libtrixi/tree/embedding_python_with_mpi

Right now, trixi_version prints out the version of libtrixi, e.g., 0.1.0-dev. I think it would be helpful for debugging purposes if we had an API function trixi_version_julia that outputs additional information about the underlying Julia packages currently used and Julia itself, e.g., something like this:

LibTrixi v0.1.0-pre
Trixi v0.5.34
OrdinaryDiffEq v5.26.3
julia v1.9.1

This could then also be automatically shown if LIBTRIXI_DEBUG is enabled.

When going to the docs, there does not seem to be a link anymore to browse the Fortran API. Has this been removed or was it just never there? And, more importantly, can we get it back?

Motivated by a separation of concerns and - possibly - by performance considerations, we should store the C function pointers inside libtrixi instead of getting them each time with a call to jl_eval_string("some_function_cfptr()"). Ideally, we'd only have calls to low-level libjulia functions in trixi_initialize and trixi_finalize, while all others functions directly call into LibTrixi.jl using C function pointers.

Following our discussion, we should make sure that we have at least -Wall -Wextra -Werror and possibly -Wpedantic enabled for both building libtrixi and our examples. This is an easy way to avoid weird errors that might be hard to debug.

The following error was reported by @bgeihe when running an executable linked against a PackageCompiler-compiled libtrixi:

*** Trixi controller ***  Initialize Trixi 
trixi_initialize: 'project_directory' is non-null but will not be used 
fatal: error thrown and no exception handler available. 
InitError(mod=:Trixi, error=ErrorException("could not load library "/home/user/install/t8code/lib/libt8.so" 
/home/user/libtrixi/LibTrixi.jl/lib/build/lib/julia/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /home/user/install/t8code/lib/libt8.so)"))

It seems like the local installation of t8code was built against a newer version of libstdc++ (/usr/lib/libstdc++.so.6.0.32) that expects GLIBCXX_3.4.32 , while Julia's own libstdc++ (libstdc++.so.6.0.30) only provides an older variant.

The problem could be remedied by preloading the system libstdc++, i.e., by calling the executable with

LD_PRELOAD=/usr/lib/libstdc++.so.6 ./simple_trixi_controller_c ...

This seems like an issue that a) could come up at other systems as well and b) that is super annoying to solve for the users. It would therefore be great if we found a solution to
a) detect this issue during the build
b) fix this issue automatically during the build

Setting up libtrixi gets involved once MPI is required in an external application. Should we make the MPI related stuff optional s.t. libtrixi can at least be tested on a simple example?