lanl / fierro

Fierro is a C++ code designed to aid the research and development of numerical methods, testing of user-specified models, and creating multi-scale models related to quasi-static solid mechanics and compressible material dynamics using low- and high-order meshes.

Fierro

Fierro (LANL code number C21030) is a modern C++ code designed to simulate quasi-static solid mechanics problems and transient, compressible material dynamic problems with Lagrangian methods, which have meshes with constant mass elements that move with the material, or with Eulerian methods, which have stationary meshes. Fierro is designed to aid a) material model research that has historically been done using commercial implicit and explicit finite element codes, b) numerical methods research, and c) computer science research. The linear Lagrangian finite element methods in Fierro supports user developed material models. Fierro is built on the ELEMENTS library that supports a diverse suite of element types, including high-order elements, and quadrature rules. The mesh class within the ELEMENTS library is designed for efficient calculations on unstructured meshes and to minimize memory usage. Fierro is designed to readily accommodate a range of numerical methods including continuous finite element, finite volume, and discontinuous Galerkin methods. Fierro is designed to support explicit and implicit time integration methods as well as implicit optimization methods.

Computer implementation

Fierro is implemented in C++ following modern programming practices. Fierro leverages the unique features of the ELEMENTS library, as such, this code serves as an example for solving a system of partial differential equations using the mesh class and geometric functions within the ELEMENTS library. Fierro registers state at material points within the element, registers polynomial fields in the element, and registers kinematic variables at element vertices. The routines for the state are designed for highly efficient computations and to minimize memory usage. The loops are written to aid fine-grained parallelization and to allow vectorization. Fierro is a light-weight software application, and cleanly written following modern programming practices, so it useful for researching computer science based technologies for software performances portability.

Spatial discretization methods

Fierro has an established conservative low-order Lagrangian finite element method, a low-order Lagrangian cell-centered finite volume method, and an arbitrary-order Lagrangian Discontinuous Galerkin method for solving the governing equations (e.g., mass, momentum, and energy evolution equations) for compressible material dynamics using unstructured hexahedral meshes. These methods are combined with a multidirectional approximate Riemann solver (MARS) for improved accuracy on smooth flows and stable solutions near velocity discontinuities and large gradients in a flow. Fierro is designed for both low and high-order Lagrangian methods research and development but other types of numerical methods can be readily added to the code. Likewise, other high-order methods can be studied within the code because it is built upon the ELEMENTS library that supports high-order elements and high-order quadrature rules. Numerical methods are being added to Fierro to simulate quasi-static solid mechanics problems. Likewise, direct Eulerian hydrodynamic methods can be investigated using Fierro.

Temporal discretization methods

Fierro supports a range of published multi-step time integration methods. The code has an explicit multi-step Runge Kutta time integration method. Implicit time integration methods can be implemented in Fierro.

Usage

Anaconda

The recommended way to use Fierro is through the provided Anaconda package. To use the anaconda package, follow the steps for your platform to install anaconda/miniconda/mamba.

Open a terminal on your machine and go to a folder where you want to run the Fierro code. Then create and activate an Anaconda environment by typing:

conda create -n FierroCode
conda activate FierroCode  

In this example, the enviroment is called FierroCode, but any name can be used. In some cases, the text to activate an enviroment is source activate FierroCode. Likewise, if an enviroment already exists, then just activate the desired environment.

To install the finite element physics solvers in Fierro, please type within the activated Anaconda environment:

conda install -c conda-forge -c fierromechanics fierro-cpu

The EVPFFT physics solver in Fierro can be installed by typing:

conda install -c conda-forge -c fierromechanics evpfft

A GUI is offered, it can be installed by typing:

conda install -c conda-forge -c fierromechanics evpfft_gui

After installing the finite element solvers, it gives you access to fierro-mesh-builder,fierro-parallel-explicit,fierro-parallel-implicit, and the fierro-voxelizer executables. These can be run by calling the appropriate executable with the desired input. For example, to call the parallel explicit hydrodynamics solver, use the following command:

fierro-parallel-explicit input.yaml

A sample input file for the explicit finite element solver is here: ./src/Parallel-Solvers/Parallel-Explicit/example_simple.yaml

The GUI can be run in the anaconda enviroment by typing:

evpfft-gui

The anaconda distributions of Fierro are located here.

Material models

The classical ideal gas model is the only material model implemented in the code, and it is useful for verification tests of the software and simple gas dynamic simulations. The linear Lagrangian finite element methods for explicit material dynamics have an interface to user developed material models. The interface is to enable Fierro to be used for model research and development that has historically been done with commercial explicit finite element codes.

To include your own custom material models, you need to implement them under Fierro/Parallel-Solvers/User-Material-Interface and re-build the project. Steps:

1. Create an anaconda environment for your build
2. Install Fierro anaconda dependencies with conda install elements fierro-trilinos-cpu elements -c fierromechanics
3. Clone the code
4. Build the code

Now, if all went correctly, you should be able to run your custom Fierro build by calling the executable located at Fierro/build/bin/fierro. The executables can be installed into your system directories with the make install command as well.

Cloning the code

If the user has set up ssh keys with GitHub, type

git clone --recursive ssh://[email protected]/lanl/Fierro.git

The code can also be cloned using

git clone --recursive https://github.com/lanl/Fierro.git

Building the code

Building the code from source allows you to compile with more targeted hardware optimizations that could offer a potentially faster executable. To build it yourself, run the following from the root directory. The native CPU architecture will automatically be taken into account.

GPU hardware will be leveraged according to the distribution of Trilinos that Fierro is built against. You are welcome to only compile the desired solver, or all of the currently available ones.

Building dependencies

Fierro depends on both ELEMENTS and Trilinos. If you are building Fierro for hardware optimizations, you should also build ELEMENTS from source. ELEMENTS is included in the Fierro source distribution and building ELEMENTS is enabled by default when building Fierro.

As for Trilinos, we recommend installing the Anaconda package for the desired build into a new Anaconda environment to satisfy Fierro's dependency rather than building it from source. If you do wish to build it from source, however, sample build scripts for Trilinos can be found in Fierro/Trilinos-Build-Scripts. Build scripts for all Anaconda packages can be found in Fierro/Anaconda-Packages/.

Alternative Build Workflows

In addition to the primary build workflow described above, there are build scripts for a variety of alternative workflows. These scripts can be found under Fierro/scripts. Although there are multiple scripts shown in this directory, the only one that will be run directly by the user is build-fierro.sh

Building the explicit and implicit Lagrangian methods with Trilinos+Kokkos

Explicit Lagrangian codes are being added to the repository that are written using MATAR+Kokkos and run with fine-grained parallellism on multi-core CPUs and GPUs. Build scripts are provided for each Lagrangian code, and those scripts follow those used in the MATAR GitHub repository. The scripts to build the Lagrangian codes (that use MATAR+Kokkos) are in the scripts folder. The user must update the modules loaded by the build scripts (for the compiler etc.), and then type Immediate help with all scripts can be had running

source {path-to-repo}/build-fierro.sh --help

The default run will build the full-app settting up the Explicit* solver with serial Kokkos for a linux computer (non-HPC machine) Running with --help option will list all parameters and their accepted arguments If the scripts fail to build a Lagrangian code, then carefully review the modules used and the computer architecture settings.
For building on a Mac, please see the extra info in the BUILD.md #A more lenghtly discussion of the build scripts is provided in the MATAR GitHub repository.

For help with Trilinos For help with Kokkos compilation

Updating submodules

The ELEMENTS library and MATAR library can be updated to the newest release using

git submodule update --remote --merge

Contributing to Fierro

As an open source software team, we greatly appreciate any and all community involvement. There are many ways to contribute to Fierro, from tidying up code sections to implementing novel solvers. To streamline the integration of community contributions, we follow the following guidelines.

Writing commit messages

Write your commit messages using these standard prefixes:

• BUG: Fix for runtime crash or incorrect result
• COMP: Compiler error or warning fix
• DOC: Documentation change
• ENH: New functionality
• PERF: Performance improvement
• STYLE: No logic impact (indentation, comments)
• WIP: Work In Progress not ready for merge

The commit message should assist in the review process and allow future developers to understand the change. To that end, please follow these few rules:

1. Try to keep the subject line below 72 characters, ideally 50.
2. Be concise, but honor the change.
3. If the change is uncharacteristically large (or diverse), consider breaking the change into multiple distinct commits.
4. If the commit references a specific issue, link it in the message

This is a great post on how to write a good commit message.

The PR Process

If you are new to Fierro development and don't have push access to the repository, here are the steps:

1. Fork and clone the repository.
2. Create a branch for your work.
3. Push the branch to your GitHub fork.
4. Build and test your changes.
5. Update any necessary documentation.
6. Create a Pull Request.

This corresponds to the Fork & Pull Model described in the GitHub collaborative development documentation.

Integrating a PR

Integrating your contributions to Fierro is relatively straightforward; here is the checklist:

• All test pass
• The changes build with no new compiler warnings/errors
• All feedback has been addressed
• Consensus is reached. This usually means that at least two reviewers approved the changes (or added a LGTM comment) and at least one business day passed without anyone objecting. LGTM is an acronym for Looks Good to Me.
• If you do NOT have push access, a Fierro core developer will integrate your PR.

Benevolent dictators for life

The benevolent dictators can integrate changes to keep the platform healthy and help interpret or address conflict related to the contribution guidelines and the platform as a whole.

These currently include:

• Nathaniel Morgan

fierro's Issues

Reorganization of FEA Modules Inputs

FEA Modules current parse their inputs from the root of the user input file. This leads to two issues:

1. We don't know at first parse time what we are parsing.
2. It is easy for key conflicts to occur, since we are trying to parse the same object as multiple different objects.
3. We cannot take advantage of "strict" parsing, where we enforce that all data in the input file gets used.

Additionally, despite containing the same information, we have two separate (slightly different) structure for our Static and Dynamic solvers.

To resolve this, we will restructure the Implicit + Explicit + FEA Module inputs according to the diagram on the website (https://lanl.github.io/Fierro/developer-guide.html). Additionally, we will take advantage of functionality introduces in PR #84 to allow for type discriminated parsing so we will parse it once up front and be able to know if there are any configuration errors.

Elastic FEA Module doesn't handle strain output

Keying off of the "output_strain" yaml flag, we allocate space for strain outputs in the elastic module, but we don't actually put anything there leading to seg faults if someone runs with with that flag set.

Make sure that the code is really unitless.

Restarts from HDF5 checkpoints

Currently some of our code pretends to support restarting. Despite it being an important functionality for HPC users that expect long running jobs (1day+), we don't really support it.

We should consolidate the solver state into a single object and support periodically creating checkpoints that we can restart from. These should probably be HDF5 files to support exporting/importing the exact solver state as structured binary data.

Support for Fortran based hooks

Many users have spent considerable effort in creating Fortran based models. We need to ensure that we support those going forward.

It is likely that this can be done without much issue if we require that the Fortran models fit a simple user interface and are compiled to the ISO C ABI. However, it will also likely take some small modification on our end to support such an adapter.

EVPFFT GUI Renders Voxelization with Misleading Scale

Issue

When we render the voxel VTK we are going to give to the EVPFFT backend, the positions of the voxels are stored in coordinate space. This can be confusing for users, as the mesh they see can change scale upon voxelization. If they enter a voxel ratio of 1x2x1, for example, it will appear that their mesh was stretched by a factor of 2 along the y dimension. Additionally, the mesh is not centered, which makes it annoying to manipulate in the viewer.

Potential fixes

Lots of potential solutions

1. Let the EVPFFT backend handle real space coordinates and not index space coordinates. Then just ask the voxelizer to give the real coordinates.
2. Voxelize it twice, once for EVPFFT backend and once for the rendering. Little stupid, but trivial to do. Performance is probably still "fine".
3. Scale the resulting voxel grid in paraview.

Consolidated Mesh IO

MPI Mesh IO is handled in the Parallel Explicit Solvers class. It also consists of a significant amount of copied code and does not load all of the mesh properties we might want to, and does not support all of the file formats that we want.

So, we will make a separate Mesh IO library with the goals of:

1. Consolidating duplicate/redundant code
2. Creating a testable Mesh IO interface

This library will support:

• Reading/writing meshes w/ and w/o MPI.
• Reading/writing scalar and vector mesh properties
• Reading from VTK, Ensight, Tecplot, and Cubit Exodus file formats

It will produce a single mesh object that can be used and modified by solvers.

Does not stop when an empty mesh file is given on input

I used cmake to build an executable. Calling ./fierro empty_file.exo does not terminate immediately.

Automatic HDF5 Serialization

Currently HDF5 serialization is a pain. It require a significant amount of boiler plate code to implement and is error prone to modify as we want to serialize new data structures.

However, for most data types it is possible to automatically generate a serialization scheme. This would be very useful for defining expected structures and supporting high resolution solver checkpointing.

IJK vs Ensight

Some parts of the code use IJK and some use the EnSight node ordering convention.

It might not technically be a problem right now, but its bound to cause us issues in the future if we don't sort that out.

We should have type guarded connectivity structures to prevent us from ever making that issue in the future.

Could also just always use the same convention internally.

Add Python package/module for our toolbox.

Distributed Mesh Class

For any kind of proper scalability, we rely on MPI. As it stands, we have separate code on top of our mesh classes that operate with MPI. We use Zoltan2 for node balancing and TPetra for MPI communications.

We should consolidate this into a single layer on top of a mesh class. The availability of out of the box distributed mesh scaling can be significantly valuable for ELEMENTS users.

The distributed mesh would handle:

1. Node balancing
2. Process mapping during read/write
3. MPI Communications

During this process, we should consider separating from TPetra MultiVectors for MPI comms and implement our own for two reasons:

1. The layout of MultiVectors is contrary to our own data layout for several arrays. This results in substantial over-communication when we have to transpose them. While only a mild slowdown now, there are future methods that we anticipate to take a much bigger hit.
2. We would like to take advantage of MPI Direct GPU comms in the future, and TPetra is unlikely to support this.

Unit Testing

Currently, Fierro has limited unit testing coverage. This needs to be remedied for the long-term health of the project. Automated unit testing allows for the safe integration of new features by warning of any unforeseen side effects of the proposed changes. The existing unit testing leverages Google Test, which is great, and I suggest we keep to that moving forward.

Unit testing is great, but it does add some requirements on how you think about structuring your code. These tests require that your functions be broken into small testable units with clearly defined inputs and outputs. Also, the inputs need to be mimicable so that they can be generated inside of the test code.

Some examples of unit tests that Fierro needs.

• Single element, single time step evolution with well-defined inputs and known expected outputs for evolving each relevant state
• Test of every step of mesh generation (unit test each connectivity structure)
• Test of volume and BMatrix calculations in geometry.

We should leave this issue open until we improve code coverage and update it as we work.

Support for dynamic user plugins

Currently user plugins have to be compiled along with Fierro. Also, they have to be written inside of the Fierro repository. Additionally, users writing such plugins have limited help in testing/debugging them.

We should create a small framework for our plugins that supports:

1. Independent compilation
2. Independent testing
3. Consistent interface across plugin locations (not terribly relevant now, as we have very few plugin hook locations. But likely relevant as we add more user controls).

User Mesh initialization hooks

With popular, commercial tooling, users do not have good option to initialize mesh properties. We believe there are important use cases when a user would want to start a simulation from a non-constant initial condition. To this end, we will create a function hook for users that is invoked just after initially loading the mesh.

The goal here is to give users ultimate control on their initial conditions.