ornl-fusion / xolotl

Spatially dependent cluster dynamics simulator

Xolotl

Xolotl is a continuum simulator under development within two DOE SciDAC programs, available at https://github.com/ORNL-Fusion/xolotl . It is a high-performance computing code using advection-reaction-diffusion (ADR) kinetic rate theory to model the time evolution of the divertor material in next generation tokamaks like ITER, as well as nuclear fuel in fission reactors.

Required Dependencies

• C++17 compiler
• CMake >= 3.13
• MPI
• Boost
• Kokkos >= 3.2
• HDF5
• PETSc >= 3.16 (configured with kokkos and kokkos-kernels; see scripts/build_petsc.sh)

Quick-Start Install

If you have the above dependencies installed

git clone https://github.com/ORNL-Fusion/xolotl $HOME/xolotl-source
mkdir $HOME/xolotl-build
cd $HOME/xolotl-build
cmake \
    -DCMAKE_BUILD_TYPE=Release \
    -DKokkos_DIR=/opt/kokkos-install \
    -DPETSC_DIR=/opt/petsc-install \
    -DCMAKE_INSTALL_PREFIX=$HOME/xolotl-build/install \
    $HOME/xolotl-source/
make
make test
make install

Wiki

You can find more information about this code on our Wiki and how to install Xolotl and its dependencies in Build Configuration.

Once installed, you can use a Benchmark file to run the code:

./xolotl <input_parameter_file> 

as described in Running Xolotl, with the description of the different options at Parameter File and PETSc Options.

More Info

If you want to contribute please check Guidelines.

A list of publications using Xolotl is available at Publications.

For any question you can send an email to our mailing list: [email protected].

xolotl's Issues

No new

Use make_shared and make_unique

Link to MPI::MPI_CXX in CMake

Bursting as reaction

Test the implementation of the bursting process as reactions in the network instead of the current disruptive process at the end of each timestep. This will also remove its random element.

Move functions on cluster data from ReactionNetwork into ClusterData

getDeviceMemorySize
syncClusterDataOnHost
ReactionNetworkWorker::generateClusterData (maybe)

Index Type from PETSc

Add a typedef to use the same type of int in Xolotl as PetscInt where needed (related to the PETSc option "--with-64-bit-indices"). Get the type from PETSc at compile time (no need for additional option): from the PETSc make.log file there is a "PETSC_USE_64BIT_INDICES" variable set to 1.
This variable should probably be defined in the network as it is what everything else depends on.

Just in case, here is a link to the current patch to switch to 64bit indices in the mooseApps-coupling branch: https://github.com/ORNL-Fusion/xolotl/blob/mooseApps-coupling/64bit.patch

compilation error

I am getting the following error when trying to compile with PETSC 3.9

[ 25%] Building CXX object xolotlSolver/CMakeFiles/xolotlSolver.dir/solverhandler/PetscSolver3DHandler.cpp.o
/Users/a242393/Dropbox/CODES/XOLOTL/xolotl/xolotlSolver/solverhandler/PetscSolver3DHandler.cpp:127:9: error: use of undeclared identifier 'DMDASetBlockFillsSparse'; did
you mean 'DMDASetBlockFills'?
ierr = DMDASetBlockFillsSparse(da, dfillsparse.data(), ofillsparse.data());
^~~~~~~~~~~~~~~~~~~~~~~
DMDASetBlockFills
/Users/a242393/Dropbox/CODES/PETSC-INSTALL/include/petscdmda.h:194:29: note: 'DMDASetBlockFills' declared here
PETSC_EXTERN PetscErrorCode DMDASetBlockFills(DM,const PetscInt*,const PetscInt*);

Using the function DMDASetBlockFills works and the code compiles but I haven't checked if the output is correct.

Use ClusterData* instead of ClusterDataRef in Cluster

Disappearing rate missing from trap mutation partials

Redefine some options

Like for the TRIDYN flux, give the path to the file to read from in the parameter file. Go through the full option list to see which ones can be improved and made more explicit.

Let CMake propagate target include directories automatically

Use target_include_directories instead of include_directories so that we don't have to duplicate lists of include directories. Let the CMake magic work for us. :)

Unable to pass kokkos command line options to xolotl or system tests

Homogenize all the read and write methods in XFile

Take 1D as an example and do the same for 2D/3D.

Revisit the TokenizedLineReader

Look at the current code and/or see if it can be done using boost or other.

Store reference to Jacobian indices in reactions

Construct connectivity mappings up-front and store in reactions (as a starting index and size ?). This could potentially allow us to discard connectivity after network construction. The details of this in its final form will also depend on Phil's work so that we can index into the PETSC Jacobian directly.

Change main branch name

Discuss names for new working git branches

Add time column to the retention file for PSI

Reactions may not need cluster data as a member

Except for trap mutation, reactions use cluster data for rates, connectivity, and coefficients. These processes could use cluster data as a function parameter, instead of Reaction storing (a pointer to) the cluster data as a member.

Add different BC for temperature

For instance the possibility to switch between constant T and constant flux in the bulk.

Update memory management for individual values in ClusterData

Use single managed view(s) for ownership (e.g., View<double[5]> and View<bool[4]>) and unmanaged views for individual value access (like, Unmanaged<View<double>>).

Segfault in PSI with bursting and grouping

Refactor cmake code for tests

• code duplication
• file(GLOB)

List target sources explicitly

Using file(GLOB) for listing target sources can sometimes have confusing results.

Check that the formation energies are set correctly in the PSI network.

Some tests failing with respect to "deviceMemorySize" after storing volumes in reactions

Add analysis scripts

Add scripts (mostly python) to post-process or plot Xolotl standard outputs.

Create IHandler class

Create IHandler class so that the factory prints information automatically once a handler is registered.

Introduce logging mechanism

It would be useful to have a log file where we write extra information that we might want to see (especially in Debug mode), but that we don't want printed to the console.

Probably it would make the most sense to use Boost.Log since we already depend on boost.

Separate grid for temperature

Use a different spacing for solving the heat equations so that the smaller spacing doesn't slow down the solver.

Add a Wiki page about the analysis python scripts

@

Print git version info from code

Use CMake to collect git version info and make it available to the code. Print code version at start-up (along with date/time, etc...).

At first this can simply be the commit hash. But this clears the way for version numbers when we're ready to start an actual release process.

Prefer raw pointers to std::reference_wrapper for non-owning references

Define interface target for collections of headers that multiple other targets depend on

This is in xolotlCore, possible elsewhere.

Add options for viz and temperature handlers

Viz And Temperature factories need work. This requires modifying the options.

Add tests with grouped network for PSI (in core)

Recent Kokkos doesn't build network correctly

Clusters are created but their composition doesn't get set properly, they stay at 0, -1 for any cluster other than the one with id = 0.

Improve plotting options

For instance make it possible to choose which type of data will be plotted at runtime through PETSc options and monitors (in the same way as the custom flux handler?).

Update README and Wiki

Throw exceptions instead of setting exit code in Options

log grid generation

For reference:

This is logarithmic grid generation in FACE:
Note: useful to have a dichotomy method to find out alpha automatically when giving dx0 (initial cell grid size).

subroutine init_grid()

      integer:: j,ngrd2

      real(DP)::dx0

!      ------------------------------------------------------------------

!      initialization of grid arrays

!     ------------------------------------------------------------------


      x(0)=0.d0

      if (alpha .eq. 1.d0) then

       dx0=length/ngrd

       do j=1,ngrd

        x  (j  )=j*dx0

        dx (j-1)=x(j)-x(j-1)

       enddo

       dx (ngrd)=dx(ngrd-1)

      else

       ngrd2=ngrd/2

       if (mod(ngrd,2) .ne. 0) then

        dx0=length*(alpha-1.d0)/((1.d0+alpha)*alpha**ngrd2-2.d0)

        x (0)=0.d0

        dx(0)=dx0

        do j=1,ngrd2

         x (j)=x(j-1)+dx(j-1)

         dx(j)=dx(j-1)*alpha

        enddo

        do j=ngrd2+1,ngrd

         x (j)=x(j-1)+dx(j-1)

         dx(j)=dx(j-1)/alpha

        enddo

       else

        dx0=0.5d0*length*(alpha-1.d0)/(alpha**ngrd2-1.d0)

        x (0)=0.d0

        dx(0)=dx0

        do j=1,ngrd2-1

         x (j)=x(j-1)+dx(j-1)

         dx(j)=dx(j-1)*alpha

        enddo

        x (ngrd2)=x (ngrd2-1)+dx(ngrd2-1)

        dx(ngrd2)=dx(ngrd2-1)

        do j=ngrd2+1,ngrd

         x (j)=x(j-1)+dx(j-1)

         dx(j)=dx(j-1)/alpha

        enddo

       endif

       dx (ngrd)=dx(ngrd-1)

      endif

      write(iout,*) "Initialization x grid: DONE"

      end subroutine init_grid

Store the volumes in the reactions to save time when flux and partials are computed.

Add tests for bursting and other processes

Set the RNG seed to a given value to be able to reproduce the results.

Get rid of ClusterDataRef altogether

Someone should own a View so that an instance of the ClusterData has a permanent home on the device. This then could be the one referred (pointed) to by both clusters and reactions.

Create a network for alpha-zirconium material

Use submodule for plsm

That way specific tags of Xolotl can point to specific release of plsm. Add the option in cmake to checkout the develop plsm branch if needed.

More versatile performance report

See #30 for initial discussion.

My thought here is that we could have a performance report that is separate from the general log file. At the end of a simulation, the PerfHandler::reportStatistics function could generate a PerfReport object which the interface could hang onto. The PerfReport could optionally append itself to a file in a human-readable (and machine-parse-able) format. It could also send itself (as @rothpc suggested) to another tool without touching the filesystem. This design, I think, would give the appropriate flexibility to enable a runtime decision as to how to report the performance data for a given run or coupled simulation.

Error running benchmark

I am trying to run the params_PSI2.txt benchmark, but it gives an error after 75 timesteps:

mpiexec -n 10 ~/XOLOTL/xolotl-build/xolotl params_PSI2.txt 

75 TS dt 0.002 time 0.032
Time: 0.03199999741738022
Helium content = 429.1373945691204
Deuterium content = 0
Tritium content = 0
Fluence = 12799.99896695209

xolotlSolver::Monitor1D: The surface is trying to go outside of the grid!!
Aborting.

enum-indexed array

For anything we want to index by Species:

• Template class with overloaded subscript operator
• Should work with any array-like structure, specifically std::array and Kokkos::Array
• Template parameter for array template
• Template parameter for scoped enum type

Convert util::Array to use Kokkos::Array and be usable in device code

HDF5 network and plsm

Should we be able to read the network from the HDF5 file at restart when the plsm library is used?

Just use Boost.Filesystem

Until the time when we can reasonably require full (i.e., non-experimental) C++17 support from compilers on all machines where we run Xolotl, we should just use the Boost.Filesystem library. Trying to handle the different experimental versions from various versions of libstdc++ and libc++ in CMake is messy and error-prone.