wdmapp / wdmapp-config

WDMapp support: spack configurations, etc

WDMapp build and run configurations

This repository contains the necessary bits to use Spack to build WDMapp and its constituent codes.

Step-by-step documentation is available at https://wdmapp.readthedocs.io/en/latest/.

This is a work-in-progress!

Repository Layout

• bin/: Some useful tools to help with Spack setup on a new machine
• spack/wdmapp/: An add-on Spack repository which adds WDMapp-specific packages
• docs/: Sources for the documentation.
• summit/: Configuration files and job scripts for Summit at OLCF.
• longhorn/: Configuration files and job scripts for Longhorn at TACC.

wdmapp-config's Issues

XGC-Devel build with -adios2

Is there any particular reason xgc-devel is using -adios2 in wdmapp? I've turned it on in xgc-devel@wdmapp +adios2 for XGC and it doesn't seem to cause any issues. (I know Scott will want us to turn ADIOS-2 on instead of ADIOS-1 for everything we can.)

Using TAU on Rhea

Has anyone else tried TAU on Rhea with +papi? I'm getting this in XGC when I do that:

Error: To use TAU's PAPI RAPL Perf interface please ensure that /proc/sys/kernel/perf_event_paranoid has a -1 in it.

I haven't seen that when I turn PAPI off. Do we want to turn it off in general? I haven't tried with +papi on Summit either, but it might work there.

OpenMPI on Rhea

The reason OpenMPI doesn't work like usual on Rhea, is because OLCF installed it with Spack, where the package came from a repo namespace called olcf. When one configures something that was installed with Spack as a module or path in packages.yaml, it tries to match up with the metadata in the .spack installation directory. Our OpenMPI is the builtin namespace, so it's expecting builtin/ to contain openmpi under .spack/ but it does't. It's under olcf/.

FileNotFoundError: [Errno 2] No such file or directory: '/autofs/nccs-svm1_sw/rhea/.swci/0-core/opt/spack/20191017/linux-rhel7-x86_64/gcc-8.4.0/openmpi-3.1.4-bmn7or3eq2xi2sol4f7yvlpnwniut7yd/.spack/repos/builtin/packages/openmpi/'

Instead of copying OpenMPI to a different directory without the .spack directory, or installing a non-OLCF built version, this can be fixed by having an olcf namespace with an openmpi package. Previously, I had just symbolically linked builtin packages under olcf, but I don't know a general way to do that, without assuming fully resolved file paths, or forking all of Spack. So probably the easiest thing to do is just copy the releases/v0.14 versions of any relevant packages in an olcf repo namespace. I can make a relevant pull request.

External packages on Rhea

@germasch @cwsmith Have we settled that on Rhea we're using a spack packages.yaml that has the extended list of system software. If so, entries need to be added for Python modules py-mpi4py, py-numy, py-matplotlib, and py-pyyaml that point to the anaconda directory. Otherwise there are conflicts when a spack activate is used, e.g. for EFFIS.

I can make a pull request, I mostly asking if the extended set of system software is preferred. It's basically an issue of speed of install vs. a less verbose packages.yaml that's less prone to getting outdated. (I probably slightly prefer using fewer system packages.) We can also have a file for both, like on Summit.

missing adios2 variant in tau package

We are hitting a error during concretization with master (45dd8e2) when trying to install wdmapp on summit:

+ spack spec -I wdmapp
Input spec
--------------------------------
 -   wdmapp

Concretized
--------------------------------
==> Error: trying to set variant "adios2" in package "tau", but the package has no such variant [happened during concretization of [email protected]%[email protected]+effis~passthrough+tau~xgc1_legacy arch=linux-rhel7-power9le ^tau@develop+adios2~libunwind+mpi~pdt]

Running the following script will reproduce the problem. Note, the script renames your existing ~/.spack/linux-summit and ~/.spack/linux with .bk appended.

#!/bin/bash -x
wdmDir=$SCRATCH/wdmInstallTest2
mkdir -p $wdmDir
cd $wdmDir
git clone -b v0.15.4 https://github.com/spack/spack.git
git clone [email protected]:wdmapp/wdmapp-config.git
cd spack
export SPACK_ROOT=$PWD
source $SPACK_ROOT/share/spack/setup-env.sh
which spack # sanity check
mv ~/.spack/linux-summit ~/.spack/linux-summit.bk
mv ~/.spack/linux ~/.spack/linux.bk
mkdir -p ~/.spack/linux-summit
ln -snf ~/.spack/linux-summit ~/.spack/linux
cp $wdmDir/wdmapp-config/summit/spack/*.yaml ~/.spack/linux
spack repo add $wdmDir/wdmapp-config/spack/wdmapp
spack repo list # sanity check
spack spec -I wdmapp